PALMNUT: An Enhanced Proximal Alternating Linearized Minimization Algorithm with Application to Separate Regularization of Magnitude and Phase

Abstract

We introduce a new algorithm for complex image reconstruction with separate regularization of the image magnitude and phase. This optimization problem is interesting in many different image reconstruction contexts, although is nonconvex and can be difficult to solve. In this work, we first describe a novel implementation of the previous proximal alternating linearized minimization (PALM) algorithm to solve this optimization problem. We then make enhancements to PALM, leading to a new algorithm named PALMNUT that combines the PALM together with Nesterov’s momentum and a novel approach that relies on uncoupled coordinatewise step sizes derived from coordinatewise Lipschitz-like bounds. Theoretically, we establish that a version of PALMNUT (without Nesterov’s momentum) monotonically decreases the objective function, guaranteeing convergence of the cost function value. Empirical results obtained in the context of magnetic resonance imaging demonstrate that PALMNUT has computational advantages over common existing approaches like alternating minimization. Although our focus is on the application to separate magnitude and phase regularization, we expect that the same approach may also be useful in other nonconvex optimization problems with similar objective function structure.

I. Introduction

Many model-based computational imaging methods assume that data acquisition can be represented as a linear system of equations, such that

$$\mathbf{b}=\mathbf{Af}+\mathbf{n},$$ (1)

where b is the length-M vector of measured data samples, f is the length-N vector of unknown image voxel values, A is an M × N matrix modeling the data acquisition operator, and n is a length-M vector of additive noise perturbations. Based on this forward model, it is relatively popular to formulate image reconstruction as a regularized linear least-squares optimization problem:

$$\widehat{\mathbf{f}}=\text{arg}\underset{\mathbf{f}}{\text{min}}\frac{1}{2}\Vert \mathbf{Af}-\mathbf{b}{\Vert }_2^2+R(\mathbf{f}),$$ (2)

where ∥ · ∥₂ is the standard Euclidean (ℓ₂) norm, and R(·) is a regularization functional that is designed to reduce illposedness by encouraging the reconstructed image to have desirable characteristics. For example, it is common to encourage the reconstructed image to be spatially-smooth by choosing R(·) as a penalty on the norm of the image gradient or as a penalty on the norm of the high-frequency subbands in a wavelet representation of the image.

Although the image f may be real-valued in many scenarios (i.e., $\mathbf{f}\in {\mathbb{R}}^N$), there are several imaging modalities (including certain kinds of optical imaging, ultrasound imaging, magnetic resonance imaging (MRI), synthetic aperture radar, etc. [1]-[10]) for which the image of interest is most naturally modeled as complex-valued (i.e., $\mathbf{f}\in {\mathbb{C}}^N$). In such cases, the image magnitude and image phase often represent manifestations of different physical phenomena, and have distinct spatial characteristics from one another that are not easily captured by a single unified regularization penalty. This has led several authors, working on image reconstruction across a variety of different modalities and application domains, to consider optimization problems in which the magnitude and phase of the image are regularized separately [1]-[10].

While these approaches were developed independently in many cases, the optimization problems can largely be unified by the following general formulation:

$$\{\widehat{\mathbf{m}},\widehat{\mathbf{p}}\}=\text{arg}\underset{\begin{array}{c}\hfill \mathbf{m}\in {\mathbb{R}}^N\hfill \\ \hfill \mathbf{p}\in {\mathbb{R}}^N\hfill \end{array}}{\text{min}}\tilde{J}(\mathbf{m},\mathbf{p})+R_1(\mathbf{m})+{\tilde{R}}_2(\mathbf{p}),$$ (3)

where

$$\tilde{J}(\mathbf{m},\mathbf{p})\triangleq \frac{1}{2}\Vert \mathbf{A}\left(\mathbf{m}\odot e^{i\mathbf{p}}\right)-\mathbf{b}{\Vert }_2^2.$$ (4)

In this expression, m and p respectively represent the magnitude¹ and phase of the complex image f such that f = mʘe^ip, and R₁(·) and ${\tilde{R}}_2(\cdot )$ respectively represent separate regularization penalties for the magnitude and phase. We have used the notation ʘ to represent the Hadamard product (elementwise multiplication) of two vectors, have used the notation e^z to denote the elementwise exponentiation of the vector z, and have used i to denote the unit imaginary number ($i=\sqrt{-1}$).

On one hand, Eq. (3) is desirable because it allows independent customization of the regularization penalties applied to the magnitude and phase components of the image, which has previously been shown to be beneficial [1]-[10]. On the other hand, this formulation can be substantially more difficult to solve than Eq. (2). For example, while Eq. (2) is convex if R(·) is chosen to be convex (and therefore can be globally optimized from arbitrary initializations using standard optimization methods!), the formulation in Eq. (3) is generally non-convex and is thus more challenging to optimize. As a result, many authors have decided to make use of a simple alternating minimization (AM) strategy that alternates between optimizing m for a fixed value of p and optimizing p for a fixed value of m [2], [4], [5], [7], [8], [10]. However, while AM can successfully decrease the objective function, it is not necessarily computationally efficient.

In this work, we propose and evaluate a new algorithm named PALMNUT to solve Eq. (3). This algorithm is based on combining the proximal alternating linearized minimization (PALM) algorithm [11] with Nesterov’s momentum [12] and a novel approach that uses uncoupled coordinatewise step sizes derived from coordinatewise Lipschitz-like bounds. PALMNUT is evaluated in the context of several different MRI-related inverse problems, where it is demonstrated to outperform popular existing methods. A preliminary account of portions of this work was previously presented in a recent conference [13].

During the review process for this paper, we were made aware of the existence of the Block Proximal Extrapolated Gradient method using a Majorizer (BPEG-M) algorithm, which was originally developed in the context of dictionary and operator learning [14], [15]. Although PALMNUT and BPEG-M were developed independently from one another, the algorithms share some similar features, including the alternating application of proximal gradient methods to different sets of variables and the use of momentum. In addition, the use of uncoupled step-sizes in PALMNUT has similarities to the use of “majorization matrices” in BPEG-M, and both approaches are derived using similar proof techniques. A nuanced description of the differences between the two algorithms is outside the purview of this paper, although is available from the authors on request. However, at a high level, PALMNUT’s approach is based on a slightly more general² majorization relationship (based on inner products) than the one used by BPEG-M (based on norms). Conversely, BPEG-M has proposed the use of a positive definite matrix for majorization which is slightly more general than the coordinatewise (diagonal) majorization described herein for PALMNUT, although we note that PALMNUT would be easily generalized to use positive definite majorization if so desired. While a full comparison of PALMNUT and BPEG-M is beyond the scope of this paper, our preliminary analysis and numerical experiments (not shown, but available from the authors on request) suggest that PALMNUT generally uses larger step-sizes and has faster convergence than the most recent and most relevant published BPEG-M variation [15], although this variation of BPEG-M also possesses a strong form of convergence guarantee (stronger than what we have attempted to derive for PALMNUT).

II. Background

The following subsections review the AM approach (one of the dominant existing algorithms for solving Eq. (3)) as well as the related PALM algorithm (which serves as the foundation for our proposed PALMNUT approach).

Since both AM and PALM are generic algorithms that are broadly applicable beyond just the magnitude and phase problem from Eq. (3), we will start by introducing both of these algorithms in a more general setting before specializing to our problem of interest. Specifically, we will consider the generic optimization problem of the form

$$\{\widehat{\mathbf{x}},\widehat{\mathbf{y}}\}=\text{arg}\underset{\begin{array}{c}\hfill \mathbf{x}\in {\mathbb{R}}^{N_1}\hfill \\ \hfill \mathbf{y}\in {\mathbb{R}}^{N_2}\hfill \end{array}}{\text{min}}\Psi (\mathbf{x},\mathbf{y}),$$ (5)

where $\mathbf{x}\in {\mathbb{R}}^{N_1}$ and $\mathbf{y}\in {\mathbb{R}}^{N_2}$ are optimization variables, and the objective function Ψ(·, ·) can be decomposed as

$$\Psi (\mathbf{x},\mathbf{y})=H(\mathbf{x},\mathbf{y})+F(\mathbf{x})+G(\mathbf{y})$$ (6)

for some real-valued or extended-real-valued scalar functions H(·, ·), F(·), and G(·).

A. Alternating Minimization

Given the problem setting described above, the AM algorithm proceeds according to Alg. 1.

Algorithm 1 Alternating Minimization for Eq. (5)

$$\begin{array}{cc}\hfill & \mathrm{Initialization}:\text{Set}k=1\text{and initialize}{\widehat{\mathbf{x}}}_0\text{and}{\widehat{\mathbf{y}}}_0.\hfill \\ \hfill & \mathbf{while}\text{not converge}\mathbf{do}\hfill \\ \hfill & {\widehat{\mathbf{x}}}_k=\underset{\mathbf{x}\in {\mathbb{R}}^{N_1}}{\text{arg}\text{min}}\Psi (\mathbf{x},{\widehat{\mathbf{y}}}_{k-1})\hfill \\ \hfill & =\underset{\mathbf{x}\in {\mathbb{R}}^{N_1}}{\text{arg}\text{min}}H(\mathbf{x},{\widehat{\mathbf{y}}}_{k-1})+F(\mathbf{x})\hfill \\ \hfill & {\widehat{\mathbf{y}}}_k=\underset{\mathbf{y}\in {\mathbb{R}}^{N_2}}{\text{arg}\text{min}}\Psi ({\widehat{\mathbf{x}}}_k,\mathbf{y})\hfill \\ \hfill & =\underset{\mathbf{y}\in {\mathbb{R}}^{N_2}}{\text{arg}\text{min}}H({\widehat{\mathbf{x}}}_k,\mathbf{y})+G(\mathbf{y})\hfill \\ \hfill & k\leftarrow k+1\hfill \\ \hfill & \mathbf{end}\mathbf{while}\hfill \end{array}$$

As can be seen, AM alternates between optimizing the estimate of x for a fixed estimated value of y and optimizing the estimate of y for a fixed estimated value of x. This can be viewed as a block coordinate descent algorithm, and thus has well-studied theoretical characteristics [16]. The algorithm leads to monotonic decrease of the objective function value, and therefore (by the monotone convergence theorem) convergence of the objective function value if Ψ(x, y) is bounded from below. In practice, it is not necessary to solve the optimization subproblems for ${\widehat{\mathbf{x}}}_k$ and ${\widehat{\mathbf{y}}}_k$ exactly in each step, as the overall objective function will still decrease monotonically as long as the subproblem objective function values always decrease in each iteration.

In the case of optimizing Eq. (3) with AM [2], [4], [5], [7], [8], [10], one should associate x with m, y with p, H(·, ·) with $\tilde{J}(\cdot ,\cdot )$, F(·) with R₁(·), and G(·) with ${\tilde{R}}_2(\cdot )$.

When defining $\tilde{J}(\mathbf{m},\mathbf{p})$ as in Eq. (4), the magnitude subproblem has the form of a classical regularized linear least-squares problem. As such, there exist many different algorithms to solve this kind of problem efficiently. However, the phase subproblem is more complicated, as it is generally nonconvex. Existing methods have frequently relied on either the nonlinear conjugate gradient (NCG) algorithm [2], [4], [5], [7], [10] or a phase cycling heuristic [8] to address this subproblem.

B. Proximal Alternating Linearized Minimization (PALM)

The PALM algorithm considers the same setup described previously, but with the additional assumptions that H(·, ·) is a smooth real-valued function and F(·) and G(·) are proper and lower semicontinuous (but potentially nonsmooth) extended-real-valued functions. None of these functions are required to be convex.

The structure of PALM is strongly motivated by the AM algorithm, and similar to AM, the PALM algorithm for Eq. (5) proceeds by updating the x and y variables in alternation. However, rather than directly using the AM update formulas, PALM first applies proximal linearization of H to each subproblem prior to computing each update step. Specifically, PALM uses

$$\begin{gathered} {\widehat{\mathbf{x}}}_k=\underset{\mathbf{x}\in {\mathbb{R}}_1^N}{\text{argmin}}H({\widehat{\mathbf{x}}}_{k-1},{\widehat{\mathbf{y}}}_{k-1}) \\ +\langle \mathbf{x}-{\widehat{\mathbf{x}}}_{k-1},{\nabla }_{\mathbf{x}}H({\widehat{\mathbf{x}}}_{k-1},{\widehat{\mathbf{y}}}_{k-1})\rangle \\ +\frac{c_k}{2}\Vert \mathbf{x}-{\widehat{\mathbf{x}}}_{k-1}{\Vert }_2^2+F(\mathbf{x})+G({\widehat{\mathbf{y}}}_{k-1}) \end{gathered}$$ (7)

and

$$\begin{gathered} {\widehat{\mathbf{y}}}_k=\underset{\mathbf{y}\in {\mathbb{R}}^{N_2}}{\text{arg min}}H({\widehat{\mathbf{x}}}_k,{\widehat{\mathbf{y}}}_{k-1}) \\ +\langle \mathbf{y}-{\widehat{\mathbf{y}}}_{k-1},{\nabla }_{\mathbf{y}}H({\widehat{\mathbf{x}}}_k,{\widehat{\mathbf{y}}}_{k-1})\rangle \\ +\frac{d_k}{2}\Vert \mathbf{y}-{\widehat{\mathbf{y}}}_{k-1}{\Vert }_2^2+F({\widehat{\mathbf{x}}}_k)+G(\mathbf{y}), \end{gathered}$$ (8)

where c_k and d_k are real-valued positive scalars, ∇_x represents the gradient with respect to x, and ⟨·, ·⟩ denotes the standard dot product. These steps can be viewed as alternatingly performing proximal gradient descent on each set of variables.

One way to justify Eqs. (7) and (8) is to invoke the majorize-minimize algorithmic framework [17]. Specifically, for any real-valued objective function Q(x) with $\mathbf{x}\in {\mathbb{R}}^N$, we say that the function S_k(x) is a “majorant” of Q(x) at the point ${\widehat{\mathbf{x}}}_{k-1}$ if it satisfies two conditions: $(\mathrm{i})S_k({\widehat{\mathbf{x}}}_{k-1})=Q({\widehat{\mathbf{x}}}_{k-1})$; and (ii) Q(x) ≤ S_k (x) for $\forall \mathbf{x}\in {\mathbb{R}}^N$. An important feature of majorant functions is that if $S_k({\widehat{\mathbf{x}}}_k)<S_k({\widehat{\mathbf{x}}}_{k-1})$, then it is guaranteed that $Q({\widehat{\mathbf{x}}}_k)<Q({\widehat{\mathbf{x}}}_{k-1})$. This fact allows the majorant S_k(x) to be used as a surrogate for the original objective function Q(x), since descending on S_k(x) guarantees descent on Q(x). The use of a surrogate can be beneficial whenever the surrogate is easier to optimize than the original function [17].

With these concepts defined, we return to Eqs. (7) and (8). Assume that for a fixed value of y, ∇_xH(x, y) is Lipschitz in x such that

$$\Vert {\nabla }_{\mathbf{x}}H({\mathbf{x}}_1,\mathbf{y})-{\nabla }_{\mathbf{x}}H({\mathbf{x}}_2,\mathbf{y}){\Vert }_2\le L_1(\mathbf{y})\Vert {\mathbf{x}}_1-{\mathbf{x}}_2{\Vert }_2$$ (9)

for ∀x₁, ${\mathbf{x}}_2\in {\mathbb{R}}^{N_1}$, where L₁(y) is the corresponding Lipschitz constant. Similarly, assume that for a fixed value of x, ∇_yH(x, y) is Lipschitz in y such that

$$\Vert {\nabla }_{\mathbf{y}}H(\mathbf{x},{\mathbf{y}}_1)-{\nabla }_{\mathbf{y}}H(\mathbf{x},{\mathbf{y}}_2){\Vert }_2\le L_2(\mathbf{x})\Vert {\mathbf{y}}_1-{\mathbf{y}}_2{\Vert }_2$$ (10)

for ∀y₁, ${\mathbf{y}}_2\in {\mathbb{R}}^{N_2}$, where L₂(x) is the corresponding Lipschitz constant. Under these assumptions, it can be shown that the objective function in Eq. (7) will be a majorant of $\Psi (\mathbf{x},{\widehat{\mathbf{y}}}_{k-1})$ at the point $\mathbf{x}={\widehat{\mathbf{x}}}_{k-1}$ whenever $c_k\ge L_1({\widehat{\mathbf{y}}}_{k-1})$, and the objective function in Eq. (8) will be a majorant of $\Psi ({\widehat{\mathbf{x}}}_k,\mathbf{y})$ at the point $\mathbf{y}={\widehat{\mathbf{y}}}_{k-1}$ whenever $d_k\ge L_2({\widehat{\mathbf{x}}}_k)$ [11]. This means that if c_k and d_k are chosen large enough, then Eqs. (7) and (8) can be interpreted as majorize-minimize steps. This will guarantee monotonic decrease of the objective function value, and therefore convergence of the objective function value will be guaranteed for PALM if Ψ(x, y) is bounded from below [17]. Under some additional conditions (e.g., that the cost function satisfies something called the Kurdyka-Lojasiewicz property), the iterates of PALM are guaranteed to converge to a critical point of Ψ(x, y) [11].

The solutions to the optimization problems in Eqs. (7) and (8) can be written in a more concise “proximal operator” form [11]. Specifically, Eq. (7) is equivalent to the “proximal operator”

$$\begin{gathered} {\widehat{\mathbf{x}}}_k=\underset{\mathbf{x}\in {\mathbb{R}}^{N_1}}{\text{argmin}}F(\mathbf{x})+\frac{c_k}{2}\Vert \mathbf{x}-{\mathbf{w}}_k{\Vert }_2^2 \\ \triangleq {\text{prox}}_F({\mathbf{w}}_k,c_k) \end{gathered}$$ (11)

with

$${\mathbf{w}}_k={\widehat{\mathbf{x}}}_{k-1}-(1∕c_k){\nabla }_{\mathbf{x}}H({\widehat{\mathbf{x}}}_{k-1},{\widehat{\mathbf{y}}}_{k-1}),$$ (12)

and Eq. (8) is equivalent to the “proximal operator”

$$\begin{gathered} {\widehat{\mathbf{y}}}_k=\underset{\mathbf{y}\in {\mathbb{R}}^{N_2}}{\text{argmin}}G(\mathbf{y})+\frac{d_k}{2}\Vert \mathbf{y}-{\mathbf{z}}_k{\Vert }_2^2, \\ \triangleq {\text{prox}}_G({\mathbf{z}}_k,d_k) \end{gathered}$$ (13)

with

$${\mathbf{z}}_k={\widehat{\mathbf{y}}}_{k-1}-(1∕d_k){\nabla }_{\mathbf{y}}H({\widehat{\mathbf{x}}}_k,{\widehat{\mathbf{y}}}_{k-1}).$$ (14)

These representations are useful, because for many common regularization penalties, the corresponding proximal operators often have simple closed-form solutions [18], [19].

In summary, the PALM algorithm proceeds according to Alg. 2.

Algorithm 2 PALM for Eq. (5)

$$\begin{array}{cc}\hfill & \mathrm{Initialization}:\text{Set}k=1\text{and initialize}{\widehat{\mathbf{x}}}_0\text{and}{\widehat{\mathbf{y}}}_0.\hfill \\ \hfill & \mathbf{while}\text{not}\text{converge}\mathbf{do}\hfill \\ \hfill & \text{Choose}{c}_k\ge L_1({\widehat{\mathbf{y}}}_{k-1})\hfill \\ \hfill & {\mathbf{w}}_k={\widehat{\mathbf{x}}}_{k-1}-(1∕c_k){\nabla }_{\mathbf{x}}H({\widehat{\mathbf{x}}}_{k-1},{\widehat{\mathbf{y}}}_{k-1})\hfill \\ \hfill & {\widehat{\mathbf{x}}}_k={\text{prox}}_F({\mathbf{w}}_k,c_k)\hfill \\ \hfill & \text{Choose}{d}_k\ge L_2({\widehat{\mathbf{x}}}_k)\hfill \\ \hfill & {\mathbf{z}}_k={\widehat{\mathbf{y}}}_{k-1}-(1∕d_k){\nabla }_{\mathbf{y}}H({\widehat{\mathbf{x}}}_k,{\widehat{\mathbf{y}}}_{k-1})\hfill \\ \hfill & {\widehat{\mathbf{y}}}_k={\text{prox}}_G({\mathbf{z}}_k,d_k)\hfill \\ \hfill & k\leftarrow k+1\hfill \\ \hfill & \mathbf{end}\mathbf{while}\hfill \end{array}$$

Applying PALM to Eq. (3) is nontrivial and, to the best of our knowledge, has never been done before. We will describe our implementation (as well as the enhancements needed for PALMNUT) in the next section.

III. Methods

This section is organized as follows. In Subsection III-A, we first demonstrate how to apply PALM to solve Eq. (3). Next, Subsection III-B will introduce the principles of our novel approach that relies on uncoupled coordinatewise step sizes. Afterwards, Subsection III-C will describe the incorporation of Nesterov’s momentum strategy and describe the full PALMNUT algorithm.

For the sake of concreteness, the remainder of this paper will make the further assumption that the phase regularization term ${\tilde{R}}_2(\mathbf{p})$ can be written as

$${\tilde{R}}_2(\mathbf{p})=R_2(e^{i\mathbf{p}})$$ (15)

for some appropriate penalty function R₂(·), such that we are regularizing the exponentiated phase. This choice is beneficial because it means that p always appears in exponentiated form in every place it appears in the objective function, which will enable simplifications later on. This choice has also been used in previous work [4], [5], [10] because it offers several other benefits. First, the exponentiated phase e^ip is unique, even though the phase vector p is never unique due to the 2π-periodicity of phase. Second, the exponentiated phase e^ip can be spatially-smooth even if the phase p is nonsmooth due to issues associated with phase wrapping. And finally, the spatial derivatives of the exponentiated phase image can be shown to have the same magnitude as the spatial derivatives of the optimally-unwrapped non-exponentiated phase image under common regularity conditions [20], which is important because phase derivatives are often used by regularization strategies that are designed to promote smooth phase [2], [4], [5], [8], [10]. However, it should also be noted that while regularizing the exponentiated phase is often beneficial in cases where phase wrapping has no consequence, it might not be the preferred choice in certain situations where the absolute (unwrapped) phase is important, e.g., in experiments where phase serves as a surrogate for an absolute physical quantity like temperature or velocity.

A. Applying PALM to Eq. (3)

In order to apply PALM to Eq. (3), it may be tempting to associate H(·, ·) with $\tilde{J}(\cdot ,\cdot )$, F(·) with R₁(·), and G(·) with R₂(·) as we had also done for AM. However, it turns out that $\tilde{J}(\cdot ,\cdot )$ does not have favorable Lipschitz bounds for the phase subproblem, and a different approach may be preferred.

To see this, note for the magnitude that

$$\begin{gathered} {\nabla }_{\mathbf{m}}\tilde{J}(\mathbf{m},\mathbf{p})= \\ \Re \left\{e^{-i\mathbf{p}}\odot \left[{\mathbf{A}}^H\mathbf{A}\left(\mathbf{m}\odot e^{i\mathbf{p}}\right)-{\mathbf{A}}^H\mathbf{b}\right]\right\} \end{gathered}$$ (16)

and that

$$\begin{gathered} \Vert {\nabla }_{\mathbf{m}}\tilde{J}({\mathbf{m}}_1,\mathbf{p})-{\nabla }_{\mathbf{m}}\tilde{J}({\mathbf{m}}_2,\mathbf{p}){\Vert }_2\le \\ \Vert \Re \left\{\text{diag}(e^{-i\mathbf{p}}){\mathbf{A}}^H\mathbf{A}\text{diag}(e^{i\mathbf{p}})\right\}\Vert \Vert {\mathbf{m}}_1-{\mathbf{m}}_2{\Vert }_2, \end{gathered}$$ (17)

where $\Re \{\cdot \}$ denotes the operator that extracts the real part of its input, ∥ · ∥ denotes the spectral norm, and diag(e^ip) is the square diagonal matrix whose diagonal entries are equal to the elements of e^ip. Due to the characteristics of spectral norms, Eq. (17) provides a tight Lipschitz bound.

However, for the phase, note that

$$\begin{gathered} {\nabla }_{\mathbf{p}}\tilde{J}(\mathbf{m},\mathbf{p})= \\ \Im \left\{e^{-i\mathbf{p}}\odot \mathbf{m}\odot \left({\mathbf{A}}^H\mathbf{A}\left(\mathbf{m}\odot e^{i\mathbf{p}}\right)-{\mathbf{A}}^H\mathbf{b}\right)\right\}, \end{gathered}$$ (18)

where $\Im \{\cdot \}$ denotes the operator that extracts the imaginary part of its input. Because of the form of this gradient expression (i.e., p appears nonlinearly in Eq. (18)), deriving a good Lipschitz bound for the phase subproblem is nontrivial. We have been partially successful in deriving valid Lipschitz upper bounds for this case (results not shown), but none of the bounds we’ve derived have been anywhere close to tight. This is problematic for the implementation of PALM, because these loose Lipschitz bounds could cause us to choose d_k values that are much larger than necessary, which will result in smaller-than-necessary step sizes in the phase update problem, which will ultimately lead to slow convergence speed.

To avoid this issue, instead of working directly with the original variable p and its exponentiated version e^ip, we will instead consider an equivalent formulation using the change of variables $\mathbf{q}\triangleq e^{i\mathbf{p}}$ under the constraint that all entries of q have magnitude one. This allows us to equivalently express Eq. (3) as

$$\{\widehat{\mathbf{m}},\widehat{\mathbf{q}}\}=\text{arg}\underset{\begin{array}{c}\hfill \mathbf{m}\in {\mathbb{R}}^N\hfill \\ \hfill \mathbf{q}\in \mathcal{V}\hfill \end{array}}{\text{min}}J(\mathbf{m},\mathbf{q})+R_1(\mathbf{m})+R_2(\mathbf{q}),$$ (19)

where

$$J(\mathbf{m},\mathbf{q})\triangleq \frac{1}{2}\Vert \mathbf{A}\left(\mathbf{m}\odot \mathbf{q}\right)-\mathbf{b}{\Vert }_2^2$$ (20)

and

$$\mathcal{V}\triangleq \{\mathbf{q}\in {\mathbb{C}}^N:\mid q_n\mid =1,n=1,2,\dots ,N\}.$$ (21)

Once $\widehat{\mathbf{m}}$ and $\widehat{\mathbf{q}}$ are obtained, the corresponding value of $\widehat{\mathbf{p}}$ can be obtained, if so desired, by computing the angle of each of the entries of q.

This reformulation is beneficial, because the gradients simplify substantially. For the magnitude, we now have that

$${\nabla }_{\mathbf{m}}J(\mathbf{m},\mathbf{q})=\Re \left\{\overline{\mathbf{q}}\odot \left[{\mathbf{A}}^H\mathbf{A}(\mathbf{m}\odot \mathbf{q})-{\mathbf{A}}^H\mathbf{b}\right]\right\}$$ (22)

where $\overline{\mathbf{q}}$ denotes the elementwise complex conjugation of q, which gives the tight Lipschitz bound

$$\begin{gathered} \Vert {\nabla }_{\mathbf{m}}J({\mathbf{m}}_1,\mathbf{q})-{\nabla }_{\mathbf{m}}J({\mathbf{m}}_2,\mathbf{q}){\Vert }_2\le \\ \Vert \Re \left\{\text{diag}(\overline{\mathbf{q}}){\mathbf{A}}^H\mathbf{A}\text{diag}(\mathbf{q})\right\}\Vert \Vert {\mathbf{m}}_1-{\mathbf{m}}_2{\Vert }_2. \end{gathered}$$ (23)

For the phase, we now have that

$${\nabla }_{\mathbf{q}}J(\mathbf{m},\mathbf{q})=\mathbf{m}\odot [{\mathbf{A}}^H\mathbf{A}(\mathbf{m}\odot \mathbf{q})-{\mathbf{A}}^H\mathbf{b}]$$ (24)

which gives the tight Lipschitz bound

$$\begin{gathered} \Vert {\nabla }_{\mathbf{q}}J(\mathbf{m},{\mathbf{q}}_1)-{\nabla }_{\mathbf{q}}J(\mathbf{m},{\mathbf{q}}_2){\Vert }_2\le \\ \Vert \mathbf{A}\text{diag}(\mathbf{m}){\Vert }^2\Vert {\mathbf{q}}_1-{\mathbf{q}}_2{\Vert }_2. \end{gathered}$$ (25)

Although this reformulation simplifies the calculation of Lipschitz constants, the introduction of the constraint set $\mathcal{V}$ makes the optimization problem more complicated. To avoid the need for constrained optimization, we will further rewrite the constrained problem from Eq. (19) in an equivalent unconstrained form using indicator functions [21] as

$$\begin{gathered} \{\widehat{\mathbf{m}},\widehat{\mathbf{q}}\}=\text{arg}\underset{\begin{array}{c}\hfill \mathbf{m}\in {\mathbb{R}}^N\hfill \\ \hfill \mathbf{q}\in {\mathbb{C}}^N\hfill \end{array}}{\text{min}}J(\mathbf{m},\mathbf{q})+R_1(\mathbf{m}) \\ +R_2(\mathbf{q})+{\mathcal{I}}_{\mathcal{V}}(\mathbf{q}), \end{gathered}$$ (26)

where

$${\mathcal{I}}_{\mathcal{V}}(\mathbf{q})\triangleq \{\begin{array}{cc}0,\hfill & \mathbf{q}\in \mathcal{V}\hfill \\ +\infty ,\hfill & \mathbf{q}\notin \mathcal{V}.\hfill \end{array}\phantom{\}}$$ (27)

The new objective function from Eq. (26) now has four terms in it (i.e., J(·, ·), R₁(·), R₂(·), and ${\mathcal{I}}_{\mathcal{V}}(\cdot )$), and to apply PALM, it is necessary to associate these with the PALM terms H(·, ·), F(·), and G(·). The function J(·, ·) is always smooth and involves both m and q, so we will associate it with H(·, ·). The function ${\mathcal{I}}_{\mathcal{V}}(\cdot )$ is always nonsmooth and only involves q, so we will associate it with G(·). While these associations are straightforward (we have no other options), we potentially have options for R₁(·) and R₂(·) depending on their smoothness characteristics. If R₁(·) is nonsmooth, then it has to be associated with F(·), but if it is smooth (Lipschitz) then we could either choose to associate it with F(·) or incorporate it into H(·, ·). Similarly, if R₂(·) is nonsmooth, then it has to be associated with G(·), but if it is smooth (Lipschitz) then we could either choose to associate it with G(·) or incorporate it into H(·, ·).

Although we have different options, the remainder of this paper will assume (for simplicity and without loss of generality) that smooth regularization functions will always be incorporated into H(·, ·). This choice leads to function associations that are summarized in Table I.

TABLE I:

Associations between PALM and Eq. (26) depending on whether R₁(·) and R₂(·) are smooth (Lipschitz).

R1(·) is Smooth?	R2(·) is Smooth?	H(·, ·)	F(·)	G(·)
Y	Y	J(m, q) + R1(m) + R2(q)	0	${\mathcal{I}}_{\mathcal{V}}(\mathbf{q})$
N	Y	J(m, q) + R2(q)	R1(m)	${\mathcal{I}}_{\mathcal{V}}(\mathbf{q})$
Y	N	J(m, q) + R1(q)	0	$R_2(\mathbf{q})+{\mathcal{I}}_{\mathcal{V}}(\mathbf{q})$
N	N	J(m, q)	R1(m)	$R_2(\mathbf{q})+{\mathcal{I}}_{\mathcal{V}}(\mathbf{q})$

With these assignments, the PALM algorithm from Alg. 2 can be directly applied, although it is still necessary to specify the computation of c_k, d_k, prox_F(·, ·), and prox_G(·, ·). Although these computations will necessarily depend on the characteristics of R₁(·) and R₂(·), we will provide concrete illustrations of these calculations for two typical choices of regularization penalties (we will use these same choices of regularization penalties in the validation study presented later in the paper). Of course, these two illustrations do not encompass every possibility, and interested readers are referred to Refs. [18], [19], [22] for further discussion and examples of computating proximal operators. However, it should be noted that the two illustrations below focus on the case where R₂(·) is smooth (the first two rows of Table I), as we have found that it is frequently nontrivial to derive the prox_G(·, ·) operator when G(·) incorporates both R₂(·) and ${\mathcal{I}}_{\mathcal{V}}(\cdot )$ (the last two rows of Table I).

1). Huber-function regularization of m with Tikhonov regularization of q:

For our first illustration, we will consider the case where magnitude regularization takes the form of either

$$R_1(\mathbf{m})={\lambda }_1\sum _{\ell =1}^L{h}_{\xi }([\mathbf{Bm}{]}_{\ell })$$ (28)

or

$$R_1(\mathbf{m})={\lambda }_1\sum _{\ell =1}^L{h}_{\xi }\left(\sqrt{\sum _{t=1}^T\mid [{\mathbf{B}}_t\mathbf{m}{]}_{\ell }{\mid }^2}\right),$$ (29)

and phase regularization takes the form of

$$R_2(\mathbf{q})=\frac{{\lambda }_2}{2}\Vert \mathbf{Cq}{\Vert }_2^2.$$ (30)

In these expressions, λ₁ and λ₂ are positive scalar regularization parameters that can be respectively adjusted to control the influence of the magnitude and phase regularization terms, [z]_ℓ is used to denote the ℓth entry of the vector z, and h_ξ(·) is the Huber function (with parameter ξ > 0) defined as

$$h_{\xi }(t)=\{\begin{array}{cc}\frac{1}{2\xi }\mid t{\mid }^2,\hfill & \mid t\mid \le \xi \hfill \\ \mid t\mid -\frac{1}{2}\xi ,\hfill & \mid t\mid >\xi .\hfill \end{array}\phantom{\}}$$ (31)

The Huber function is a smooth, convex function that is commonly used for both robust statistics (to mitigate the effects of outliers) [23] and for edge/discontinuity/sparsity-preserving image regularization [24], [25]. As can be seen from Eq. (31), the Huber function is similar to the ℓ₁ -norm for large values of its argument, but unlike the ℓ₁-norm, is also smooth at the origin because it behaves like a squared ℓ₂-norm for small values of its argument. The Huber function with a small value of ξ is frequently chosen as a differentiable surrogate for sparsity-promoting ℓ₁-norm regularization, while choosing larger values of ξ can make the Huber function more tolerant to smoothly-varying image regions, more resilient to noise, and easier to characterize theoretically [24], [26], [27].

The regularization in Eq. (28) is based on applying the Huber function to a single transformation $\mathbf{B}\in {\mathbb{C}}^{L\times N}$ (e.g., a wavelet transform, a finite-difference transform, etc.) of the magnitude vector m. The more general regularization in Eq. (29) applies the Huber function to a combination of T different transforms ${\mathbf{B}}_t\in {\mathbb{C}}^{L\times N}$ of m, which can be useful for imposing additional transform-domain structure. For example, combining a horizontal finite-difference transform with a vertical finite-difference transform within Eq. (29) is a common way to achieve isotropic regularization [28]. In addition, Eq. (29) is related to concepts of simultaneous sparsity [29], and our previous work has used regularization penalties of this form to impose the constraint that multi-contrast images of the same anatomy will frequently have correlated edge characteristics [10], [26], [27], [30]. Since Eq. (28) is a special case of Eq. (29) with T = 1, our description below will assume use of the more general form of Eq. (29).

The regularization in Eq. (30) corresponds to standard quadratic (Tikhonov) regularization. If C is chosen as a spatial finite-difference operator, this type of regularization can be good at imposing the constraint that the image phase should be spatially smooth without major discontinuities [2], [4], [5], [7], [10]. While not every MRI image will have smooth phase characteristics, most do, and smooth phase is a common constraint within the image reconstruction literature [31], [32].

For this case, both R₁(·) and R₂(·) are smooth, corresponding to the situation in the first row of Table I. As such, to implement PALM, we use the assignments:

$$\begin{gathered} H(\cdot ,\cdot )\leftarrow \frac{1}{2}\Vert \mathbf{A}(\mathbf{m}\odot \mathbf{q})-\mathbf{b}{\Vert }_2^2 \\ +{\lambda }_1\sum _{\ell =1}^L{h}_{\xi }\left(\sqrt{\sum _{t=1}^T\mid [{\mathbf{B}}_t\mathbf{m}{]}_{\ell }{\mid }^2}\right) \\ +\frac{{\lambda }_2}{2}\Vert \mathbf{Cq}{\Vert }_2^2 \\ F(\cdot )\leftarrow 0 \\ G(\cdot )\leftarrow {\mathcal{I}}_{\mathcal{V}}(\mathbf{q}). \end{gathered}$$ (32)

The gradients of H(·, ·) needed for PALM are

$$\begin{gathered} {\nabla }_{\mathbf{m}}H(\mathbf{m},\mathbf{q})=\Re \{\overline{\mathbf{q}}\odot \left[{\mathbf{A}}^H\mathbf{A}(\mathbf{m}\odot \mathbf{q})-{\mathbf{A}}^H\mathbf{b}\right]\phantom{\}} \\ \phantom{\{}+{\lambda }_1\sum _{t=1}^T{\mathbf{B}}_t^H\mathbf{W}(\mathbf{m}){\mathbf{B}}_t\mathbf{m}\} \end{gathered}$$ (33)

and

$$\begin{gathered} {\nabla }_{\mathbf{q}}H(\mathbf{m},\mathbf{q})=\mathbf{m}\odot \left[{\mathbf{A}}^H\mathbf{A}(\mathbf{m}\odot \mathbf{q})-{\mathbf{A}}^H\mathbf{b}\right] \\ +{\lambda }_2{\mathbf{C}}^H\mathbf{Cq}, \end{gathered}$$ (34)

where W(m) is an L × L diagonal matrix depending on m with ℓth diagonal entry given by

$$[\mathbf{W}(\mathbf{m}){]}_{\ell \ell }=1∕\max \left\{\xi ,\sqrt{\sum _{t=1}^T\mid [{\mathbf{B}}_t\mathbf{m}{]}_{\ell }{\mid }^2}\right\}.$$ (35)

These gradient expressions give rise to Lipschitz-type upper bounds in the form of Eqs. (9) and (10), such that the majorization and descent conditions will be satisfied whenever

$$c_k\ge \Vert \mathbf{A}{\Vert }^2+\frac{{\lambda }_1}{\xi }\Vert \sum _{t=1}^T{\mathbf{B}}_t^H{\mathbf{B}}_t\Vert$$ (36)

and

$$d_k\ge \Vert \mathbf{A}{\Vert }^2\Vert {\widehat{\mathbf{m}}}_k{\Vert }_{\infty }^2+{\lambda }_2\Vert \mathbf{C}{\Vert }^2,$$ (37)

where ∥ · ∥_∞ denotes the ℓ_∞-norm. Please refer to [33]-[35] for insight on obtaining Lipschitz constants for this kind of Huber function. The spectral norms in these expressions are not iteration-dependent, and can be precomputed and reused throughout the iterative process. If it is difficult to analytically calculate the spectral norm values, they can also be evaluated using standard computationally-efficient numerical methods like Lanczos iteration [36].

Finally, the proximal operators needed for PALM are given by

$${\text{prox}}_F({\mathbf{w}}_k,c_k)={\mathbf{w}}_k$$ (38)

and

$${\text{prox}}_G({\mathbf{z}}_k,d_k)=\frac{{\mathbf{z}}_k}{\mid {\mathbf{z}}_k\mid },$$ (39)

where division is performed elementwise and we choose the convention that $\frac{0}{0}=1$.

2). ℓ₁regularization of m with Huber-function regularization of q:

For our second illustration, we will consider the case where

$$R_1(\mathbf{m})={\lambda }_1\Vert \mathbf{Tm}{\Vert }_1$$ (40)

and

$$R_2(\mathbf{q})={\lambda }_2\sum _{\ell =1}^L{h}_{\xi }\left(\sqrt{\sum _{t=1}^T\mid [{\mathbf{B}}_t\mathbf{q}{]}_{\ell }{\mid }^2}\right).$$ (41)

The ℓ₁-norm penalty with sparsifying transform matrix T from Eq. (40) is standard for promoting transform-domain sparsity, and has been previously used to regularize the magnitude vector m in several applications [1], [3], [5]-[9]. The characteristics of the Huber function from Eq. (41) have been discussed previously. By taking a small value of the parameter ξ, the Huber function can be used as a smooth approximation of the ℓ₁-norm in order to enable sparsity-promoting and/or edge-preserving regularization of the phase image, which can useful for some applications with more complicated phase characteristics [5], [8]. As indicated earlier, choosing R₂(·) to be a smooth function and including it within H(·, ·) enables simplified computation of prox_G(·, ·).

In this case, R₁(·) is non-smooth while R₂(·) is smooth, corresponding to the second row of Table I. As such, to implement PALM, we use the assignments:

$$\begin{gathered} H(\cdot ,\cdot )\leftarrow \frac{1}{2}\Vert \mathbf{A}(\mathbf{m}\odot \mathbf{q})-\mathbf{b}{\Vert }_2^2 \\ +{\lambda }_2\sum _{\ell =1}^L{h}_{\xi }\left(\sqrt{\sum _{t=1}^T\mid [{\mathbf{B}}_t\mathbf{q}{]}_{\ell }{\mid }^2}\right) \\ F(\cdot )\leftarrow {\lambda }_1\Vert \mathbf{Tm}{\Vert }_1 \\ G(\cdot )\leftarrow {\mathcal{I}}_{\mathcal{V}}(\mathbf{q}). \end{gathered}$$ (42)

The gradients of H(·, ·) needed for PALM are

$${\nabla }_{\mathbf{m}}H(\mathbf{m},\mathbf{q})=\Re \left\{\overline{\mathbf{q}}\odot \left[{\mathbf{A}}^H\mathbf{A}(\mathbf{m}\odot \mathbf{q})-{\mathbf{A}}^H\mathbf{b}\right]\right\}$$ (43)

and

$$\begin{gathered} {\nabla }_{\mathbf{q}}H(\mathbf{m},\mathbf{q})=\mathbf{m}\odot \left[{\mathbf{A}}^H\mathbf{A}(\mathbf{m}\odot \mathbf{q})-{\mathbf{A}}^H\mathbf{b}\right] \\ +{\lambda }_2\sum _{t=1}^T{\mathbf{B}}_t^H\mathbf{W}(\mathbf{q}){\mathbf{B}}_t\mathbf{q}. \end{gathered}$$ (44)

These gradient expressions give rise to Lipschitz-type upper bounds in the form of Eqs. (9) and (10), such that the majorization and descent conditions will be satisfied whenever

$$c_k\ge \Vert \mathbf{A}{\Vert }^2$$ (45)

and

$$d_k\ge \Vert \mathbf{A}{\Vert }^2\Vert {\widehat{\mathbf{m}}}_k{\Vert }_{\infty }^2+\frac{{\lambda }_2}{\xi }\Vert \sum _{t=1}^T{\mathbf{B}}_t^H{\mathbf{B}}_t\Vert .$$ (46)

As before, the spectral norms in these expressions are not iteration-dependent, and can be precomputed and reused throughout the iterative process.

Finally, assuming that T is a unitary transform such that T^H = T⁻¹, the proximal operator for F(·) is given by [19]

$$\begin{gathered} {\text{prox}}_F({\mathbf{w}}_k,c_k)= \\ {\mathbf{T}}^H\text{diag}\left(\frac{\max \{\mid {\mathbf{Tw}}_k\mid -{\lambda }_1∕c_k,0\}}{\mid {\mathbf{Tw}}_k\mid }\right){\mathbf{Tw}}_k, \end{gathered}$$ (47)

where maximization, absolute value, and division operations are performed elementwise. Note that G(·) is the same as in the previous illustration, and therefore has the same proximal operator (Eq. (39)).

B. Uncoupled Step Sizes

Although the PALM algorithm described in the previous section provides a novel and effective approach for solving Eq. (3), we observe in this section that it may be very conservative and computationally inefficient for PALM to use the same value of d_k (and therefore, the same step size 1/d_k) for all elements of the phase vector p. This inefficiency stems from the fact that d_k is set based on the global Lipschitz constant (effectively, the worst-case rate of change of the gradient along any possible direction), while we have observed that the rate of change of the gradient can be much smaller than the worst-case along specific directions. Concretely, using the global Lipschitz constant means that the step size will depend on the maximum value of m, while we observe that it can be much better for the step size for each coordinate to instead depend on the coordinatewise values of m. This observation motivates us to investigate and utilize coordinatewise bounds on the rate of change of the gradient, enabling uncoupled coordinatewise step sizes. Some readers might view our use of uncoupled step sizes as a form of iteration-dependent preconditioning.

For the sake of generality, we will first describe this approach for the general setting of Section II, where we are given a generic smooth real-valued function H(x, y). The PALM approach utilized majorants of H(x, y) that were derived based on the global (scalar-valued) Lipschitz constants L₁(y) and L₂(x) of Eqs. (9) and (10). In this section, we instead make the assumption that a vector ${\mathbf{L}}_1(\mathbf{y})\in {\mathbb{R}}^{N_1}$ can be found such that, for a fixed value of y, we have

$$\begin{gathered} \langle {\nabla }_{\mathbf{x}}H({\mathbf{x}}_1,\mathbf{y})-{\nabla }_{\mathbf{x}}H({\mathbf{x}}_2,\mathbf{y}),{\mathbf{x}}_1-{\mathbf{x}}_2\rangle \\ \le {\Vert \sqrt{{\mathbf{L}}_1(\mathbf{y})}\odot ({\mathbf{x}}_1-{\mathbf{x}}_2)\Vert }_2^2 \end{gathered}$$ (48)

for ∀x₁, ${\mathbf{x}}_2\in {\mathbb{R}}_1^N$, where the square-root operation is applied elementwise. Similarly, we assume that a vector ${\mathbf{L}}_2(\mathbf{x})\in {\mathbb{R}}^{N_2}$ can be found such that, for a fixed value of x, we have

$$\begin{gathered} \langle {\nabla }_{\mathbf{y}}H(\mathbf{x},{\mathbf{y}}_1)-{\nabla }_{\mathbf{y}}H(\mathbf{x},{\mathbf{y}}_2),{\mathbf{y}}_1-{\mathbf{y}}_2\rangle \\ \le {\Vert \sqrt{{\mathbf{L}}_2(\mathbf{x})}\odot ({\mathbf{y}}_1-{\mathbf{y}}_2)\Vert }_2^2 \end{gathered}$$ (49)

for ∀y₁, ${\mathbf{y}}_2\in {\mathbb{R}}^{N_2}$.

It should be noted that if the global Lipschitz continuity conditions of Eqs. (9) and (10) are known to be satisfied, then a vector L₁(y) satisfying Eq. (48) can be trivially obtained by setting all of its entries equal to L₁(y), with an analogous argument holding true for L₂(x). In particular, the Cauchy-Schwarz inequality combined with Eq. (9) implies that

$$\begin{gathered} \langle {\nabla }_{\mathbf{x}}H({\mathbf{x}}_1,\mathbf{y})-{\nabla }_{\mathbf{x}}H({\mathbf{x}}_2,\mathbf{y}),{\mathbf{x}}_1-{\mathbf{x}}_2\rangle \\ \le \Vert {\nabla }_{\mathbf{x}}H({\mathbf{x}}_1,\mathbf{y})-{\nabla }_{\mathbf{x}}H({\mathbf{x}}_2,\mathbf{y}){\Vert }_2\Vert {\mathbf{x}}_1-{\mathbf{x}}_2{\Vert }_2 \\ \le L_1(\mathbf{y})\Vert {\mathbf{x}}_1-{\mathbf{x}}_2{\Vert }_2^2 \\ ={\Vert \sqrt{{\mathbf{L}}_1(\mathbf{y})}({\mathbf{x}}_1-{\mathbf{x}}_2)\Vert }_2^2. \end{gathered}$$ (50)

However, Eqs. (48) and (49) are more flexible than Eqs. (9) and (10) because a different Lipschitz-like constant can be used for every coordinate, and many of these entries can be potentially much smaller than the global Lipschitz constant (because the function gradient may change much more slowly along these directions).

From an optimization perspective, Eqs. (48) and (49) are important because they enable the use of potentially better majorants than were used by PALM, as described by the following theorem.

Theorem 1. Consider the setting described in Section II, and assume that the smooth real-valued function H(x, y) satisfies the conditions of Eq. (48). Then given a vector ${\mathbf{c}}_k\in {\mathbb{R}}^{N_1}$ and assuming y is held fixed at $\mathbf{y}={\widehat{\mathbf{y}}}_{k-1}$, the function

$$\begin{gathered} H({\widehat{\mathbf{x}}}_{k-1},{\widehat{\mathbf{y}}}_{k-1})+\langle \mathbf{x}-{\widehat{\mathbf{x}}}_{k-1},{\nabla }_{\mathbf{x}}H({\widehat{\mathbf{x}}}_{k-1},{\widehat{\mathbf{y}}}_{k-1})\rangle \\ +\frac{1}{2}\Vert \sqrt{{\mathbf{c}}_k}\odot (\mathbf{x}-{\widehat{\mathbf{x}}}_{k-1}){\Vert }_2^2+F(\mathbf{x})+G({\widehat{\mathbf{y}}}_{k-1}) \end{gathered}$$ (51)

is a majorant of $\Psi (\mathbf{x},{\widehat{\mathbf{y}}}_{k-1})$ at the point $\mathbf{x}={\widehat{\mathbf{x}}}_{k-1}$ whenever ${\mathbf{c}}_k\ge {\mathbf{L}}_1({\widehat{\mathbf{y}}}_{k-1})$ (elementwise). Similarly, given a vector ${\mathbf{d}}_k\in {\mathbb{R}}^{N_2}$, assuming H(x, y) satisfies the conditions of Eq. (49), and assuming x is held fixed at $\mathbf{x}={\widehat{\mathbf{x}}}_k$, the function

$$\begin{gathered} H({\widehat{\mathbf{x}}}_k,{\widehat{\mathbf{y}}}_{k-1})+\langle \mathbf{y}-{\widehat{\mathbf{y}}}_{k-1},{\nabla }_{\mathbf{y}}H({\widehat{\mathbf{x}}}_k,{\widehat{\mathbf{y}}}_{k-1})\rangle \\ +\frac{1}{2}{\Vert \sqrt{{\mathbf{d}}_k}\odot (\mathbf{y}-{\widehat{\mathbf{y}}}_{k-1})\Vert }_2^2+F({\widehat{\mathbf{x}}}_k)+G(\mathbf{y}) \end{gathered}$$ (52)

is a majorant of $\Psi ({\widehat{\mathbf{x}}}_k,\mathbf{y})$ at the point $\mathbf{y}={\widehat{\mathbf{y}}}_{k-1}$ whenever ${\mathbf{d}}_k\ge {\mathbf{L}}_2({\widehat{\mathbf{x}}}_k)$ (elementwise).

The proof of this theorem is given in Appendix A. Although this theorem is stated for real-valued vectors (for consistency with previous descriptions), the same also holds true for complex-valued vectors x and y.

Following the same approach as PALM but replacing the original PALM majorants (Eqs. (7) and (8)) with the new majorants from Thm. 1 results in the new PALM algorithm with uncoupled step sizes given in Alg. 3. This algorithm uses proximal operators with vector-valued c_k and d_k, which we define as

$$\begin{gathered} {\text{prox}}_F({\mathbf{w}}_k,{\mathbf{c}}_k)\triangleq \underset{\mathbf{x}\in {\mathbb{R}}^{N_1}}{\text{arg}\text{min}}F(\mathbf{x}) \\ +\frac{1}{2}\Vert \sqrt{{\mathbf{c}}_k}\odot (\mathbf{x}-{\mathbf{w}}_k){\Vert }_2^2 \end{gathered}$$ (53)

and

$$\begin{gathered} {\text{prox}}_G({\mathbf{z}}_k,{\mathbf{d}}_k)\triangleq \underset{\mathbf{y}\in {\mathbb{R}}^{N_2}}{\text{arg}\text{min}}G(\mathbf{y}) \\ +\frac{1}{2}{\Vert \sqrt{{\mathbf{d}}_k}\odot (\mathbf{y}-{\mathbf{z}}_k)\Vert }_2^2. \end{gathered}$$ (54)

Just like for the original PALM algorithm, the use of valid majorants in our new algorithm guarantees that it will monotonically decrease the objective function value, and that the objective function value converge if Ψ(x, y) is bounded from below. Further, this new algorithm reduces to the original PALM algorithm if the c_k and d_k are treated as scalars (i.e., the values in different entries are always the same) instead of choosing different values in each of the entries. We hypothesize that making additional assumptions about the structure of the cost function (e.g., the Kurdyka-Lojasiewicz property that has been used with the PALM algorithm [11]) may enable a proof that this new algorithm has guaranteed convergence to a critical point, although such a proof is beyond the scope of the present work.

Algorithm 3 PALM for Eq. (5) with Uncoupled Step Sizes

$$\begin{array}{cc}\hfill & \mathrm{Initialization}:\text{Set}k=1\text{and initialize}{\widehat{\mathbf{x}}}_0\text{and}{\widehat{\mathbf{y}}}_0.\hfill \\ \hfill & \mathbf{while}\text{not}\text{converge}\mathbf{do}\hfill \\ \hfill & \text{Choose}{\mathbf{c}}_k\ge {\mathbf{L}}_1({\widehat{\mathbf{y}}}_{k-1})(\text{elementwise})\hfill \\ \hfill & {\mathbf{w}}_k={\widehat{\mathbf{x}}}_{k-1}-\text{diag}({\mathbf{c}}_k{)}^{-1}{\nabla }_{\mathbf{x}}H({\widehat{\mathbf{x}}}_{k-1},{\widehat{\mathbf{y}}}_{k-1})\hfill \\ \hfill & {\widehat{\mathbf{x}}}_k={\text{prox}}_F({\mathbf{w}}_k,{\mathbf{c}}_k)\hfill \\ \hfill & \text{Choose}{\mathbf{d}}_k\ge {\mathbf{L}}_2({\widehat{\mathbf{x}}}_k)(\text{elementwise})\hfill \\ \hfill & {\mathbf{z}}_k={\widehat{\mathbf{y}}}_{k-1}-\text{diag}({\mathbf{d}}_k{)}^{-1}{\nabla }_{\mathbf{y}}H({\widehat{\mathbf{x}}}_k,{\widehat{\mathbf{y}}}_{k-1})\hfill \\ \hfill & {\widehat{\mathbf{y}}}_k={\text{prox}}_G({\mathbf{z}}_k,{\mathbf{d}}_k)\hfill \\ \hfill & k\leftarrow k+1\hfill \\ \hfill & \mathbf{end}\mathbf{while}\hfill \end{array}$$

Now that the general approach has been described, let’s consider the application to the specific magnitude and phase optimization problem of interest from Eq. (3). Without further assumptions about the structure of A, we do not observe any special coordinatewise structure for the magnitude subproblem. As such, we can simply utilize the global Lipschitz constant in this case, setting c_k = c_k1, where 1 is a vector whose entries are all equal to 1 and c_k is the value obtained for the PALM algorithm as described in the previous section.

However, for the phase subproblem with a fixed value of m, we observe that

$$\begin{gathered} \langle {\nabla }_{\mathbf{q}}J(\mathbf{m},{\mathbf{q}}_1)-{\nabla }_{\mathbf{q}}J(\mathbf{m},{\mathbf{q}}_2),{\mathbf{q}}_1-{\mathbf{q}}_2\rangle \le \\ \Vert \Vert \mathbf{A}\Vert \mid \mathbf{m}\mid \odot ({\mathbf{q}}_1-{\mathbf{q}}_2){\Vert }_2^2, \end{gathered}$$ (55)

where the magnitude operation is applied elementwise to the vector m. Thus, for the first illustration from the previous subsection (Huber-function regularization of m with Tikhonov regularization of p), our new algorithm can adopt

$${\mathbf{d}}_k\ge \Vert \mathbf{A}{\Vert }^2\mid {\widehat{\mathbf{m}}}_k{\mid }^2+{\lambda }_2\Vert \mathbf{C}{\Vert }^2$$ (56)

instead of the previous expression from Eq. (37). Similarly, for the second illustration from the previous subsection (ℓ₁-regularization of m with Huber-function regularization of p), our new algorithm can adopt

$${\mathbf{d}}_k\ge \Vert \mathbf{A}{\Vert }^2\mid {\widehat{\mathbf{m}}}_k{\mid }^2+\frac{{\lambda }_2}{\xi }\Vert \sum _{t=1}^T{\mathbf{B}}_t^H{\mathbf{B}}_t\Vert$$ (57)

instead of the previous expression from Eq. (46). Note also that for both of the previous illustrations, the prox_G (·, ·) expressions had no dependence on the actual value of d_k, which allows us to simply reuse the same proximal operators for the new algorithm without modification. As such, for these illustrations, the main difference between the PALM algorithm and our new algorithm is that PALM takes a uniform step size for the phase update that depends on the maximium value of m, while the new algorithm can take larger step sizes for coordinates where the corresponding value of m is small.

C. Nesterov’s Momentum Acceleration and PALMNUT

Our proposed PALMNUT (PALM with Nesterov’s momentum and Uncoupled sTep sizes) algorithm is obtained by combining Alg. 3 with Nesterov’s momentum technique. The basic idea of Nesterov’s technique is that, instead of computing the next iterate ${\widehat{\mathbf{x}}}_k$ based on values derived from ${\widehat{\mathbf{x}}}_{k-1}$, it can be better to instead find the next iterate using values derived from a combination of the previous iterates [12], [37], which can be interpreted as using “momentum” from previous iterations. For convex optimization problems, this approach can even result in convergence rates that have optimal order [12], [37]. Of course, the problem of interest in this work is not convex, although it has been shown empirically (often-times without rigorous theoretical justification) that Nesterov’s momentum technique can often substantially accelerate the iterative solution of nonconvex optimization problems.

The idea of applying momentum to accelerate the convergence of PALM has been studied in Ref. [38], where the resulting algorithm was called inertial PALM (iPALM). Theoretical convergence results for iPALM were proven with restrictive parameter choices (different from Nesterov’s original parameter choices), although it was also shown empirically that using Nesterov’s original parameter choices generally led to much faster convergence, despite the lack of theoretical guarantees. Our empirical experience is also consistent with these previous observations, so our proposed PALMNUT algorithm similarly utilizes Nesterov’s original parameter choices (following the concise form described by Ref. [39]). The final PALMNUT algorithm, incorporating both uncoupled coordinatewise step sizes and Nesterov’s momentum, is given in Alg. 4.

Algorithm 4 PALMNUT

$$\begin{array}{cc}\hfill & \mathrm{Initialization}:\text{Set}k=1\text{and initialize}{\widehat{\mathbf{x}}}_0\text{and}{\widehat{\mathbf{y}}}_0.\hfill \\ \hfill & \text{Set}{\mathbf{u}}_0={\widehat{\mathbf{x}}}_0\text{and}{\mathbf{v}}_0={\widehat{\mathbf{y}}}_0.\hfill \\ \hfill & \mathbf{while}\text{not}\text{converge}\mathbf{do}\hfill \\ \hfill & \text{Choose}{\mathbf{c}}_k\ge {\mathbf{L}}_1({\mathbf{v}}_{k-1})(\text{elementwise})\hfill \\ \hfill & {\mathbf{w}}_k={\mathbf{u}}_{k-1}-\text{diag}({\mathbf{c}}_k{)}^{-1}{\nabla }_{\mathbf{x}}H({\mathbf{u}}_{k-1},{\mathbf{v}}_{k-1})\hfill \\ \hfill & {\widehat{\mathbf{x}}}_k={\text{prox}}_F({\mathbf{w}}_k,{\mathbf{c}}_k)\hfill \\ \hfill & {\mathbf{u}}_k={\widehat{\mathbf{x}}}_k+\frac{k-1}{k+2}({\widehat{\mathbf{x}}}_k-{\widehat{\mathbf{x}}}_{k-1})\hfill \\ \hfill & \text{Choose}{\mathbf{d}}_k\ge {\mathbf{L}}_2({\mathbf{u}}_k)(\text{elementwise})\hfill \\ \hfill & {\mathbf{z}}_k={\mathbf{v}}_{k-1}-\text{diag}({\mathbf{d}}_k{)}^{-1}{\nabla }_{\mathbf{y}}H({\mathbf{u}}_k,{\mathbf{v}}_{k-1})\hfill \\ \hfill & {\widehat{\mathbf{y}}}_k={\text{prox}}_G({\mathbf{z}}_k,{\mathbf{d}}_k)\hfill \\ \hfill & {\mathbf{v}}_k={\widehat{\mathbf{y}}}_k+\frac{k-1}{k+2}({\widehat{\mathbf{y}}}_k-{\widehat{\mathbf{y}}}_{k-1})\hfill \\ \hfill & k\leftarrow k+1\hfill \\ \hfill & \mathbf{end}\mathbf{while}\hfill \end{array}$$

IV. Numerical Experiments

In the following subsections, we evaluate PALMNUT in three different simulations that are representative of a diverse set of real problems in MRI: sparsity-promoting reconstruction of undersampled k-space data [5], [7], [8], regularization-based denoising of a complex image [10], [26], [30], and using phase correction to enable the combination of multiple images acquired in the presence of experimental phase instabilities [10], [40]-[44].

In each of these cases, we compare PALMNUT against AM combined with NCG [2], [4], [5], [7], [10] and AM combined with phase cycling [8]. PALMNUT and AM combined with NCG are directly comparable, because they can both be used to regularize the exponentiated phase e^ip, and therefore can both be applied to the exact same optimization problem. As a result, for both of these algorithms, the phase was regularized with R₂(e^ip) as described previously, and we used the cost function value and the computation time to judge algorithm performance. However, the phase cycling heuristic [8] is intended to be used for the direct regularization of p. For this algorithm, we therefore had to instead use phase regularization of the form R₂(p). In addition, by its nature, the phase cycling heuristic employs a different phase-cycled cost function at each iteration, which makes it difficult to plot a meaningful cost function value. To allow for comparisons with this different approach, we therefore also computed a normalized root-mean-squared error (NRMSE) metric for each algorithm. If f represents the ground truth vector and $\widehat{\mathbf{f}}$ represents an estimate, the NRMSE of the estimate is given by

$$\text{NRMSE}\triangleq \Vert \widehat{\mathbf{f}}-\mathbf{f}{\Vert }_2∕\Vert \mathbf{f}{\Vert }_2.$$ (58)

In order to focus on consequential errors, the NRMSE values were always computed after masking out the empty (background) parts of the field-of-view. This was particularly beneficial for AM with phase cycling, which frequently showed higher error levels in the image background compared to the other two approaches.

Regularization parameters for each cost function in each scenario were empirically optimized to achieve the smallest possible final NRMSE values for the two AM algorithms. Our implementation of AM with NCG used the Polak-Ribiere version of NCG [45]. Our implementation of AM with phase cycling was based on code provided by the authors of Ref. [8] (available from https://mrirecon.github.io/bart/).

A. Undersampled MRI Reconstruction

In the first set of evaluations, we considered the reconstruction of an MR image from 8×-undersampled k-space data. The gold-standard magnitude and phase images, which were obtained from a real fully-sampled in vivo T1-weighted MRI acquisition with 256 × 256 in-plane matrix size, are shown in Fig. 1. This figure also shows the k-space sampling mask (corresponding to 8× undersampling) that we used to simulate an accelerated acquisition.

Fig. 1:

The ground truth (a) magnitude and (b) phase images used for the undersampled MRI reconstruction scenario, along with (c) the 8×-accelerated k-space sampling mask used to simulate the undersampled acquisition.

For this simulation, the original fully-sampled k-space data (originally measured with 32 channels) was coil-compressed down to 8 virtual channels to reduce computational complexity, and was then retrospectively undersampled using the afore-mentioned k-space sampling mask. For reconstruction, the A matrix was chosen according to the standard SENSE model [46], with sensitivity maps estimated using ESPIRiT [47]. The magnitude regularization took the form of an ℓ₁ penalty as given by Eq. (40), where, following Ref. [8], the sparsifying transform T was chosen to be the unitary Daubechies-4 wavelet transform. The phase regularization took the form of a Huber-function penalty as given by Eq. (41), where the Huber parameter ξ was chosen to be a small number (i.e., ξ = 0.001) in order to approximate the ℓ₁-norm. Following Ref. [8], the transform we used for phase regularization was also a unitary Daubechies-4 wavelet transform. All three algorithms were initialized by applying SENSE-based coil-combination to the multi-channel images obtained by zero-filling the unmeasured data.

Convergence results for all three algorithms are shown in Fig. 2, along with representative image reconstruction results. As can be seen, the cost function and NRMSE values converge to similar levels for both PALMNUT and AM with NCG, although PALMNUT converged much faster. Although AM with phase cycling converged to a result with a reasonably-good NRMSE, it was substantially worse than PALMNUT in both convergence speed and the final achieved NRMSE value.

Fig. 2:

Convergence plots and reconstructed images for the undersampled MRI reconstruction scenario. The convergence plots show (a) the cost function value as a function of computation time and (b) the NRMSE value for the complex image as a function of computation time. Also shown are the magnitude and phase images corresponding to (c) zero-filled reconstruction and (d) PALMNUT.

B. Regularization-based MRI Denoising

In the second set of evaluations, we considered regularization-based denoising of a 230×180 single-channel T1-weighted MR image, obtained by applying complex coil-combination to an 8-channel dataset and subsequently adding simulated complex Gaussian noise. The ground truth and noisy images are shown in Fig. 3.

Fig. 3:

The (a) ground truth and (b) noisy magnitude and phase images used for the regularization-based MRI denoising scenario.

For reconstruction, the A matrix was an identity matrix. Following Refs. [10], [26], [27], [30], the magnitude was regularized using a Huber-function penalty as given by Eq. (28), where a finite difference transformation was used to enforce spatial smoothness of the image. Following Ref. [2], [4], [5], [7], [10], the phase was regularized using a Tikhonov penalty as given by Eq. (29), also using a finite difference transformation to enforce spatial smoothness. All algorithms were initialized with the noisy image.

Convergence results are shown in Fig. 4, along with representative reconstruction results. As can be seen, the results in this case are consistent with the previous case: PALMNUT and AM with NCG converged to similar NRMSE values, although PALMNUT was substantially faster, while AM with phase cycling was reasonably successful yet still substantially worse than the others in both speed and NRMSE.

Fig. 4:

Convergence plots and reconstructed images for the regularization-based MRI denoising scenario. The convergence plots show (a) the cost function value as a function of computation time and (b) the NRMSE value for the complex image as a function of computation time. Also shown are the (c) magnitude and phase images corresponding to PALMNUT.

C. Phase-Corrected MR Image Combination

In the third experiment, we simulated a scenario that is common in diffusion MRI, in which multiple measurements are made of the same image to enable averaging to improve SNR, but the phase varies randomly with each measurement due to experimental instabilities. This case was simulated based on actual diffusion MRI magnitude and phase data from Ref. [10]. We simulated a case with four repetitions, based on one ground truth magnitude image and four ground truth phase images, all with matrix size 180 × 332. The magnitude image was combined with each of the different phase images to yield four different complex images, and then complex Gaussian noise was added to each result. The ground truth images and a representative noisy image are shown in Fig. 5.

Fig. 5:

(a) The ground truth magnitude and (b) a representative noisy magnitude image, along with (c) the four different ground truth phase images used for the scenario with phase-corrected combination of multiple images.

For reconstruction, we estimated a single shared magnitude image m and four different phase images p_j corresponding to the four different noisy measured images b_j, j = 1, 2, 3, 4, by minimizing the cost function

$$R_1(\mathbf{m})+\sum _{j=1}^4\frac{1}{2}\Vert \mathbf{m}\odot e^{i{\mathbf{p}}_j}-{\mathbf{b}}_j{\Vert }_2^2+R_2(e^{i{\mathbf{p}}_j}),$$ (59)

where the magnitude and phase regularization penalties R₁ (·) and R₂(·) were chosen in exactly the same way as for the denoising scenario from the previous subsection. All optimization algorithms were initialized with the noisy phase images and by taking the average of the noisy magnitude images.

The convergence plots shown in Fig. 6 show similar characteristics to those observed in the previous scenarios, with PALMNUT having a distinct advantage over the two alternative algorithms.

Fig. 6:

Convergence plots and reconstructed images for the phase-corrected MRI image combination scenario. The convergence plots show (a) the cost function value as a function of computation time and (b) the NRMSE value as a function of computation time. Also shown is (c) the combined magnitude image obtained from PALMNUT.

V. Discussion

The results of the previous section demonstrated that PALMNUT can have major advantages relative to AM methods. However, PALMNUT represents a combination of three different ideas (PALM, Nesterov’s momentum, and uncoupled stepsizes), and the previous experiments did not investigate which parts of PALMNUT contribute most to its performance. In order to gain more insight, we did another set of experiments in which we compared PALMNUT against the original PALM algorithm (Alg. 2), PALM with Nesterov’s momentum but without uncoupled stepsizes (iPALM)³, and PALM with uncoupled stepsizes but without Nesterov’s momentum (Alg. 3). Results are shown in Fig. 7 for both the undersampled MRI reconstruction (Section IV-A) and the regularization-based MRI denoising (Section IV-B) scenarios described previously. From these plots, we observe that both uncoupled step sizes and Nesterov’s momentum independently improve the convergence speed relative to the original PALM method, with Nesterov’s momentum contributing a little more than the use of uncoupled step sizes. However, PALMNUT’s combination of all of these elements leads to the best overall results (i.e., reaching small cost function values sooner than the other algorithms and achieving the fastest empirical convergence of the cost function value).

Fig. 7:

The convergence plots show the cost function value and the NRMSE value as a function of computation time for (a) the undersampled MRI reconstruction scenario and (b) the regularization-based MRI denoising scenario.

Interestingly, it was also observed that the NRMSE curves for PALMNUT and iPALM followed similar trends in this set of experiments, and that the NRMSE values appeared to converge sooner than the cost function values did. This is likely due to the well-known fact that NRMSE can be insensitive to some kinds of image features [48], although it should be clear from the cost function behavior that the image estimates are still changing even after the NRMSE curves appear to have converged. In addition, it is important to keep in mind that NRMSE values cannot generally be computed in practical applications, and unlike the cost function behavior, the NRMSE behavior could not be easily used as an algorithm stopping criterion.

An interesting phenomenon we observed with PALMNUT is that the magnitude estimate frequently converged faster than the phase estimate (results not shown). As a result, it could be potentially beneficial to update the phase estimate more frequently than the magnitude estimate, although we believe that such an exploration is beyond the scope of this paper.

One of the key ingredients of PALMNUT is the use of coordinatewise step sizes based on coordinatewise Lipschitz-like bounds in the form of Eqs. (48) and (49). Although these bounds were formulated in the context of PALM, we believe that Lemma 1 (found in the Appendix and used in the proof of Theorem 1) represents a novel coordinatewise majorization relationship, which could also potentially be useful to construct majorants in more general optimization scenarios.

Finally, we should mention that while PALM combined with uncoupled step sizes has guaranteed monotonic convergence properties, the convergence of the PALMNUT approach (which additionally includes Nesterov’s momentum) has not yet been theoretically proven. This represents another potentially interesting topic for future research.

VI. Conclusion

We proposed and evaluated a new algorithm called PALMNUT, which combines the PALM algorithm with Nesterov’s momentum with uncoupled coordinatewise step sizes derived from coordinatewise Lipschitz-like bounds. Although our approach is general and can be applied to other computational imaging scenarios, our evaluation studies focused on MRI scenarios involving separate regularization of the image magnitude and phase. Applying algorithms like PALM and PALMNUT to this problem also required us to reformulate this optimization problem in a novel way that is more compatible with them than the standard formulation. our empirical results demonstrated that PALMNUT consistently had substantial advantages over previous approaches based on alternating minimization across several different MRI scenarios. As a result, we expect that PALMNUT will be useful for these kinds of MRI scenarios, and may also prove useful for more general computational imaging problems with similar optimization structure.

Appendix A

proof of Thm. 1

Theorem 1 is an immediate consequence of the following Lemma.

Lemma 1. Let $Q(\mathbf{x}):{\mathbb{R}}^N\to \mathbb{R}$ be smooth, and assume that a non-negative vector $\mathbf{L}\in {\mathbb{R}}^N$ exists such that

$$\langle \nabla Q({\mathbf{x}}_1)-\nabla Q({\mathbf{x}}_2),{\mathbf{x}}_1-{\mathbf{x}}_2\rangle \le {\Vert \sqrt{\mathbf{L}}\odot ({\mathbf{x}}_1-{\mathbf{x}}_2)\Vert }_2^2$$ (60)

for ∀x₁, ${\mathbf{x}}_2\in {\mathbb{R}}^N$. Then for ∀x, ${\widehat{\mathbf{x}}}_k\in {\mathbb{R}}^N$ and ∀c ≥ L (elementwise), we have

$$Q(\mathbf{x})\le Q({\widehat{\mathbf{x}}}_k)+\langle \mathbf{x}-{\widehat{\mathbf{x}}}_k,\nabla Q({\widehat{\mathbf{x}}}_k)\rangle +\frac{1}{2}\Vert \sqrt{\mathbf{c}}\odot (\mathbf{x}-{\widehat{\mathbf{x}}}_k){\Vert }_2^2.$$ (61)

Further, the right hand side of Eq. (61) can be written compactly as

$$Q({\widehat{\mathbf{x}}}_k)+\langle \mathbf{x}-{\widehat{\mathbf{x}}}_k,\nabla Q({\widehat{\mathbf{x}}}_k)\rangle +\frac{1}{2}\Vert \sqrt{\mathbf{c}}\odot (\mathbf{x}-{\widehat{\mathbf{x}}}_k){\Vert }_2^2={\tau }_k+\frac{1}{2}\Vert \sqrt{\mathbf{c}}\odot (\mathbf{x}-{\mathbf{w}}_k){\Vert }_2^2,$$ (62)

where

$${\mathbf{w}}_k={\widehat{\mathbf{x}}}_k-\text{diag}(\mathbf{c}{)}^{-1}\nabla Q({\widehat{\mathbf{x}}}_k)$$ (63)

and τ_k is a constant that does not depend on the variable x, given by

$${\tau }_k=Q({\widehat{\mathbf{x}}}_k)-\langle {\widehat{\mathbf{x}}}_k,\nabla Q({\widehat{\mathbf{x}}}_k)\rangle +\frac{1}{2}\Vert \sqrt{\mathbf{c}}\odot {\widehat{\mathbf{x}}}_k{\Vert }_2^2-\frac{1}{2}\Vert \sqrt{\mathbf{c}}\odot {\mathbf{w}}_k{\Vert }_2^2.$$ (64)

Proof. Inspired by the proof of Proposition A.24 from [49], define $\alpha (t)\triangleq Q(t\mathbf{x}+(1-t){\widehat{\mathbf{x}}}_k)$. Then $\alpha (0)=Q({\widehat{\mathbf{x}}}_k)$, α(1) = Q(x), and

$$\frac{d}{dt}\alpha (t)=\langle \nabla Q(t\mathbf{x}+(1-t){\widehat{\mathbf{x}}}_k),\mathbf{x}-{\widehat{\mathbf{x}}}_k\rangle .$$ (65)

We have that

$$\begin{gathered} Q(\mathbf{x})-Q({\widehat{\mathbf{x}}}_k)={\int }_0^1\frac{d}{dt}\alpha (t)dt \\ ={\int }_0^1\langle \nabla Q(t\mathbf{x}+(1-t){\widehat{\mathbf{x}}}_k),\mathbf{x}-{\widehat{\mathbf{x}}}_k\rangle dt \\ ={\int }_0^1\langle \nabla Q(t\mathbf{x}+(1-t){\widehat{\mathbf{x}}}_k)-\nabla Q({\widehat{\mathbf{x}}}_k),\mathbf{x}-{\widehat{\mathbf{x}}}_k\rangle dt+\langle \nabla Q({\widehat{\mathbf{x}}}_k),\mathbf{x}-{\widehat{\mathbf{x}}}_k\rangle \\ ={\int }_0^1\frac{1}{t}\langle \nabla Q(t\mathbf{x}+(1-t){\widehat{\mathbf{x}}}_k)-\nabla Q({\widehat{\mathbf{x}}}_k),t\mathbf{x}-t{\widehat{\mathbf{x}}}_k\rangle dt+\langle \mathbf{x}-{\widehat{\mathbf{x}}}_k,\nabla Q({\widehat{\mathbf{x}}}_k)\rangle \\ ={\int }_0^1\frac{1}{t}\langle \nabla Q(t\mathbf{x}+(1-t){\widehat{\mathbf{x}}}_k)-\nabla Q({\widehat{\mathbf{x}}}_k),t\mathbf{x}+(1-t){\widehat{\mathbf{x}}}_k-{\widehat{\mathbf{x}}}_k\rangle dt+\langle \mathbf{x}-{\widehat{\mathbf{x}}}_k,\nabla Q({\widehat{\mathbf{x}}}_k)\rangle \\ \le {\int }_0^1\frac{1}{t}{\Vert \sqrt{\mathbf{L}}\odot (t\mathbf{x}+(1-t){\widehat{\mathbf{x}}}_k-{\widehat{\mathbf{x}}}_k)\Vert }_2^{2}dt+\langle \mathbf{x}-{\widehat{\mathbf{x}}}_k,\nabla Q({\widehat{\mathbf{x}}}_k)\rangle \\ ={\int }_0^{1}t{\Vert \sqrt{\mathbf{L}}\odot (\mathbf{x}-{\widehat{\mathbf{x}}}_k)\Vert }_2^{2}dt+\langle \mathbf{x}-{\widehat{\mathbf{x}}}_k,\nabla Q({\widehat{\mathbf{x}}}_k)\rangle \\ =\frac{1}{2}{\Vert \sqrt{\mathbf{L}}\odot (\mathbf{x}-{\widehat{\mathbf{x}}}_k)\Vert }_2^2+\langle \mathbf{x}-{\widehat{\mathbf{x}}}_k,\nabla Q({\widehat{\mathbf{x}}}_k)\rangle \\ \le \frac{1}{2}\Vert \sqrt{\mathbf{c}}\odot (\mathbf{x}-{\widehat{\mathbf{x}}}_k){\Vert }_2^2+\langle \mathbf{x}-{\widehat{\mathbf{x}}}_k,\nabla Q({\widehat{\mathbf{x}}}_k)\rangle . \end{gathered}$$ (66)

This derivation proves Eq. (61), while the simplifications leading to Eq. (62) come simply from completing the square. □