Momentum-Net: Fast and convergent iterative neural network for inverse problems

Abstract

Iterative neural networks (INN) are rapidly gaining attention for solving inverse problems in imaging, image processing, and computer vision. INNs combine regression NNs and an iterative model-based image reconstruction (MBIR) algorithm, often leading to both good generalization capability and outperforming reconstruction quality over existing MBIR optimization models. This paper proposes the first fast and convergent INN architecture, Momentum-Net, by generalizing a block-wise MBIR algorithm that uses momentum and majorizers with regression NNs. For fast MBIR, Momentum-Net uses momentum terms in extrapolation modules, and noniterative MBIR modules at each iteration by using majorizers, where each iteration of Momentum-Net consists of three core modules: image refining, extrapolation, and MBIR. Momentum-Net guarantees convergence to a fixed-point for general differentiable (non)convex MBIR functions (or data-fit terms) and convex feasible sets, under two asymptomatic conditions. To consider data-fit variations across training and testing samples, we also propose a regularization parameter selection scheme based on the “spectral spread” of majorization matrices. Numerical experiments for light-field photography using a focal stack and sparse-view computational tomography demonstrate that, given identical regression NN architectures, Momentum-Net significantly improves MBIR speed and accuracy over several existing INNs; it significantly improves reconstruction quality compared to a state-of-the-art MBIR method in each application.

1. INTRODUCTION

Deep regression neural network (NN) methods have been actively studied for solving diverse inverse problems, due to their effectiveness at mapping noisy signals into clean signals. Examples include image denoising [1]–[4], image deconvolution [5], [6], image super-resolution [7], [8], magnetic resonance imaging (MRI) [9], [10], X-ray computational tomography (CT) [11]–[13], and light-field (LF) photography [14], [15]. However, regression NNs with a greater mapping capability have increased overfitting/hallucination risks [16]–[19]. An alternative approach to solving inverse problems is an iterative NN (INN) that combines regression NNs – called “refiners” or denoisers – with an unrolled iterative model-based image reconstruction (MBIR) algorithm [20]–[27]. This alternative approach can regulate overfitting of regression NNs, by balancing physical data-fit of MBIR and prior information estimated by refining NNs [16], [18]. This “soft-refiner” approach has been successfully applied to several extreme imaging systems, e.g., highly undersampled MRI [20], [25], [28]–[30], low-dose or sparse-view CT [16], [19], [24], [27], [31], and low-count emission tomography [18], [32]–[34].

1.1. Notation

This section provides mathematical notations. We use f(x; y) to denote a function f of x given y. We use ∥·∥_p to denote the ℓ^p-norm and write ⟨·,·⟩ for the standard inner product on ${ℂ}^N$. The weighted ℓ²-norm with a Hermitian positive definite matrix A is denoted by $\Vert \cdot {\Vert }_A={\Vert A^{\frac{1}{2}}(\cdot )\Vert }_2$. The Frobenius norm of a matrix is denoted by ∥ · ∥_F. (·)^T, (·)^H, and (·)^∗ indicate the transpose, complex conjugate transpose (Hermitian transpose), and complex conjugate, respectively. diag(·) denotes the conversion of a vector into a diagonal matrix or diagonal elements of a matrix into a vector. For (self-adjoint) matrices $A,B\in {ℂ}^{N\times N}$, the notation B ⪯ A denotes that A − B is a positive semidefinite matrix.

1.2. From block-wise optimization to INN

To recover signals $x\in {ℂ}^N$ from measurements $y\in {ℂ}^m$, consider the following MBIR optimization problem:

$$\underset{x\in \mathcal{X}}{\mathrm{argmin}}F(x;y,z),F(x;y,z)\triangleq f(x;y)+\frac{\gamma }{2}\Vert x-z{\Vert }_2^2,$$ (P0)

where $\mathcal{X}$ is a set of feasible points, f(x; y) is data-fit function, γ is a regularization parameter, and $z\in {ℂ}^N$ is some high-quality approximation to the true unknown signal x. The data-fit f(x; y) measures deviations of model-based predictions of x from data y, considering models of imaging physics (or image formation) and noise statistics in y. In (P0), the signal recovery accuracy increases as the quality of z improves [17, Prop. 3]; however, it is difficult to obtain such z in practice. Alternatively, there has been a growing trend in learning sparsifying regularizers (e.g., convolutional regularizers [24], [35]–[38]) from training datasets and applying the trained regularizers to the following block-wise MBIR problem: ${\mathrm{argmin}}_{x\in \mathcal{X}}f(x;y)+{\min }_{\zeta }r(x,\zeta ;\mathcal{O})$. Here, a learned regularizer ${\min }_{\zeta }r(x,\zeta ;\mathcal{O})$ quantifies consistency between x and refined sparse signal ζ via some learned operators $\mathcal{O}$. Recently, we have constructed INNs by generalizing the corresponding block-wise MBIR updates with regression NNs without convergence analysis [25], [27]. In existing INNs, two major challenges exist: convergence and acceleration.

1.3. Challenges in existing INNs: Convergence

Existing convergence analysis has some practical limitations. The original form of plug-and-play (PnP [23], [39]–[41]) is motivated by the alternating direction method of multipliers (ADMM [42]), and its fixed-point convergence has been analyzed with consensus equilibrium perspectives [23]. However, similar to ADMM, its practical convergence depends on how one selects ADMM penalty parameters. For example, [22] reported unstable convergence behaviors of PnP-ADMM with fixed ADMM parameters. To moderate this problem, [41] proposed a scheme that adaptively controls the ADMM parameters based on relative residuals. Similar to the residual balancing technique [42, §3.4.1], the scheme in [41] requires tuning initial parameters. Regularization by Denoising (RED [22]) is an alternative that moderates some such limitations. In particular, RED aims to make a clear connection between optimization and a denoiser $\mathcal{D}$, by defining its prior term by (scaled) $x^T(x-\mathcal{D}(x))$. Nonetheless, [43] showed that many practical denoisers do not satisfy the Jacobian symmetry in [22], and proposed a less restrictive method, score-matching by denoising.

The convergence analysis of the INN inspired by the relaxed projected gradient descent (RPGD) method in [31] has the least restrictive conditions on the regression NN among the existing INNs. This method replaces the projector of a projected gradient descent method with an image refining NN. However, the RPGD-inspired INN directly applies an image refining NN to gradient descent updates of data-fit; thus, this INN relies heavily on the mapping performance of a refining NN and can have overfitting risks, similar to non-MBIR regression NNs, e.g., FBPConvNet [12]. In addition, it exploits the data-fit term only for the first few iterations [31, Fig. 5(c)]. We refer the perspective used in RPGD-inspired INN and its related works [26], [44] as “hard-refiner”: different from soft-refiners, these methods do not use a refining NN as a regularizer. More recently, [26] presented convergence analysis for an INN inspired by a proximal gradient descent method. However, their analysis is based on noiseless measurements, which is typically impractical.

Fig. 5.

RMSE minimization comparisons between different INNs for sparse-view CT (fan-beam geometry with 12.5% projections views and 10⁵ incident photons; averaged RMSE values across two test reconstructed images).

Broadly speaking, existing convergence analysis largely depends on the (firmly) nonexpansive property of image refining NNs [22], [23], [43], [31, PGD], [26]. However, except for a single-hidden layer convolutional NN (CNN), it is yet unclear which analytical conditions guarantee the non-expansiveness of general refining NNs [27]. To guarantee convergence of INNs even when using possibly expansive image refining NNs, we proposed a method that normalizes the output signal of image refining NNs by their Lipschitz constants [27]. However, if one uses expansive NNs that are identical across iterations, it is difficult to obtain “best” image recovery with that normalization scheme. The spectral normalization based training [45], [46] can ensure the non-expansiveness of refining NNs by single-step power iteration. However, similar to the normalization method in [27], refining NNs trained with the spectral normalization method [46] degraded the image reconstruction accuracy for an INN using iteration-wise refining NNs [19]. In addition, there does not yet exist theoretical convergence results when refining NNs change across iterations, yet iteration-wise refining NNs are widely studied [20], [21], [25], [28]. Finally, existing analysis considers only a narrow class of data-fit terms: most analyses consider a quadratic function with a linear imaging model [26], [31] or more generally, a convex cost function [23], [43], [46] that can be minimized with a practical closed-form solution. No theoretical convergence results exist for general (non)convex data-fit terms, iteration-wise NN denoisers, and a general set of feasible points.

1.4. Challenges in existing INNs: Acceleration

Compared to non-MBIR regression NNs that do not exploit the data-fit f(x; y) in (P0), INNs require more computation because they consider the imaging physics. Computation increases as the imaging system or image formation model becomes larger-scale, e.g., LF photography from a focal stack, 3D CT, parallel MRI using many receive coils, and image super-resolution. Thus, acceleration becomes crucial for INNs.

First, consider the existing methods motivated by ADMM or block coordinate descent (BCD) method: examples include PnP-ADMM [23], [41], RED-ADMM [22], [43], MoDL [30], BCD-Net [16], [18], [25], etc. These methods can require multiple inner iterations to balance data-fit and prior information estimated by trained refining NNs, increasing total MBIR time. For example, in solving such problems, each outer iteration involves x⁽ⁱ⁺¹⁾ = argmin_x F(x; y, z⁽ⁱ⁺¹⁾), where F is given as in (P0) and z⁽ⁱ⁺¹⁾ is the output from the ith image refining NN. For LF imaging system using a focal stack data [47], solving the above problem requires multiple iterations, and the total computational cost scale with the numbers of photosensors and sub-aperture images. In addition, nonconvexity of the data-fit term f(x; y) can break convergence guarantees of these methods, because in general, the proximal mapping ${\mathrm{argmin}}_{x}f(x;y)+\gamma {\Vert x-z^{(i+1)}\Vert }_2^2$ is no longer nonexpansive.

Second, consider the existing works motivated by gradient descent methods [21], [26], [28], [31]. These methods resolve the inner iteration issue; however, they lack a sophisticated step-size control or backtracking scheme that influences convergence guarantee and acceleration. Accelerated proximal gradient (APG) methods using momentum terms can significantly accelerate convergence rates for solving composite convex problems [48], [49], so we expect that INN methods in the second class have yet to be maximally accelerated. The work in [44] applied PnP to the APG method [49]; [50] applied PnP to the primal-dual splitting (PDS) algorithm [51]. However, similar to RPGD [31], these are hard-refiner methods using some state-of-the-art denoisiers (e.g., BM3D [52]) but not trained NNs. Those methods lack convergence analyses and guarantees may be limited to convex data-fit function.

1.5. Contributions and organization of the paper

This paper proposes Momentum-Net, the first INN architecture that aims for fast and convergent MBIR. The architecture of Momentum-Net is motivated by applying the Block Proximal Extrapolated Gradient method using a Majorizer (BPEG-M) [24], [35] to MBIR using trainable convolutional autoencoders [24], [25], [37]. Specifically, each iteration of Momentum-Net consists of three core modules: image refining, extrapolation, and MBIR. At each Momentum-Net iteration, an extrapolation module uses momentum from previous updates to amplify the changes in subsequent iterations and accelerate convergence, and an MBIR module is noniterative. In addition, Momentum-Net resolves the convergence issues mentioned in §1.3: for general differentiable (non)convex data-fit terms and convex feasible sets, it guarantees convergence to a point that satisfies fixed-point and critical point conditions, under some mild conditions and two asymptotic conditions, i.e., asymptotically nonexpansive paired refining NNs and asymptotically block-coordinate minimizer.

The remainder of this paper is organized as follows. §2 constructs the Momentum-Net architecture motivated by BPEG-M algorithm that solves MBIR problem using a learnable convolutional regularizer, describes its relation to existing works, analyzes its convergence, and summarizes the benefits of Momentum-Net over existing INNs. §3 provides details of training INNs, including image refining NN architectures, single-hidden layer or “shallow” CNN (sCNN) and multi-hidden layer or “deep” CNN (dCNN), and training loss function, and proposes a regularization parameter selection scheme to consider data-fit variations across training and testing samples. §4 considers two extreme imaging applications: sparse-view CT and LF photography using a focal stack. §4 reports numerical experiments of applications where the proposed Momentum-Net using extrapolation significantly improves MBIR speed and accuracy, over the existing INNs, BCD-Net [22], [25], [30], Momentum-Net using no extrapolation [21], [28], ADMM-Net [20], [23], [41], and PnP-PDS [50] using refining NNs. Furthermore, §4 reports numerical experiments where Momentum-Net significantly improves reconstruction quality compared to a state-of-the-art MBIR method in each application.

2. MOMENTUM-NET: WHERE BPEG-M MEETS NNS FOR INVERSE PROBLEMS

2.1. Motivation: BPEG-M algorithm for MBIR using learnable convolutional regularizer

This section motivates the proposed Momentum-Net architecture, based on our previous works [24], [37]. Consider the following approach for recovering signal x from measurements y (see the setup of block multi-(non)convex problems in §A.1.1):

$$\begin{gathered} \underset{x\in \mathcal{X}}{\mathrm{argmin}}f(x;y)+\gamma \left(\underset{\left\{{\zeta }_k\right\}}{\min }r\left(x,\left\{{\zeta }_k\right\};\left\{h_k\right\}\right)\right), \\ r\left(x,\left\{{\zeta }_k\right\};\left\{h_k\right\}\right)\triangleq \sum _{k=1}^K\frac{1}{2}{\Vert h_k\ast x-{\zeta }_k\Vert }_2^2+{\beta }_k{\Vert {\zeta }_k\Vert }_1, \end{gathered}$$ (1)

where $\mathcal{X}$ is a closed set, f(x; y) + γr(x, {ζ_k}; {h_k}) is a (continuosly) differentiable (non)convex function in x, ${\min }_{\left\{{\zeta }_k\right\}}r\left(x,\left\{{\zeta }_k\right\};\left\{h_k\right\}\right)$ is a learnable convolutional regularizer [24], [36], {ζ_k : k=1, …, K} is a set of sparse features that correspond to {h_k ∗x}, $\left\{h_k\in {ℂ}^R:k=1,\dots ,K\right\}$ is a set of trainable filters, and R and K denote the size and number of trained filters, respectively.

Problem (1) can be viewed as a two-block optimization problem in terms of the image x and the features {ζ_k}. We solve (1) using the recent BPEG-M optimization framework [24], [35] that has attractive convergence guarantee and rapidly solved several block optimization problems [24], [35], [53]–[55]. BPEG-M has the following key ideas for each block optimization problem (see details in §A.1):

• M_b-Lipschitz continuity for the gradient of the bth block optimization problem, ∀b:

  Definition 1 (M-Lipschitz continuity [24]).

  A function $g:{ℝ}^n\to {ℝ}^n$ is M-Lipschitz continuous on ${ℝ}^n$ if there exists a (symmetric) positive definite matrix M such that

  $$\Vert g(u)-g(v){\Vert }_{M^{-1}}\le \Vert u-v{\Vert }_M,\forall u,v\in {ℝ}^n.$$

  Definition 1 is a more general concept than the classical Lipschitz continuity.

• A sharper majorization matrix M that gives a tighter bound in Definition 1 leads to a tighter quadratic majorization bound in the following lemma:

  Lemma 2 (Quadratic majorization via M-Lipschitz continuous gradients [24]). Let $f(u):{ℝ}^n\to ℝ$. If ∇f is M-Lipschitz continuous, then

  $$f(u)\le f(v)+\langle {\nabla }_{u}f(v),u-v\rangle +\frac{1}{2}\Vert u-v{\Vert }_M^2,\forall u,v\in {ℝ}^n.$$

  Having tighter majorization bounds, sharper majorization matrices tend to accelerate BPEG-M convergence.

• The majorized block problems are “proximable”, i.e., proximal mapping of majorized function is “easily” computable depending on the properties of bth block majorizer and regularizer, M_b and r_b, where the proximal mapping operator is defined by

  $${\mathrm{Prox}}_{r_b}^{M_b}(z)\triangleq \underset{u}{\arg \min }\frac{1}{2}\Vert u-z{\Vert }_{M_b}^2+r_b(u),\forall b.$$ (2)

• Block-wise extrapolation and momentum terms to accelerate convergence.

Suppose that 1) gradient of f(x; y) + γr(x, {ζ_k}; {h_k}) is M-Lipschitz continuous at an extrapolated point ${\acute{x}}^{(i+1)}$, ∀i; 2) filters in (1) satisfy the tight-frame (TF) condition, ${\sum }_{k=1}^K{\Vert h_k\ast u\Vert }_2^2=\Vert u{\Vert }_{2^{\prime }}^2$, ∀u for some boundary conditions [24]. Applying the BPEG-M framework (see Algorithm A.1) to solving (1) leads to the following block updates:

$$z^{(i+1)}=\sum _{k=1}^K\mathrm{flip}(h_k^{\ast })\ast {\mathcal{T}}_{{\beta }_k}\left(h_k\ast x^{(i)}\right),$$ (3)

$${\acute{x}}^{(i+1)}=x^{(i)}+E^{(i+1)}(x^{(i)}-x^{(i-1)}),$$ (4)

$$x^{(i+1)}={\mathrm{Prox}}_{{\mathbb{I}}_{\mathcal{X}}}^{{\tilde{M}}^{(i+1)}}\left({\acute{x}}^{(i+1)}-{({\tilde{M}}^{(i+1)})}^{-1}\nabla F({\acute{x}}^{(i+1)};y,z^{(i+1)})\right),$$ (5)

where E⁽ⁱ⁺¹⁾ is an extrapolation matrix that is given in (8) or (9) below, ${\tilde{M}}^{(i+1)}$ is a (scaled) majorization matrix for ∇F(x; y, z⁽ⁱ⁺¹⁾) that is given in (7) below, ∀i, the proximal operator ${\mathrm{Prox}}_{{\mathbb{I}}_{\mathcal{X}}}^{{\tilde{M}}^{(i+1)}}(\cdot )$ in (5) is given by (2), and ${\mathbb{I}}_{\mathcal{X}}(x)$ is the characteristic function of set $\mathcal{X}$ (i.e., ${\mathbb{I}}_{\mathcal{X}}$ equals to 0 if $x\in \mathcal{X}$, and ∞ otherwise).

Proximal mapping update (3) has a single-hidden layer convolutional autoencoder architecture that consists of encoding convolution, nonlinear thresholding, and decoding convolution, where flip(·) flips a filter along each dimension, and the soft-thresholding operator ${\mathcal{T}}_{\alpha }(u):{ℂ}^N\to {ℂ}^N$ is defined by

$${\left({\mathcal{T}}_{\alpha }(u)\right)}_n\triangleq \{\begin{array}{cc}{u}_n-\alpha \cdot \mathrm{sign}\left(u_n\right),& \left|u_n\right|>\alpha ,\\ 0,& \text{otherwise,}\end{array}$$ (6)

for n = 1, …, N, in which sign(·) is the sign function. See details of deriving BPEG-M updates (3)–(5) in §A.1.4. The BPEG-M updates in (3)–(5) guarantee convergence to a critical point, when MBIR problem (1) satisfies some mild conditions, e.g., lower-boundedness and existence of critical points; see Assumption S.1 in §A.1.3.

The following section generalizes the BPEG-M updates in (3)–(5) and constructs the Momentum-Net architecture.

2.2. Architecture

This section establishes the INN architecture of Momentum-Net by generalizing BPEG-M updates (3)–(5) that solve (1). Specifically, we replace the proximal mapping in (3) with a general image refining NN ${\mathcal{R}}_{\theta }(\cdot )$, where θ denotes the trainable parameters. To effectively remove iteration-wise artifacts and give “best” signal estimates at each iteration, we further generalize a refining NN ${\mathcal{R}}_{\theta }(\cdot )$ to iteration-wise image refining NNs $\{{\mathcal{R}}_{{\theta }^{(i+1)}}(\cdot ):i=0,\dots ,N_{\text{iter}}-1\}$, where θ⁽ⁱ⁺¹⁾ denotes the parameters for the ith iteration refining NN ${\mathcal{R}}_{\theta (i+1)}$, and N_iter is the number of Momentum-Net iterations. The iteration-wise NNs are particularly useful for reducing overfitting risks in regression, because ${\mathcal{R}}_{\theta (i+1)}$ is responsible for removing noise features only at the ith iteration, and thus one does not need to greatly increase dimensions of its parameter θ⁽ⁱ⁺¹⁾ [16], [18]. In low-dose CT reconstruction, for example, the refining NNs at the early and later iterations remove streak artifacts and Gaussian-like noise, respectively [16].

Algorithm 1.

Momentum-Net

Require: $\left\{{\mathcal{R}}_{{\theta }^{(i)}}:i=1,\dots ,N_{\text{iter}}\right\}$, ρ ∈ (0, 1), γ > 0, x(0) = x(−1), y

for i = 0, …, Niter−1 do

Calculate ${\tilde{M}}^{(i+1)}$ by (7), and E(i+1) by (8) or (9)

Image refining: $$z^{(i+1)}=(1-\rho )x^{(i)}+\rho {\mathcal{R}}_{\theta (i+1)}\left(x^{(i)}\right)$$ (Alg.1.1)

Extrapolation: $${\acute{x}}^{(i+1)}=x^{(i)}+E^{(i+1)}\left(x^{(i)}-x^{(i-1)}\right)$$ (Alg.1.2)

MBIR: $$\begin{gathered} x^{(i+1)} \\ ={\mathrm{Prox}}_{{\mathbb{I}}_{\mathcal{X}}}^{{\tilde{M}}^{(i+1)}}\left({\acute{x}}^{(i+1)}-{({\tilde{M}}^{(i+1)})}^{-1}\nabla F({\acute{x}}^{(i+1)};y,z^{(i+1)})\right) \end{gathered}$$ (Alg.1.3)

end for

Each iteration of Momentum-Net consists of 1) image refining, 2) extrapolation, and 3) MBIR modules, corresponding to the BPEG-M updates (3), (4), and (5), respectively. See the architecture of Momentum-Net in Fig. 1(a) and Algorithm 1. At the ith iteration, Momentum-Net performs the following three processes:

Fig. 1.

Architectures of different INNs for MBIR. (a–b) The architectures of Momentum-Net and BCD-Net [25] are constructed by generalizing BPEG-M and BCD algorithms that solve MBIR problem using a convolutional regularizer trained via convolutional analysis operator learning (CAOL) [24], [36], respectively. (a) Removing extrapolation modules (i.e., setting the extrapolation matrices {E⁽ⁱ⁺¹⁾ : ∀i} as a zero matrix), Momentum-Net specializes to the existing gradient-descent-inspired INNs [21], [28]. When the MBIR cost function F(x; y, z⁽ⁱ⁺¹⁾) in (P1) has a sharp majorizer ${\tilde{M}}^{(i+1)}$, ∀i, Momentum-Net (using ρ≈1) specializes to BCD-Net; see Examples 5–6. (b) BCD-Net is a general version of the existing INNs in [20], [22], [23], [30], [39]–[41] by using iteration-wise image refining NNs, i.e., $\{{\mathcal{R}}_{{\theta }^{(i+1)}}:\forall i\}$, or considering general convex data-fit f(x; y).

• Refining: The ith image refining module gives the “refined” image z⁽ⁱ⁺¹⁾, by applying the ith refining NN, ${\mathcal{R}}_{\theta (i+1)}$, to an input image at the ith iteration, x⁽ⁱ⁾ (i.e., image estimate from the (i − 1)th iteration). Different from existing INNs, e.g., ADMM-Net [20], PnP-ADMM [23], [41], RED [22], MoDL [30], BCD-Net [25] (see Fig. 1(b)), TNRD [21], [28], we apply ρ-relaxation with ρ ∈ (0, 1); see (Alg.1.1). The parameter ρ controls the strength of inference from refining NNs, but does not affect the convergence guarantee of Momentum-Net. Proper selection of ρ can improve MBIR accuracy (see §A.10).

• Extrapolation: The ith extrapolation module gives the extrapolated point ${\acute{x}}^{(i+1)}$, based on momentum terms x⁽ⁱ⁾ −x⁽ⁱ⁻¹⁾; see (Alg.1.2). Intuitively speaking, momentum is information from previous updates to amplify the changes in subsequent iterations. Its effectiveness has been shown in diverse optimization literature, e.g., convex optimization [48], [49] and block optimization [24], [35].

• MBIR: Given a refined image z⁽ⁱ⁺¹⁾ and a measurement vector y, the ith MBIR module (Alg.1.3) applies the proximal operator ${\mathrm{Prox}}_{{\mathbb{I}}_{\mathcal{X}}}^{{\tilde{M}}^{(i+1)}}(\cdot )$ to the extrapolated gradient update using a quadratic majorizer of F(x; y, z⁽ⁱ⁺¹⁾), where F is defined in (P0). Intuitively speaking, this step solves a majorized version of the following MBIR problem at the extrapolated point ${\acute{x}}^{(i+1)}$:

$$\underset{x\in \mathcal{X}}{\min }F(x;y,z^{(i+1)}),$$ (P1)

and gives a reconstructed image x⁽ⁱ⁺¹⁾. In Momentum-Net, we consider (non)convex differentiable MBIR cost functions F with M-Lipschitz continuous gradients, and a convex and closed set $\mathcal{X}$. For a wide range of large-scale inverse imaging problems, the majorized MBIR problem (Alg.1.3) has a practical closed-form solution and thus, does not require an iterative solver, depending on the properties of practically invertible majorization matrices M⁽ⁱ⁺¹⁾ and constraints. Examples of $\mathcal{X}$ combinations that give a noniterative solution for (Alg.1.3) include scaled identity and diagonal matrices with a box constraint and the non-negativity constraint, and matrices decomposable by unitary transforms, e.g., a circulant matrix [56], [57], with $\mathcal{X}={ℂ}^N$. The updated image x⁽ⁱ⁺¹⁾ is the input to the next Momentum-Net iteration.

The followings are details of Momentum-Net in Algorithm 1. A scaled majorization matrix is

$${\tilde{M}}^{(i+1)}=\lambda \cdot M^{(i+1)}\succ 0,\lambda \ge 1,$$ (7)

where $M^{(i+1)}\in {ℝ}^{N\times N}$ is a symmetric positive definite majorization matrix of ∇F(x; y, z⁽ⁱ⁺¹⁾) in the sense of M-Lipschitz continuity (see Definition 1). In (7), λ = 1 and λ > 1 for convex and nonconvex F(x; y, z⁽ⁱ⁺¹⁾) (or convex and nonconvex f(x; y)), respectively. We design the extrapolation matrices as follows:

for convex F,

$$E^{(i+1)}={\delta }^2{m}^{(i)}\cdot {\left(M^{(i+1)}\right)}^{-\frac{1}{2}}{\left(M^{(i)}\right)}^{\frac{1}{2}};$$ (8)

for nonconvex F,

$$E^{(i+1)}={\delta }^2{m}^{(i)}\cdot \frac{\lambda -1}{2(\lambda +1)}\cdot {(M^{(i+1)})}^{-\frac{1}{2}}{(M^{(i)})}^{\frac{1}{2}},$$ (9)

for some δ < 1 and {0 ≤ m⁽ⁱ⁾ ≤ 1 : ∀i}. We update the momentum coefficients {m⁽ⁱ⁺¹⁾ : ∀i} by the following formula [24], [35]:

$$m^{(i+1)}=\frac{{\theta }^{(i)}-1}{{\theta }^{(i+1)}},{\theta }^{(i+1)}=\frac{1+\sqrt{1+4{({\theta }^{(i)})}^2}}{2};$$ (10)

if F(x; y, z⁽ⁱ⁺¹⁾) has a sharp majorizer, i.e., ∇F(x; y, z⁽ⁱ⁺¹⁾) has M⁽ⁱ⁺¹⁾ such that the corresponding bound in Definition 1 is tight, then we set m⁽ⁱ⁺¹⁾ = 0, ∀i. §A.11 lists parameters of Momentum-Net, and summarizes selection guidelines or gives default values.

2.3. Relations to previous works

Several existing MBIR methods can be viewed as a special case of Momentum-Net:

Example 3. (MBIR model (1) using convolutional autoencoders that satisfy the TF condition [24]). The BPEG-M updates in (3)–(5) are special cases of the modules in Momentum-Net (Algorithm 1), with $\{{\mathcal{R}}_{{\theta }^{(i+1)}}(\cdot )={\sum }_{k=1}^K\text{flip}\left(h_k^{\ast }\right)\ast {\mathcal{T}}_{{\beta }_k}\left(h_k\ast (\cdot )\right):\forall i\}$ flip (i.e., single hidden-layer convolutional autoencoder [24]) and ρ ≈ 1. These give a clear mathematical connection between a denoiser (3) and cost function (1). One can find a similar relation between a multi-layer CNN and a multi-layer convolutional regularizer [24, Appx.].

Example 4. (INNs inspired by gradient descent method, e.g., TNRD [21], [28]). Removing extrapolation modules, i.e., setting {E⁽ⁱ⁺¹⁾ = 0 : ∀i} in (Alg.1.2), and setting ρ ≈ 1, Momentum-Net becomes the existing INN in [21], [28].

Example 5. (BCD-Net for image denoising [25]). To obtain a clean image $x\in {ℝ}^N$ from a noisy image $y\in {ℝ}^N$ corrupted by an additive white Gaussian noise (AWGN), MBIR problem (P1) considers the data-fit $f(x;y)=\frac{1}{2}\Vert y-x{\Vert }_W^2$ with the inverse covariance matrix $W=\frac{1}{{\sigma }^2}I$, where σ² is a variance of AWGN, and the box constraint $\mathcal{X}={[0,U]}^N$ with an upper bound U > 0. For this f(x; y), the MBIR module (Alg.1.3) can use the exact majorizer $\{{\tilde{M}}^{(i+1)}=(\frac{1}{{\sigma }^2}+\gamma )I\}$ and one does not need to use the extrapolation module (Alg.1.2), i.e., {E⁽ⁱ⁺¹⁾ = 0}. Thus, Momentum-Net (with ρ ≈ 1) becomes BCD-Net.

Example 6. (BCD-Net for undersampled single-coil MRI [25]). To obtain an object magnetization $x\in {ℝ}^N$ from a k-space data $y\in {ℂ}^m$ obtained by undersampling (e.g., compressed sensing [58]) MRI, MBIR problem (P1) considers the data-fit $f(x;y)=\frac{1}{2}\Vert y-Ax{\Vert }_W^2$ with an undersampling Fourier operator A (disregarding relaxation effects and considering Cartesian k-space), the inverse covariance matrix $W=\frac{1}{{\sigma }^2}I$, where σ² is a variance of complex AWGN [59], and $\mathcal{X}={ℂ}^N$. For this f(x; y), the MBIR module (Alg.1.3) can use the exact majorizer $\{{\tilde{M}}^{(i+1)}=F_{\text{disc}}^H(\frac{1}{{\sigma }^2}P+\gamma I)F_{\text{disc}}\}$ that is practically invertible, where F_disc is the discrete Fourier transform and P is a diagonal matrix with either 0 or 1 (their positions correspond to sampling pattern in k-space), and the extrapolation module (Alg.1.2) uses the zero extrapolation matrices {E⁽ⁱ⁺¹⁾ = 0}. Thus, Momentum-Net (with ρ ≈ 1) becomes BCD-Net.

The following section analyzes the convergence of Momentum-Net.

2.4. Convergence analysis

In practice, INNs, i.e., “unrolled” or PnP methods using refining NNs, are trained and used with a specific number of iterations. Nevertheless, similar to optimization algorithms, studying convergence properties of INNs with N_iter → ∞ [23], [31], [46] is important; in particular, it is crucial to know if a given INN tends to converge as N_iter increases. For INNs using iteration-wise refining NNs, e.g., BCD-Net [25] and proposed Momentum-Net, we expect that refiners converge, i.e., their image refining capacity converges, because information provided by data-fit function f(x; y) in MBIR (e.g., likelihood) reaches some “bound” after a certain number of iterations. Fig. 2 illustrates that dCNN parameters of Momentum-Net tend to converge for different applications. (The similar behavior was reported for sCNN refiners in BCD-Net [16].) Although refiners do not completely converge, in practice, one could use a refining NN at a sufficiently large iteration number, e.g., N_iter = 100 in Momentum-Net, for the later iterations.

Fig. 2.

Convergence behavior of Momentum-Net’s dCNNs refiners $\{{\mathcal{R}}_{{\theta }^{(i)}}\}$ in different applications (θ⁽ⁱ⁾ denotes the parameter vector of the ith iteration refiner ${\mathcal{R}}_{{\theta }^{(i)}}$, for i=1, …, N_iter; see details of $\{{\mathcal{R}}_{{\theta }^{(i)}}\}$ in (19) and §4.2.1; N_iter = 100). Sparse-view CT (fan-beam geometry with 12.5% projections views): ${\mathcal{R}}_{{\theta }^{(i)}}$ quickly converges, where majorization matrices of training data-fits have similar condition numbers. LF photography using a focal stack (five detectors and reconstructed LFs consists of 9×9 sub-aperture images): ${\mathcal{R}}_{{\theta }^{(i)}}$ has slower convergence, where majorization matrices of training data-fits have largely different condition numbers.

There are two key challenges in analyzing the convergence of Momentum-Net in Algorithm 1: both challenges relate to its image refining modules (Alg.1.1). First, image refining NNs ${\mathcal{R}}_{{\theta }^{(i+1)}}$ change across iterations; even if they are identical across iterations, they are not necessarily nonexpansive operators [60], [61] in practice. Second, the iteration-wise refining NNs are not necessarily proximal mapping operators, i.e., they are not written explicitly in the form of (2). This section proposes two new asymptotic definitions to overcome these challenges, and then uses those conditions to analyze convergence properties of Momentum-Net in Algorithm 1.

2.4.1. Preliminaries

To resolve the challenge of iteration-wise refining NNs and the practical difficulty in guaranteeing their non-expansiveness, we introduce the following generalized definition of the non-expansiveness [60], [61].

Definition 7 (Asymptotically nonexpansive paired operators). A sequence of paired operators $({\mathcal{R}}_{{\theta }^{(i)}},{\mathcal{R}}_{{\theta }^{(i+1)}})$ is asymptotically nonexpansive if there exist a summable nonnegative sequence $\left\{{ϵ}^{(i+1)}\ge 0:{\sum }_{i=0}^{\infty }{ϵ}^{(i+1)}<\infty \right\}$ such that¹

$${\Vert {\mathcal{R}}_{{\theta }^{(i+1)}}(u)-{\mathcal{R}}_{{\theta }^{(i)}}(v)\Vert }_2^2\le \Vert u-v{\Vert }_2^2+{ϵ}^{(i+1)},\forall u,v,i.$$ (11)

When ${\mathcal{R}}_{\theta (i+1)}={\mathcal{R}}_{\theta }$ and ${ϵ}^{(i+1)}=0,\forall i$, Definition 7 becomes the standard non-expansiveness of a mapping operator ${\mathcal{R}}_{\theta }$. If we replace the inequality (≤) with the strict inequality (<) in (11), then we say that the sequence of paired operators $({\mathcal{R}}_{{\theta }^{(i+1)}},{\mathcal{R}}_{{\theta }^{(i+1)}})$ is asymptotically contractive. (This stronger assumption is used to prove convergence of BCD-Net in Proposition A.5.) Definition 7 also implies that mapping operators ${\mathcal{R}}_{{\theta }^{(i+1)}}$ converge to some nonexpansive operator, if the corresponding parameters θ⁽ⁱ⁺¹⁾ converge.

Definition 7 incorporates a pairing property because Momentum-Net uses iteration-wise image refining NNs. Specifically, the pairing property helps prove convergence of Momentum-Net, by connecting image refining NNs at adjacent iterations. Furthermore, the asymptotic property in Definition 7 allows Momentum-Net to use expansive refining NNs (i.e., mapping operators having a Lipschitz constant larger than 1) for some iterations, while guaranteeing convergence; see Figs. 3(a3) and 3(b3). Suppose that refining NNs are identical across iterations, i.e., ${\mathcal{R}}_{\theta (i+1)}={\mathcal{R}}_{\theta }$, ∀i, similar to some existing INNs, e.g., PnP [23], RED [22], and other methods in §1.3. In such cases, if ${\mathcal{R}}_{\theta }$ is expansive, Momentum-Net may diverge; this property corresponds to the limitation of existing methods described in §1.3. Momentum-Net moderates this issue by using iteration-wise refining NNs that satisfy the asymptotic paired non-expansiveness in Definition 7.

Fig. 3.

Empirical measures related to Assumption 4 for guaranteeing convergence of Momentum-Net using dCNNs refiners (for details, see (19) and §4.2.1), in different applications. See estimation procedures in §A.2. (a) The sparse-view CT reconstruction experiment used fan-beam geometry with 12.5% projections views. (b) The LF photography experiment used five detectors and reconstructed LFs consisting of 9×9 sub-aperture images. (a1, b1) For both the applications, we observed that Δ⁽ⁱ⁾ → 0. This implies that the z⁽ⁱ⁺¹⁾-updates in (Alg.1.1) satisfy the asymptotic block-coordinate minimizer condition in Assumption 4. (Magenta dots denote the mean values and black vertical error bars denote standard deviations.) (a2) Momentum-Net trained from training data-fits, where their majorization matrices have mild condition number variations, shows that ϵ⁽ⁱ⁾ → 0. This implies that paired NNs $({\mathcal{R}}_{{\theta }^{(i+1)}},{\mathcal{R}}_{{\theta }^{(i)}})$ in (Alg.1.1) are asymptotically nonexpansive. (b2) Momentum-Net trained from training training data-fits, where their majorization matrices have mild condition number variations, shows that ϵ⁽ⁱ⁾ becomes close to zero, but does not converge to zero in one hundred iterations. (a3, b3) The NNs, ${\mathcal{R}}_{{\theta }^{(i+1)}}$ in (Alg.1.1), become nonexpansive, i.e., its Lipschitz constant κ⁽ⁱ⁾ becomes less than 1, as i increases.

Because the sequence {z⁽ⁱ⁺¹⁾ : ∀i} in (Alg.1.1) is not necessarily updated with a proximal mapping, we introduce a generalized definition of block-coordinate minimizers [53, (2.3)] for z⁽ⁱ⁺¹⁾-updates:

Definition 8 (Asymptotic block-coordinate minimizer). The update z⁽ⁱ⁺¹⁾ is an asymptotic block-coordinate minimizer if there exists a summable nonnegative sequence $\{{\mathrm{\Delta }}^{(i+1)}\ge 0:{\sum }_{i=0}^{\infty }{\mathrm{\Delta }}^{(i+1)}<\infty \}$ such that

$${\Vert z^{(i+1)}-x^{(i)}\Vert }_2^2\le {\Vert z^{(i)}-x^{(i)}\Vert }_2^2+{\mathrm{\Delta }}^{(i+1)},\forall i.$$ (12)

Definition 8 implies that as i → ∞, the updates {z⁽ⁱ⁺¹⁾ : i ≥ 0} approach a block-coordinate minimizer trajectory that satisfies (12) with {Δ⁽ⁱ⁺¹⁾ = 0 : i ≥ 0}. In particular, Δ⁽ⁱ⁺¹⁾ quantifies how much the update z⁽ⁱ⁺¹⁾ in (Alg.1.1) perturbs a block-coordinate minimizer trajectory. The bound ${\Vert z^{(i+1)}-x^{(i)}\Vert }_2^2\le {\Vert z^{(i)}-x^{(i)}\Vert }_2^2$ always holds, ∀i, when one uses the proximal mapping in (3) within the BPEG-M framework. While applying trained Momentum-Net, (12) is easy to examine empirically, whereas (11) is harder to check.

2.4.2. Assumptions

This section introduces and interprets the assumptions for convergence analysis of Momentum-Net in Algorithm 1:

• Assumption 1) In MBIR problems (P1), (non)convex F(x; y, z⁽ⁱ⁺¹⁾) is (continuously) differentiable, proper, and lower-bounded in dom(F),² ∀i, and $\mathcal{X}$ is convex and closed. Algorithm 1 has a fixed-point.

• Assumption 2) ∇F(x; y, z⁽ⁱ⁺¹⁾) is M⁽ⁱ⁺¹⁾-Lipschitz continuous with respect to x (see Definition 1), where M⁽ⁱ⁺¹⁾ is a iteration-wise majorization matrix that satisfies $m_{F,\min }{I}_N⪯M^{(i+1)}⪯m_{F,\max }{I}_N$ with 0 < m_F,min ≤ m _F,max < ∞, ∀i.

• Assumption 3) The extrapolation matrices E⁽ⁱ⁺¹⁾ ⪰ 0 in (8)–(9) satisfy the following conditions:

for convex F,

$${(E^{(i+1)})}^T{M}^{(i+1)}{E}^{(i+1)}⪯{\delta }^2\cdot M^{(i)},\delta <1;$$ (13)

for nonconvex F,

$${(E^{(i+1)})}^T{M}^{(i+1)}{E}^{(i+1)}⪯\frac{{\delta }^2{(\lambda -1)}^2}{4{(\lambda +1)}^2}\cdot M^{(i)},\delta <1.$$ (14)

• Assumption 4) The sequence of paired operators $({\mathcal{R}}_{{\theta }^{(i+1)}},{\mathcal{R}}_{{\theta }^{(i)}})$ is asymptotically nonexpansive with a summable sequence {ϵ⁽ⁱ⁺ⁱ⁾ ≥ 0}; the update z⁽ⁱ⁺¹⁾ is an asymptotic block-coordinate minimizer with a summable sequence {Δ⁽ⁱ⁺ⁱ⁾ ≥ 0}. The mapping functions $\{{\mathcal{R}}_{{\theta }^{(i+1)}}:\forall i\}$ are continuous with respect to input points and the corresponding parameters {θ⁽ⁱ⁺¹⁾ : ∀i} are bounded.

Assumption 1 is a slight modification of Assumption S.1 of BPEG-M, and Assumptions 2–3 are identical to Assumptions S.2–S.3 of BPEG-M; see Assumptions S.1–S.3 in §A.1.3. The extrapolation matrix designs (8) and (9) satisfy conditions (13) and (14) in Assumption 3, respectively.

We provide empirical justifications for the first two conditions in Assumption 4. First, Figs. 3(a2) and A.1(a2) illustrate that paired refining NNs $({\mathcal{R}}_{{\theta }^{(i+1)}},{\mathcal{R}}_{{\theta }^{(i)}})$ of Momentum-Net appear to be asymptotically nonexpansive in an application that has mild condition number variations across training data-fit majorization matrices. Figs. 3(a3), 3(b3), A.1(a3), and A.1(b3) illustrate for different applications that refining NNs $\{{\mathcal{R}}_{{\theta }^{(i+1)}}\}$ become nonexpansive: their Lipschitz constants at the first several iterations are larger than 1, and their Lipschitz constants in later iterations become less than 1. Alternatively, the asymptotic non-expansiveness of paired operators $({\mathcal{R}}_{{\theta }^{(i+1)}},{\mathcal{R}}_{{\theta }^{(i)}})$ can be satisfied by a stronger assumption that the sequence $\{{\mathcal{R}}_{{\theta }^{(i+1)}}\}$ converges to some nonexpansive operator. (Fig. 2 illustrates that dCNN parameters of Momentum-Net appear to converge.)

Figs. 3(a3), 3(b3), A.1(a3), and A.1(b3) illustrate for different applications that the z⁽ⁱ⁺¹⁾-updates are asymptotic block-coordinate minimizers. Lemma A.4 and §A.3 in the appendices provide a probabilistic justification for the asymptotic block-coordinate minimizer condition.

2.4.3. Main convergence results

This section analyzes fixed-point and critical point convergence of Momentum-Net in Algorithm 1, under the assumptions in the previous section. We first show that differences between two consecutive iterates generated by Momentum-Net converge to zero:

Proposition 9 (Convergence properties). Under Assumptions 1–4, let {x⁽ⁱ⁺¹⁾, z⁽ⁱ⁺¹⁾ : i ≥ 0} be the sequence generated by Algorithm 1. Then, the sequence satisfies

$$\sum _{i=0}^{\infty }{\Vert \left[\begin{array}{l}{x}^{(i+1)}\hfill \\ z^{(i+1)}\hfill \end{array}\right]-\left[\begin{array}{l}{x}^{(i)}\hfill \\ z^{(i)}\hfill \end{array}\right]\Vert }_2^2<\infty ,$$ (15)

and hence ${\Vert \left[\begin{array}{l}{x}^{(i+1)}\hfill \\ z^{(i+1)}\hfill \end{array}\right]-\left[\begin{array}{l}{x}^{(i)}\hfill \\ z^{(i)}\hfill \end{array}\right]\Vert }_2\to 0$.

Proof. See §A.4 in the appendices.

Using Proposition 9, our main theorem provides that any limit points of the sequence generated by Momentum-Net satisfy critical point and fixed-point conditions:

Theorem 10 (A limit point satisfies both critical point and fixed-point conditions). Under Assumptions 1–4 above, let {x⁽ⁱ⁺¹⁾, z⁽ⁱ⁺¹⁾ : i ≥ 0} be the sequence generated by Algorithm 1. Consider either a fixed majorization matrix with general structure, i.e., M⁽ⁱ⁺¹⁾ = M for i ≥ 0, or a sequence of diagonal majorization matrices, i.e., {M⁽ⁱ⁺¹⁾ : i ≥ 0}. Then, any limit point $\overline{x}$ of {x⁽ⁱ⁺¹⁾} satisfies both the critical point condition:

$$\langle \nabla F(\overline{x};y,\overline{z}),x-\overline{x}\rangle \ge 0,\forall x\in \mathcal{X},$$ (16)

where $\overline{z}$ is a limit point of {z⁽ⁱ⁺¹⁾}, and the fixed-point condition:

$$\left[\begin{array}{c}\overline{x}\\ \overline{x}\end{array}\right]={\mathcal{A}}_{{\mathcal{R}}_{\overline{\theta }}}^{\overline{M}}\left(\left[\begin{array}{l}\overline{x}\hfill \\ \overline{x}\hfill \end{array}\right]\right),$$ (17)

where $\left[\begin{array}{c}{x}^{(i+1)}\\ x^{(i)}\end{array}\right]={\mathcal{A}}_{{\mathcal{R}}_{\theta (i+1)}}^{M(i+1)}\left(\left[\begin{array}{c}{x}^{(i)}\\ x^{(i-1)}\end{array}\right]\right)$, ${\mathcal{A}}_{{\mathcal{R}}_{\theta (i+1)}}^{M(i+1)}(\cdot )$ denotes performing the ith updates in Algorithm 1, and $\overline{\theta }$ and $\overline{M}$ is a limit point of {θ⁽ⁱ⁺¹⁾} and {M⁽ⁱ⁺¹⁾}, respectively.

Proof. See §A.5 in the appendices.

Observe that, if $\mathcal{X}={ℝ}^N$ or $\overline{x}$ is an interior point of $\mathcal{X}$, (16) reduces to the first-order optimality condition $0\in \partial F(\overline{x};y,\overline{z})$, where ∂F(x) denotes the limiting subdifferential of F at x. With additional isolation and boundedness assumptions for the points satisfying (16) and (17), we obtain whole sequence guarantees:

Corollary 11 (Whole sequence convergence). Consider the construction in Theorem 10. Let $\mathcal{S}$ be the set of points satisfying the critical point condition in (16) and the fixed-point condition in (17). If {x⁽ⁱ⁺¹⁾ : i ≥ 0} is bounded, then $\mathrm{dist}(x^{(i+1)},\mathcal{S})\to 0$, where $\mathrm{dist}(u,\mathcal{V})\triangleq \inf \{\Vert u-v\Vert :v\in \mathcal{V}\}$ denotes the distance from u to $\mathcal{V}$, for any point $u\in {ℝ}^N$ and any subset $\mathcal{V}\subset {ℝ}^N$. If $\mathcal{S}$ contains uniformly isolated points, i.e., there exists η > 0 such that ∥u−v∥ ≥ η for any distinct points $u,v\in \mathcal{S}$, then {x⁽ⁱ⁺¹⁾} converges to a point in $\mathcal{S}$.

Proof. See §A.6 in the appendices.

The boundedness assumption for {x⁽ⁱ⁺¹⁾} in Corollary 11 is standard in block-wise optimization, e.g., [24], [35], [53], [55], [62]. The assumption can be satisfied if the set $\mathcal{X}$ is bounded (e.g., box constraints), one chooses appropriate regularization parameters in Algorithm 1 [24], [35], [55], the function F(x; y, z) is coercive [62], or the level set is bounded [53]. However, for general F(x; y, z), it is hard to verify the isolation condition for the points in $\mathcal{S}$ in practice. Instead, one may use Kurdyka-Łojasiewicz property [53], [62] to analyze the whole sequence convergence with some appropriate modifications.

For simplicity, we focused our discussion to noniterative MBIR module (Alg.1.3). However, Momentum-Net practically converges with any proximable MRIR function (Alg.1.3) that may need an iterative solver, if sufficient inner iterations are used. To maximize the computational benefit of Momentum-Net, one needs to make sure that majorized MBIR function (Alg.1.3) is better proximable over its original form (P1).

2.5. Benefits of Momentum-Net

Momentum-Net has several benefits over existing INNs:

• Benefits from refining module: The image refining module (Alg.1.1) can use iteration-wise image refining NNs $\{{\mathcal{R}}_{{\theta }^{(i+1)}}:i\ge 0\}$: those are particularly useful to reduce overfitting risks by reducing dimensions of their parameters θ⁽ⁱ⁺¹⁾ at each iteration [16], [18], [19]. Iteration-wise refining NNs require less memory for training, compared to methods that use a single refining NN for all iterations, e.g., [63]. Different from the existing methods mentioned in §1.3, Momentum-Net does not require (firmly) nonexpansive mapping operators $\{{\mathcal{R}}_{{\theta }^{(i+1)}}\}$ to guarantee convergence. Instead, $\{{\mathcal{R}}_{{\theta }^{(i+1)}}\}$ in (Alg.1.1) assumes a generalized notion of the (firm) non-expansiveness condition assumed for convergence of the existing methods that use identical refining NNs across iterations, including PnP [20], [23], [39]–[41], [46], RED [22], [43], etc. The generalized concept is the first practical condition to guarantee convergence of INNs using iteration-wise refining NNs; see Definition 7.

• Benefits from extrapolation module: The extrapolation module (Alg.1.2) uses the momentum terms x⁽ⁱ⁾ − x⁽ⁱ⁻¹⁾ that accelerate the convergence of Momentum-Net. In particular, compared to the existing gradient-descent-inspired INNs, e.g., TNRD [21], [28], Momentum-Net converges faster. (Note that the way the authors of [43] used momentum is less conventional. The corresponding method, RED-APG [43, Alg. 6], still can require multiple inner iterations to solve its quadratic MBIR problem, similar to BCD-Net-type methods.)

• Benefits from MBIR module: The MBIR module (Alg.1.3) does not require multiple inner iterations for a wide range of imaging problems and has both theoretical and practical benefits. Note first that convergence analysis of INNs (including Momentum-Net) assumes that their MBIR operators are noniterative. In other words, related convergence theory (e.g., Proposition A.5) is inapplicable if iterative methods, particularly with insufficient number of iterations, are applied to MBIR modules. Different from the existing BCD-Net-type methods [20], [22], [23], [25], [30], [39]–[41], [43] that can require iterative solvers for their MBIR modules, MBIR module (Alg.1.3) of Momentum-Net can have practical close-form solution (see examples in §2.2), and its corresponding convergence analysis (see §2.4) can hold stably for a wide range of imaging applications. Second, combined with extrapolation module (Alg.1.2), noniterative MBIR modules (Alg.1.3) lead to faster MBIR, compared to the existing BCD-Net-type methods that can require multiple inner iterations for their MBIR modules for convergence. Third, Momentum-Net guarantees convergence even for nonconvex MBIR cost function F(x; y, z) or nonconvex data-fit f(x; y) of which the gradient is M-Lipschitz continuous (see Definition 1), while existing INNs overlooked nonconvex F(x; y, z) or f(x; y).

Furthermore, §A.7 analyzes the sequence convergence of BCD-Net [25], and describes the convergence benefits of Momentum-Net over BCD-Net.

3. TRAINING INNS

This section describes training of all the INNs compared in this paper.

3.1. Architecture of refining NNs and their training

For all INNs in this paper, we train the refining NN at each iteration to remove artifacts from the input image x⁽ⁱ⁾ that is fed from the previous iteration. For the ith iteration NN, we first consider the following sCNN architecture, residual single-hidden layer convolutional autoencoder:

$${\mathcal{R}}_{{\theta }^{(i+1)}}(u)=\sum _{k=1}^K{d}_k^{(i+1)}\ast {\mathcal{T}}_{\exp ({\alpha }_k^{(i+1)})}(e_k^{(i+1)}\ast u)+u,$$ (18)

where ${\theta }^{(i+1)}=\{d_k^{(i+1)},{\alpha }_k^{(i+1)},e_k^{(i+1)}:\forall k\}$ is the parameter set of the ith image refining NN, $\{d_k^{(i+1)},e_k^{(i+1)}\in {ℂ}^R:k=1,\dots ,K\}$ is a set of K decoding and encoding filters of size R, $\{\exp ({\alpha }_k^{(i+1)}):k=1,\dots ,K\}$ is a set of K thresholding values, and ${\mathcal{T}}_{\alpha }(u)$ is the soft-thresholding operator with parameter α defined in (6), for i = 0, …, N_iter −1. We use the exponential function exp(·) to prevent the thresholding parameters {α_k} from becoming negative during training. We observed that the residual convolutional autoencoder in (18) gives better results compared to the convolutional autoencoder, i.e., (18) without the second term [18], [25]. This corresponds to the empirical result in [64], [65] that having skip connections (e.g., the second term in (18)) can improve generalization. The sequence of paired sCNN refiners (18) can satisfy the asymptotic non-expansiveness, if its convergent refiner satisfies that

$${\sigma }_{\max }({\overline{D}}^H\overline{D})\le 1/R,{\sigma }_{\max }({\overline{E}}^H\overline{E})\le 1/R,$$

where σ_max(·) is the largest eigenvalue of a matrix, $\overline{D}\triangleq \left[{\overline{d}}_1,\dots ,{\overline{d}}_K,{\delta }_R\right],\overline{E}\triangleq \left[{\overline{e}}_1,\dots ,{\overline{e}}_K,{\delta }_R\right],\{{\overline{d}}_k,{\overline{e}}_k:\forall k\}$ are limit point filters, and δ_R is the Kronecker delta filter of size R.

For dCNN refiners, we use the following residual multi-layer CNN architecture, a simplified DnCNN [4] using fewer layers, no pooling, and no batch normalization [46] (we drop superscript indices (·)⁽ⁱ⁾ for simplicity):

$${\mathcal{R}}_{\theta }(u)=u-\sum _{k=1}^K{e}_k^{[L]}\ast u_k^{[L-1]},$$ (19)

$$u_k^{[1]}=\mathrm{ReLU}\left(e_k^{[1]}\ast u\right),u_k^{[l]}=\mathrm{ReLU}\left(\sum _{k^{\prime }=1}^K{e}_{k,k^{\prime }}^{[l]}\ast z_{k^{\prime }}^{[l-1]}\right),$$

for k = 1, …, K and l = 2, …, L−1, where $\theta =\{e_k^{[l]},e_{k,k^{\prime }}^{[l]}:\forall k,k^{\prime },l\}$ is the parameter set of each refining NN, K is the number of feature maps, L is the number of layers, $\{e_k^{[l]}\in {ℝ}^R:k=1,\dots ,K,l=1,L\}$ is a set of filters at the first and last dCNN layer, $\{e_{k,k^{\prime }}^{[l]}\in {ℝ}^R:k,k^{\prime }=1,\dots ,K,l=2,\dots ,L-1\}$ is a set of filters for remaining dCNN layer, and the rectified linear unit activation function is defined by $\mathrm{ReLU}(u)\triangleq \max (0,u)$.

The training process of Momentum-Net requires S high-quality training images, {x_s : s=1, …, S}, S training measurements simulated via imaging physics, {y_s : s=1, …, S}, and S data-fits {f_s(x; y_s) : s = 1, …, S} and the corresponding majorization matrices $\{M_s^{(i)},{\tilde{M}}_s^{(i)}:s=1,\dots ,S,i=1,\dots ,N_{\text{iter}}\}$. Different from [16], [25], [66] that train convolutional autoencoders from the patch perspective, we train the image refining NNs in (18)–(19) from the convolution perspective (that does not store many overlapping patches, e.g., see [24]). From S training pairs $(x_s,x_s^{(i)})$, where $\{x_s^{(i)}:s=1,\dots ,S\}$ is a set of S reconstructed images at the (i−1)th Momentum-Net iteration, we train the ith iteration image refining NN in (18) by solving the following optimization problem:

$${\theta }^{(i+1)}=\underset{\theta }{\mathrm{argmin}}\frac{1}{2S}\sum _{s=1}^S{\Vert x_s-{\mathcal{R}}_{\theta }(x_s^{(i)})\Vert }_2^2,$$ (P2)

where θ⁽ⁱ⁺¹⁾ is given as in (18), for i = 0, …, N_iter − 1 (see some related properties in §A.8). We solve the training optimization problems (P2) by mini-batch stochastic optimization with the subdifferentials computed by the PyTorch Autograd package.

3.2. Regularization parameter selection based on “spectral spread”

When majorization matrices of training data-fits {f_s(x; y_s) : s=1, …, S} have similar spectral properties, e.g., condition numbers, the regularization parameter γ in (P1) is trainable by substituting (Alg.1.1) into (Alg.1.3) and modifying the training cost (P2). However, the condition numbers of data-fit majorizers can greatly differ due a variety of imaging geometries or image formation systems, or noise levels in training measurements, etc. See such examples in §4.1–4.2.

To train Momentum-Net with diverse training data-fits, we propose a parameter selection scheme based on the “spectral spread” of their majorization matrices $\{M_{f_s}^{(i)}\}$. For simplicity, consider majorization matrices of the form ${\tilde{M}}_s^{(i)}={\tilde{M}}_s=\lambda (M_{f_s}+{\gamma }_{s}I)\forall i$, where the factor λ is selected by (7) and $M_{f_s}$ is a symmetric positive semidefinite majorization matrix for f_s(x; y_s), ∀s. We select the regularization parameter γ_s for the sth training sample as

$${\gamma }_s=\frac{{\sigma }_{\text{spread}}(M_{f_s})}{\chi },$$ (20)

where the spectral spread of a symmetric positive definite matrix is defined by ${\sigma }_{\text{spread}}(\cdot )\triangleq {\sigma }_{\max }(\cdot )-{\sigma }_{\min }(\cdot )$ for ${\sigma }_{\max }(M_{f_s})>{\sigma }_{\min }(M_{f_s})\ge 0$, and σ_min(·) is the smallest eigenvalue of a matrix. For the sth training sample, a tunable factor χ controls γ_s in (20) according to ${\sigma }_{\text{spread}}(M_{f_s})$, ∀s. The proposed parameter selection scheme also applies to testing Momentum-Net, based on the tuned factor χ^⁎ in its training. We observed that the proposed parameter selection scheme (20) gives better MBIR accuracy than the condition number based selection scheme that is similarly used in selecting ADMM parameters [17] (for the two applications in §4). One may further apply this scheme to iteration-wise majorization matrices ${\tilde{M}}_s^{(i)}$ and select iteration-wise regularization parameters ${\gamma }_s^{(i)}$ accordingly. For comparing different INNs, we apply (20) to all INNs.

4. EXPERIMENTAL RESULTS AND DISCUSSION

We investigated two extreme imaging applications: sparse-view CT and LF photography using a focal stack. In particular, these two applications lack a practical closed-form solution for the MBIR modules of BCD-Net and ADMM-Net [20], e.g., solving (Alg.2.2). For these applications, we compared the performances of the following five INNs: BCD-Net [25] (i.e., generalization of RED [22] and MoDL [30]), ADMM-Net [20], i.e., PnP-ADMM [23], [41] using iteration-wise refining NNs, Momentum-Net without extrapolation (i.e., generalization of TNRD [21], [28]), PDS-Net, i.e., PnP-PDS [50] using iteration-wise refining NNs, and the proposed Momentum-Net using extrapolation.

4.1. Experimental setup: Imaging

4.1.1. Sparse-view CT

To reconstruct a linear attenuation coefficient image $x\in {ℝ}^N$ from post-log sinogram $y\in {ℝ}^m$ in sparse-view CT, the MBIR problem (P1) considers a data-fit $f(x;y)=\frac{1}{2}\Vert y-Ax{\Vert }_W^2$ and the non-negativity constraint $\mathcal{X}={[0,\infty )}^N$, where $A\in {ℝ}^{m\times N}$ is an undersampled CT system matrix, $W\in {ℝ}^{m\times m}$ is a diagonal weighting matrix with elements $\{W_{m^{\prime },m^{\prime }}=p_m^{\prime 2}/(p_m^{\prime }+{\sigma }^2):\forall m^{\prime }\}$ based on a Poisson-Gaussian model [17], [67] for the pre-log raw measurements $p\in {ℝ}^m$ with electronic readout noise variance σ².

We simulated 2D sparse-view sinograms of size m = 888×123 – ‘detectors or rays’ × ‘regularly spaced projection views or angles’, where 984 is the number of full views – with GE LightSpeed fan-beam geometry corresponding to a monoenergetic source with 10⁵ incident photons per ray and no background events, and electronic noise variance σ² =5². We avoided an inverse crime in imaging simulation and reconstructed images of size N =420×420 with a coarser grid Δ_x=Δ_y =0.9766 mm; see details in [37, §V-A2].

4.1.2. LF photography using a focal stack

To reconstruct a LF $x={\left[x_1^T,\dots ,x_C^{\prime }\right]}^T\in {ℝ}^{S{N}^{\prime }}$ that consists of C′ sub-aperture images from focal stack measurements $y={\left[y_1^T,\dots ,y_C^T\right]}^T\in {ℝ}^{C{N}^{\prime }}$ that are collected by C photosensors, the MBIR problem (P1) considers a data-fit $f\text{(}x\text{;}y\text{)=}\frac{1}{2}\Vert y-Ax{\Vert }_2^2$ and a box constraint $\mathcal{X}={[0,U]}^{C^{\prime }{N}^{\prime }}$ with U =1 (or 255 without rescaling), where $A\in {ℝ}^{C{N}^{\prime }\times C^{\prime }{N}^{\prime }}$ is a system matrix of LF imaging system using a focal stack that is constructed blockwise with C · C′ different convolution matrices $\{{\tau }_c{A}_{c,c^{\prime }}^{\prime }\in {ℝ}^{N^{\prime }\times N^{\prime }}:c=1,\dots ,C,c^{\prime }=1,\dots ,C^{\prime }\}$ [47], [68], τ_c ∈ (0, 1] is a transparency coefficient for the cth detector, and N′ is the size of sub-aperture images, $x_{c^{\prime }}$, ∀c^′.³

Algorithm 2.

BCD-Net [25]

Require: $\{{\mathcal{R}}_{{\theta }^{(i)}}:i=1,\dots ,N_{\text{iter}}\}$, γ > 0, x(0) = x(−1), y

for i = 0, …, Niter−1 do

Image refining: $$z^{(i+1)}={\mathcal{R}}_{{\theta }^{(i+1)}}(x^{(i)})$$ (Alg.2.1)

MBIR: $$x^{(i+1)}=\underset{x\in \mathcal{X}}{\mathrm{argmin}}F(x;y,z^{(i+1)})$$ (Alg.2.2)

end for

In general, a LF photography system using a focal stack is extremely under-determined, because C ≪ C′.

To avoid an inverse crime, our imaging simulation used higher-resolution synthetic LF dataset [70] (we converted the original RGB sub-aperture images to grayscale ones by the “rgb2gray.m” function in MATLAB, for simplicity and smaller memory requirements in training). We simulated C =5 focal stack images of size N′ =255×255 with 40 dB AWGN that models electronic noise at sensors, and setting transparency coefficients τ_c as 1, for c=1, …, C. The sensor positions were chosen such that five sensors focus at equally spaced depths; specifically, the closest sensor to scenes and farthest sensor from scenes focus at two different depths that correspond to ‘disp_min+ 0.2’ and ‘disp_max− 0.2’, respectively, where disp_max and disp_min are the approximate maximum and minimum disparity values specified in [70]. We reconstructed 4D LFs that consist of S = 9 × 9 sub-aperture images of size N′ =255×255, with a coarser grid Δ_x=Δ_y =0.13572 mm.

4.2. Experimental setup: INNs

4.2.1. Parameters of INNs

The parameters for the INNs compared in sparse-view CT experiments were defined as follows. We considered two BCD-Nets (see Algorithm 2): for one BCD-Net, we applied the APG method [49] with 10 inner iterations to (Alg.2.2), and set N_iter = 30; for the other BCD-Net, we applied the APG method with 3 inner iterations to (Alg.2.2), and set N_iter = 45. For ADMM-Net, we used the identical configurations as BCD-Net and set the ADMM penalty parameter to γ in (P1), similar to [16]. For Momentum-Net without extrapolation, we chose N_iter = 100 and ρ = 1 − ε. For the proposed Momentum-Net, we chose N_iter = 100 and ρ = 0.5. For PDS-Net, we set the first step size to γ₁ = γ⁻¹ and the second step size to ${\gamma }_2={\gamma }_1^{-1}{\sigma }_{\max }^{-1}(M)$, per [50]. For performance comparisons between different INNs, all the INNs used sCNN refiners (18) with {R,K = 7²} to avoid the overfitting/hallucination risks. For Momentum-Net using dCNN refiners, we chose L=4 layer dCNN (19) using R = 3² filters and K = 64 feature maps, following [46]. (The chosen parameters gave lower RMSE values than {L=6, R=3³, K=64}, for identical regularization parameters.) For comparing different MBIR methods, Momentum used extrapolation, i.e., (Alg.1.2) with (8) and (10), and {R = 7², K = 9²} for (18). We designed the majorization matrices as $\{{\tilde{M}}^{(i+1)}=\mathrm{diag}(A^{T}WA1)+\gamma I:i\ge 0\}$, using Lemma A.7 (A and W have nonnegative entries) and setting λ=1 by (7). We set an initial point of INNs, x⁽⁰⁾, to filtered-back projection (FBP) using a Hanning window. The regularization parameters of all INNs were selected by the scheme in §3.2 with χ^⁎ = 167.64. (This factor was estimated from the carefully chosen regularization parameter for sparse-view CT MBIR experiments using learned convolutional regularizers in [24].)

The parameters for the INNs compared in experiments of LF photography using a focal stack were defined as follows. We considered two BCD-Nets and two ADMM-Nets with the identical parameters listed above. For Momentum-Net without extrapolation and the proposed Momentum-Net, we set N_iter = 100 and ρ = 1 − ε. For PDS-Net, we used the identical parameter setup described above. For performance comparisons between different INNs, all the INNs used sCNN refiners (18) with {R = 5², K = 32} (to avoid the overfitting risks) in the epipolar domain. For Momentum-Net using dCNN refiners, we chose L=6 layer dCNN (19) using R = 3² filters and K = 16 feature maps. (The chosen parameters gave most accurate performances over the following setups, {L = 4, R = 3², K = 16, 32, 64}, given the identical regularization parameters.) To generate ${\mathcal{R}}_{{\theta }^{(i+1)}}(x^{(i)})$ in (Alg.1.1), we applied a sCNN (18) with {R=5², K =32} or a dCNN (19) with {L=6, R=3², K = 16} to two sets of horizontal and vertical epipolar plane images, and took the average of two LFs that were permuted back from refined horizontal and vertical epipolar plane image sets, ∀i.⁴ We designed the majorization matrices as $\{{\tilde{M}}^{(i+1)}=\mathrm{diag}(A^{T}A1)+\gamma I:i\ge 0\}$, using Lemma A.7 and setting λ = 1 by (7). We set an initial point of INNs, x⁽⁰⁾, to A^Ty rescaled in the interval [0,1] (i.e., dividing by its max value). The regularization parameters (i.e., γ in BCD-Net/Momentum-Net, the ADMM penalty parameter in ADMM-Net, and the first step size in PDS-Net) were selected by the proposed scheme in §3.2 with χ^⁎ =1.5. (We tuned the factor to achieve the best performances).

For different combinations of INNs and sCNN refiner (18)/dCNN refiner (19), we use the following naming convention: ‘the INN name’-’sCNN’ or ‘dCNN’.

4.2.2. Training INNs

For sparse-view CT experiments, we trained all the INNs from the chest CT dataset with $\{x_s,y_s,f_s(x;y_s)=\frac{1}{2}\Vert y_s-Ax{\Vert }_{W_s}^2,{\tilde{M}}_s:s=1,\dots ,S,S=142\}$; we constructed the dataset by using XCAT phantom slices [71]. The CT experiment has mild data-fit variations across training samples: the standard deviation of the condition numbers $(\triangleq {\sigma }_{\max }(\cdot )/{\sigma }_{\min }(\cdot ))$ of $\{M_{f_s}=\mathrm{diag}(A^T{W}_{s}A1):\forall s\}$ is 1.1. For experiments of LF photography using a focal stack, we trained all the INNs from the LF photography dataset with $\{y_s,f_s(x;y_s)=\frac{1}{2}{\Vert y_s-A_{s}x\Vert }_2^2,{\tilde{M}}_s:s=1,\dots ,S,S=21\}$ and two sets of ground truth epipolar images, {x_s,epi-h, x_s,epi-v : s = 1, …, S, S = 21·(255·9)}; we constructed the dataset by excluding four unrealistic “stratified” scenes from the original LF dataset in [70] that consists of 28 LFs with diverse scene parameter and camera settings. The LF experiment has large data-fit variations across training samples: the standard deviation of the condition numbers of $\{M_{f_s}=\mathrm{diag}(A_s^T{A}_{s}1):\forall s\}$ is 2245.5.

In training INNs for both the applications, if not specified, we used identical training setups. At each iteration of INNs, we solved (P2) with the mini-batch version of Adam [72] and trained iteration-wise sCNNs (18) or dCNNs (19). We selected the batch size and the number of epochs as follows: for sparse-view CT reconstruction, we chose them as 20 & 300, and 20 & 200 for sCNN and dCNN refiners, respectively; for LF photography using a focal stack, we chose them as 200 & 200, and 200 & 100, for sCNN and dCNN refiners, respectively. We chose the learning rates for (encoding/decoding) filters and thresholding values $\{{\alpha }_k^{(i+1)}:\forall k\}$ in sCNNs (18), as 10⁻³ and 10⁻¹, respectively; we reduced the learning rates by 10% every 10 epochs. At the first iteration, we initialized filter coefficients with Kaiming uniform initialization [73]; in the later iteration, i.e., at the ith INN iteration, for i ≥ 2, we initialized filter coefficients from those learned from the previous iteration, i.e., (i−1)th iteration (this also applies to initializing thresholding values).

4.2.3. Testing trained INNs

In sparse-view CT reconstruction experiments, we tested trained INNs to two samples where ground truth images and the corresponding inverse covariance matrices (i.e., W in §4.1.1) sufficiently differ from those in training samples (i.e., they are a few cm away from training images). We evaluated the reconstruction quality by the most conventional error metric in CT application, RMSE (in HU), in a region of interest (ROI), where RMSE and HU stand for root-mean-square error and (modified) Hounsfield unit, respectively, and the ROI was a circular region that includes all the phantom tissues. The RMSE is defined by $\mathrm{RMSE}(x^{\star },x^{\text{true}})\triangleq {({\sum }_{j=1}^{N_{\text{ROI}}}{(x_j^{\star }-x_j^{\text{true}})}^2/N_{\text{ROI}})}^{1/2}$, where x^⁎ is a reconstructed image, x^true is a ground truth image, and N_ROI is the number of pixels in a ROI. In addition, we compared the trained Momentum-Net (using extrapolation) to a standard MBIR method using a hand-crafted EP regularizer, and an MBIR model using a learned convolutional regularizer [24], [37] which is the state-of-the-art MBIR method within an unsupervised learning setup. We finely tuned their regularization parameters to achieve the lowest RMSE. See details of these two MBIR models in §A.9.2.

In experiments of LF photography using a focal stack, we tested trained INNs to three samples of which scene parameter and camera settings are different from those in training samples (all training and testing samples have different camera and scene parameters). We evaluated the reconstruction quality by the most conventional error metric in LF photography application, PSNR (in dB), where PSNR stands for peak signal-to-noise. In addition, we compared the trained Momentum-Net (using extrapolation) to MBIR method using the state-of-the-art non-trained regularizer, 4D EP introduced in [47]. (The low-rank plus sparse tensor decomposition model [68], [74] failed when inverse crimes and measurement noise are considered.) We finely tuned its regularization parameter to achieve the lowest RMSE values. See details of this MBIR model in §A.9.3. We further investigated impacts of the LF MBIR quality on a higher-level depth estimation application, by applying the robust Spinning Parallelogram Operator (SPO) depth estimation method [75] to reconstructed LFs.

For comparing Momentum-Net with PDS-Net, we measured quality of refined images, z⁽ⁱ⁺¹⁾ in (Alg.1.1), because PDS-Net is a hard-refiner.

The imaging simulation and reconstruction experiments were based on the Michigan image reconstruction toolbox [76], and training INNs, i.e., solving (P2), was based on PyTorch (for sparse-view CT, we used ver. 1.2.0; for LF photography using a focal stack, we used ver. 0.3.1). For sparse-view CT, single-precision MATLAB and PyTorch implementations were tested on 2.6 GHz Intel Core i7 CPU with 16 GB RAM, and 1405 MHz Nvidia Titan Xp GPU with 12 GB RAM, respectively. For LF photography using a focal stack, they were tested on 3.5 GHz AMD Threadripper 1920X CPU with 32 GB RAM, and 1531 MHz Nvidia GTX 1080 Ti GPU with 11 GB RAM, respectively.

4.3. Comparisons between different INNs

First, compare sCNN results in Figs. 4–5 and Figs. 6–7, for sparse-view CT and LF photography using a focal stack, respectively. For both applications, the proposed Momentum-Net using extrapolation significantly improves MBIR speed and accuracy, compared to the existing soft-refining INNs, [21]–[23], [28], [30] that correspond to BCD-Net [25] or Momentum-Net using no extrapolation, and ADMM-Net [20], [23], [41], and the existing hard-refining INN PDS-Net [50]. (Note that BCD-Net and Momentum-Net require slightly less computational complexity per INN iteration, compared to ADMM-Net and PDS-Net, respectively, due to having fewer modules.) Fig. 5 shows that to reach the 24 HU RMSE value in sparse-view CT reconstruction, the proposed Momentum-Net decreases MBIR time by 53.3% and 62.5%, compared to Momentum-Net without extrapolation and BCD-Net using three inner iterations, respectively. Fig. 7 shows that to reach the 32 dB PSNR value in LF reconstruction from a focal stack, the proposed Momentum-Net decreases MBIR time by 36.5% and 61.5%, compared to Momentum-Net without extrapolation and BCD-Net using three inner iterations, respectively. In addition, Figs. 5 and 7 show that using extrapolation, i.e., (Alg.1.2) with (8)–(10), improves the performance of Momentum-Net versus iterations.

Fig. 4.

RMSE minimization comparisons between different INNs for sparse-view CT (fan-beam geometry with 12:5% projections views and 10⁵ incident photons; (a) averaged RMSE values across two test refined images; (b) averaged RMSE values across two test reconstructed images).

Fig. 6.

PSNR maximization comparisons between different INNs in LF photography using a focal stack (LF photography systems with C = 5 detectors obtain a focal stack of LFs consisting of S =81 sub-aperture images; (a) averaged RMSE values across two test refined images); (b) averaged RMSE values across two test reconstructed images.

Fig. 7.

PSNR maximization comparisons between different INNs in LF photography using a focal stack (LF photography systems with C = 5 detectors obtain a focal stack of LFs consisting of S =81 sub-aperture images; averaged PSNR values across three test reconstructed images).

We conjecture that the larger performance gap between soft-refiner Momentum-Net and hard-refiner PDS-Net, in Fig. 4(a) compared to Fig. 6(a), is because the LF problem needs stronger regularization, i.e., a smaller tuned factor χ^⁎ in (20), than the CT problem. Similarly, comparing Fig. 4(b) to Fig. 6(b) shows that the LF problem has small performance gaps between BCD-Net and ADMM-Net.

For both the applications, using dCNN refiners (19) instead of sCNN refiners (18) has a negligible effect on total run time of Momentum-Net, because reconstruction time of MBIR modules (Alg.1.3) (in CPUs) dominates inference time of image refining modules (Alg.1.1) (in GPUs). Compare results between Momentum-Net-sCNN and -dCNN in Figs. 5 & 7 and Tables A.1 & A.2.

4.4. Comparisons between different MBIR methods

In sparse-view CT using 12.5% of the full projection views, Fig. 8(b)–(e) and Table A.1(b)–(f) show that the proposed Momentum-Net achieves significantly better reconstruction quality compared to the conventional EP MBIR method and the state-of-the-art MBIR method within an unsupervised learning setup, MBIR model using a learned convolutional regularizer [24], [37]. In particular, Momentum-Net recovers both low- and high-contrast regions (e.g., soft tissues and bones, respectively) more accurately than MBIR using a learned convolutional regularizer; see Fig. 8(c)–(e). In addition, when their shallow convolutional autoencoders need identical computational complexities, Momentum-Net achieves much faster MBIR compared to MBIR using a learned convolutional regularizer; see Table A.1(c)–(d).

Fig. 8.

Comparison of reconstructed images from different MBIR methods in sparse-view CT (fan-beam geometry with 12.5% projections views and 10⁵ incident photons; images outside zoom-in boxes are magnified to better show differences; display window [800,1200] HU). See also Fig. A.2.

In LF photography using five focal sensors, regardless of scene parameters and camera settings, Momentum-Net consistently achieves significantly more accurate image recovery, compared to MBIR model using the state-of-the-art non-trained regularizer, 4D EP [47]. The effectiveness of Momentum-Net is more evident for a scene with less fine details. See Fig. 9(b)–(d) and Table A.2(b)–(d). Regardless of the scene distances from LF imaging systems, the reconstructed LFs by Momentum-Net significantly improve the depth estimation accuracy over those reconstructed by the state-of-the-art non-trained regularizer, 4D EP [47]. See Fig. 10(c)–(e) and Table A.3(c)–(e).

Fig. 9.

Error map comparisons of reconstructed sub-aperture images from different MBIR methods in LF photography using a focal stack (LF photography systems with C = 5 detectors capture a focal stack of LFs consisting of S = 81 sub-aperture images; sub-aperture images at the (5,5)th angular coordinate; the PSNR values in parenthesis were measured from reconstructed LFs). See also Fig. A.2.

Fig. 10.

Comparisons of estimated depths from LFs reconstructed by different MBIR methods in LF photograph using a focal stack (LF photography systems with C = 5 detectors capture a focal stack of LFs consisting of S = 81 sub-aperture images; SPO depth estimation [75] was applied to reconstructed LFs in Fig. 9; display window in meters). See also Fig. A.2.

In general, Momentum-Net needs more computations per iteration than EP MBIR, because its refining NNs use more and larger filters than the small finite-difference filters in EP MBIR, and EP MBIR algorithms can be often further accelerated by gradient approximations, e.g., ordered-subsets methods [77], [78].

5. CONCLUSIONS

Developing rapidly converging INNs is important, because 1) it leads to fast MBIR by reducing the computational complexity in calculating data-fit gradients or applying refining NNs, and 2) training INNs with many iterations requires long training time or it is challenging when refining NNs are fixed across INN iterations. The proposed Momentum-Net framework is applicable for a wide range of inverse problems, while achieving fast and convergent MBIR. To achieve fast MBIR, Momentum-Net uses momentum in extrapolation modules, and noniterative MBIR modules at each iteration via majorizers. For sparse-view CT and LF photography using a focal stack, Momentum-Net achieves faster and more accurate MBIR compared to the existing soft-refining INNs, [21]–[23], [28], [30] that correspond to BCD-Net [25] or Momentum-Net using no extrapolation, and ADMM-Net [20], [23], [41], and the existing hard-refining INN PDS-Net [50]. When an application needs strong regularization strength, e.g., LF photography using limited detectors, using dCNN refiners with moderate depth significantly improves the MBIR accuracy of Momentum-Net compared to sCNNs, only marginally increasing total MBIR time. In addition, Momentum-Net guarantees convergence to a fixed-point for general differentiable (non)convex MBIR functions (or data-fit terms) and convex feasible sets, under some mild conditions and two asymptotic conditions. The proposed regularization parameter selection scheme uses the “spectral spread” of majorization matrices, and is useful to consider data-fit variations across training/testing samples.

There are a number of avenues for future work. First, we expect to further improve performances of Momentum-Net (e.g., MBIR time and accuracy) by using sharper majorizer designs. Second, we expect to further reduce MBIR time of Momentum-Net with the stochastic gradient perspective (e.g., ordered subset [77], [78]). On the regularization parameter selection side, our future work is learning the factor χ in (20) from datasets while training refining NNs.

Biography

Il Yong Chun (S’14–M’16) is a tenure-track Assistant Professor of EE at the University of Hawai’i, Mānoa (UHM). He received B.Eng. degree from Korea University in 2009, and the Ph.D. degree from Purdue University in 2015, both in electrical engineering. During his Ph.D., he worked with Intel Labs, Samsung Advanced Institute of Technology, and Neuroscience Research Institute, as a Research Intern or a Visiting Lecturer. Prior to joining UHM, he was a Postdoctoral Research Associate in Mathematics, Purdue University, and a Research Fellow in EECS, the University of Michigan, from 2015 to 2016 and from 2016 to 2019, respectively. His research interests include machine learning & artificial intelligence, optimization, compressed sensing, and adaptive signal processing, applied to medical imaging, computational photography, and biomedical image computing.

Zhengyu Huang received the B.S. degree in Physics from Hong Kong Baptist University in 2015 and is now pursuing Ph.D. degree in electrical and computer engineering at the University of Michigan. His research interests include image reconstruction, optics and deep learning. In particular, he is interested in applying deep learning methods to novel imaging systems for light field reconstruction and depth estimation.

Hongki Lim (S’17) received the B.S.E.E. degree from Inha University in 2012 and the Ph.D. degree in electrical and computer engineering from the University of Michigan in 2020. Before starting PhD program at University of Michigan, he worked at the strategic planning team of Samsung Electronics for two years. His research interest includes image reconstruction, medical image analysis, and deep learning.

Jeffrey A. Fessler (F’06) is the William L. Root Professor of EECS at the University of Michigan. He received the BSEE degree from Purdue University in 1985, the MSEE degree from Stanford University in 1986, and the M.S. degree in Statistics from Stanford University in 1989. From 1985 to 1988 he was a National Science Foundation Graduate Fellow at Stanford, where he earned a Ph.D. in electrical engineering in 1990. He has worked at the University of Michigan since then. From 1991 to 1992 he was a Department of Energy Alexander Hollaender Post-Doctoral Fellow in the Division of Nuclear Medicine. From 1993 to 1995 he was an Assistant Professor in Nuclear Medicine and the Bioengineering Program. He is now a Professor in the Departments of Electrical Engineering and Computer Science, Radiology, and Biomedical Engineering. He became a Fellow of the IEEE in 2006, for contributions to the theory and practice of image reconstruction. He received the Francois Erbsmann award for his IPMI93 presentation, and the Edward Hoffman Medical Imaging Scientist Award in 2013. He has served as an associate editor for the IEEE Transactions on Medical Imaging, the IEEE Signal Processing Letters, the IEEE Transactions on Image Processing, the IEEE Transactions on Computational Imaging, and is currently serving as an associate editor for SIAM J. on Imaging Science. He has chaired the IEEE T-MI Steering Committee and the ISBI Steering Committee. He was co-chair of the 1997 SPIE conference on Image Reconstruction and Restoration, technical program co-chair of the 2002 IEEE International Symposium on Biomedical Imaging (ISBI), and general chair of ISBI 2007. His research interests are in statistical aspects of imaging problems, and he has supervised doctoral research in PET, SPECT, X-ray CT, MRI, and optical imaging problems.