Convolutional Analysis Operator Learning: Acceleration and Convergence

Abstract

Convolutional operator learning is gaining attention in many signal processing and computer vision applications. Learning kernels has mostly relied on so-called patch-domain approaches that extract and store many overlapping patches across training signals. Due to memory demands, patch-domain methods have limitations when learning kernels from large datasets – particularly with multi-layered structures, e.g., convolutional neural networks – or when applying the learned kernels to high-dimensional signal recovery problems. The so-called convolution approach does not store many overlapping patches, and thus overcomes the memory problems particularly with careful algorithmic designs; it has been studied within the “synthesis” signal model, e.g., convolutional dictionary learning. This paper proposes a new convolutional analysis operator learning (CAOL) framework that learns an analysis sparsifying regularizer with the convolution perspective, and develops a new convergent Block Proximal Extrapolated Gradient method using a Majorizer (BPEG-M) to solve the corresponding block multi-nonconvex problems. To learn diverse filters within the CAOL framework, this paper introduces an orthogonality constraint that enforces a tight-frame filter condition, and a regularizer that promotes diversity between filters. Numerical experiments show that, with sharp majorizers, BPEG-M significantly accelerates the CAOL convergence rate compared to the state-of-the-art block proximal gradient (BPG) method. Numerical experiments for sparse-view computational tomography show that a convolutional sparsifying regularizer learned via CAOL significantly improves reconstruction quality compared to a conventional edge-preserving regularizer. Using more and wider kernels in a learned regularizer better preserves edges in reconstructed images.

I. Introduction

LEARNING convolutional operators from large datasets is a growing trend in signal/image processing, computer vision, and machine learning. The widely known patch-domain approaches for learning kernels (e.g., filter, dictionary, frame, and transform) extract patches from training signals for simple mathematical formulation and optimization, yielding (sparse) features of training signals [1]–[9]. Due to memory demands, using many overlapping patches across the training signals hinders using large datasets and building hierarchies on the features, e.g., deconvolutional neural networks [10], convolutional neural network (CNN) [11], and multi-layer convolutional sparse coding [12]. For similar reasons, the memory requirement of patch-domain approaches discourages learned kernels from being applied to large-scale inverse problems.

To moderate these limitations of the patch-domain approach, the so-called convolution perspective has been recently introduced by learning filters and obtaining (sparse) representations directly from the original signals without storing many overlapping patches, e.g., convolutional dictionary learning (CDL) [10], [13]–[17]. For large datasets, CDL using careful algorithmic designs [16] is more suitable for learning filters than patch-domain dictionary learning [1]; in addition, CDL can learn translation-invariant filters without obtaining highly redundant sparse representations [16]. The CDL method applies the convolution perspective for learning kernels within “synthesis” signal models. Within “analysis” signal models, however, there exist no prior frameworks using the convolution perspective for learning convolutional operators, whereas patch-domain approaches for learning analysis kernels are introduced in [3], [4], [6]–[8]. (See brief descriptions about synthesis and analysis signal models in [4, Sec. I].)

Researchers interested in dictionary learning have actively studied the structures of kernels learned by the patch-domain approach [3], [4], [6]–[8], [18]–[20]. In training CNNs (see Appendix A), however, there has been less study of filter structures having non-convex constraints, e.g., orthogonality and unit-norm constraints in Section III, although it is thought that diverse (i.e., incoherent) filters can improve performance for some applications, e.g., image recognition [9]. On the application side, researchers have applied (deep) NNs to signal/image recovery problems. Recent works combined model-based image reconstruction (MBIR) algorithm with image refining networks [21]–[30]. In these iterative NN methods, refining NNs should satisfy the non-expansiveness for fixed-point convergence [29]; however, their trainings lack consideration of filter diversity constraints, e.g., orthogonality constraint in Section III, and thus it is unclear whether the trained NNs are nonexpansive mapping [30].

This paper proposes 1) a new convolutional analysis operator learning (CAOL) framework that learns an analysis sparsifying regularizer with the convolution perspective, and 2) a new convergent Block Proximal Extrapolated Gradient method using a Majorizer (BPEG-M [16]) for solving block multi-nonconvex problems [31]. To learn diverse filters, we propose a) CAOL with an orthogonality constraint that enforces a tight-frame (TF) filter condition in convolutional perspectives, and b) CAOL with a regularizer that promotes filter diversity. BPEG-M with sharper majorizers converges significantly faster than the state-of-the-art technique, Block Proximal Gradient (BPG) method [31] for CAOL. This paper also introduces a new X-ray computational tomography (CT) MBIR model using a convolutional sparsifying regularizer learned via CAOL [32].

The remainder of this paper is organized as follows. Section II reviews how learned regularizers can help solve inverse problems. Section III proposes the two CAOL models. Section IV introduces BPEG-M with several generalizations, analyzes its convergence, and applies a momentum coefficient formula and restarting technique from [16]. Section V applies the proposed BPEG-M methods to the CAOL models, designs two majorization matrices, and describes memory flexibility and applicability of parallel computing to BPEG-M-based CAOL. Section VI introduces the CT MBIR model using a convolutional regularizer learned via CAOL [32], along with its properties, i.e., its mathematical relation to a convolutional autoencoder, the importance of TF filters, and its algorithmic role in signal recovery. Section VII reports numerical experiments that show 1) the importance of sharp majorization in accelerating BPEG-M, and 2) the benefits of BPEG-M-based CAOL – acceleration, convergence, and memory flexibility. Additionally, Section VII reports sparse-view CT experiments that show 3) the CT MBIR using learned convolutional regularizers significantly improves the reconstruction quality compared to that using a conventional edge-preserving (EP) regularizer, and 4) more and wider filters in a learned regularizer better preserves edges in reconstructed images. Finally, Appendix A mathematically formulates unsupervised training of CNNs via CAOL, and shows that its updates attained via BPEG-M correspond to the three important CNN operators. Appendix B introduces some potential applications of CAOL to image processing, imaging, and computer vision.

II. Backgrounds: MBIR Using Learned Regularizers

To recover a signal $x\in {ℂ}^{N^{\prime }}$ from a data vector $y\in {ℂ}^m$, one often considers the following MBIR optimization problem (Appendix C provides mathematical notations): ${\text{argmin}}_{x\in \mathcal{X}}f\left(x;y\right)+\gamma g\left(x\right)$, where $\mathcal{X}$ is a feasible set, f(x; y) is data fidelity function that models imaging physics (or image formation) and noise statistics, γ > 0 is a regularization parameter, and g(x) is a regularizer, such as total variation [33, §2–3]. However, when inverse problems are extremely ill-conditioned, the MBIR approach using hand-crafted regularizers g(x) has limitations in recovering signals. Alternatively, there has been a growing trend in learning sparsifying regularizers (e.g., convolutional regularizers [16], [17], [32], [34], [35]) from training datasets and applying the learned regularizers to the following MBIR problem [33]:

$$\underset{x\in \mathcal{X}}{\text{argmin}}f\left(x;y\right)+\gamma g\left(x;{\mathcal{O}}^{\star }\right),$$ (B1)

where a learned regularizer $g\left(x;{\mathcal{O}}^{\star }\right)$ quantifies consistency between any candidate x and training data that is encapsulated in some trained sparsifying operators ${\mathcal{O}}^{\star }$. The diagram in Fig. 1 shows the general process from training sparsifying operators to solving inverse problems via (B1). Such models (B1) arise in a wide range of applications. See some examples in Appendix B.

Fig. 1.

A general flowchart from learning sparsifying operators $\mathcal{O}$ to solving inverse problems via MBIR using learned operators ${\mathcal{O}}^{\star }$; see Section II. For the lth training sample $x_l,\hspace{0.17em}F\left(\mathcal{O};x_l\right)$ measures its sparse representation or sparsification errors, and sparsity of its representation generated by $\mathcal{O}$.

This paper describes multiple aspects of learning convolutional regularizers. The next section first starts with proposing a new convolutional regularizer.

III. CAOL: Models Learning Convolutional Regularizers

The goal of CAOL is to find a set of filters that “best” sparsify a set of training images. Compared to hand-crafted regularizers, learned convolutional regularizers can better extract “true” features of estimated images and remove “noisy” features with thresholding operators. We propose the following CAOL model:

$$\begin{array}{l}\underset{D=\left[d_1,\dots ,d_K\right]}{\text{argmin}}\underset{\left\{z_{l,k}\right\}}{\min }F\left(D,\left\{z_{l,k}\right\}\right)+\beta g\left(D\right),\\ \hspace{1em}\hspace{0.17em}F\left(D,\left\{z_{l,k}\right\}\right):=\sum _{l=1}^L\sum _{k=1}^K\frac{1}{2}{\Vert d_k⊛x_l-z_{l,k}\Vert }_2^2+\alpha {\Vert z_{l,k}\Vert }_0,\end{array}$$ (P0)

where ⊛ denotes a convolution operator (see details about boundary conditions in the supplementary material), $\left\{x_l\in {ℂ}^N:l=1,\dots ,L\right\}$ is a set of training images, $\left\{d_k\in {ℂ}^R:k=1,\dots ,K\right\}$ is a set of convolutional kernels, $\left\{z_{l,k}\in {ℂ}^N:l=1,\dots ,L,k=1,\dots ,K\right\}$ is a set of sparse codes, and g(D) is a regularizer or constraint that encourages filter diversity or incoherence, α > 0 is a thresholding parameter controlling the sparsity of features {z_l,k}, and β > 0 is a regularization parameter for g(D). We group the K filters into a matrix $D\in {ℂ}^{R\times K}$:

$$D:=\left[\begin{array}{ccc}{d}_1& \dots & d_K\end{array}\right].$$ (1)

For simplicity, we fix the dimension for training signals, i.e., $\left\{x_l,z_{l,k}\in {ℂ}^N\right\}$, but the proposed model (P0) can use training signals of different dimension, i.e., $\left\{x_l,z_{l,k}\in {ℂ}^{N_l}\right\}$. For sparse-view CT in particular, the diagram in Fig. 2 shows the process from CAOL (P0) to solving its inverse problem via MBIR using learned convolutional regularizers.

Fig. 2.

A flowchart from CAOL (P0) to MBIR using a convolutional sparsifying regularizer learned via CAOL (P3) in sparse-view CT. See details of the CAOL process (P0) and its variants (P1)–(P2), and the CT MBIR process (P3) in Section III and Section VI, respectively.

The following two subsections design the constraint or regularizer g(D) to avoid redundant filters (without it, all filters could be identical).

A. CAOL with Orthogonality Constraint

We first propose a CAOL model with a nonconvex orthogonality constraint on the filter matrix D in (1):

$$\underset{D}{\text{argmin}}\underset{\left\{z_{l,k}\right\}}{\min }F\left(D,\left\{z_{l,k}\right\}\right)\hspace{1em}\text{subj}.\hspace{0.17em}\text{to}\hspace{0.17em}D{D}^H=\frac{1}{R}\cdot I.$$ (P1)

The orthogonality condition $D{D}^H=\frac{1}{R}I$ in (P1) enforces a TF condition on the filters {d_k} in CAOL (P0). Proposition 3.1 below formally states this relation.

Proposition 3.1 (Tight-frame filters). Filters satisfying the orthogonality constraint $D{D}^H=\frac{1}{R}I$ satisfy the following TF condition in a convolution perspective:

$$\sum _{k=1}^K{\Vert d_k⊛x\Vert }_2^2={\Vert x\Vert }_2^2,\hspace{1em}\forall x\in {ℂ}^N,$$ (2)

for both circular and symmetric boundary conditions.

Proof: See Section S.I of the supplementary material. ■

Proposition 3.1 corresponds to a TF result from patch-domain approaches; see Section S.I. (Note that the patch-domain approach in [6, Prop. 3] requires R = K.) However, we constrain the filter dimension to be R ≤ K to have an efficient solution for CAOL model (P1); see Proposition 5.4 later. The following section proposes a more flexible CAOL model in terms of the filter dimensions R and K.

B. CAOL with Diversity Promoting Regularizer

As an alternative to the CAOL model (P1), we propose a CAOL model with a diversity promoting regularizer and a nonconvex norm constraint on the filters {d_k}:

$$\begin{array}{l}\underset{D}{\text{argmin}}\underset{\left\{z_{l,k}\right\}}{\min }F\left(D,\left\{z_{l,k}\right\}\right)+\frac{\beta }{2}\stackrel{=:g_{\text{div}}\left(D\right)}{\overbrace{{\Vert D^{H}D-\frac{1}{R}\cdot I\Vert }_{\text{F}}^2}},\\ \text{ subject to }{\Vert d_k\Vert }_2^2=\frac{1}{R},\hspace{1em}k=1,\dots ,K.\end{array}$$ (P2)

In the CAOL model (P2), we consider the following:

• The constraint in (P2) forces the learned filters {d_k} to have uniform energy. In addition, it avoids the “scale ambiguity” problem [36].

• The regularizer in (P2), g_div(D), promotes filter diversity, i.e., incoherence between d_k and $\left\{d_{k^{\prime }}:k^{\prime }\ne k\right\}$, measured by ${\left|\langle d_k,d_{k^{\prime }}\rangle \right|}^2$ for k ≠ k^′.

When R = K and β → ∞, the model (P2) becomes (P1) since $D^{H}D=\frac{1}{R}I$ implies $D{D}^H=\frac{1}{R}I$ (for square matrices A and B, if AB = I then BA = I). Thus (P2) generalizes (P1) by relaxing the off-diagonal elements of the equality constraint in (P1). (In other words, when R = K, the orthogonality constraint in (P1) enforces the TF condition and promotes the filter diversity.) One price of this generalization is the extra tuning parameter β.

(P1)–(P2) are challenging nonconvex optimization problems and block optimization approaches seem suitable. The following section proposes a new block optimization method with momentum and majorizers, to rapidly solve the multiple block multi-nonconvex problems proposed in this paper, while guaranteeing convergence to critical points.

IV. BPEG-M: Solving Block Multi-Nonconvex Problems with Convergence Guarantees

This section describes a new optimization approach, BPEG-M, for solving block multi-nonconvex problems like a) CAOL (P1)–(P2),¹ b) CT MBIR (P3) using learned convolutional regularizer via (P1) (see Section VI), and c) “hierarchical” CAOL (A1) (see Appendix A).

A. BPEG-M – Setup

We treat the variables of the underlying optimization problem either as a single block or multiple disjoint blocks. Specifically, consider the following block multi-nonconvex optimization problem:

$$\min F\left(x_1,\dots ,x_B\right):=f\left(x_1,\dots ,x_B\right)+\sum _{b=1}^B{g}_b\left(x_b\right),$$ (3)

where variable x is decomposed into B blocks $x_1,\dots ,x_B\left(\left\{x_b\in {ℝ}^{n_b}:b=1,\dots ,B\right\}\right)$, f is assumed to be continuously differentiable, but functions {g_b : b = 1, …, B} are not necessarily differentiable. The function g_b can incorporate the constraint $x_b\in {\mathcal{X}}_b$, by allowing any g_b to be extended-valued, e.g., g_b(x_b) = ∞ if $x_b\notin {\mathcal{X}}_b$, for b = 1, …, B. It is standard to assume that both f and {g_b} are closed and proper and the sets $\left\{{\mathcal{X}}_b\right\}$ are closed and nonempty. We do not assume that f, {g_b}, or $\left\{{\mathcal{X}}_b\right\}$ are convex. Importantly, g_b can be a nonconvex ℓ^p quasi-norm, p ∈ [0, 1). The general block multi-convex problem in [16], [38] is a special case of (3).

The BPEG-M framework considers a more general concept than Lipschitz continuity of the gradient as follows:

Definition 4.1 (M-Lipschitz continuity). A function $g:{ℝ}^n\to {ℝ}^n$ is M-Lipschitz continuous on ${ℝ}^n$ if there exist a (symmetric) positive definite matrix M such that

$${\Vert g\left(x\right)-g\left(y\right)\Vert }_{M^{-1}}\le {\Vert x-y\Vert }_M,\hspace{1em}\forall x,y,$$

where ${\Vert x\Vert }_M^2:=x^{T}Mx$.

Lipschitz continuity is a special case of M-Lipschitz continuity with M equal to a scaled identity matrix with a Lipschitz constant of the gradient ∇f (e.g., for $f\left(x\right)=\frac{1}{2}{\Vert Ax-b\Vert }_2^2$, the (smallest) Lipschitz constant of ∇f is the maximum eigenvalue of A^TA). If the gradient of a function is M-Lipschitz continuous, then we obtain the following quadratic majorizer (i.e., surrogate function [39], [40]) at a given point y without assuming convexity:

Lemma 4.2 (Quadratic majorization (QM) via M-Lipschitz continuous gradients). Let $f:{ℝ}^n\to ℝ$. If ∇f is M-Lipschitz continuous, then

$$f\left(x\right)\le f\left(y\right)+\langle \nabla f\left(y\right),x-y\rangle +\frac{1}{2}{\Vert x-y\Vert }_M^2,\hspace{1em}\forall x,y\in {ℝ}^n.$$

Proof: See Section S.II of the supplementary material. ■

Exploiting Definition 4.1 and Lemma 4.2, the proposed method, BPEG-M, is given as follows. To solve (3), we minimize a majorizer of F cyclically over each block x₁, …, x_B, while fixing the remaining blocks at their previously updated variables. Let $x_b^{\left(i+1\right)}$ be the value of x_b after its ith update, and define

$$f_b^{\left(i+1\right)}\left(x_b\right):=f\left(x_1^{\left(i+1\right)},\dots ,x_{b-1}^{\left(i+1\right)},x_b,x_{b+1}^{\left(i\right)},\dots x_B^{\left(i\right)}\right),\hspace{1em}\forall b,i.$$

At the bth block of the ith iteration, we apply Lemma 4.2 to functional $f_b^{\left(i+1\right)}\left(x_b\right)$ with a $M_b^{\left(i+1\right)}$-Lipschitz continuous gradient, and minimize the majorized function.² Specifically, BPEG-M uses the updates

$$\begin{array}{l}{x}_b^{\left(i+1\right)}=\underset{x_b}{\text{argmin}}\langle {\nabla }_{x_b}{f}_b^{\left(i+1\right)}({\stackrel{\prime }{x}}_b^{\left(i+1\right)}),x_b-{\stackrel{\prime }{x}}_b^{\left(i+1\right)}\rangle \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}+\frac{1}{2}{\Vert x_b-{\stackrel{\prime }{x}}_b^{\left(i+1\right)}\Vert }_{{\tilde{M}}_b^{\left(i+1\right)}}^2+g_b\left(x_b\right)\\ \hspace{1em}\hspace{1em}=\underset{x_b}{\text{argmin}}\frac{1}{2}\Vert x_b-({\stackrel{\prime }{x}}_b^{\left(i+1\right)}-{\left({\tilde{M}}_b^{\left(i+1\right)}\right)}^{-1}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\cdot {\nabla }_{x_b}{f}_b^{\left(i+1\right)}({\stackrel{\prime }{x}}_b^{\left(i+1\right)}))\Vert }_{{\tilde{M}}_b^{\left(i+1\right)}}^2+g_b\left(x_b\right)\\ \hspace{1em}\hspace{1em}={\text{Prox}}_{g_b}^{{\tilde{M}}_b^{\left(i+1\right)}}\underset{extrapolated\hspace{0.17em}gradient\hspace{0.17em}step\hspace{0.17em}using\hspace{0.17em}a\hspace{0.17em}majorizer\hspace{0.17em}\text{of }{f}_b^{\left(i+1\right)}}{\underbrace{\left({\stackrel{\prime }{x}}_b^{\left(i+1\right)}-{\left({\tilde{M}}_b^{\left(i+1\right)}\right)}^{-1}{\nabla }_{x_b}{f}_b^{\left(i+1\right)}\left({\stackrel{\prime }{x}}_b^{\left(i+1\right)}\right)\right)}},\end{array}$$ (4)

where

$${\stackrel{\prime }{x}}_b^{\left(i+1\right)}=x_b^{\left(i\right)}+E_b^{\left(i+1\right)}\left(x_b^{\left(i\right)}-x_b^{\left(i-1\right)}\right),$$ (5)

the proximal operator is defined by

$${\text{Prox}}_g^M\left(y\right):=\underset{x}{\text{argmin}}\frac{1}{2}{\Vert x-y\Vert }_M^2+g\left(x\right),$$

$\nabla f_b^{\left(i+1\right)}({\stackrel{\prime }{x}}_b^{\left(i+1\right)})$ is the block-partial gradient of f at ${\stackrel{\prime }{x}}_b^{\left(i+1\right)}$, an upper-bounded majorization matrix is updated by

$${\tilde{M}}_b^{\left(i+1\right)}={\lambda }_b\cdot M_b^{\left(i+1\right)}\succ 0,\hspace{1em}{\lambda }_b>1,$$ (6)

$$\underset{\_}{\overline{\begin{array}{l}\underset{\_}{\text{Algorithm 1 BPEG-M}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}}\\ \mathbf{\text{Require: }}\left\{x_b^{\left(0\right)}=x_b^{\left(-1\right)}:\forall b\right\},\left\{E_b^{\left(i\right)}\in \left[0,1\right],\forall b,i\right\},\hspace{0.17em}i=0\\ \mathbf{\text{while a stopping criterion is not satisfied}} \mathbf{\text{do}}\\ \mathbf{\text{for }}b=1,\dots ,B \mathbf{\text{do}}\\ \text{ Calculate }{M}_b^{\left(i+1\right)},{\tilde{M}}_b^{\left(i+1\right)}\text{ by }\left(6\right),\text{ and }{E}_b^{\left(i+1\right)}\text{ by }\left(7\right)\\ {\stackrel{\prime }{x}}_b^{\left(i+1\right)}=x_b^{\left(i\right)}+E_b^{\left(i+1\right)}\left(x_b^{\left(i\right)}-x_b^{\left(i-1\right)}\right)\\ {\stackrel{\prime }{x}}_b^{\left(i+1\right)}=\dots \\ {\text{ Prox}}_{g_b}^{{\tilde{M}}_b^{\left(i+1\right)}}\left({\stackrel{\prime }{x}}_b^{\left(i+1\right)}-{\left({\tilde{M}}_b^{\left(i+1\right)}\right)}^{-1}\nabla f_b^{\left(i+1\right)}({\stackrel{\prime }{x}}_b^{\left(i+1\right)})\right)\\ \mathbf{\text{end for}}\\ i=i+1\\ \mathbf{\text{end while}}\end{array}}}$$

and $M_b^{\left(i+1\right)}\in {ℝ}^{n_b\times n_b}$ is a symmetric positive definite majorization matrix of $\nabla f_b^{\left(i+1\right)}$. In (5), the ${ℝ}^{n_b\times n_b}$ matrix $E_b^{\left(i+1\right)}\underset{\_}{\succ }0$ is an extrapolation matrix that accelerates convergence in solving block multi-convex problems [16]. We design it in the following form:

$$E_b^{\left(i+1\right)}=e_b^{\left(i\right)}\cdot \frac{\delta \left({\lambda }_b-1\right)}{2\left({\lambda }_b+1\right)}\cdot {\left(M_b^{\left(i+1\right)}\right)}^{-1/2}{\left(M_b^{\left(i\right)}\right)}^{1/2},$$ (7)

for some $\left\{0\le e_b^{\left(i\right)}\le 1:\forall b,\hspace{0.17em}i\right\}$ and δ < 1, to satisfy condition (9) below. In general, choosing λ_b values in (6)–(7) to accelerate convergence is application-specific. Algorithm 1 summarizes these updates.

The majorization matrices $M_b^{\left(i\right)}$ and ${\tilde{M}}_b^{\left(i+1\right)}$ in (6) influence the convergence rate of BPEG-M. A tighter majorization matrix (i.e., a matrix giving tighter bounds in the sense of Lemma 4.2) provided faster convergence rate [41, Lem. 1], [16, Fig. 2–3]. An interesting observation in Algorithm 1 is that there exists a tradeoff between majorization sharpness via (6) and extrapolation effect via (5) and (7). For example, increasing λ_b (e.g., λ_b = 2) allows more extrapolation but results in looser majorization; setting λ_b → 1 results in sharper majorization but provides less extrapolation.

Fig. 3.

Cost minimization comparisons in CAOL (P1) with different BPG-type algorithms and datasets (R = K = 49 and α = 2.5 × 10⁻⁴; solution (31) was used for sparse code updates; BPG (Xu & Ying ‘17) [31] used the maximum eigenvalue of Hessians for Lipschitz constants; the cross mark x denotes a termination point). A sharper majorization leads to faster convergence of BPEG-M; for all the training datasets considered in this paper, the majorization matrix in Proposition 5.1 is sharper than those in Lemmas 5.2–5.3.

Remark 4.3. The proposed BPEG-M framework – with key updates (4)–(5) – generalizes the BPG method [31], and has several benefits over BPG [31] and BPEG-M introduced earlier in [16]:

The BPG setup in [31] is a particular case of BPEG-M using a scaled identity majorization matrix M_b with a Lipschitz constant of $\nabla f_b^{\left(i+1\right)}({\stackrel{\prime }{x}}_b^{\left(i+1\right)})$. The BPEG-M framework can significantly accelerate convergence by allowing sharp majorization; see [16, Fig. 2–3] and Fig. 3. This generalization was first introduced for block multi-convex problems in [16], but the proposed BPEG-M in this paper addresses the more general problem, block multi-(non)convex optimization.

• BPEG-M is useful for controlling the tradeoff between majorization sharpness and extrapolation effect in different blocks, by allowing each block to use different λ_b values. If tight majorization matrices can be designed for a certain block b, then it could be reasonable to maintain the majorization sharpness by setting λ_b very close to 1. When setting λ_b = 1 + ϵ (e.g., ϵ is a machine epsilon) and using $E_b^{\left(i+1\right)}=0$ (no extrapolation), solutions of the original and its upper-bounded problem become (almost) identical. In such cases, it is unnecessary to solve the upper bounded problem (4), and the proposed BPEG-M framework allows using the solution of $f_b^{\left(i+1\right)}\left(x_b\right)$ without QM; see Section V-B. This generalization was not considered in [31].

• The condition for designing the extrapolation matrix (7), i.e., (9) in Assumption 3, is more general than that in [16, (9)] (e.g., (10)). Specifically, the matrices $E_b^{\left(i+1\right)}$ and $M_b^{\left(i+1\right)}$ in (7) need not be diagonalized by the same basis.

The first two generalizations lead to the question, “Under the sharp QM regime (i.e., having tight bounds in Lemma 4.2), what is the best way in controlling {λ_b} in (6)–(7) in Algorithm 1?” Our experiments show that, if sufficiently sharp majorizers are obtained for partial or all blocks, then giving more weight to sharp majorization provides faster convergence compared to emphasizing extrapolation; for example, λ_b = 1 + ϵ gives faster convergence than λ_b = 2.

B. BPEG-M – Convergence Analysis

This section analyzes the convergence of Algorithm 1 under the following assumptions.

Assumption 1) F is proper and lower bounded in dom(F), f is continuously differentiable, g_b is proper lower semicontinuous, ∀b.³ (3) has a critical point $\overline{x}$, i.e., $0\in \partial F\left(\overline{x}\right)$, where ∂F(x) denotes the limiting subdifferential of F at x (see [42, §1.9], [43, §8]).

Assumption 2) The block-partial gradients of f, $\nabla f_b^{\left(i+1\right)}$, are $M_b^{\left(i+1\right)}$-Lipschitz continuous, i.e.,

$$\begin{array}{l}{\Vert {\nabla }_{x_b}{f}_b^{\left(i+1\right)}\left(u\right)-{\nabla }_{x_b}{f}_b^{\left(i+1\right)}\left(v\right)\Vert }_{{\left(M_b^{\left(i+1\right)}\right)}^{-1}}\\ \le {\Vert u-v\Vert }_{M_b^{\left(i+1\right)}},\end{array}$$ (8)

for $u,v\in {ℝ}^{n_b}$, and (unscaled) majorization matrices satisfy $m_b{I}_{n_b}\underset{\_}{\prec }{M}_b^{\left(i+1\right)}$ with 0 < m_b < ∞, ∀b, i.

Assumption 3) The extrapolation matrices $E_b^{\left(i+1\right)}\underset{\_}{\succ }0$ satisfy

$${\left(E_b^{\left(i+1\right)}\right)}^T{M}_b^{\left(i+1\right)}{E}_b^{\left(i+1\right)}\underset{\_}{\prec }\frac{{\delta }^2{\left({\lambda }_b-1\right)}^2}{4{\left({\lambda }_b+1\right)}^2}\cdot M_b^{\left(i\right)},$$ (9)

for any δ < 1, ∀b, i.

Condition (9) in Assumption 3 generalizes that in [16, Assumption 3]. If eigenspaces of $E_b^{\left(i+1\right)}$ and $M_b^{\left(i+1\right)}$ coincide (e.g., diagonal and circulant matrices), ∀i [16, Assumption 3], (9) becomes

$$E_b^{\left(i+1\right)}\underset{\_}{\prec }\frac{\delta \left({\lambda }_b-1\right)}{2\left({\lambda }_b+1\right)}\cdot {\left(M_b^{\left(i\right)}\right)}^{1/2}{\left(M_b^{\left(i+1\right)}\right)}^{-1/2},$$ (10)

as similarly given in [16, (9)]. This generalization allows one to consider arbitrary structures of $M_b^{\left(i\right)}$ across iterations.

Lemma 4.4 (Sequence bounds). Let $\left\{{\tilde{M}}_b:b=1,\dots ,B\right\}$ and {E_b: b = 1, …, B} be as in (6)–(7), respectively. The cost function decrease for the ith update satisfies:

$$\begin{array}{l}{F}_b\left(x_b^{\left(i\right)}\right)-F_b\left(x_b^{\left(i+1\right)}\right)\ge \frac{{\lambda }_b-1}{4}{\Vert x_b^{\left(i\right)}-x_b^{\left(i+1\right)}\Vert }_{M_b^{\left(i+1\right)}}^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}-\frac{\left({\lambda }_b-1\right){\delta }^2}{4}{\Vert x_b^{\left(i-1\right)}-x_b^{\left(i\right)}\Vert }_{M_b^{\left(i\right)}}^2\end{array}$$ (11)

Proof: See Section S.III of the supplementary material. ■

Lemma 4.4 generalizes [31, Lem. 1] using {λ_b = 2}. Taking the majorization matrices in (11) to be scaled identities with Lipschitz constants, i.e., $M_b^{\left(i+1\right)}=L_b^{\left(i+1\right)}\cdot I$ and $M_b^{\left(i\right)}=L_b^{\left(i\right)}\cdot I$, where $L_b^{\left(i+1\right)}$ and $L_b^{\left(i\right)}$ are Lipschitz constants, the bound (11) becomes equivalent to that in [31, (13)]. Note that BPEG-M for block multi-convex problems in [16] can be viewed within BPEG-M in Algorithm 1, by similar reasons in [31, Rem. 2] – bound (11) holds for the block multi-convex problems by taking $E_b^{\left(i+1\right)}$ in (10) as $E_b^{\left(i+1\right)}\underset{\_}{\prec }\delta \cdot {\left(M_b^{\left(i\right)}\right)}^{1/2}{\left(M_b^{\left(i+1\right)}\right)}^{-1/2}$ in [16, Prop. 3.2].

Proposition 4.5 (Square summability). Let $\left\{x^{\left(i+1\right)}:i\ge 0\right\}$ be generated by Algorithm 1. We have

$$\sum _{i=0}^{\infty }{\Vert x^{\left(i\right)}-x^{\left(i+1\right)}\Vert }_2^2<\infty .$$ (12)

Proof: See Section S.IV of the supplementary material. ■

Proposition 4.5 implies that

$${\Vert x^{\left(i\right)}-x^{\left(i+1\right)}\Vert }_2^2\to 0,$$ (13)

and (13) is used to prove the following theorem:

Theorem 4.6 (A limit point is a critical point). Under Assumptions 1–3, let {x⁽ⁱ⁺¹⁾ : i ≥ 0} be generated by Algorithm 1. Then any limit point $\overline{x}$ of {x⁽ⁱ⁺¹⁾ : i ≥ 0} is a critical point of (3). If the subsequence $\left\{x^{\left(i_j+1\right)}\right\}$ converges to $\overline{x}$, then

$$\underset{j\to \infty }{\lim }F\left(x^{\left(i_j+1\right)}\right)=F\left(\overline{x}\right).$$

Proof: See Section S.V of the supplementary material. ■

Finite limit points exist if the generated sequence {x⁽ⁱ⁺¹⁾ : i ≥ 0} is bounded; see, for example, [44, Lem. 3.2–3.3]. For some applications, the boundedness of {x⁽ⁱ⁺¹⁾ : i ≥ 0} can be satisfied by choosing appropriate regularization parameters, e.g., [16].

C. Restarting BPEG-M

BPG-type methods [16], [31], [38] can be further accelerated by applying 1) a momentum coefficient formula similar to those used in fast proximal gradient (FPG) methods [45]– [47], and/or 2) an adaptive momentum restarting scheme [48], [49]; see [16]. This section applies these two techniques to further accelerate BPEG-M in Algorithm 1.

First, we apply the following increasing momentum-coefficient formula to (7) [45]:

$$e_b^{\left(i+1\right)}=\frac{{\theta }^{\left(i\right)}-1}{{\theta }^{\left(i+1\right)}},\hspace{1em}{\theta }^{\left(i+1\right)}=\frac{1+\sqrt{1+4{\left({\theta }^{\left(i\right)}\right)}^2}}{2}.$$ (14)

This choice guarantees fast convergence of FPG method [45]. Second, we apply a momentum restarting scheme [48], [49], when the following gradient-mapping criterion is met [16]:

$$\cos \left(\Theta \left(M_b^{\left(i+1\right)}\left({\stackrel{\prime }{x}}_b^{\left(i+1\right)}-x_b^{\left(i+1\right)}\right),x_b^{\left(i+1\right)}-x_b^{\left(i\right)}\right)\right)>\omega ,$$ (15)

where the angle between two nonzero real vectors ϑ and ϑ^′ is $\Theta \left(\vartheta ,{\vartheta }^{\prime }\right):=\langle \vartheta ,{\vartheta }^{\prime }\rangle /\left({\Vert \vartheta \Vert }_2{\Vert {\vartheta }^{\prime }\Vert }_2\right)$ and ω ∈ [−1, 0]. This scheme restarts the algorithm whenever the momentum, i.e., $x_b^{\left(i+1\right)}-x_b^{\left(i\right)}$, is likely to lead the algorithm in an unhelpful direction, as measured by the gradient mapping at the $x_b^{\left(i+1\right)}$-update. We refer to BPEG-M combined with the methods (14)–(15) as restarting BPEG-M (reBPEG-M). Section S.VI in the supplementary material summarizes the updates of reBPEG-M.

To solve the block multi-nonconvex problems proposed in this paper (e.g., (P1)–(P3)), we apply reBPEG-M (a variant of Algorithm 1; see Algorithm S.1), promoting fast convergence to a critical point.

V. Fast and Convergent CAOL via BPEG-M

This section applies the general BPEG-M approach to CAOL. The CAOL models (P1) and (P2) satisfy the assumptions of BPEG-M; see Assumption 1–3 in Section IV-B. CAOL models (P1) and (P2) readily satisfy Assumption 1 of BPEG-M. To show the continuously differentiability of f and the lower boundedness of F, consider that 1) ${\sum }_l{\sum }_k\frac{1}{2}{\Vert d_k⊛x_l-z_{l,k}\Vert }_2^2$ in (P0) is continuously differentiable with respect to D and {z_l,k}; 2) the sequences {D⁽ⁱ⁺¹⁾} are bounded, because they are in the compact set ${\mathcal{D}}_{\left(\text{P}1\right)}=\left\{D:D{D}^H=\frac{1}{R}I\right\}$ and ${\mathcal{D}}_{\left(\text{P2}\right)}=\left\{d_k:{\Vert d_k\Vert }_2^2=\frac{1}{R},\forall k\right\}$ in (P1) and (P2), respectively; and 3) the positive thresholding parameter α ensures that the sequence $\left\{z_{l,k}^{\left(i+1\right)}\right\}$ is bounded (otherwise the cost would diverge). In addition, for both (P1) and (P2), the lower semicontinuity of regularizer g_b holds, ∀b. For D-optimization, the indicator function of the sets ${\mathcal{D}}_{\left(\text{P}1\right)}$ and ${\mathcal{D}}_{\left(\text{P}2\right)}$ lower semicontinuous, because the sets are compact. For {z_l,k}-optimization, the ℓ⁰-quasi-norm is a lower semicontinuous function. Assumptions 2 and 3 are satisfied with the majorization matrix designs in this section – see Sections V-A–V-B later – and the extrapolation matrix design in (7), respectively.

Since CAOL models (P1) and (P2) satisfy the BPEG-M conditions, we solve (P1) and (P2) by the reBPEG-M method with a two-block scheme, i.e., we alternatively update all filters D and all sparse codes {z_l,k : l = 1, …, L, k = 1, …, K}. Sections V-A and V-B describe details of D-block and {z_l,k}-block optimization within the BPEG-M framework, respectively. The BPEG-M-based CAOL algorithm is particularly useful for learning convolutional regularizers from large datasets because of its memory flexibility and parallel computing applicability, as described in Section V-C and Sections V-A–V-B, respectively.

A. Filter Update: $\text{D}$-Block Optimization

We first investigate the structure of the system matrix in the filter update for (P0). This is useful for 1) accelerating majorization matrix computation in filter updates (e.g., Lemmas 5.2–5.3) and 2) applying R × N-sized adjoint operators (e.g., ${\Psi }_l^H$ in (17) below) to an N-sized vector without needing the Fourier approach [16, Sec. V-A] that uses commutativity of convolution and Parseval’s relation. Given the current estimates of {z_l,k} : l = 1, …, L, k = 1, …, K}, the filter update problem of (P0) is equivalent to

$$\underset{\left\{d_k\right\}}{\text{argmin}}\frac{1}{2}\sum _{k=1}^K\sum _{l=1}^L{\Vert {\Psi }_l{d}_k-z_{l,k}\Vert }_2^2+\beta g\left(D\right),$$ (16)

where D is defined in (1), ${\Psi }_l\in {ℂ}^{N\times R}$ is defined by

$${\Psi }_l:=\left[\begin{array}{ccc}{P}_{B_1}{\widehat{x}}_l& \dots & P_{B_R}{\widehat{x}}_l\end{array}\right],$$ (17)

$P_{B_r}\in {ℂ}^{N\times \widehat{N}}$ is the rth (rectangular) selection matrix that selects N rows corresponding to the indices B_r = {r, …, r + N − 1} from $I_{\widehat{N}}$, $\left\{{\widehat{x}}_l\in {ℂ}^{\widehat{N}}:l=1,\dots ,L\right\}$ is a set of padded training data, $\widehat{N}=N+R-1$. Note that applying ${\Psi }_l^H$ in (17) to a vector of size N is analogous to calculating cross-correlation between ${\widehat{x}}_l$ and the vector, i.e., ${\left({\Psi }_l^H{\widehat{z}}_{l,k}\right)}_r={\sum }_{n=1}^N{\widehat{x}}_{n+r-1}^{\ast }{\left({\widehat{z}}_{l,k}\right)}_n$, r = 1, …, R. In general, $\stackrel{\wedge }{(\cdot )}$ denotes a padded signal vector.

1). Majorizer Design:

This subsection designs multiple majorizers for the D-block optimization and compares their required computational complexity and tightness. The next proposition considers the structure of Ψ_l in (17) to obtain the Hessian ${\sum }_{l=1}^L{\Psi }_l^H{\Psi }_l\in {ℂ}^{R\times R}$ in (16) for an arbitrary boundary condition.

Proposition 5.1 (Exact Hessian matrix M_D). The following matrix $M_D\in {ℂ}^{R\times R}$ is identical to ${\sum }_{l=1}^L{\Psi }_l^H{\Psi }_l$:

$${\left[M_D\right]}_{r,r^{\prime }}=\sum _{l=1}^L\langle P_{B_r}{\widehat{x}}_l,P_{B_{r^{\prime }}}{\widehat{x}}_l\rangle ,\hspace{1em}r,r^{\prime }=1,\dots ,R.$$ (18)

A sufficiently large number of training signals (with N ≥ R), L, can guarantee $M_D={\sum }_{l=1}^L{\Psi }_l^H{\Psi }_l\succ 0$ in Proposition 5.1. The drawback of using Proposition 5.1 is its polynomial computational complexity, i.e., O(LR²N) – see Table I. When L (the number of training signals) or N (the size of training signals) are large, the quadratic complexity with the size of filters – R² – can quickly increase the total computational costs when multiplied by L and N. (The BPG setup in [31] additionally requires O(R³) because it uses the eigendecomposition of (18) to calculate the Lipschitz constant.)

TABLE I.

Computational complexity of different majorization matrix designs for the filter update problem (16)

Lemmas 5.2–5.3	Proposition 5.1
O(LRN)	O(LR2N)

Considering CAOL problems (P0) themselves, different from CDL [13]–[17], the complexity O(LR²N) in applying Proposition 5.1 is reasonable. In BPEG-M-based CDL [16], [17], a majorization matrix for kernel update is calculated every iteration because it depends on updated sparse codes; however, in CAOL, one can precompute M_D via Proposition 5.1 (or Lemmas 5.2–5.3 below) without needing to change it every kernel update. The polynomial computational cost in applying Proposition 5.1 becomes problematic only when the training signals change. Examples include 1) hierarchical CAOL, e.g., CNN in Appendix A, 2) “adaptive-filter MBIR” particularly with high-dimensional signals [2], [6], [50], and 3) online learning [51], [52]. Therefore, we also describe a more efficiently computable majorization matrix at the cost of looser bounds (i.e., slower convergence; see Fig 3). Applying Lemma S.1, we first introduce a diagonal majorization matrix M_D for the Hessian ${\sum }_l{\Psi }_l^H{\Psi }_l$ in (16):

Lemma 5.2 (Diagonal majorization matrix M_D). The following $M_D\in {ℂ}^{R\times R}$ satisfies $M_D\underset{\_}{\succ }{\sum }_{l=1}^L{\Psi }_l^H{\Psi }_l$:

$$M_D=\text{diag}\left(\sum _{l=1}^L\left|{\Psi }_l^H\right|\left|{\Psi }_l\right|1_R\right),$$ (19)

where |·| takes the absolute values of the elements of a matrix.

The majorization matrix design in Lemma 5.2 is more efficient to compute than that in Proposition 5.1, because no R²-factor is needed for calculating M_D in Lemma 5.2, i.e., O(LRN); see Table I. Designing M_D in Lemma 5.2 takes fewer calculations than [16, Lem. 5.1] using Fourier approaches, when $R<\log \left(\widehat{N}\right)$. Using Lemma S.2, we next design a potentially sharper majorization matrix than (19), while maintaining the cost O(LRN):

Lemma 5.3 (Scaled identity majorization matrix M_D). The following matrix $M_D\in {ℂ}^{R\times R}$ satisfies $M_D\succsim {\sum }_{l=1}^L{\Psi }_l^H{\Psi }_l$:

$$M_D=\sum _{r=1}^R\left|\sum _{l=1}^L\langle P_{B_1}{\widehat{x}}_l,P_{B_r}{\widehat{x}}_l\rangle \right|\cdot I_R,$$ (20)

for a circular boundary condition.

Proof: See Section S.VII of the supplementary material. ■

For all the training datasets used in this paper, we observed that the tightness of majorization matrices in Proposition 5.1 and Lemmas 5.2–5.3 for the Hessian ${\sum }_l{\Psi }_l^H{\Psi }_l$ is given by

$$\sum _{l=1}^L{\Psi }_l^H{\Psi }_l=\left(18\right)\preccurlyeq \left(20\right)\preccurlyeq \left(19\right).$$ (21)

(Note that (18) ⪯ (19) always holds regardless of training data.) Fig. 3 illustrates the effects of the majorizer sharpness in (21) on CAOL convergence rates. As described in Section IV-A, selecting λ_D (see (22) and (26) below) controls the tradeoff between majorization sharpness and extrapolation effect. We found that using fixed λ_D = 1 + ϵ gives faster convergence than λ_D = 2; see Fig. 4 (this behavior is more obvious in solving the CT MBIR model in (P3) via BPEG-M – see [32, Fig. 3]). The results in Fig. 4 and [32, Fig. 3] show that, under the sharp majorization regime, maintaining sharper majorization is more critical in accelerating the convergence of BPEG-M than giving more weight to extrapolation.

Fig. 4.

Cost minimization comparisons in CAOL (P1) with different BPEG-M algorithms and datasets (Lemma 5.2 was used for M_D; R = K = 49; deterministic filter initialization and random sparse code initialization). Under the sharp majorization regime, maintaining sharp majorization (i.e., λ_D = 1 + ϵ) provides faster convergence than giving more weight on extrapolation (i.e., λ_D = 2). (The same behavior was found in sparse-view CT application [32, Fig. 3].) There exist no differences in convergence between solution (31) and solution (33) using {λ_Z = 1 + ϵ}.

Sections V-A2 and V-A3 below apply the majorization matrices designed in this section to proximal mappings of D-optimization in (P1) and (P2), respectively.

2). Proximal Mapping with Orthogonality Constraint:

The corresponding proximal mapping problem of (16) using the orthogonality constraint in (P1) is given by

$$\begin{array}{l}\left\{d_k^{\left(i+1\right)}\right\}=\underset{\left\{d_k\right\}}{\text{argmin}}\sum _{k=1}^K\frac{1}{2}{\Vert d_k-{\nu }_k^{\left(i+1\right)}\Vert }_{{\tilde{M}}_D}^2,\\ \hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\text{subject to }D{D}^H=\frac{1}{R}\cdot I,\end{array}$$ (22)

where

$${\nu }_k^{\left(i+1\right)}={\stackrel{\prime }{d}}_k^{\left(i+1\right)}-{\tilde{M}}_D^{-1}{\sum }_D^{-1}{\sum }_{l=1}^L{\Psi }_l^H\left({\Psi }_l{\stackrel{\prime }{d}}_k^{\left(i+1\right)}-z_{l,k}\right),$$ (23)

$${\stackrel{\prime }{d}}_k^{\left(i+1\right)}=d_k^{\left(i\right)}+E_D^{\left(i+1\right)}\left(d_k^{\left(i\right)}-d_k^{\left(i-1\right)}\right),$$ (24)

for k = 1, …, K, and ${\tilde{M}}_D={\lambda }_D{M}_D$ by (6). One can parallelize over k = 1, …, K in computing $\left\{{\nu }_k^{\left(i+1\right)}\right\}$ in (23). The proposition below provides an optimal solution to (22):

Proposition 5.4. Consider the following constrained minimization problem:

$$\underset{D}{\min }{\Vert {\tilde{M}}_D^{1/2}D-{\tilde{M}}_D^{1/2}\mathcal{V}\Vert }_{\text{F}}^2,\hspace{1em}\text{subj. to }D{D}^H=\frac{1}{R}\cdot I,$$ (25)

where D is given as (1), $\mathcal{V}=\left[{\nu }_1^{\left(i+1\right)}\cdots {\nu }_K^{\left(i+1\right)}\right]\in {ℂ}^{R\times K}$, ${\tilde{M}}_D={\lambda }_D{M}_D$ and $M_D\in {ℝ}^{R\times R}$ is given by (18), (19), or (20). The optimal solution to (25) is given by

$$D^{\star }=\frac{1}{\sqrt{R}}\cdot U\left[I_R,0_{R\times \left(K-R\right)}\right]V^H,\hspace{1em}for\hspace{0.17em}\hspace{0.17em}R\le K,$$

where ${\tilde{M}}_D\mathcal{V}$ has (full) singular value decomposition, ${\tilde{M}}_D\mathcal{V}=U\Lambda V^H$.

Proof: See Section S.VIII of the supplementary material. ■

When using Proposition 5.1, ${\tilde{M}}_D{\nu }_k^{\left(i+1\right)}$ of ${\tilde{M}}_D\mathcal{V}$ in Proposition 5.4 simplifies to the following update:

$${\tilde{M}}_D{\nu }_k^{\left(i+1\right)}=\left({\lambda }_D-1\right)M_D{\stackrel{\prime }{d}}_k^{\left(i+1\right)}+\sum _{l=1}^L{\Psi }_l^H{z}_{l,k}.$$

Similar to obtaining $\left\{{\nu }_k^{\left(i+1\right)}\right\}$ in (23), computing $\left\{{\tilde{M}}_D{\nu }_k^{\left(i+1\right)}:k=1,\dots ,K\right\}$ is parallelizable over k.

3). Proximal Mapping with Diversity Promoting Regularizer:

The corresponding proximal mapping problem of (16) using the norm constraint and diversity promoting regularizer in (P2) is given by

$$\begin{array}{l}\left\{d_k^{\left(i+1\right)}\right\}=\underset{\left\{d_k\right\}}{\text{argmin}}\sum _{k=1}^K\frac{1}{2}{\Vert d_k-{\nu }_k^{\left(i+1\right)}\Vert }_{{\tilde{M}}_D}^2+\frac{\beta }{2}{g}_{\text{div}}\left(D\right),\\ \hspace{1em}\hspace{1em}\hspace{0.17em}\text{subject to }{\Vert d_k\Vert }_2^2=\frac{1}{R},\hspace{1em}k=1,\dots ,K,\end{array}$$ (26)

where g_div(D), ${\nu }_k^{\left(i+1\right)}$, and ${\stackrel{\prime }{d}}_k^{\left(i+1\right)}$ are given as in (P2), (23), and (24), respectively. We first decompose the regularization term g_div(D) as follows:

$$\begin{array}{l}{g}_{\text{div}}\left(D\right)=\sum _{k=1}^K\sum _{k^{\prime }=1}^K\left(d_k^H{d}_{k^{\prime }}{d}_{k^{\prime }}^H{d}_k-R^{-1}\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}=\sum _{k=1}^K{d}_k^H\left(\sum _{k^{\prime }\ne k}{d}_{k^{\prime }}{d}_{k^{\prime }}^H\right)d_k+{\left(d_k^H{d}_k-R^{-1}\right)}^2\\ \hspace{1em}\hspace{1em}\hspace{1em}=\sum _{k=1}^K{d}_k^H{\Gamma }_k{d}_k,\end{array}$$ (27)

where the equality in (27) holds by using the constraint in (26), and the Hermitian matrix ${\Gamma }_k\in {ℂ}^{R\times R}$ is defined by

$${\Gamma }_k:=\sum _{k^{\prime }\ne k}{d}_{k^{\prime }}{d}_{k^{\prime }}^H.$$ (28)

Using (27) and (28), we rewrite (26) as

$$\begin{array}{l}{d}_k^{\left(i+1\right)}=\underset{d_k}{\text{argmin}}\frac{1}{2}{\Vert d_k-{\nu }_k^{\left(i\right)}\Vert }_{{\tilde{M}}_D}^2+\frac{\beta }{2}{d}_k^H{\Gamma }_k{d}_k,\\ \hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{subject to }{\Vert d_k\Vert }_2^2=\frac{1}{R},\hspace{1em}k=1,\dots ,K.\end{array}$$ (29)

This is a quadratically constrained quadratic program with $\left\{{\tilde{M}}_D+\beta {\Gamma }_k\succ 0:k=1,\dots ,K\right\}$. We apply an accelerated Newton’s method to solve (29); see Section S.IX. Similar to solving (22) in Section V-A2, solving (26) is a small-dimensional problem (K separate problems of size R).

B. Sparse Code Update: $\left\{z_{l,k}\right\}$-Block Optimization

Given the current estimate of D, the sparse code update problem for (P0) is given by

$$\underset{\left\{z_{l,k}\right\}}{\text{argmin}}\sum _{l=1}^L\sum _{k=1}^K\frac{1}{2}{\Vert d_k⊛x_l-z_{l,k}\Vert }_2^2+\alpha {\Vert z_{l,k}\Vert }_0.$$ (30)

This problem separates readily, allowing parallel computation with LK threads. An optimal solution to (30) is efficiently obtained by the well-known hard thresholding:

$$z_{l,k}^{\left(i+1\right)}={\mathcal{H}}_{\sqrt{2\alpha }}\left(d_k⊛x_l\right),$$ (31)

for k = 1, …, K and l = 1, …, L, where

$${\mathcal{H}}_a{\left(x\right)}_n:=\{\begin{array}{cc}0,& \left|x_n\right|<a_n,\\ x_n,& \left|x_n\right|\ge a_n.\end{array}$$ (32)

for all n. Considering λ_Z (in ${\tilde{M}}_Z={\lambda }_Z{M}_Z$) as λ_Z → 1, the solution obtained by the BPEG-M approach becomes equivalent to (31). To show this, observe first that the BPEG-M-based solution (using M_Z = I_N) to (30) is obtained by

$$z_{l,k}^{\left(i+1\right)}={\mathcal{H}}_{\sqrt{\frac{2\alpha }{{\lambda }_Z}}}\left({\zeta }_{l,k}^{\left(i+1\right)}\right),$$ (33)

$${\zeta }_{l,k}^{\left(i+1\right)}=\left(1-{\lambda }_Z^{-1}\right)\cdot {\stackrel{\prime }{z}}_{l,k}^{\left(i+1\right)}+{\lambda }_Z^{-1}\cdot d_k⊛x_l,$$

$${\stackrel{\prime }{z}}_{l,k}^{\left(i+1\right)}=z_{l,k}^{\left(i\right)}+E_Z^{\left(i+1\right)}\left(z_{l,k}^{\left(i\right)}-z_{l,k}^{\left(i-1\right)}\right).$$

The downside of applying solution (33) is that it would require additional memory to store the corresponding extrapolated points –$\left\{{\stackrel{\prime }{z}}_{l,k}^{\left(i+1\right)}\right\}$– and the memory grows with N, L, and K. Considering the sharpness of the majorizer in (30), i.e., M_Z = I_N, and the memory issue, it is reasonable to consider the solution (33) with no extrapolation, i.e., $\left\{E_Z^{\left(i+1\right)}=0\right\}$:

$$z_{l,k}^{\left(i+1\right)}={\mathcal{H}}_{\sqrt{\frac{2\alpha }{{\lambda }_Z}}}\left({({\lambda }_Z-1)}^{-1}{\lambda }_Z\cdot z_{l,k}^{\left(i\right)}+{\lambda }_Z^{-1}\cdot d_k⊛x_l\right)$$

becoming equivalent to (31) as λ_Z → 1.

Solution (31) has two benefits over (33): compared to (33), (31) requires only half the memory to update all $z_{l,k}^{\left(i+1\right)}$ vectors and no additional computations related to ${\stackrel{\prime }{z}}_{l,k}^{\left(i+1\right)}$. While having these benefits, empirically (31) has equivalent convergence rates as (33) using {λ_Z = 1 + ϵ}; see Fig. 4. Throughout the paper, we solve the sparse coding problems (e.g., (30) and {z_k}-block optimization in (P3)) via optimal solutions in the form of (31).

C. Lower Memory Use than Patch-Domain Approaches

The convolution perspective in CAOL (P0) requires much less memory than conventional patch-domain approaches; thus, it is more suitable for learning filters from large datasets or applying the learned filters to high-dimensional MBIR problems. First, consider the training stage (e.g., (P0)). The patch-domain approaches, e.g., [1], [6], [7], require about R times more memory to store training signals. For example, 2D patches extracted by $\sqrt{R}\times \sqrt{R}$-sized windows (with “stride” one and periodic boundaries [6], [12], as used in convolution) require about R (e.g., R = 64 [1], [7]) times more memory than storing the original image of size $\sqrt{N}\times \sqrt{N}$. For L training images, their memory usage dramatically increases with a factor LRN. This becomes even more problematic in forming hierarchical representations, e.g., CNNs – see Appendix A. Unlike the patch-domain approaches, the memory use of CAOL (P0) only depends on the LN-factor to store training signals. As a result, the BPEG-M algorithm for CAOL (P1) requires about two times less memory than the patch-domain approach [6] (using BPEG-M). See Table II–B. (Both the corresponding BPEG-M algorithms use identical computations per iteration that scale with LR²N; see Table II–A.)

TABLE II.

Comparisons of computational complexity and memory usages between CAOL and patch-domain approach

A. Computational complexity per BPEG-M iteration
	Filter update	Sparse code update
CAOL (P1)	O(LKRN) + O(R2K)	O(LKRN)
Patch-domain [6]†	O(LR2N) + O(R3)	O(LR2N)

B. Memory usage for BPEG-M algorithm
	Filter update	Sparse code update
CAOL (P1)	O(LN) + O(RK)	O(LKN)
Patch-domain [6]†	O(LRN) + O(R2)	O(LRN)

^†The patch-domain approach [6] considers the orthogonality constraint in (P1) with R = K; see Section III-A. The estimates consider all the extracted overlapping patches of size R with the stride parameter 1 and periodic boundaries, as used in convolution.

Second, consider solving MBIR problems. Different from the training stage, the memory burden depends on how one applies the learned filters. In [53], the learned filters are applied with the conventional convolutional operators – e.g., ⊛ in (P0) – and, thus, there exists no additional memory burden. However, in [2], [54], [55], the $\sqrt{R}\times \sqrt{R}$-sized learned kernels are applied with a matrix constructed by many overlapping patches extracted from the updated image at each iteration. In adaptive-filter MBIR problems [2], [6], [8], the memory issue pervades the patch-domain approaches.

VI. Sparse-View CT MBIR using Convolutional Regularizer Learned via CAOL, and BPEG-M

This section introduces a specific example of applying the learned convolutional regularizer, i.e., $F\left(D^{\star },\left\{z_{l,k}\right\}\right)$ in (P0), from a representative dataset to recover images in extreme imaging that collects highly undersampled or noisy measurements. We choose a sparse-view CT application since it has interesting challenges in reconstructing images that include Poisson noise in measurements, nonuniform noise or resolution properties in reconstructed images, and complicated (or no) structures in the system matrices. For CT, undersampling schemes can significantly reduce the radiation dose and cancer risk from CT scanning. The proposed approach can be applied to other applications (by replacing the data fidelity and spatial strength regularization terms in (P3) below).

We pre-learn TF filters $\left\{d_k^{\star }\in {ℝ}^K:k=1,\dots ,K\right\}$ via CAOL (P1) with a set of high-quality (e.g., normal-dose) CT images {x_l: l = 1, …, L}. To reconstruct a linear attenuation coefficient image $x\in {ℝ}^{N^{\prime }}$ from post-log measurement $y\in {ℝ}^m$ [54], [56], we apply the learned convolutional regularizer to CT MBIR and solve the following block multi-nonconvex problem [32], [35]:

$$\begin{array}{l}\underset{x\ge 0}{\text{argmin}}\underset{\text{data fidelity }f\left(x;y\right)}{\underbrace{\frac{1}{2}{\Vert y-Ax\Vert }_W^2}}+\\ \hspace{1em}\hspace{1em}\underset{\text{learned convolutional regularizer }g\left(x,\left\{z_k\right\};\left\{d_k\right\}\right)}{\underbrace{\gamma \cdot \underset{\left\{z_k\right\}}{\min }\sum _{k=1}^K\frac{1}{2}{\Vert d_k^{\star }⊛x-z_k\Vert }_2^2+{\alpha }^{\prime }\sum _{n=1}^{N^{\prime }}{\psi }_j\varphi \left({\left(z_k\right)}_n\right)}}.\end{array}$$ (P3)

Here, $A\in {ℝ}^{m\times N^{\prime }}$ is a CT system matrix, $W\in {ℝ}^{m\times m}$ is a (diagonal) weighting matrix with elements $\left\{W_{l,l}={\rho }_l^2/\left({\rho }_l+{\sigma }^2\right):l=1,\dots ,m\right\}$ based on a Poisson-Gaussian model for the pre-log measurements $\rho \in {ℝ}^m$ with electronic readout noise variance σ² [54]–[56], $\psi \in {ℝ}^{N^{\prime }}$ is a pre-tuned spatial strength regularization vector [57] with non-negative elements $\left\{{\psi }_n={\left({\sum }_{l=1}^m{A}_{l,n}^2{W}_{l,l}\right)}^{1/2}/{\left({\sum }_{l=1}^m{A}_{l,n}^2\right)}^{1/2}:n=1,\dots ,N^{\prime }\right\}$⁴ that promotes uniform resolution or noise properties in the reconstructed image [54, Appx.], an indicator function ϕ(a) is equal to 0 if a = 0, and is 1 otherwise, $z_k\in {ℝ}^{N^{\prime }}$ is unknown sparse code for the kth filter, and α^′ > 0 is a thresholding parameter.

We solved (P3) via reBPEG-M in Section IV with a two-block scheme [32], and summarize the corresponding BPEG-M updates as

$$\begin{array}{l}{x}^{\left(i+1\right)}=[{\left({\tilde{M}}_A+\gamma I_R\right)}^{-1}\cdot ({\tilde{M}}_A{\eta }^{\left(i+1\right)}+\\ {\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\gamma \sum _{k=1}^K\left(P_f{d}_k^{\star }\right)⊛{\mathcal{H}}_{\sqrt{2{\alpha }^{\prime }\psi }}\left(d_k^{\star }⊛x^{\left(i\right)}\right))]}_{\ge 0},\end{array}$$ (34)

where

$${\eta }^{\left(i+1\right)}={\stackrel{\prime }{x}}^{\left(i+1\right)}-{\tilde{M}}_A^{-1}{A}^{T}W\left(A{\stackrel{\prime }{x}}^{\left(i+1\right)}-y\right),$$ (35)

$${\stackrel{\prime }{x}}^{\left(i+1\right)}=x^{\left(i\right)}+E_A^{\left(i+1\right)}\left(x^{\left(i\right)}-x^{\left(i-1\right)}\right),$$

${\tilde{M}}_A={\lambda }_A{M}_A$ by (6), a diagonal majorization matrix $M_A\succcurlyeq A^{T}WA$ is designed by Lemma S.1, and $P_f\in {ℂ}^{R\times R}$ flips a column vector in the vertical direction (e.g., it rotates 2D filters by 180°). Interpreting the update (34) leads to the following two remarks:

Remark 6.1. When the convolutional regularizer learned via CAOL (P1) is applied to MBIR, it works as an autoencoding CNN:

$$\mathcal{M}\left(x\right)=\sum _{k=1}^K\left(P_f{d}_k^{\star }\right)⊛{\mathcal{H}}_{\sqrt{2{\alpha }_k^{\prime }}}\left(d_k^{\star }⊛x\right)$$ (36)

(setting $\psi =1_{N^{\prime }}$ and generalizing α^′ to $\left\{{\alpha }_k^{\prime }:k=1,\dots ,K\right\}$ in (P3)). This is an explicit mathematical motivation for constructing architectures of iterative regression CNNs for MBIR, e.g., BCD-Net [28], [58]–[60] and Momentum-Net [29], [30]. Particularly when the learned filters $\left\{d_k^{\star }\right\}$ in (36) satisfy the TF condition, they are useful for compacting energy of an input signal x and removing unwanted features via the non-linear thresholding in (36).

Remark 6.2. Update (34) improves the solution x⁽ⁱ⁺¹⁾ by weighting between a) the extrapolated point considering the data fidelity, i.e., η⁽ⁱ⁺¹⁾ in (35), and b) the “refined” update via the (ψ-weighting) convolutional autoencoder, i.e., ${\sum }_k\left(P_f{d}_k^{\star }\right)⊛{\mathcal{H}}_{\sqrt{2{\alpha }^{\prime }\psi }}\left(d_k^{\star }⊛x^{\left(i\right)}\right)$.

VII. Results and Discussion

A. Experimental Setup

This section examines the performance (e.g., scalability, convergence, and acceleration) and behaviors (e.g., effects of model parameters on filters structures and effects of dimensions of learned filter on MBIR performance) of the proposed CAOL algorithms and models, respectively.

1). CAOL:

We tested the introduced CAOL models/algorithms for four datasets: 1) the fruit dataset with L = 10 and N = 100 × 100 [10]; 2) the city dataset with L = 10 and N = 100 × 100 [14]; 3) the CT dataset of L = 80 and N = 128×128, created by dividing down-sampled 512×512 XCAT phantom slices [61] into 16 sub-images [13], [62] – referred to the CT-(i) dataset; 4) the CT dataset of with L = 10 and N = 512×512 from down-sampled 512×512 XCAT phantom slices [61] – referred to the CT-(ii) dataset. The preprocessing includes intensity rescaling to [0, 1] [10], [13], [14] and/or (global) mean substraction [63, §2], [1], as conventionally used in many sparse coding studies, e.g., [1], [10], [13], [14], [63]. For the fruit and city datasets, we trained K = 49 filters of size R = 7×7. For the CT dataset (i), we trained filters of size R = 5×5, with K = 25 or K = 20. For CT reconstruction experiments, we learned the filters from the CT-(ii) dataset; however, we did not apply mean subtraction because it is not modeled in (P3).

The parameters for the BPEG-M algorithms were defined as follows.⁵ We set the regularization parameters α, β as follows:

• CAOL (P1): To investigate the effects of α, we tested (P1) with different α’s in the case R = K. For the fruit and city datasets, we used α = 2.5×{10⁻⁵, 10⁻⁴}; for the CT-dataset (for CT reconstruction experiments), see details in [32, Sec. V1].

• CAOL (P2): Once α is fixed from the CAOL (P1) experiments above, we tested (P2) with different β’s to see its effects in the case R > K. For the CT-(i) dataset, we fixed α = 10⁻⁴, and used β = {5×10⁶, 5×10⁴}.

We set λ_D = 1 + ϵ as the default. We initialized filters in either deterministic or random ways. The deterministic filter initialization follows that in [6, Sec. 3.4]. When filters were randomly initialized, we used a scaled one-vector for the first filter. We initialize sparse codes mainly with a deterministic way that applies (31) based on $\left\{d_k^{\left(0\right)}\right\}$. If not specified, we used the random filter and deterministic sparse code initializations. For BPG [31], we used the maximum eigenvalue of Hessians for Lipschitz constants in (16), and applied the gradient-based restarting scheme in Section IV-C. We terminated the iterations if the relative error stopping criterion (e.g., [16, (44)]) is met before reaching the maximum number of iterations. We set the tolerance value as 10⁻¹³ for the CAOL algorithms using Proposition 5.1, and 10⁻⁵ for those using Lemmas 5.2–5.3, and the maximum number of iterations to 2×10⁴.

The CAOL experiments used the convolutional operator learning toolbox [64].

2). Sparse-View CT MBIR with Learned Convolutional Regularizer via CAOL:

We simulated sparse-view sinograms of size 888 × 123 (‘detectors or rays’ × ‘regularly spaced projection views or angles’, where 984 is the number of full views) with GE LightSpeed fan-beam geometry corresponding to a monoenergetic source with 10⁵ incident photons per ray and no background events, and electronic noise variance σ² = 5². We avoided an inverse crime in our imaging simulation and reconstructed images with a coarser grid with ∆_x = ∆_y = 0.9766 mm; see details in [32, Sec. V-A2].

For EP MBIR, we finely tuned its regularization parameter to achieve both good root mean square error (RMSE) and structural similarity index measurement [65] values. For the CT MBIR model (P3), we chose the model parameters {γ, α^′} that showed a good tradeoff between the data fidelity term and the learned convolutional regularizer, and set λ_A = 1 + ϵ. We evaluated the reconstruction quality by the RMSE (in a modified Hounsfield unit, HU, where air is 0 HU and water is 1000 HU) in a region of interest. See further details in [32, Sec. V-A2] and Fig. 6.

Fig. 6.

Comparisons of reconstructed images from different reconstruction methods for sparse-view CT (123 views (12.5% sampling); for the MBIR model (P3), convolutional regularizers were trained by CAOL (P1) – see [32, Fig. 2]; display window is within [800, 1200] HU) [32]. The MBIR model (P3) using convolutional sparsifying regularizers trained via CAOL (P1) shows higher image reconstruction accuracy compared to the EP reconstruction; see red arrows and magnified areas. For the MBIR model (P3), the autoencoder (see Remark 6.1) using the filter dimension R=K=49 improves reconstruction accuracy of that using R=K=25; compare the results in (d) and (e). In particular, the larger dimensional filters improve the edge sharpness of reconstructed images; see circled areas. The corresponding error maps are shown in Fig. S.5 of the supplementary material.

The imaging simulation and reconstruction experiments used the Michigan image reconstruction toolbox [66].

B. CAOL with BPEG-M

Under the sharp majorization regime (i.e., partial or all blocks have sufficiently tight bounds in Lemma 4.2), the proposed convergence-guaranteed BPEG-M can achieve significantly faster CAOL convergence rates compared with the state-of-the-art BPG algorithm [31] for solving block multi-nonconvex problems, by several generalizations of BPG (see Remark 4.3) and two majorization designs (see Proposition 5.1 and Lemma 5.3). See Fig. 3. In controlling the tradeoff between majorization sharpness and extrapolation effect of BPEG-M (i.e., choosing {λ_b} in (6)–(7)), maintaining majorization sharpness is more critical than gaining stronger extrapolation effects to accelerate convergence under the sharp majorization regime. See Fig. 4.

While using about two times less memory (see Table II), CAOL (P0) learns TF filters corresponding to those given by the patch-domain TF learning in [6, Fig. 2]. See Section V-C and Fig. S.1 with deterministic $\left\{d_k^{\left(0\right)}\right\}$. Note that BPEG-M-based CAOL (P0) requires even less memory than BPEG-M-based CDL in [16], by using exact sparse coding solutions (e.g., (31) and (34)) without saving their extrapolated points. In particular, when tested with the large CT dataset of {L = 40, N = 512×512}, the BPEG-M-based CAOL algorithm ran fine, while BPEG-M-based CDL [16] and patch-domain AOL [6] were terminated due to exceeding available memory.⁶ In addition, the CAOL models (P1) and (P2) are easily parallelizable with K threads. Combining these results, the BPEG-M-based CAOL is a reasonable choice for learning filters from large training datasets. Finally, [34] shows theoretically how using many samples can improve CAOL, accentuating the benefits of the low memory usage of CAOL.

The effects of parameters for the CAOL models are shown as follows. In CAOL (P1), as the thresholding parameter α increases, the learned filters have more elongated structures; see Figs. 5(a) and S.2. In CAOL (P2), when α is fixed, increasing the filter diversity promoting regularizer β successfully lowers coherences between filters (e.g., g_div(D) in (P2)); see Fig. 5(b).

Fig. 5.

Examples of learned filters with different CAOL models and parameters (Proposition 5.1 was used for M_D; the CT-(i) dataset with a symmetric boundary condition).

In adaptive MBIR (e.g., [2], [6], [8]), one may apply adaptive image denoising [53], [67]–[71] to optimize thresholding parameters. However, if CAOL (P0) and testing the learned convolutional regularizer to MBIR (e.g., (P3)) are separated, selecting “optimal” thresholding parameters in (unsupervised) CAOL is challenging – similar to existing dictionary or analysis operator learning methods. Our strategy to select the thresholding parameter α in CAOL (P1) (with R = K) is given as follows. We first apply the first-order finite difference filters $\left\{d_k:{\Vert d_k\Vert }_2^2=1/R,\forall k\right\}$ (e.g., $\frac{1}{\sqrt{2R}}{\left[1,-1\right]}^T$ in 1D) to all training signals and find their sparse representations, and then find α_est that corresponds to the largest 95(±1)% of non-zero elements of the sparsified training signals. This procedure defines the range $\left[\frac{1}{10}{\alpha }_{\text{est}},{\alpha }_{\text{est}}\right]$ to select desirable α^⋆ and its corresponding filter D^⋆. We next ran CAOL (P1) with multiple α values within this range. Selecting {α^⋆, D^⋆} depends on application. For CT MBIR, D^⋆ that both has (short) first-order finite difference filters and captures diverse (particularly diagonal) features of training signals, gave good RMSE values and well preserved edges; see Fig. S.2(c) and [32, Fig. 2].

C. Sparse-View CT MBIR with Learned Convolutional Sparsifying Regularizer (via CAOL) and BPEG-M

In sparse-view CT using only 12.5% of the full projections views, the CT MBIR (P3) using the learned convolutional regularizer via CAOL (P1) outperforms EP MBIR; it reduces RMSE by approximately 5.6–6.1HU. See the results in Figs. 6(c)–(e). The model (P3) can better recover high-contrast regions (e.g., bones) – see red arrows and magnified areas in Fig. 6(c)–(e). Nonetheless, the filters with R = K = 5² in the (ψ-weighting) autoencoding CNN, i.e., ${\sum }_k\left(P_f{d}_k^{\star }\right)⊛{\mathcal{H}}_{\sqrt{2{\alpha }^{\prime }\psi }}\left(d_k^{\star }⊛(\cdot )\right)$ in (36), can blur edges in low-contrast regions (e.g., soft tissues) while removing noise. See Fig. 6(d) – the blurry issues were similarly observed in [54], [55]. The larger dimensional kernels (i.e., R=K=7²) in the convolutional autoencoder can moderate this issue, while further reducing RMSE values; compare the results in Fig. 6(d)– (e). In particular, the larger dimensional convolutional kernels capture more diverse features – see [32, Fig. 2]) – and the diverse features captured in kernels are useful to further improve the performance of the proposed MBIR model (P3). (The importance of diverse features in kernels was similarly observed in CT experiments with the learned autoencoders having a fixed kernel dimension; see Fig. S.2(c).) The RMSE reduction over EP MBIR is comparable to that of CT MBIR (P3) using the {R,K=8²}-dimensional filters trained via the patch-domain AOL [7]; however, at each BPEG-M iteration, this MBIR model using the trained (non-TF) filters via patch-domain AOL [7] requires more computations than the proposed CT MBIR model (P3) using the learned convolutional regularizer via CAOL (P1). See related results and discussion in Fig. S.4 and Section S.X, respectively.

On the algorithmic side, the BPEG-M framework can guarantee the convergence of CT MBIR (P3). Under the sharp majorization regime in BPEG-M, maintaining the majorization sharpness is more critical than having stronger extrapolation effects – see [32, Fig. 3], as similarly shown in CAOL experiments (see Section VII-B).

VIII. Conclusion

Developing rapidly converging and memory-efficient CAOL engines is important, since it is a basic element in training CNNs in an unsupervised learning manner (see Appendix A). Studying structures of convolutional kernels is another fundamental issue, since it can avoid learning redundant filters or provide energy compaction properties to filters. The proposed BPEG-M-based CAOL framework has several benefits. First, the orthogonality constraint and diversity promoting regularizer in CAOL are useful in learning filters with diverse structures. Second, the proposed BPEG-M algorithm significantly accelerates CAOL over the state-of-the-art method, BPG [31], with our sufficiently sharp majorizer designs. Third, BPEG-M-based CAOL uses much less memory compared to patch-domain AOL methods [3], [4], [7], and easily allows parallel computing. Finally, the learned convolutional regularizer provides the autoencoding CNN architecture in MBIR, and outperforms EP reconstruction in sparse-view CT.

Similar to existing unsupervised synthesis or analysis operator learning methods, the biggest remaining challenge of CAOL is optimizing its model parameters. This would become more challenging when one applies CAOL to train CNNs (see Appendix A). Our first future work is developing “task-driven” CAOL that is particularly useful to train thresholding values. Other future works include further acceleration of BPEG-M in Algorithm 1, designing sharper majorizers requiring only O(LRN) for the filter update problem of CAOL (P0), and applying the CNN model learned via (A1) to MBIR.

Appendix

A. Training CNN in a unsupervised manner via CAOL

This section mathematically formulates an unsupervised training cost function for classical CNN (e.g., LeNet-5 [11] and AlexNet [72]) and solves the corresponding optimization problem, via the CAOL and BPEG-M frameworks studied in Sections III–V. We model the three core modules of CNN: 1) convolution, 2) pooling, e.g., average [11] or max [63], and 3) thresholding, e.g., RELU [73], while considering the TF filter condition in Proposition 3.1. Particularly, the orthogonality constraint in CAOL (P1) leads to a sharp majorizer, and BPEG-M is useful to train CNNs with convergence guarantees. Note that it is unclear how to train such diverse (or incoherent) filters described in Section III by the most common CNN optimization method, the stochastic gradient method in which gradients are computed by back-propagation. The major challenges include a) the non-differentiable hard thresholding operator related to ℓ⁰-norm in (P0), b) the nonconvex filter constraints in (P1) and (P2), c) using the identical filters in both encoder and decoder (e.g., W and W^H in Section S.I), and d) vanishing gradients.

For simplicity, we consider a two-layer CNN with a single training image, but one can extend the CNN model (A1) (see below) to “deep” layers with multiple images. The first layer consists of 1c) convolutional, 1t) thresholding, and 1p) pooling layers; the second layer consists of 2c) convolutional and 2t) thresholding layers. Extending CAOL (P1), we model two-layer CNN training as the following optimization problem:

$$\begin{array}{l}\underset{\left\{d_k^{\left[1\right]},d_{k,k^{\prime }}^{\left[2\right]}\right\}}{\text{argmin}}\underset{\left\{z_k^{\left[1\right]},z_{k^{\prime }}^{\left[2\right]}\right\}}{\min }{\displaystyle \sum _{k=1}^{K_1}}\frac{1}{2}{\Vert d_k^{\left[1\right]}⊛x-z_k^{\left[1\right]}\Vert }_2^2+{\alpha }_1{\Vert z_k^{\left[1\right]}\Vert }_0\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.5em}+\frac{1}{2}{\Vert \left({\displaystyle \sum _{k=1}^{K_1}}\left[\begin{array}{c}{d}_{k,1}^{\left[2\right]}⊛P{z}_k^{\left[1\right]}\\ ⋮\\ d_{k,K_2}^{\left[2\right]}⊛P{z}_k^{\left[1\right]}\end{array}\right]\right)-\left[\begin{array}{c}{z}_1^{\left[2\right]}\\ ⋮\\ z_{K_2}^{\left[2\right]}\end{array}\right]\Vert }_2^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.5em}+{\alpha }_2{\displaystyle \sum _{k^{\prime }=1}^{K_2}}{\Vert z_{k^{\prime }}^{\left[2\right]}\Vert }_0\\ \hspace{1em}\text{subject to }{D}^{\left[1\right]}{\left(D^{\left[1\right]}\right)}^H=\frac{1}{R_1}\cdot I,\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}{D}_k^{\left[2\right]}{\left(D_k^{\left[2\right]}\right)}^H=\frac{1}{R_2}\cdot I,\hspace{1em}k=1,\dots ,K_1,\end{array}$$ (A1)

where $x\in {ℝ}^N$ is the training data, $\left\{d_k^{\left[1\right]}\in {ℝ}^{R_1}:k=1,\dots ,K_1\right\}$ is a set of filters in the first convolutional layer, $\left\{z_k^{\left[1\right]}\in {ℝ}^N:k=1,\dots ,K_1\right\}$ is a set of features after the first thresholding layer, $\left\{d_{k,k^{\prime }}^{\left[2\right]}\in {ℝ}^{R_2}:k^{\prime }=1,\dots ,K_2\right\}$ is a set of filters for each of $\left\{z_k^{\left[1\right]}\right\}$ in the second convolutional layer, $\left\{z_{k^{\prime }}^{\left[2\right]}\in {ℝ}^{N/\omega }:k=1,\dots ,K_2\right\}$ is a set of features after the second thresholding layer, D^[1] and $\left\{D_k^{\left[2\right]}\right\}$ are similarly given as in (1), $P\in {ℝ}^{N/\omega \times \omega }$ denotes an average pooling [11] operator (see its definition below), and ω is the size of pooling window. The superscripted number in the bracket of vectors and matrices denotes the (·)th layer. Here, we model a simple average pooling operator $P\in {ℝ}^{\left(N/\omega \right)\times \omega }$ by a block diagonal matrix with row vector $\frac{1}{\omega }{1}_{\omega }^T\in {ℝ}^{\omega }$: $P:=\frac{1}{\omega }{\oplus }_{j=1}^{N/\omega }{1}_{\omega }^T$. We obtain a majorization matrix of P^TP by $P^{T}P\preccurlyeq \text{diag}\left(P^{T}P{1}_N\right)=\frac{1}{\omega }{I}_N$ (using Lemma S.1). For 2D case, the structure of P changes, but $P^{T}P\preccurlyeq \frac{1}{\omega }{I}_N$ holds.

We solve the CNN training model in (A1) via the BPEG-M techniques in Section V, and relate the solutions of (A1) and modules in the two-layer CNN training. The symbols in the following items denote the CNN modules.

• 1c)Filters in the first layer, $\left\{d_k^{\left[1\right]}\right\}$: Updating the filters is straightforward via the techniques in Section V-A2.
• 1t)
  Features at the first layers, $\left\{z_k^{\left[1\right]}\right\}$: Using BPEG-M with the kth set of TF filters $\left\{d_{k,k^{\prime }}^{\left[2\right]}:k^{\prime }\right\}$ and $P^{T}P\preccurlyeq \frac{1}{\omega }{I}_N$ (see above), the proximal mapping for $z_k^{\left[1\right]}$ is

  $$\underset{z_k^{\left[1\right]}}{\min }\frac{1}{2}{\Vert d_k^{\left[1\right]}⊛x-z_k^{\left[1\right]}\Vert }_2^2+\frac{1}{2{\omega }^{\prime }}{\Vert z_k^{\left[1\right]}-{\zeta }_k^{\left[k\right]}\Vert }_2^2+{\alpha }_1{\Vert z_k^{\left[1\right]}\Vert }_0,$$ (37)

  where ${\omega }^{\prime }=\omega /{\lambda }_Z$ and ${\zeta }_k^{\left[k\right]}$ is given by (4). Combining the first two quadratic terms in (37) into a single quadratic term leads to an optimal update for (37):

  $$z_k^{\left[1\right]}={\mathcal{H}}_{\sqrt{2\frac{{\omega }^{\prime }{\alpha }_1}{{\omega }^{\prime }+1}}}\left(d_k^{\left[1\right]}⊛x+\frac{1}{{\omega }^{\prime }}{\zeta }_k^{\left[k\right]}\right),\hspace{1em}k\in \left[K\right],$$

  where the hard thresholding operator ${\mathcal{H}}_a(\cdot )$ with a thresholding parameter a is defined in (32).
• 1p)Pooling, P : Applying the pooling operator P to $\left\{z_k^{\left[1\right]}\right\}$ gives input data –$\left\{P{z}_k^{\left[1\right]}\right\}$– to the second layer.
• 2c)Filters in the second layer, $\left\{d_{k,k^{\prime }}^{\left[2\right]}\right\}$: We update the kth set filters $\left\{d_{k,k^{\prime }}^{\left[2\right]}:\forall k^{\prime }\right\}$ in a sequential way. Updating the kth set filters is straightforward via the techniques in Section V-A2.
• 2t)
  Features at the second layers, $\left\{z_{k^{\prime }}^{\left[2\right]}\right\}$: The corresponding update is given by

  $$z_{k^{\prime }}^{\left[2\right]}={\mathcal{H}}_{\sqrt{2{\alpha }_2}}\left(\sum _{k=1}^{K_1}{d}_{k,k^{\prime }}^{\left[1\right]}⊛P{z}_k^{\left[1\right]}\right),\hspace{1em}{k}^{\prime }\in \left[K_2\right].$$

Considering the introduced mathematical formulation of training CNNs [11] via CAOL, BPEG-M-based CAOL has potential to be a basic engine to rapidly train CNNs with big data (i.e., training data consisting of many (high-dimensional) signals).

B. Examples of $\left\{f\left(x;y\right),\mathcal{X}\right\}$ in MBIR model (B1) using learned regularizers

This section introduces some potential applications of using MBIR model (B1) using learned regularizers in imaging processing, imaging, and computer vision. We first consider quadratic data fidelity function in the form of $f\left(x;y\right)=\frac{1}{2}{\Vert y-Ax\Vert }_W^2$. Examples include

• Image debluring (with W = I for simplicity), where y is a blurred image, A is a blurring operator, and $\mathcal{X}$ is a box constraint;

• Image denoising (with A = I), where y is a noisy image corrupted by additive white Gaussian noise (AWGN), W is the inverse covariance matrix corresponding to AWGN statistics, and $\mathcal{X}$ is a box constraint;

• Compressed sensing (with $\left\{W=I,\mathcal{X}\in {ℂ}^{N^{\prime }}\right\}$ for simplicity) [74], [75], where y is a measurement vector, and A is a compressed sensing operator, e.g., subgaussian random matrix, bounded orthonormal system, subsampled isometries, certain types of random convolutions;

• Image inpainting (with W = I for simplicity), where y is an image with missing entries, A is a masking operator, and $\mathcal{X}$ is a box constraint;

• Light-field photography from focal stack data with $f\left(x;y\right)={\sum }_c{\Vert y_c-{\sum }_s{A}_{c,s}{x}_s\Vert }_2^2$, where y_c denotes measurements collected at the cth sensor, A_c,s models camera imaging geometry at the sth angular position for the cth detector, x_s denotes the sth sub-aperture image, ∀c, s, and $\mathcal{X}$ is a box constraint [29], [76].

Examples that use nonlinear data fidelity function include image classification using the logistic function [77], magnetic resonance imaging considering unknown magnetic field variation [78], and positron emission tomography [59].

C. Notation

We use ${\Vert \cdot \Vert }_p$ to denote the ℓ^p-norm and write $\langle \cdot ,\cdot \rangle$ for the standard inner product on ${ℂ}^N$. The weighted ℓ²-norm with a Hermitian positive definite matrix A is denoted by ${\Vert \cdot \Vert }_A={\Vert A^{1/2}(\cdot )\Vert }_2$. ${\Vert \cdot \Vert }_0$ denotes the ℓ⁰-quasi-norm, i.e., the number of nonzeros of a vector. The Frobenius norm of a matrix is denoted by ${\Vert \cdot \Vert }_{\text{F}}$.${(\cdot )}^T$, ${(\cdot )}^H$, and ${(\cdot )}^{\ast }$ indicate the transpose, complex conjugate transpose (Hermitian transpose), and complex conjugate, respectively. diag(·) denotes the conversion of a vector into a diagonal matrix or diagonal elements of a matrix into a vector. ⴲ denotes the matrix direct sum of matrices. [C] denotes the set {1, 2, …, C}. Distinct from the index i, we denote the imaginary unit $\sqrt{-1}$ by i. For (self-adjoint) matrices $A,B\in {ℂ}^{N\times N}$, the notation $B\preccurlyeq A$ denotes that A − B is a positive semi-definite matrix.