Discretization of Learned NETT Regularization for Solving Inverse Problems

Abstract

Deep learning based reconstruction methods deliver outstanding results for solving inverse problems and are therefore becoming increasingly important. A recently invented class of learning-based reconstruction methods is the so-called NETT (for Network Tikhonov Regularization), which contains a trained neural network as regularizer in generalized Tikhonov regularization. The existing analysis of NETT considers fixed operators and fixed regularizers and analyzes the convergence as the noise level in the data approaches zero. In this paper, we extend the frameworks and analysis considerably to reflect various practical aspects and take into account discretization of the data space, the solution space, the forward operator and the neural network defining the regularizer. We show the asymptotic convergence of the discretized NETT approach for decreasing noise levels and discretization errors. Additionally, we derive convergence rates and present numerical results for a limited data problem in photoacoustic tomography.

1. Introduction

In this paper, we are interested in neural network based solutions to inverse problems of the form

$$\mathrm{Find}x\mathrm{from}\mathrm{data}{y}^{\delta }=\mathbf{A}x+\eta .$$ (1)

Here $\mathbf{A}$ is a potentially non-linear operator between Banach spaces $\mathbb{X}$ and $\mathbb{Y}$, $y^{\delta }$ are the given noisy data, x is the unknown to be recovered, $\eta$ is the unknown noise perturbation and $\delta \ge 0$ indicates the noise level. Numerous image reconstruction problems, parameter identification tasks and geophysical applications can be stated as such inverse problems [1,2,3,4]. Special challenges in solving inverse problems are the non-uniqueness of the solutions and the instability of the solutions with respect to the given data. To overcome these issues, regularization methods are needed, which select specific solutions and at the same time stabilize the inversion process.

1.1. Reconstruction with Learned Regularizers

One of the most established class of methods for solving inverse problems is variational regularization where regularized solutions are defined as minimizers of the generelaized Tikhonov functional [2,5,6]

$${\mathcal{T}}_{y^{\delta },\alpha }:\mathbb{X}\to [0,\infty ]:x\mapsto \mathcal{D}(\mathbf{A}x,y^{\delta })+\alpha \mathcal{R}\left(x\right).$$ (2)

Here $\mathcal{D}$ is a distance like function measuring closeness of the data, $\mathcal{R}$ a regularization term enforcing regularity of the minimizer and $\alpha$ is the regularization parameter. Taking minimizers of this functional as regularized solution is also called (generalized) Tikhonov regularization. In the case that $\mathcal{D}$ and the regularizer are defined by the Hilbert space norms, (2) is classical Tikhonov regularization for which the theory is quite complete [1,7]. In particular, in this case, convergence rates, which name quantitative estimates for the distance between the true noise-free solution and regularized solutions from noisy data, are well known. Convergence rates for non-convex regularizers are derived in [8].

Typical regularization techniques are based on simple hand crafted regularization terms such as the total variation $\parallel f{\parallel }_{\mathrm{TV}}=\int \left|\nabla f\right|$ or quadratic Sobolev norms ${\parallel \nabla f\parallel }_2^2=\int {\left|\nabla f\right|}^2$ on some function space. However, these regularizers are quite simplistic and might not well reflect the actual complexity of the underlying class of functions. Therefore, recently, it has been proposed and analyzed in [9] to use machine learning to construct regularizers in a data driven manner. In particular, the strategy in [9] is to construct a data-driven regularizer via the following consecutive steps:

• (T1)Choose a family of desired reconstructions ${\left(x_i\right)}_{i=1}^n$.
• (T2)For some $\mathbf{B}:\mathbb{Y}\to \mathbb{X}$, construct undesired reconstructions ${\left(\mathbf{B}\mathbf{A}{x}_i\right)}_{i=1}^n$.
• (T3)Choose a class ${\left({\Phi }_{\theta }\right)}_{\theta \in \Theta }$ of functions (networks) ${\Phi }_{\theta }:\mathbb{X}\to \mathbb{X}$.
• (T4)Determine ${\theta }^{\star }\in \Theta$ with ${\Phi }_{{\theta }^{\star }}\left(x_i\right)\simeq x_i\wedge {\Phi }_{{\theta }^{\star }}\left(\mathbf{B}\mathbf{A}{x}_i\right)\simeq x_i$.
• (T5)Define $\mathcal{R}\left(x\right)=r(x,\Phi (x\left)\right)$ with $\Phi ={\Phi }_{{\theta }^{\star }}$ for some $r:\mathbb{Y}\times \mathbb{Y}\to [0,\infty ]$.

For imaging applications, the function class ${\left({\Phi }_{\theta }\right)}_{\theta \in \Theta }$ can be chosen as convolutional neural networks which have demonstrated to give powerful classes of mappings between image spaces. The function r measures distance between a potential reconstruction x and the output of the network $\Phi \left(x\right)$, and possibly contains additional regularization [10,11]. According to the training strategy in item (T4) the value of the regularizer will be small if the reconstruction is similar to elements in ${\left(x_i\right)}_{i=1}^n$ and large for elements in ${\left(\mathbf{B}\mathbf{A}{x}_i\right)}_{i=1}^n$. A simple example that we will use for our numerical results is the learned regularizer $\mathcal{R}\left(x\right)=\parallel {x-\Phi \left(x\right)\parallel }^2+{\parallel x\parallel }_{\mathrm{TV}}$.

Convergence analysis and convergence rates for NETT (which stands for Network Tikhonov; referring to variants of (2), where the regularization term is given by a neural network) as well as training strategies have been established in [9,11,12]. A different training strategy for learning a regularizer has been proposed in [13,14]. Note that learning the regularizer first and then minimizing the Tikhonov functional is different from variational and iterative networks [15,16,17,18,19,20] where an iterative scheme is applied to enroll the functional ${\mathcal{D}}_{\theta }(\mathbf{A}x,y^{\delta })+\alpha {\mathcal{R}}_{\theta }\left(x\right)$ which is then trained in an end to end fashion. Training the regularizer first has the advantage of being more modular, sharing some similarity with plug and play techniques [21], and the network training is independent of the forward operator $\mathbf{A}$. Moreover, it enables to derive a convergence analysis as the noise level tends to zero and therefore comes with theoretical recovery guarantees.

1.2. Discrete NETT

The existing analysis of NETT considers minimizers of the Tikhonov functional (2) with regularizer of the form $\mathcal{R}\left(x\right)=r(x,\Phi (x\left)\right)$ before discretization, typically in an infinite dimensional setting. However, in practice, only finite dimensional approximations of the unknown, the operator and the neural network are given. To address these issues, in this paper, we study discrete NETT regularization which considers minimizers of

$${\mathcal{T}}_{y^{\delta },\alpha ,n}:{\mathbb{X}}_n\to \mathbb{Y}:x\mapsto \mathcal{D}({\mathbf{A}}_{n}x,y^{\delta })+\alpha {\mathcal{R}}_n\left(x\right).$$ (3)

Here ${\left({\mathbb{X}}_n\right)}_{n\in \mathbb{N}}$, ${\left({\mathbf{A}}_n\right)}_{n\in \mathbb{N}}$ and ${\left({\mathcal{R}}_n\right)}_{n\in \mathbb{N}}$ are families of subspaces of ${\mathbb{X}}_n\subseteq \mathbb{X}$, mappings ${\mathbf{A}}_n:\mathbb{X}\to \mathbb{Y}$ and regularizers ${\mathcal{R}}_n:\mathbb{X}\to [0,\infty ]$, respectively, which reflect discretization of all involved operations. We present a full convergence analysis as the noise level $\delta$ converges to zero and $n,\alpha$ are chosen accordingly. Discretization of variational regularization has studied in [22] for the case that $\mathcal{D}$ is given by the norm distance and the regularizer $\mathcal{R}$ is taken convex and fixed. However, in the case of discrete NETT regularization it is natural to consider the case where the regularization depends on the discretization as regularization is learned in a discretized setting based on actual data. For that purpose our analysis includes non-convex regularizers that are allowed to depend on the discretization and the noise level.

1.3. Outline

The convergence analysis including convergence rates is presented in Section 2. In Section 3 we will present numerical results for a non-standard limited data problem in photoacoustic tomography that can be considered as simultaneous inpainting and artifact removal problem. We conclude the paper with a short summary and conclusion presented in Section 4.

2. Convergence Analysis

In this section we study the convergence of (3) and derive convergence rates.

2.1. Well-Posedness

First we state the assumptions that we will use for well-posedness (existence and stability) of minimizing NETT.

Assumption 1

(Conditions for well-posedness).

• (W1) $\mathbb{X}$, $\mathbb{Y}$ are Banach spaces, $\mathbb{X}$ reflexive, $\mathbb{D}\subseteq \mathbb{X}$ weakly sequentially closed.
• (W2) The distance measure $\mathcal{D}:\mathbb{Y}\times \mathbb{Y}\to [0,\infty ]$ satisfies
  • (a) $\exists \tau \ge 1:\forall y_1,y_2,y_3\in \mathbb{Y}:\mathcal{D}(y_1,y_2)\le \tau \mathcal{D}(y_1,y_3)+\tau \mathcal{D}(y_3,y_2)$.
  • (b) $\forall y_1,y_2\in \mathbb{Y}:\mathcal{D}(y_1,y_2)=0\iff y_1=y_2$.
  • (c) $\forall y,\tilde{y}\in \mathbb{Y}:\mathcal{D}(y,\tilde{y})<\infty \wedge \parallel \tilde{y}-y_k\parallel \to 0\Rightarrow \mathcal{D}(y,y_k)\to \mathcal{D}(y,\tilde{y})$.
  • (d) $\forall y\in \mathbb{Y}:\parallel y_k-y\parallel \to 0\Rightarrow \mathcal{D}(y_k,y)\to 0$.
  • (e) $\mathcal{D}$ is weakly sequentially lower semi-continuous (wslsc).
• (W3) $\mathcal{R}:\mathbb{X}\to [0,\infty ]$ is proper and wslsc.
• (W4) $\mathbf{A}:\mathbb{D}\subseteq \mathbb{X}\to \mathbb{Y}$ is weakly sequentially continuous.
• (W5) $\forall y,\alpha ,C:\{x\in \mathbb{X}\mid {\mathcal{T}}_{y,\alpha }\le C\}$ is nonempty and bounded.
• (W6) ${\left({\mathbb{X}}_n\right)}_{n\in \mathbb{N}}$ is a sequence of subspaces of $\mathbb{X}$.
• (W7) ${\left({\mathbf{A}}_n\right)}_{n\in \mathbb{N}}$ is a family of weakly sequentially continuous ${\mathbf{A}}_n:\mathbb{D}\to \mathbb{Y}$.
• (W8) ${\left({\mathcal{R}}_n\right)}_{n\in \mathbb{N}}$ is a family of proper wslsc regularizers ${\mathcal{R}}_n:\mathbb{X}\to [0,\infty ]$.
• (W9) $\forall y,\alpha ,C,n:\{x\in {\mathbb{X}}_n\mid {\mathcal{T}}_{y,\alpha ,n}\le C\}$ is nonempty and bounded.

Conditions (W2)–(W5) are quite standard for Tikhonov regularization in Banach spaces to guarantee the existence and stability of minimizers of the Tikhonov functional and the given conditions are similar to [2,8,9,10,12,23,24]. In particular, (W2) describes the properties that the distance measure $\mathcal{D}$ should have. Clearly, the norm distance on $\mathbb{Y}$ fulfills these properties. Moreover, (W2a) holds for the norm with $\tau =1$ since it then corresponds to the triangle inequality. Item (W2c) is the continuity of $\mathcal{D}(y,·)$ while (W2d) considers the continuity of $\mathcal{D}(·,y)$ at y. While (W2c) is not needed for existence and convergence of NETT it is required for the stability result as shown in [10] (Example 2.7). On the other hand (W2e) implies that the Tikhonov functional is wslsc which is needed for existence. Assumption (W5) is a coercivity condition; see [9] (Remark 2.4f.) on how to achieve this for a regularizer defined by neural networks. Item (W8) poses some restrictions on the regularizers. For NETT this is not an issue as neural networks used in practice are continuous. Note that for convergence and convergence rates we will require additional conditions that concern the discretization of the reconstruction space, the forward operator and regularizer.

The references [8,9,10,23] all consider general distance measures and allow non-convex regularizers. However, existence and stability of minimizing (2) are shown under assumptions slightly different from (W1)–(W5). Below we therefore give a short proof of the existence and stability results.

Theorem 1

(Existence and Stability). Let Assumption 1 hold. Then for all $y^{\delta }\in \mathbb{Y}$, $\alpha >0$, $n\in \mathbb{N}$ the following assertions hold true:

• (a) $argmin{\mathcal{T}}_{y,\alpha ,n}\ne \varnothing$.
• (b) Let ${\left(y_k\right)}_{k\in \mathbb{N}}\in {\mathbb{Y}}^{\mathbb{N}}$ with $y_k\to y$ and consider $x_k\in argmin{\mathcal{T}}_{y_k,\alpha ,n}$.

  • ${\left(x_k\right)}_{k\in \mathbb{N}}$ has at least one weak accumulation point.

  • Every weak accumulation point ${\left(x_k\right)}_{k\in \mathbb{N}}$ is a minimizer of ${\mathcal{T}}_{y,\alpha ,n}$.

• (c) The statements in (a),(b) also hold for ${\mathcal{T}}_{y,\alpha }$ in place of ${\mathcal{T}}_{y,\alpha ,n}$,

Proof. 

Since (W1), (W6)–(W9) for ${\mathcal{T}}_{y,\alpha ,n}$ when $n\in \mathbb{N}$ are fixed give the same assumption as (W1), (W3)–(W5) for the non-discrete counterpart ${\mathcal{T}}_{y,\alpha }$, it is sufficient to verify (a), (b) for the latter. Existence of minimizers follows from (W1), (W2e), (W3)–(W5), because these items imply that the ${\mathcal{T}}_{y,\alpha }$ is a wslsc coercive functional defined on a nonempty weakly sequentially closed subset of a reflexive Banach space. To show stability one notes that according to (W2a) for all $x\in \mathbb{X}$ we have

$$\begin{array}{c}\mathcal{D}(\mathbf{A}{x}_k,y)+\alpha \mathcal{R}\left(x_k\right)\le \tau \left(\mathcal{D}(\mathbf{A}{x}_k,y_k)+\alpha \mathcal{R}\left(x_k\right)\right)+\tau \mathcal{D}(y,y_k)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\le \tau \left(\mathcal{D}(\mathbf{A}x,y_k)+\alpha \mathcal{R}\left(x\right)\right)+\tau \mathcal{D}(y,y_k).\hfill \end{array}$$

According to (W2c), (W2d), (W5) there exists $x\in \mathbb{X}$ such that the right hand side is bounded, which by (W5) shows that ${\left(x_k\right)}_k$ has a weak accumulation point. Following the standard proof [2] (Theorem 3.23) shows the weak accumulation points ${\left(x_k\right)}_k$ are minimizers of ${\mathcal{T}}_{y,\alpha }$. This uses the fact that the weak topology is indeed weaker than the norm topology, and that the involved functionals are wslsc.    □

In the following we write $x_{\alpha ,n}^{\delta }$ for minimizers of ${\mathcal{T}}_{y^{\delta },\alpha ,n}$. For $y\in \mathbb{Y}$ we call $x^{+}\in argmin\{\mathcal{R}\left(x\right)\mid x\in \mathbb{X}\wedge \mathbf{A}x=y\}$ an $\mathcal{R}$-minimizing solution of $\mathbf{A}x=y$.

Lemma 1

(Existence of $\mathcal{R}$-minimizing solutions). Let Assumption 1 hold. For any $y\in \mathbf{A}\left(\mathbb{D}\right)$ an $\mathcal{R}$-minimizing solution of $\mathbf{A}x=y$ exists. Likewise, if $n\in \mathbb{N}$ and $y\in {\mathbf{A}}_n\left(\mathbb{D}\right)$ an ${\mathcal{R}}_n$-minimizing solution of ${\mathbf{A}}_{n}x=y$ exists.

Proof. 

Again is is sufficient the verify the claim for $\mathcal{R}$-minimizing solution. Because $y\in \mathbf{A}\left(\mathbb{D}\right)$, the set ${\mathbf{A}}^{-1}\left(\left\{y\right\}\right)=\{x\in \mathbb{X}\mid \mathbf{A}x=y\}$ is non-empty. Hence we can choose a sequence ${\left(x_k\right)}_{k\in \mathbb{N}}$ in ${\mathbf{A}}^{-1}\left(\left\{y\right\}\right)$ with $\mathcal{R}\left(x_k\right)\to inf\{\mathcal{R}\left(x\right)\mid x\in \mathbb{X}\wedge \mathbf{A}x=y\}$. Due to (W2b), ${\left(x_k\right)}_{k\in \mathbb{N}}$ is contained in $\{x\in \mathbb{X}\mid \mathcal{D}(\mathbf{A}\left(x\right),y)+\alpha \mathcal{R}(x)\le C\}$ for some $C>0$ which is bounded according to (W5). By (W1) $\mathbb{X}$ is reflexive and therefore ${\left(x_k\right)}_{k\in \mathbb{N}}$ has a weak accumulation point $x^{+}$. From (W1), (W4), (W3) we conclude that $x^{+}$ is an $\mathcal{R}$-minimizing solution of $\mathbf{A}x=y$. The case of ${\mathcal{R}}_n$-minimizing solutions follows analogous.    □

2.2. Convergence

Next we proof that discrete NETT converges as the noise level goes to zero and the discretization as well as the regularization parameter are chosen properly. We write ${\mathbb{D}}_{n,M}:=\{x\in \mathbb{D}\cap {\mathbb{X}}_n\mid {\mathcal{R}}_n\left(x\right)\le M\}$ and formulate the following approximation conditions for obtaining convergence.

Assumption 2

(Conditions for convergence).

Element $x^{+}\in \mathbb{D}$ satisfies the following for all $M>0$:

• (C1)  $\exists \left(z_n\right)\in {\prod }_{n\in \mathbb{N}}(\mathbb{D}\cap {\mathbb{X}}_n)$ with ${\lambda }_n:=\left|{\mathcal{R}}_n\left(z_n\right)-\mathcal{R}\left(x^{+}\right)\right|\to 0$.
• (C2)  ${\rho }_n:={sup}_{x\in {\mathbb{D}}_{n,M}}|{\mathcal{R}}_n\left(x\right)-\mathcal{R}\left(x\right)|\to 0$.
• (C3)  ${\gamma }_n:=\mathcal{D}({\mathbf{A}}_n{z}_n,\mathbf{A}{x}^{+})\to 0$.
• (C4)  $a_n:={sup}_{x\in {\mathbb{D}}_{n,M}}|\mathcal{D}({\mathbf{A}}_{n}x,\mathbf{A}{x}^{+})-\mathcal{D}(\mathbf{A}x,\mathbf{A}{x}^{+})|\to 0$.

Conditions (C1) and (C3) concerns the approximation of the true unknown x with elements in the discretization space, that is compatible with the discretization of the forward operator and regularizer. Conditions (C2) and (C4) are uniform approximation properties of the operator and the regularizer on ${\mathcal{R}}_n$-bounded sets.

Theorem 2

(Convergence). Let (W1)–(W9) hold, $y\in \mathbf{A}\left(\mathbb{D}\right)$ and let $x^{+}$ be an $\mathcal{R}$-minimizing solution of $\mathbf{A}x=y$ that satisfies (C1)–(C4). Moreover, suppose ${\left({\delta }_k\right)}_{k\in \mathbb{N}}\in {(0,\infty )}^{\mathbb{N}}$ converges to zero and ${\left(y_k\right)}_{k\in \mathbb{N}}\in {\mathbb{Y}}^{\mathbb{N}}$ satisfies $\mathcal{D}(y,y_k)\le {\delta }_k$. Choose ${\left({\alpha }_k\right)}_{k\in \mathbb{N}}$ and ${\left(n_k\right)}_{k\in \mathbb{N}}$ such that as $k\to \infty$ we have

$$\begin{array}{cc}\hfill & {\alpha }_k\to 0\hfill \end{array}$$ (4)

$$\begin{array}{cc}\hfill & n_k\to \infty \hfill \end{array}$$ (5)

$$\begin{array}{cc}\hfill & ({\delta }_k+\mathcal{D}({\mathbf{A}}_{n_k}{z}_{n_k},y))/{\alpha }_k\to 0.\hfill \end{array}$$ (6)

Then for $x_k\in argmin{\mathcal{T}}_{y_k,{\delta }_k,n_k}$ the following hold:

• (a) ${\left(x_k\right)}_{k\in \mathbb{N}}$ has a weakly convergent subsequence ${\left(x_{\sigma \left(k\right)}\right)}_{k\in \mathbb{N}}$
• (b) The weak limit of ${\left(x_{\sigma \left(k\right)}\right)}_{k\in \mathbb{N}}$ is an $\mathcal{R}$-minimizing solution of $\mathbf{A}x=y$.
• (c) ${\mathcal{R}}_{\sigma \left(k\right)}\left(x_{\sigma \left(k\right)}\right)\to \mathcal{R}\left(x^{\star }\right)$, where $x^{\star }$ is the weak limit of ${\left(x_{\sigma \left(k\right)}\right)}_{k\in \mathbb{N}}$.
• (d) If the $\mathcal{R}$-minimizing solution of $\mathbf{A}x=y$ is unique, then ${\left(x_k\right)}_{k\in \mathbb{N}}\rightharpoonup x^{+}$.

Proof. 

For convenience and some abuse of notation we use the abbreviations ${\mathcal{R}}_k:={\mathcal{R}}_{n_k}$, ${\mathbf{A}}_k:={\mathbf{A}}_{n_k}$, $a_k:=a_{n_k}$, $z_k:=z_{n_k}$ and ${\rho }_k:={\rho }_{n_k}$. Because $x_k$ is a minimizer of the discrete NETT functional ${\mathcal{T}}_{y_k,{\delta }_k,n_k}$ by (W2) we have

$$\begin{array}{cc}\hfill \mathcal{D}({\mathbf{A}}_k{x}_k,y_k)& +{\alpha }_k{\mathcal{R}}_k\left(x_k\right)\le \mathcal{D}({\mathbf{A}}_k{z}_k,y_k)+{\alpha }_k{\mathcal{R}}_k\left(z_k\right)\hfill \\ & \le \tau \mathcal{D}({\mathbf{A}}_k{z}_k,y)+\tau \mathcal{D}(y,y_k)+{\alpha }_k{\mathcal{R}}_k\left(z_k\right)=\tau \mathcal{D}({\mathbf{A}}_k{z}_k,y)+\tau {\delta }_k+{\alpha }_k{\mathcal{R}}_k\left(z_k\right)\hfill \end{array}$$

According to (C1), (4), we get

$$\begin{array}{c}\hfill \mathcal{D}({\mathbf{A}}_k{x}_k,y_k)\le \tau (\mathcal{D}({\mathbf{A}}_k{z}_k,y)+{\delta }_k),\end{array}$$ (7)

$$\begin{array}{c}\hfill {\mathcal{R}}_k\left(x_k\right)\le \tau ·\frac{\mathcal{D}({\mathbf{A}}_k{z}_k,y_k)+{\delta }_k}{{\alpha }_k}+{\mathcal{R}}_k\left(z_k\right).\end{array}$$ (8)

According to (C1), (C3), (5), (6) the right hand side in (7) converges to zero and the right hand side in (8) to $\mathcal{R}\left(x_{+}\right)$. Together with (C2) we obtain $\mathcal{R}\left(x_k\right)\le {\mathcal{R}}_k\left(x_k\right)+{\rho }_k\to \mathcal{R}\left(x_{+}\right)$ and $\mathcal{D}(\mathbf{A}{x}_k,y)\le \mathcal{D}({\mathbf{A}}_k{x}_k,y)+a_k\le \tau \mathcal{D}({\mathbf{A}}_k{x}_k,y)+a_k+\tau {\delta }_k\to 0$. This shows that ${(\mathcal{D}(\mathbf{A}{x}_k,y)+\mathcal{R}\left(x_k\right))}_{k\in \mathbb{N}}$ is bounded and by (W1), (W9) there exists a weakly convergent subsequence ${\left(x_{\sigma \left(k\right)}\right)}_{k\in \mathbb{N}}$. We denote the weak limit by $x^{\star }\in \mathbb{X}$. From (W2), (W4) we obtain $\mathbf{A}x=y$. The weak lower semi-continuity of $\mathcal{R}$ assumed in (W3) shows

$$\begin{array}{c}\mathcal{R}\left(x^{\star }\right)\le \underset{k}{lim\; inf}\mathcal{R}\left(x_{\sigma \left(k\right)}\right)\le \underset{k}{lim\; sup}\mathcal{R}\left(x_{\sigma \left(k\right)}\right)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\le \underset{k}{lim\; sup}({\mathcal{R}}_{\sigma \left(k\right)}\left(x_{\sigma \left(k\right)}\right)+{\rho }_k)\le \mathcal{R}\left(x^{+}\right).\hfill \end{array}$$

Consequently, $x^{\star }$ is an $\mathcal{R}$-minimizing solution of $\mathbf{A}x=y$ and $\mathcal{R}\left(x_{\sigma \left(k\right)}\right)\to \mathcal{R}\left(x^{\star }\right)$. If the $\mathcal{R}$-minimizing solution is unique then $x^{+}$ is the only weak accumulation point of ${\left(x_k\right)}_{k\in \mathbb{N}}$ which concludes the proof.    □

2.3. Convergence Rates

Next we derive quantitative error estimates (convergence rates) in terms of the absolute Bregman distance. Recall that a function $\mathcal{R}:\mathbb{X}\to [0,\infty ]$ is Gâteaux differentiable at some $x^{\star }\in \mathbb{X}$ if the directional derivative ${\mathcal{R}}^{\prime }\left(x^{\star }\right)\left(h\right):=(\mathcal{R}(x^{\star }+th)-\mathcal{R}\left(x^{\star }\right))/t$ exist for every $h\in \mathbb{X}$. We denote by ${\mathcal{R}}^{\prime }\left(x^{\star }\right)$ the Gâteaux derivative of $\mathcal{R}$ at x. In [9] we introduced the absolute Bregman distance ${\mathcal{B}}_{\mathcal{R}}(·,x^{\star }):\mathbb{X}\to [0,\infty ]$ of a Gâteaux differentiable functional $\mathcal{R}:\mathbb{X}\to [0,\infty ]$ at $x^{\star }\in \mathbb{X}$ with respect to $\mathcal{R}$ defined by

$$\forall x\in \mathbb{X}:{\mathcal{B}}_{\mathcal{R}}(x,x^{\star }):=\left|\mathcal{R}\left(x\right)-\mathcal{R}\left(x^{\star }\right)-{\mathcal{R}}^{\prime }\left(x^{\star }\right)(x-x^{\star })\right|.$$ (9)

We write ${sup}_{y^{\delta }}H\left(y^{\delta }\right):=sup\{H\left(y^{\delta }\right)\mid y^{\delta }\in \mathbb{X}\wedge \mathcal{D}(\mathbf{A}{x}^{+},y^{\delta })\le \delta \}$. Convergence rates in terms of the Bregman distance are derived under a smoothness assumption on the true solution in the form of a certain variational inequality. More precisely we assume the following:

Assumption 3

(Conditions for convergence rates).

Element $x^{+}\in \mathbb{D}$ satisfies the following for all $M,\delta >0$:

• (R1)  Items (C1), (C2) hold.
• (R2)  ${\gamma }_{n,\delta }:={sup}_{y^{\delta }}\left|\mathcal{D}({\mathbf{A}}_n{z}_n,y^{\delta })-\mathcal{D}(\mathbf{A}{x}^{+},y^{\delta })\right|\to 0$.
• (R3)  $a_{n,\delta }:={sup}_{y^{\delta }}{sup}_{x\in {\mathbb{D}}_{n,M}}|\mathcal{D}({\mathbf{A}}_{n}x,y^{\delta })-\mathcal{D}(\mathbf{A}x,y^{\delta })|\to 0$.
• (R4)  $\mathcal{R}$ is Gâteaux differentiable at $x^{+}$
• (R5) 
  There exist a concave, continuous, strictly increasing $\phi :[0,\infty )\to [0,\infty )$ with $\phi \left(0\right)=0$ and $ϵ,\beta >0$ such that for all $x\in \mathbb{X}$

  $$|\mathcal{R}\left(x\right)-\mathcal{R}\left(x^{+}\right)|\le ϵ\Rightarrow \beta {\mathcal{B}}_{\mathcal{R}}(x,x^{+})\le \mathcal{R}\left(x\right)-\mathcal{R}\left(x^{+}\right)+\phi \left(\mathcal{D}(\mathbf{A}x,\mathbf{A}{x}^{+})\right).$$

According to (R5) the inverse function ${\phi }^{-1}:[0,\infty )\to [0,\infty )$ exists and is convex. We denote by ${\phi }^{-*}\left(s\right):=sup\{st-{\phi }^{-1}\left(t\right)\mid t\ge 0\}$ its Fenchel conjugate.

Proposition 1

(Error estimates). Let $y\in \mathbf{A}\left(\mathbb{D}\right)$ and $x^{+}$ be an $\mathcal{R}$-minimizing solution of $\mathbf{A}x=y$ such that (W1)–(W9) and (R1)–(R5) are satisfied. For $y^{\delta }\in \mathbb{Y}$ with $\mathcal{D}(y,y^{\delta })\le \delta$ let $x_{\alpha ,n}^{\delta }\in argmin{\mathcal{T}}_{y^{\delta },\alpha ,n}$. Then for sufficient small $\delta ,\alpha >0$ and sufficiently large $n\in \mathbb{N}$, we have the error estimate

$${\mathcal{B}}_{\mathcal{R}}(x_{\alpha ,n}^{\delta },x^{+})\le \frac{a_{n,\delta }+{\gamma }_{n,\delta }+\delta }{\alpha }+{\rho }_n+{\lambda }_n+\phi \left(\tau \delta \right)+\frac{{\phi }^{-*}\left(\tau \alpha \right)}{\tau \alpha }.$$ (10)

Proof. 

According to Theorem 2 we can assume $|\mathcal{R}\left(x_{\alpha ,n}^{\delta }\right)-\mathcal{R}\left(x^{+}\right)|<ϵ$ and with (R5) we obtain

$$\begin{array}{cc}\hfill & \alpha \beta {\mathcal{B}}_{\mathcal{R}}(x_{\alpha ,n}^{\delta },x^{+})\hfill \\ & \le \alpha \mathcal{R}\left(x_{\alpha ,n}^{\delta }\right)-\alpha \mathcal{R}\left(x^{+}\right)+\alpha \phi \left(\mathcal{D}(\mathbf{A}{x}_{\alpha ,n}^{\delta },y)\right)\hfill \\ & \le \alpha {\mathcal{R}}_n\left(x_{\alpha ,n}^{\delta }\right)-\alpha {\mathcal{R}}_n\left(z_n\right)+\alpha {\rho }_n+\alpha {\lambda }_n+\alpha \phi \left(\mathcal{D}(\mathbf{A}{x}_{\alpha ,n}^{\delta },y)\right)\hfill \\ & \le \mathcal{D}({\mathbf{A}}_n{z}_n,y^{\delta })-\mathcal{D}({\mathbf{A}}_n{x}_{\alpha ,n}^{\delta },y^{\delta })+\alpha {\rho }_n+\alpha {\lambda }_n+\alpha \phi \left(\mathcal{D}(\mathbf{A}{x}_{\alpha ,n}^{\delta },y)\right)\hfill \\ & \le \delta -\mathcal{D}(\mathbf{A}{x}_{\alpha ,n}^{\delta },y^{\delta })+{\gamma }_{n,\delta }+a_{n,\delta }+\alpha {\rho }_n+\alpha {\lambda }_n+\alpha \phi \left(\tau \delta \right)+\alpha \phi \left(\tau \mathcal{D}(\mathbf{A}{x}_{\alpha ,n}^{\delta },y^{\delta })\right)\hfill \\ & \le \delta +{\gamma }_{n,\delta }+a_{n,\delta }+\alpha {\rho }_n+\alpha {\lambda }_n+\alpha \phi \left(\tau \delta \right)+{\tau }^{-1}{\phi }^{-*}\left(\tau \delta \right).\hfill \end{array}$$

For the second inequality we used (C1) and (C2). We have $\mathcal{D}({\mathbf{A}}_n{x}_{\alpha ,n}^{\delta },y^{\delta })+\alpha {\mathcal{R}}_n\left(x_{\alpha ,n}^{\delta }\right)\le \mathcal{D}({\mathbf{A}}_n{z}_n,y^{\delta })+\alpha {\mathcal{R}}_n\left(z_n\right)$ and thus we get an estimate for ${\mathcal{R}}_n\left(x_{\alpha ,n}^{\delta }\right)-{\mathcal{R}}_n\left(z_n\right)$ which we used for the third inequality. For the next inequality we used (R2) and (R3). Finally we used Young’s inequality $\alpha \phi \left(\tau t\right)\le t+{\tau }^{-1}{\phi }^{-*}\left(\tau \alpha \right)$ for the last step.    □

Remark 1.

The error estimate (10) includes the approximation quality of the discrete or inexact forward operator ${\mathbf{A}}_n$ and the discrete or inexact regularizer ${\mathcal{R}}_n$ described by $a_{n,\delta }$ and ${\rho }_n$, respectively. What might be unexpected at first is the inclusion of two new parameters ${\lambda }_n$ and ${\gamma }_{n,\delta }$. These factors both arise from the approximation of $\mathbb{X}$ by the finite dimensional spaces ${\mathbb{X}}_n$, where ${\gamma }_{n,\delta }$ reflects approximation accuracy in the image of the operator $\mathbf{A}$ and ${\lambda }_n$ approximation accuracy with respect to the true regularization functional $\mathcal{R}$. Note that in the case where the forward operator, the regularizer, and the solution space $\mathbb{X}$ are given precisely, we have $a_{n,\delta }={\gamma }_{n,\delta }={\lambda }_n={\rho }_n=0$. In this particular case we recover the estimate derived for the NETT in [9].

Theorem 3

(Convergence rates). Let the assumptions of Proposition 1 hold and consider the parameter choice rule $\alpha \left(\delta \right)\asymp \delta /\phi \left(\delta \right)$ and let the approximation errors satisfy $a_{n,\delta }+{\gamma }_{n,\delta }=\mathcal{O}\left(\delta \right)$, ${\rho }_n+{\lambda }_n=\mathcal{O}\left(\phi \left(\tau \delta \right)\right)$. Then we have the convergence rate

$${\mathcal{B}}_{\mathcal{R}}(x_{\alpha \left(\delta \right),n\left(\delta \right)}^{\delta },x^{+})=\mathcal{O}\left(\phi \left(\tau \delta \right)\right).$$ (11)

Proof. 

Noting that ${\phi }^{-*}(\tau \delta /\phi \left(\tau \delta \right))/\delta$ remains bounded as $\delta \to 0$, this directly follows from Proposition 1.    □

Next we verify that a variational inequality of the form (R5) is satisfied with $\phi \left(t\right)=c\sqrt{t}$ under a typical source like condition.

Lemma 2

(Variational inequality under source condition). Let $\mathcal{R}$, $\mathbf{A}$ be Gâteaux differentiable at $x^{+}\in \mathbb{X}$, consider the distance measure $\mathcal{D}(y_1,y_2)={\parallel y_1-y_2\parallel }^2$ and assume there exist $\eta \in {\mathbb{X}}^{\star }$ and $c_1,c_2,ϵ>0$ with $c_1\parallel \eta \parallel <1$ such that for all $x\in \mathbb{X}$ with $|\mathcal{R}\left(x\right)-\mathcal{R}\left(x^{+}\right)|\le ϵ$ we have

$$\begin{array}{cc}\hfill & {\mathcal{R}}^{\prime }\left(x^{+}\right)={\mathbf{A}}^{\prime }{\left(x^{+}\right)}^{*}\eta \hfill \\ \hfill & \parallel \mathbf{A}x-\mathbf{A}{x}^{+}-{\mathbf{A}}^{\prime }\left(x^{+}\right)(x-x^{+})\parallel \le c_1{\mathcal{B}}_{\mathcal{R}}(x,x^{+})\hfill \\ \hfill & \mathcal{R}\left(x^{+}\right)-\mathcal{R}\left(x\right)\le c_2\parallel \mathbf{A}x-\mathbf{A}{x}^{+}\parallel .\hfill \end{array}$$ (12)

Then (R5) holds with $\phi \left(t\right)=(\parallel \eta \parallel +2c_2)\sqrt{t}$ and $\beta =1-c_1\parallel \eta \parallel$.

Proof. 

Let $x\in \mathbb{X}$ with $|\mathcal{R}\left(x\right)-\mathcal{R}\left(x^{+}\right)|\le ϵ$. Using the Cauchy-Schwarz inequality and Equation (12), we can estimate

$$\begin{array}{cc}\hfill |\langle {\mathcal{R}}^{\prime }\left(x^{+}\right),x-x^{+}\rangle |\le & \parallel {\mathbf{A}}^{\prime }\left(x^{+}\right)(x-x^{+})\parallel \parallel \eta \parallel \hfill \\ \hfill \le & \parallel \mathbf{A}x-\mathbf{A}{x}^{+}\parallel \parallel \eta \parallel +\parallel \mathbf{A}x-\mathbf{A}{x}^{+}-{\mathbf{A}}^{\prime }\left(x^{+}\right)(x-x^{+})\parallel \parallel \eta \parallel \hfill \\ \hfill \le & \parallel \mathbf{A}x-\mathbf{A}{x}^{+}\parallel \parallel \eta \parallel +c_1\parallel \eta \parallel {\mathcal{B}}_{\mathcal{R}}(x,x^{+}).\hfill \end{array}$$

Additionally, if $\mathcal{R}\left(x\right)\ge \mathcal{R}\left(x^{+}\right)$, we have $|\mathcal{R}\left(x\right)-\mathcal{R}\left(x^{+}\right)|=\mathcal{R}\left(x\right)-\mathcal{R}\left(x^{+}\right)$, and on the other hand if $\mathcal{R}\left(x\right)<\mathcal{R}\left(x^{+}\right)$, we have $|\mathcal{R}\left(x\right)-\mathcal{R}\left(x^{+}\right)|\le \mathcal{R}\left(x\right)-\mathcal{R}\left(x^{+}\right)+2(\mathcal{R}\left(x^{+}\right)-\mathcal{R}\left(x\right))\le \mathcal{R}\left(x\right)-\mathcal{R}\left(x^{+}\right)+2c_2\parallel \mathbf{A}x-\mathbf{A}{x}^{+}\parallel$. Putting this together we get

$$\begin{array}{cc}\hfill {\mathcal{B}}_{\mathcal{R}}(x,x^{+})\le |\mathcal{R}\left(x\right)-& \mathcal{R}\left(x^{+}\right)|+|\langle {\mathcal{R}}^{\prime }\left(x^{+}\right),x-x^{+}\rangle |\hfill \\ & \hfill \le \mathcal{R}\left(x\right)-\mathcal{R}\left(x^{+}\right)+(\parallel \eta \parallel +2c_2)\parallel \mathbf{A}x-\mathbf{A}{x}^{+}\parallel +c_1\parallel \eta \parallel {\mathcal{B}}_{\mathcal{R}}(x,x^{+}),\end{array}$$

and thus $(1-c_1\parallel \eta \parallel ){\mathcal{B}}_{\mathcal{R}}(x,x^{+})\le \mathcal{R}\left(x\right)-\mathcal{R}\left(x^{+}\right)+(\parallel \eta \parallel +2c_2)\parallel \mathbf{A}x-\mathbf{A}{x}^{+}\parallel$.    □

Corollary 1

(Convergence rates under source condition). Let the conditions of Lemma 2 hold and suppose

$$\begin{array}{cc}\hfill & \alpha \left(\delta \right)\asymp \sqrt{\delta }\hfill \\ \hfill & |{\mathcal{R}}_{n\left(\delta \right)}\left(z_{n\left(\delta \right)}\right)-\mathcal{R}\left(x^{+}\right)|=\mathcal{O}\left(\sqrt{\delta }\right)\hfill \\ \hfill & sup\{|{\mathcal{R}}_{n\left(\delta \right)}\left(x\right)-\mathcal{R}\left(x\right)|\mid x\in {\mathbb{D}}_{n\left(\delta \right),M}\}=\mathcal{O}\left(\sqrt{\delta }\right)\hfill \\ \hfill & \parallel {\mathbf{A}}_{n\left(\delta \right)}{z}_{n\left(\delta \right)}-\mathbf{A}{x}^{+}\parallel =\mathcal{O}\left(\sqrt{\delta }\right)\hfill \\ \hfill & sup\{\parallel {\mathbf{A}}_{n\left(\delta \right)}x-\mathbf{A}x\parallel \mid x\in {\mathbb{D}}_{n\left(\delta \right),M}\}=\mathcal{O}\left(\sqrt{\delta }\right)\hfill \\ \hfill & sup\{\parallel {\mathbf{A}}_{n\left(\delta \right)}x\parallel \mid x\in {\mathbb{D}}_{n\left(\delta \right),M}\}<\infty .\hfill \end{array}$$

Then we have the convergence rates result

$${\mathcal{B}}_{\mathcal{R}}(x_{\alpha \left(\delta \right),n\left(\delta \right)}^{\delta },x^{+})=\mathcal{O}\left(\sqrt{\delta }\right).$$ (13)

Proof. 

Follows from Theorem 3 and Lemma 2. Note that we use $\parallel ·\parallel$ in the theorem, while $\mathcal{D}(y_1,y_1)={\parallel y_1-y_2\parallel }^2$ uses the squared norm ${\parallel ·\parallel }^2$ and thus the approximation rates for the terms concerning ${\mathbf{A}}_{n\left(\delta \right)}$ are order $\sqrt{\delta }$ instead of $\delta$ as in Theorem 3.    □

In Corollary 1, the approximation quality of the discrete operator ${\mathbf{A}}_n$ and the discrete and inexact regularization functional ${\mathcal{R}}_n$ need to be of the same order.

3. Application to a Limited Data Problem in PAT

Photoacoustic Tomography (PAT) is an emerging non-invasive coupled-physics biomedical imaging technique with high contrast and high spatial resolution [25,26]. It works by illuminating a semi-transparent sample with short optical pulses which causes heating of the sample followed by expansion and the subsequent emission of an acoustic wave. Sensors on the outside of the sample measure the acoustic wave and these measurements are then used to reconstruct the initial pressure $f:{\mathbb{R}}^d\to \mathbb{R}$, which provides information about the interior of the object. The cases $d=2$ and $d=3$ are relevant for applications in PAT. Here we only consider the case $d=2$ and assume a circular measurement geometry. The 2D case arises for example when using integrating line detectors in PAT [26].

3.1. Discrete Forward Operator

The pressure data $p:{\mathbb{R}}^2\times [0,\infty )\to \mathbb{R}$ satisfies the wave equation $({\partial }_t^2-\Delta )p(\mathbf{r},t)=0\mathrm{for}(\mathbf{r},t)\in {\mathbb{R}}^2\times (0,\infty )$ with initial data $p(·,0)=$ and ${\partial }_{t}p(·,0)=0$. In the case of circular measurement geometry one assumes that f vanishes outside the unit disc $D_1:=\{\mathbf{r}\in {\mathbb{R}}^2\mid \parallel \mathbf{r}\parallel <1\}$ and the measurement sensors are located on the boundary $\partial D_1={\mathbb{S}}^1$. We assume that the phantom will not generate any data for some region $I\subseteq D_1$, for example when the acoustic pressure generated inside I is too small to be recorded. This masked PAT problem consists in the recovery of the function f from sampled noisy measurements of $g=\mathcal{W}\left({\mathbb{1}}_{I^c}f\right)$ where $\mathcal{W}$ denotes the solution operator of the wave equation and ${𝟙}_{I^c}$ the indicator function on $I^c:={\mathbb{R}}^2\setminus I$. Note that the resulting inverse problem can be seen of the combination of an inpainting problem and in inverse problems for the wave equation.

In order to implement the PAT forward operator we use a basis ansatz $f\left(\mathbf{r}\right)={\sum }_{i=1}^{N\times N}{x}_i\psi (\mathbf{r}-{\mathbf{r}}_i)$ where $x_i\in \mathbb{R}$ are basis coefficients and $\psi :{\mathbb{R}}^2\to \mathbb{R}$ a generalized Kaiser-Bessel (KB) and ${\mathbf{r}}_i=(i-1)/N$ with $i=(i_1,i_2)\in {\{1,\cdots ,N\}}^2$. The generalized KB functions are popular in tomographic inverse problems [27,28,29,30] and denote radially symmetric functions with support in $D_R$ defined by

$$\psi \left(\mathbf{r}\right):={\left(1-{\parallel \mathbf{r}\parallel }^2/R^2\right)}^{m/2}\frac{I_m\left(\gamma \sqrt{1-{\parallel \mathbf{r}\parallel }^2/R^2}\right)}{I_m\left(\gamma \right)}\mathrm{for}\parallel \mathbf{r}\parallel \le R.$$ (14)

Here $I_m$ is the modified Bessel function of the first kind of order $n\in \mathbb{N}$ and the parameters $\gamma >0$ and R denote the window taper and support radius, respectively. Since $\mathcal{W}$ is linear we have $\mathcal{W}f={\sum }_{i=1}^{N\times N}{x}_i\mathcal{W}\left(\psi (·-{\mathbf{r}}_i)\right)$. For convenience we will use a pseudo-3D approach where use the 3D solution of $\mathcal{W}\psi$ for which there exists an analytical representation [29]. Denote by ${\mathbf{s}}_k$ uniformly spaced sensor locations on ${\mathbb{S}}^1$ and by $t_j>0$ uniformly sampled measurement times in $[0,2]$. Define the $N_t{N}_s\times N^2$ model matrix by ${\mathbf{W}}_{N_t(k-1)+j,N(i_1-1)+i_2}=\mathcal{W}\left(\psi (·-{\mathbf{r}}_i)\right)({\mathbf{s}}_k,t_j)$ and an $N^2\times N^2$ diagonal matrix by ${\left({\mathbf{M}}_I\right)}_{N(i_1-1)+i_2,N(i_1-1)+i_2}=1$ if ${\mathbf{r}}_i\in I^c$ and zero otherwise. Let $\mathbf{W}{\mathbf{M}}_I=\mathbf{U}\Sigma {\mathbf{V}}^{⊺}$ be the singular valued decomposition. We then consider the discrete forward matrix $\mathbf{A}=\mathbf{U}{\Sigma }_{\star }{\mathbf{V}}^{⊺}$ where ${\Sigma }_{\star }$ is the diagonal matrix derived from $\Sigma$ by setting singular values smaller than some ${\sigma }_{\star }$ to zero. This allows us to easily calculate ${\mathbf{A}}^{+}=\mathbf{V}{\Sigma }_{\star }^{+}{\mathbf{U}}^{⊺}$ where ${\Sigma }_{\star }^{+}$ is calculated by inverting all diagonal elements of ${\Sigma }_{\star }$ that are greater than zero. In our experiments we use $N=N_t=128$, $N_s=150$ and take I fixed as a diagonal stripe of width $0.34$.

3.2. Discrete NETT

We consider the discrete NETT with discrepancy term $\mathcal{D}(\mathbf{A}x,y^{\delta })={\parallel \mathbf{A}x-y^{\delta }\parallel }_2^2/2$ and regularizer given by

$${\mathcal{R}}^{\left(m\right)}\left(x\right)=\parallel {x-{\Phi }^{\left(m\right)}\left(x\right)\parallel }_2^2+\beta {\parallel \nabla x\parallel }_{1,ϵ},$$ (15)

where $\parallel {\nabla x\parallel }_{1,ϵ}:={\sum }_{i_1,i_2=1}^{128}\sqrt{{|{\mathbf{r}}_{i_1+1,i_2}-{\mathbf{r}}_i|}^2+{|{\mathbf{r}}_{i_1,i_2+1}-{\mathbf{r}}_{i_1,i_2}|}^2+{ϵ}^2}$ with $ϵ>0$ is a smooth version of the total variation [31] and ${\Phi }^{\left(m\right)}$ is a learnable network. We take ${\Phi }^{\left(m\right)}$ as the U-Net [32] with residual connection, which has first been applied to PAT image reconstruction in [33]. Here m stands for the number of down-/upsampling steps performed in the U-Net (the original one had $m=4$ i.e., $m\in \mathbb{N}$. This means that larger m yield a deeper network with more parameters. We generate training data that consist of square shaped rings with random profile and random location. See Figure 1 for an example of one such phantom (note that all plots in signal space use the same colorbar) and the corresponding data. We get a set of phantoms $x_1,\cdots ,x_{1000}$ and corresponding basic reconstructions $h_a:={\mathbf{A}}^{+}(\mathbf{A}{x}_a+{\eta }_a)$, where ${\mathbf{A}}^{+}$ is the pseudo-inverse and ${\eta }_a$ is Gaussian noise with standard deviation of $\sigma \parallel {\mathbf{A}{x}_a\parallel }_{\infty }$ with $\sigma =0.01$. The networks are trained by minimizing ${\sum }_{a=1}^{1000}{\parallel {\Phi }^{\left(m\right)}\left(h_a\right)-x_a\parallel }_1+\gamma {\parallel {\Phi }^{\left(m\right)}\left(x_a\right)-x_a\parallel }_1$ where we used the Adam optimizer with learning rate 0.01 and $\gamma =0.1$. The considered loss is that we want the trained regularizer to give small values for $x_a$ and large values for $h_a$. The strategy is similar to [9] but we use the final output of the network for the regularizer as proposed in [34]. To minimize (15) we use Algorithm 1 which implements a forward-backward scheme [35]. The most expensive step of this algorithm is the matrix inversion but since we use constant stepsize one also has to option to only calculate the inverse of the matrix once and reuse it. Thus one only has to perform two matrix-vector multiplications which are of the order $\mathcal{O}\left(N^2{N}_t{N}_s\right)$ and $\mathcal{O}\left(N^4\right)$ since $N_2$ is the dimension of our phantoms. On the other hand calculating the gradient has similar complexity than applying the neural network which is in the order $\mathcal{O}\left(F^{2}L{N}^2\right)$ with F the number of convolution channels and L the number of layers.

Algorithm 1: NETT optimization.

Figure 1.

Top from left to right: phantom, masked phantom, and initial reconstruction ${\mathbf{A}}^{+}\mathbf{A}x$. The difference between the phantoms on the left and the middle one shows the mask region $I\subseteq D_1$ where no data is generated. Bottom from left to right: data without noise, low noise $\sigma =0.01$, and high noise $\sigma =0.1$.

3.3. Numerical Results

For the numerical results we train two regularizers ${\mathcal{R}}^{\left(1\right)}$ and ${\mathcal{R}}^{\left(3\right)}$ as described in Section 3.2. The networks are implemented using PyTorch [36]. We also use PyTorch in order to calculate the gradient ${\nabla }_x{\mathcal{R}}^{\left(m\right)}$. We take $N_{\mathrm{iter}}=15$, $s=0.25$ and $x_0={\Phi }^{\left(m\right)}{\mathbf{A}}^{⊺}y$ in Algorithm 1 and compute the inverse ${({\mathbf{A}}^{⊺}\mathbf{A}-sId)}^{-1}$ only once and then use it for all examples. We set $\alpha =0.015$ for the noise-free case, $\alpha =0.016$ for the low noise case and $\alpha =0.02$ for the high noise cases, respectively, and selected a fixed $\beta =15$. We expect that the NETT functional will yield better results due to data consistency, which is mainly helpful outside the masked center diagonal.

First we use the phantom from the testdata shown in Figure 1. The results using post processing and NETT are shown in Figure 2. One sees that all results with higher noise than used during training are not very good. This indicates that one should use similar noise as in the later applications even for the NETT. Figure 3 shows the average error using 10 test phantoms similar to the on in Figure 1. Careful numerical comparison of the numerical convergence rates and the theoretical results of Theorem 1 is an interesting aspect of further research. To investigate the stability of our method with respect to phantoms that are different from the training data we create a phantom with different structures as seen in Figure 4. As expected, the post processing network ${\Phi }^{\left(3\right)}$ is not really able to reconstruct the circles object, since it is quite different from the training data, but it also does not break down completely. On the other hand, the NETT approach yields good results due to data consistency.

Figure 2.

Top row: reconstructions using post-processing network ${\Phi }^{\left(1\right)}$. Middle row: NETT reconstructions using ${\mathcal{R}}^{\left(1\right)}$. Bottom row: NETT reconstructions using ${\mathcal{R}}^{\left(3\right)}$. From Left to Right: Reconstructions from data without noise, low noise ($\sigma =0.01$) and high noise ($\sigma =0.1)$.

Figure 3.

Semilogarithmic plot of the mean squared errors of the NETT using ${\mathcal{R}}^{\left(1\right)}$ and ${\mathcal{R}}^{\left(3\right)}$ depending on the noise level. The crosses are the values for the phantoms in Figure 2.

Figure 4.

Left column: phantom with a structure not contained in the training data (top) and pseudo inverse reconstruction (bottom). Middle column: Post-processing reconstructions with ${\Phi }^{\left(3\right)}$ using exact (top) and noisy data (bottom). Right column: NETT reconstructions with ${\mathcal{R}}^{\left(3\right)}$ using exact (top) and noisy data (bottom).

4. Conclusions

We have analyzed the convergence a discretized NETT approach and derived the convergence rates under certain assumptions on the approximation quality of the involved operators. We performed numerical experiments using a limited data problem for PAT that is the combination of an inverse problem for the wave equation and an inpainting problem. To the best of our knowledge this is the first such problem studied with deep learning. The NETT approach yields better results that post processing for phantoms different from the training data. NETT still fails to recover some missing parts of the phantom in cases the data contains more noise than the training data. This highlights the relevance of using different regularizers for different noise levels. Finding ways to make the regularizers less dependent on the noise level used during training is a possible future research direction. Another interesting question is if this results can be combined with with approximation error estimates for neural networks e.g., [37,38]. It seems not obvious how these two approaches can be combined. Furthermore, studying how one can define neural network based regularizers that fulfill (12) might also be an interesting line of future research.

Author Contributions

M.H. prosed the conceptualization, framework and the long term vision of the work. M.H and S.A. developed the ideas, performed the formal analysis and have written and edited the paper. S.A. conducted the numerical experiments and has written the software. All authors have read and agreed to the published version of the manuscript.

Funding

This work has been supported bay the Austrian Science Fund (FWF), project P 30747-N32.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data and code are freely available upon request.

Conflicts of Interest

The authors declare no conflict of interest.