Solving 2D Fredholm Integral from Incomplete Measurements Using Compressive Sensing

Abstract

We present an algorithm to solve the two-dimensional Fredholm integral of the first kind with tensor product structure from a limited number of measurements, with the goal of using this method to speed up nuclear magnetic resonance spectroscopy. This is done by incorporating compressive sensing–type arguments to fill in missing measurements, using a priori knowledge of the structure of the data. In the first step we recover a compressed data matrix from measurements that form a tight frame, and establish that these measurements satisfy the restricted isometry property. Recovery can be done from as few as 10% of the total measurements. In the second and third steps, we solve the zeroth-order regularization minimization problem using the Venkataramanan–Song–Hürlimann algorithm. We demonstrate the performance of this algorithm on simulated data and show that our approach is a realistic approach to speeding up the data acquisition.

1. Introduction

We present a method of solving the two-dimensional (2D) Fredholm integral of the first kind from a limited number of measurements. This is particularly useful in the field of nuclear magnetic resonance (NMR), in which making a sufficient number of measurements takes several hours. Our work is an extension of the algorithm in [56] based on the new idea of matrix completion; cf. [10, 29, 52].

A 2D Fredholm integral of the first kind is written as

$$g(x,y)=\int \int k_1(x,s)k_2(y,t)f(s,t)\mathit{dsdt},$$

where k₁ and k₂ are continuous Hilbert–Schmidt kernel functions and f, g ∈ L²(ℝ²); cf. [27]. 2D Fourier, Laplace, and Hankel transforms are all common examples of Fredholm integral equations. Applications of these transformations arise in any number of fields, including methods for solving PDEs [31], image deblurring [4, 38], and moment generating functions [40]. This paper specifically focuses on Laplace-type transforms, where the kernel singular values decay quickly to zero.

To present the main idea of the problem, the data M is measured over sampling times τ₁ and τ₂ and is related to the object of interest ℱ(x, y) by a 2D Fredholm integral of the first kind with a tensor product kernel,

$$M({\tau }_1,{\tau }_2)=\int \int k_1(x,{\tau }_1)k_2(y,{\tau }_2)\mathcal{F}(x,y)\mathit{dxdy}+\epsilon ({\tau }_1,{\tau }_2),$$

where ε(τ₁, τ₂) is assumed to be Gaussian white noise. In most applications, including NMR, the kernels k₁ and k₂ are explicit functions that are known to be smooth and continuous a priori. Solving a Fredholm integral with smooth kernels is an ill-conditioned problem, since the kernel’s singular values decay quickly to zero [34]. This makes the problem particularly interesting, as small variations in the data can lead to large fluctuations in the solution.

For our purposes, ℱ(x, y) represents the joint probability density function of the variables x and y. Specifically in NMR, x and y can be the measurements of the two combination of the longitudinal relaxation time T1, transverse relaxation time T2, diffusion D, and other dynamic properties. Knowledge of the correlation of these properties of a sample is used to identify its microstructure properties and dynamics [6].

This paper focuses on the discretized version of the 2D Fredholm integral,

$$M=K_{1}F{K}_2^{\prime }+E,$$ (1.1)

where our data is the matrix M ∈ ℝ^N₁×N₂, matrices K₁ ∈ ℝ^N₁×N_x and K₂ ∈ ℝ^N₂×N_y are discretized versions of the smooth kernels k₁ and k₂, and the matrix F ∈ ℝ^N_x×N_y is the discretized version of the probability density function ℱ(x, y) which we are interested in recovering. We also assume that each element of the Gaussian noise matrix E has zero mean and constant variance. And since we have assumed that ℱ(x, y) is a joint probability density function, each element of F is nonnegative.

Venkataramanan, Song, and Hürlimann [56] laid out an efficient strategy for solving this problem given complete knowledge of the data matrix M. The approach centers around finding an intelligent way to solve the Tikhonov regularization problem,

$$\widehat{F}=\underset{F\ge 0}{argmin}{\Vert M-K_{1}F{K}_2^{\prime }\Vert }_F^2+\alpha {\Vert F\Vert }_F^2,$$ (1.2)

where || · ||_F is the Frobenius norm.

There are three steps to the algorithm in [56] for solving (1.2).

1. Compress the data. Let the SVD of K_i be

   $$K_i=U_i{S}_i{V}_i^{\prime },i\in \{1,2\}.$$

   Because K₁ and K₂ are sampled from smooth functions k₁ and k₂, the singular values decay quickly to 0. Let s₁ be the number of nonzero singular values of K₁, and let s₂ be the number of nonzero singular values of K₂. Then U_i ∈ ℝ^N_i×s_i and S_i ∈ ℝ^s_i×s_i for i = 1, 2, as well as V₁ ∈ ℝ^N_x×s₁ and V₂ ∈ ℝ^N_y×s₂.

   The data matrix M can be projected onto the column space of K₁ and the row space of K₂ by $U_1{U}_1^{\prime }M{U}_2{U}_2^{\prime }$. We denote this as $\stackrel{\sim }{M}=U_1^{\prime }M{U}_2$. The Tikhonov regularization problem (1.2) is now rewritten as

   $$\widehat{F}=\underset{F\ge 0}{argmin}{\Vert U_1\stackrel{\sim }{M}{U}_2^{\prime }-U_1{U}_1^{\prime }{K}_{1}F{K}_2^{\prime }{U}_2{U}_2^{\prime }\Vert }_F^2+{\Vert M\Vert }_F^2-{\Vert U_1\stackrel{\sim }{M}{U}_2^{\prime }\Vert }_F^2+\alpha {\Vert F\Vert }_F^2$$ (1.3)

   $$=\underset{F\ge 0}{argmin}{\Vert \stackrel{\sim }{M}-(S_1{V}_1^{\prime })F{(S_2{V}_2^{\prime })}^{\prime }\Vert }_F^2+\alpha {\Vert F\Vert }_F^2,$$ (1.4)

   where (1.4) comes from U₁ and U₂ having orthogonal columns, and the second and third terms in (1.3) being independent of F. The key note here is that M̃ ∈ ℝ^s₁×s₂, which significantly reduces the complexity of the computations.

2. Optimization. For a given value of α, (1.4) has a unique solution due to the second term being quadratic. We shall detail the method of finding this solution in section 4.

3. Choosing α. Once (1.4) has been solved for a specific α, an update for α is chosen based on the characteristics of the solution in step 2. Repeat steps 2 and 3 until convergence. Again, this is detailed in section 4.

The approach in [56] assumes complete knowledge of the data matrix M. However, in applications with NMR, there is a cost associated with collecting all the elements of M, namely, time. With the microstructure-related information contained in the multidimensional diffusion-relaxation correlation spectrum of the biological sample [49, 22, 25, 20, 35, 54] and high-resolution spatial information that magnetic resonance imaging (MRI) techniques can provide, there is a need to combine the multidimensional correlation spectra NMR with 2D/3D MRI for preclinical and clinical applications [21]. Without any acceleration, however, it could take several days to acquire this data.

In practice, the potential pulse sequences for the combined multidimensional diffusion-relaxation MRI would be single spin echo (90°–180° acquisition and spatial localization) with saturation, inversion recovery, driven-equilibrium preparation to measure T1-T2 correlation, and diffusion weighting preparation for D-T2 measurements. With these MRI pulse sequences, a single point in the 2D T1-T2 or D-T2 space is acquired for each “shot,” and the total time for the sampling of the T1-T2 or D-T2 space is determined directly by the number of measurements required to recover F from (1.2). A vastly reduced number of sample points in M, together with rapid MRI acquisition techniques, which can include, e.g., parallel imaging [50], echo planar imaging (EPI) [24], gradient-recalled echo [36], and sparse sampling with compressed sensing [45], could reduce the total experiment time sufficiently to make this promising technique practicable for preclinical and clinical in vivo studies.

Notice that, despite collecting all N₁ × N₂ data points in M, step 1 of the algorithm immediately throws away a large amount of that information, reducing the number of data points to a matrix of size s₁ × s₂. M̃ is effectively a compressed version of the original M, containing the same information in a smaller number of entries. But this raises the question of why all of M must be collected when a large amount of information is immediately thrown away, since we are interested only in M̃.

This question is what motivates the introduction of a compressive sensing–type approach. The task is to undersample signals that are “compressible,” meaning that the signal is sparse in some basis representation [12, 11, 23]. The problem of recovering M falls into a subset of this field known as low-rank matrix completion; see [10, 29, 14].

An n × n matrix X that is rank r requires approximately nr parameters to be completely specified. If r ≪ n, then X is seen as being compressible, as the number of parameters needed to specify it is much less than its n² entries. It is less clear how to recover X from a limited number of coefficients efficiently. But the results of [10] showed that it is possible to recover X from, up to a constant, nr log(n) measurements by employing a simple optimization problem. These findings were inspired, at least in part, by [12, 11]. Also, the types of measurements we utilize in this paper, operator bases with bounded norm, originated via quantum state tomography [30].

Compressive sensing has been used in various forms of medical imaging for several years. The authors of [45] originally proposed speeding up MRI acquisition. The authors of [47] introduced group sparsity into consideration for accelerating T2-weighted MR. Finally, the authors of [55, 39] both introduced basic ideas of compressive sensing into the NMR framework and attained promising results, but the results did not utilize the matrix completion aspect of the physical problem and did not introduce any theoretical guarantees for reconstruction.

This paper develops an alternative to [56] which incorporates matrix completion in order to recover M̃ from significantly fewer measurements. It is organized as follows. Section 2 examines how recovery of M̃ fits into existing theory and shows that data from the 2D Fredholm integral can be recovered from 10% of the measurements. Section 3 covers the practical considerations of the problem and discusses the error created by our reconstruction. Section 4 covers the algorithms used to solve the low-rank minimization problem and invert the 2D Fredholm integral to obtain F. Section 5 shows the effectiveness of this reconstruction on simulated data. Appendix A contains a detailed proof of the central theorem of this paper.

2. Data recovery using matrix completion

2.1. Background for matrix completion

The problem of matrix completion has been in the center of scientific interest and activity in recent years [10, 26, 29, 51, 5, 9, 13, 7]. The basic problem revolves around trying to recover a matrix X₀ ∈ ℝ^n₁×n₂ from only a fraction of the N₁ × N₂ measurements required to observe each element of M. Without any additional assumptions, this is an ill-posed problem. However, there have been a number of attempts to add natural assumptions to make this problem well-posed. Other than assuming that X₀ is low rank, as we mentioned in section 1, there are assumptions that X₀ is positive definite [28, 42], or that X₀ is a distance matrix [2], or that X₀ has a nonnegative factorization [57]. A survey of some of these other methods can be found in [37].

For our purposes, we shall focus on low-rank matrix completion, as that is the most natural. Let X₀ be rank r. Consider a linear operator 𝒜: ℝ^n₁×n₂ → ℝ^m. Then our observations take the form

$$y=\mathcal{A}(X_0)+z,{\Vert z\Vert }_2\le \epsilon ,$$ (2.1)

where z represents a noise vector that is typically white noise, though not necessarily.

The naive way to proceed would be to solve the nonlinear optimization problem

$$\begin{array}{ll}min\hfill & \text{rank}(Z)\hfill \\ \text{such}\text{that}\hfill & {\Vert \mathcal{A}(Z)-y\Vert }_2\le \epsilon .\hfill \end{array}$$ (2.2)

However, the objective function rank(Z) makes the problem NP-hard. So instead we define the convex envelope of the rank function.

Definition 2.1

Let σ_i(X) be the ith singular value of a rank r matrix X. Then the nuclear norm of X is

$${\Vert X\Vert }_{\ast }:=\sum _{i=1}^r{\sigma }_i(X).$$

We now proceed by attempting to solve the convex relaxation of (2.2),

$$\begin{array}{ll}min\hfill & {\Vert Z\Vert }_{\ast }\hfill \\ \text{such}\text{that}\hfill & {\Vert \mathcal{A}(Z)-y\Vert }_2\le \epsilon .\hfill \end{array}$$ (2.3)

As with traditional compressive sensing, there exists a restricted isometry property (RIP) over the set of matrices of rank r.

Definition 2.2

A linear operator 𝒜: ℝ^n₁×n₂ → ℝ^m satisfies the RIP of rank r with isometry constant δ_r if, for all rank r matrices X,

$$(1-{\delta }_r){\Vert X\Vert }_F\le {\Vert \mathcal{A}(X)\Vert }_2\le (1+{\delta }_r){\Vert X\Vert }_F.$$

The RIP has been shown to be a sufficient condition for solving (2.3) [52, 8, 26]. These papers build on each other to establish the following theorem.

Theorem 2.3

Let X₀ be an arbitrary matrix in ℂ^m×n. Assume that δ_5r < 1/10. Then the X̂ obtained from solving (2.3) obeys

$${\Vert \widehat{X}-X_0\Vert }_F\le C_0\frac{{\Vert X_0-X_{0,r}\Vert }_{\ast }}{\sqrt{r}}+C_1\epsilon ,$$ (2.4)

where X_0,r is the best r rank approximation to X₀, and C₀,C₁ are small constants depending only on the isometry constant.

This means that, if the measurement operator 𝒜 satisfies RIP, then reconstruction via convex optimization behaves stably in the presence of noise. This result is very important in the context of the 2D Fredholm problem, as inversion of the Fredholm integral is very sensitive to noise. The bound in (2.4) guarantees that our reconstructed data behaves stably and will not create excess noise that would cause issues in the inversion process.

2.2. Matrix completion applied to NMR

For the NMR problem, let us say that

$$\begin{array}{l}M=K_{1}F{K}_2^{\prime }+E\\ =U_1{\stackrel{\sim }{M}}_0{U}_2^{\prime }+E,\end{array}$$ (2.5)

where U_i ∈ ℝ^N_i×s_i, M̃ ∈ ℝ^s₁×s₂, and E ∈ ℝ^N₁×N₂. This means that

$${\stackrel{\sim }{M}}_0=S_1{V}_1^{\prime }F{V}_2{S}_2.$$ (2.6)

To subsample the data matrix M, we shall observe it on random entries. Let Ω ⊂ {1, …, N₁} × {1, …, N₂} be the set of indices where we observe M. For |Ω| = m, let the indices be ordered as $\mathrm{\Omega }={\{(i_k,j_k)\}}_{k=1}^m$. Then we define the masking operator 𝒜_Ω as

$$\begin{array}{r}{\mathcal{A}}_{\mathrm{\Omega }}:{ℝ}^{N_1\times N_2}\to {ℝ}^m,\\ {({\mathcal{A}}_{\mathrm{\Omega }}(X))}_k=X_{i_k,j_k}.\end{array}$$

Recall that the goal is to recover M̃₀. This means that our actual sampling operator is

$$\begin{array}{r}{\mathcal{R}}_{\mathrm{\Omega }}:{ℝ}^{s_1\times s_2}\to {ℝ}^m,\\ {\mathcal{R}}_{\mathrm{\Omega }}(X)={\mathcal{A}}_{\mathrm{\Omega }}(U_{1}X{U}_2^{\prime }).\end{array}$$

Now the problem of speeding up NMR can be written as an attempt to recover M̃₀ from measurements

$$y={\mathcal{R}}_{\mathrm{\Omega }}({\stackrel{\sim }{M}}_0)+e,{\Vert e\Vert }_2\le \epsilon .$$ (2.7)

Note that [56] is assuming Ω = {1, …, N₁} × {1, …, N₂}, making the sampling operator ${\mathcal{R}}_{\mathrm{\Omega }}(\stackrel{\sim }{M})=U_1\stackrel{\sim }{M}{U}_2^{\prime }$.

Then in the notation of this NMR problem, our recovery step takes the form

$$\begin{array}{ll}min\hfill & {\Vert Z\Vert }_{\ast }\hfill \\ \text{such}\text{that}\hfill & {\Vert {\mathcal{R}}_{\mathrm{\Omega }}(Z)-y\Vert }_2\le \epsilon .\hfill \end{array}$$ (2.8)

Now the key question becomes whether ℛ_Ω satisfies the RIP. As we said in section 2.1, the RIP is a sufficient condition for an operator to satisfy the noise bounds of Theorem 2.3. Without this, there is no guarantee that solving (2.8) yields an accurate prediction of M̃. For this reason, the rest of this section shall focus on proving that ℛ_Ω is an RIP operator.

First, we must define the notion of a Parseval tight frame.

Definition 2.4

A Parseval tight frame for a d-dimensional Hilbert space ℋ is a collection of elements {ϕ_j}_j∈J ⊂ ℋ for an index set J such that

$$\sum _{j\in J}{\mid \langle f,{\varphi }_j\rangle \mid }^2={\Vert f\Vert }^2\forall f\in H.$$

This automatically forces |J| ≥ d.

This definition is very closely related to the idea of an orthonormal basis. In fact, if |J| = d, then {ϕ_j}_j∈J would be an orthonormal basis. This definition can be thought of as a generalization. Frames have the benefit of giving overcomplete representations of the function f, making them much more robust to errors and erasures than orthonormal bases [17, 41, 15]. This redundancy is exactly what we will be taking advantage of in Theorem 2.6.

Further, we introduce a definition used in [29].

Definition 2.5

A bounded norm Parseval tight frame with incoherence μ is a Parseval tight frame {ϕ_j}_j∈J on ℂ^d×d that also satisfies

$${\Vert {\varphi }_j\Vert }^2\le \mu \frac{d}{\mid J\mid }\forall j\in J.$$ (2.9)

The paper [29] defines this type of bound on an orthonormal basis. Note that, in the case of {ϕ_j}_j∈J being an orthonormal basis, |J| = d², reducing the bound in (2.9) to ||ϕ_j||² ≤ μ/d, as in the case of [29].

Now notice that in our problem, ignoring noise, each observation can be written as

$$\begin{array}{l}{M}_{j,k}=(u_1^j){\stackrel{\sim }{M}}_0{(u_2^k)}^{\prime }\\ =\langle {(u_1^j)}^{\prime }(u_2^k),{\stackrel{\sim }{M}}_0\rangle ,\end{array}$$

where $u_1^j$ (resp., $u_2^j$) is the jth row of U₁ (resp., U₂). Noting that U₁ and U₂ are left orthogonal (i.e., $U_i^{\prime }{U}_i=I{d}_{s_i}$), one can immediately show that ${\{{(u_1^j)}^{\prime }(u_2^k)\}}_{(j,k)\in {\mathbb{Z}}_{N_1}\times {\mathbb{Z}}_{N_2}}$ forms a Parseval tight frame for ℝ^s₁×s₂. Also, because K₁ and K₂ are discretized versions of smooth continuous functions, { ${(u_1^i)}^{\prime }(u_2^j)$} are a bounded norm frame for a reasonable constant μ (see further discussion of μ in section 3.2). Thus, ℛ_Ω is generated by randomly selecting measurements from a bounded norm Parseval tight frame.

We now have the necessary notation to state our central theorem, which establishes bounds on the quality of reconstruction from (2.8) in the presence of noise. The theorem and proof rely on a generalization of [44], which only assumes the measurements to be orthonormal basis elements.

It is interesting to note that, because our measurements are overcomplete (|J| > s₁s₂), our system of equations is not necessarily underdetermined. However, Theorem 2.6 still gives guarantees on how the reconstruction scales with the noise, regardless of this detail. This is a difference from most compressive sensing literature. Generally the goal is to show that an underdetermined system has a stable solution. In our case we are showing that, regardless of whether or not the system is underdetermined, our reconstruction is stable in the presence of noise and the reconstruction error decreases monotonically with the number of measurements.

Theorem 2.6

Let {ϕ_j}_j∈J ⊂ ℂ^s₁×s₂ be a bounded norm Parseval tight frame, with incoherence parameter μ. Let n = max(s₁, s₂), and let the number of measurements m satisfy

$$m\ge C\mu rn{log}^{5}n·log\mid J\mid ,$$

where C is a constant. Let the sampling operator ℛ_Ω be defined for Ω ⊂ J, with Ω = {i₁, …, i_m}, as

$$\begin{array}{c}{\mathcal{R}}_{\mathrm{\Omega }}:{ℂ}^{s_1\times s_2}\to {ℂ}^m,\\ {({\mathcal{R}}_{\mathrm{\Omega }}(X))}_j=\langle {\varphi }_{i_j},X\rangle ,j=1,\dots ,m.\end{array}$$

Let measurements y satisfy (2.7). Then, with probability greater than 1−e^−Cδ2 over the choice of Ω, the solution M̃ to (2.8) satisfies

$${\Vert \stackrel{\sim }{M}-{\stackrel{\sim }{M}}_0\Vert }_F\le C_0\frac{{\Vert {\stackrel{\sim }{M}}_0-{\stackrel{\sim }{M}}_{0,r}\Vert }_{\ast }}{\sqrt{r}}+C_1{p}^{-1/2}\epsilon ,$$ (2.10)

where $p=\frac{m}{\mid J\mid }$.

To prove this result, we need a key lemma, which establishes that our measurements satisfy the RIP.

Lemma 2.7

Let {ϕ_j}_j∈J ⊂ ℂ^s₁×s₂ be a bounded norm Parseval tight frame, with incoherence parameter μ. Fix some 0 < δ < 1. Let n = max(s₁, s₂), and let the number of measurements m satisfy

$$m\ge C\mu rn{log}^{5}n·log\mid J\mid ,$$ (2.11)

where C ∝ 1/δ². Let the sampling operator ℛ_Ω be defined for Ω ⊂ J, with Ω = {i₁, …, i_m}, as

$$\begin{array}{r}{\mathcal{R}}_{\mathrm{\Omega }}:{ℂ}^{s_1\times s_2}\to {ℂ}^m,\\ {({\mathcal{R}}_{\mathrm{\Omega }}(X))}_j=\langle {\varphi }_{i_j},X\rangle ,j=1,\dots ,m.\end{array}$$

Then, with probability greater than 1 − e^−Cδ2 over the choice of Ω, $\sqrt{\frac{\mid J\mid }{m}}{\mathcal{R}}_{\mathrm{\Omega }}$ satisfies the RIP of rank r with isometry constant δ.

The proof of this lemma is found in Appendix A and follows [44], where the claim is proved for an orthonormal basis. The main point here is to generalize the measurements to a bounded norm Parseval tight frame (also mentioned in [48], however, not considering when m > n²).

Proof of Theorem 2.6

We assume that Lemma 2.7 is true. Lemma 2.7 states that $\sqrt{\frac{\mid J\mid }{m}}{\mathcal{R}}_{\mathrm{\Omega }}$ satisfies the RIP. However, (2.8) is stated using only ℛ_Ω as the measurement operator.

This means we must include a scaling factor of $\sqrt{\frac{\mid J\mid }{m}}$ to understand the noise bound. Let $p=\frac{m}{\mid J\mid }=\frac{m}{N_1{N}_2}$ be the percentage of elements observed. Then, to utilize the RIP, we must try to solve the problem

$$\begin{array}{ll}min\hfill & {\Vert Z\Vert }_{\ast }\hfill \\ \text{such}\text{that}\hfill & {\Vert p^{-1/2}{\mathcal{R}}_{\mathrm{\Omega }}(Z)-p^{-1/2}y\Vert }_2\le p^{-1/2}\epsilon .\hfill \end{array}$$ (2.12)

While scaling by a constant does not affect the result of the minimization problem, it does help us better understand the error in our reconstruction.

Theorem 2.3 tells us that our reconstruction error is bounded by a constant multiple of the error bound. But (2.12) means we can rewrite the error bound as

$${\Vert \stackrel{\sim }{M}-{\stackrel{\sim }{M}}_0\Vert }_F\le C_0\frac{{\Vert {\stackrel{\sim }{M}}_0-{\stackrel{\sim }{M}}_{0,r}\Vert }_{\ast }}{\sqrt{r}}+C_1{p}^{-1/2}\epsilon ,$$

thus attaining the desired inequality.

Remark 1

Examination of the proof of Lemma 2.7 shows that the bound on m in (2.11) is actually not sharp. If one refers to (A.1) in Appendix A, m is actually bounded below by a factor of logm. In (A.2) we simply overestimate this term with log |J| for simplicity. However, in reality the bound is

$$m\ge C\lambda \mu rn{log}^{5}n·logm.$$

Let N = Cλμrn log⁵ n. This would give the bound m ≥ e^{−W₋₁(−1/N)}, where W₋₁ is the lower branch of the Lambert W function [19]. Taking the first three terms of a series approximation of W₋₁ in terms of log(1/N) and log(log(N)) [18] gives us

$$\begin{array}{l}m\ge e^{-log(1/N)}{e}^{log(log(N))}{e}^{-\frac{log(log(N))}{log(1/N)}}\\ =Nlog(N)e^{-\frac{log(log(N))}{log(1/N)}}\\ =C\lambda \mu rn{log}^{5}n·log(C\lambda \mu rn{log}^{5}n)·e^{\frac{log(log(C\lambda \mu rn{log}^{5}n))}{log(C\lambda \mu rn{log}^{5}n)}}.\end{array}$$ (2.13)

Note that taking three terms is sufficient as each subsequent term is asymptotically small compared to the previous. The bound in (2.13) is clearly much more intricate than simply bounding by m ≥ Cλμrn log⁵ n log |J|, but for typical sizes of |J| in the Fredholm integral setting, this results in m decreasing by less than 5% from its original size.

3. Numerical considerations

Section 2 gives theoretical guarantees about the error of estimating M̃₀ with the recovered M̃. We shall address several issues related to practical applications in this section. We shall let M̃₀ be the original compressed data matrix we are hoping to recover, and let M̃ be the approximation obtained by solving (2.8) for the sampling operator ℛ_Ω. We consider the guarantee given in (2.4) term by term.

For the rest of this paper, we take the kernels K₁ and K₂ to be Laplace-type kernels with quickly decaying singular values. For our purposes, we shall use the kernels k₁(τ₁, x) = 1 − e^−τ₁/x and k₂(τ₂, y) = e^−τ₂/y to represent the general data structure of most multiexponential NMR spectroscopy measurements. The same kernels shall be used in section 5 for simulations and experiments. Also, τ₁ is logarithmically sampled between 0.0005 and 4, and τ₂ is linearly sampled between 0.0002 and 0.4, as these are typical values in practice. Also for this section, F is taken to be a two-peak distribution, namely Model 3 from section 5.

When needed, we set s₁ = s₂ = 20. This choice is determined by the discrete Picard condition (DPC) [33]. For ill-conditioned kernel problems Kf = g, with {u_i} denoting left singular vectors of K and {σ_i} the corresponding singular values, the DPC guarantees that the best reconstruction of f is given by keeping all σ_i ≠ 0 such that $\frac{\mid u_1^{\ast }g\mid }{{\sigma }_i}$ on average decays to zero as σ_i decrease. For our kernels with tensor product structure in (1.1), Figure 1 shows the relevant singular values and vectors to keep. The s₁ = s₂ = 20 rectangle provides a close estimate for what fits inside this curve, implying that at a minimum we could set s₁ = s₂ = 20 to satisfy the DPC. The DPC provides a stronger condition than simply keeping the largest singular values or attempting to preserve some large percentage of the energy [32].

Figure 1.

Points denote which singular values of K₁ (rows of plot) and K₂ (columns of plot) to keep in order to satisfy the DPC for stable inversion.

3.1. Noise bound in practice

Theorem 2.3 hinges on the assumption that δ_5r < 1/10, where δ_r is the isometry constant for rank r. This puts a constraint on the maximum size of r. Let us denote that maximal rank by r₀. If we knew a priori that M̃₀ was at most rank r₀, then this term of $\frac{{\Vert {\stackrel{\sim }{M}}_0-{\stackrel{\sim }{M}}_{0,r}\Vert }_{\ast }}{\sqrt{r}}$ would have zero contribution, as M̃₀ = M̃_0,r. However, because of (2.6), M̃₀ could theoretically be full rank, since S₁ and S₂ are decaying but not necessarily 0.

This problem is rectified by utilizing the knowledge that K₁ and K₂ have rapidly decaying singular values. Figure 2 shows just how rapidly the singular values decay for a typical choice of kernels and discretization points. This means M̃₀ from (2.6) must have even more rapidly decaying singular values, as $V_1^{\prime }F{V}_2$ is multiplied by both S₁ and S₂. Figure 3 shows that the singular values of M̃₀ drop to zero almost immediately for a typical compressed data matrix.

Figure 2.

Plots of the singular value decay of the kernels. Left: K₁. Right: K₂.

Figure 3.

Plot of the singular value decay for data matrix M.

This means that even for small r₀, $\frac{{\Vert {\stackrel{\sim }{M}}_0-{\stackrel{\sim }{M}}_{0,r}\Vert }_{\ast }}{\sqrt{r}}\le \Vert {\sum }_{i=r_0+1}^{min(s_1,s_2)}{\sigma }_i({\stackrel{\sim }{M}}_0)\Vert$ is very close to zero, as the tail singular values of M̃₀ are almost exactly zero.

Figure 4 shows how the relative error decays for larger percentages of measurement, and how that curve matches the predicted curve of p^−1/2||e||₂. One can see from this curve that the rank r error does not play any significant role in the reconstruction error.

Figure 4.

Plot of the error in reconstruction.

3.2. Incoherence

The incoherence parameter μ to bound the number of measurements in (2.11) plays a vital role in determining m in practice. It determines whether the measurements { $u_i^{\prime }{v}_j$} are viable for reconstruction from significantly reduced m, even though they form a Parseval tight frame.

To show that μ does not make reconstruction prohibitive, we demonstrate on a typical example of K₁ and K₂, as described at the beginning of this section.

Figure 5 shows the ${\Vert {\varphi }_j\Vert }^2\frac{\mid J\mid }{n}$ for each measurement { $u_i^{\prime }{v}_j$} from the above description, making $\mu =max{\Vert {\varphi }_j\Vert }^2\frac{\mid J\mid }{n}=89.9$. While this bound on μ is not ideal, as it makes m > n², there are two important points to consider. First, as was mentioned in section 2.2, Theorem2.3 guarantees strong error bounds regardless of the system being underdetermined. Second, as is shown in section 3.3, the estimate M̃ is still significantly better than a simple least squares minimization, which in theory applies as the system isn’t underdetermined.

Figure 5.

Plot of $\Vert u_i^{\prime }{v}_j\Vert \frac{\mid J\mid }{n}$ for each measurement element from the NMR problem.

Also note from Figure 5 the fact that $\text{mean}({\Vert {\varphi }_j\Vert }^2\frac{\mid J\mid }{n})$ and $\text{median}({\Vert {\varphi }_j\Vert }^2\frac{\mid J\mid }{n})$ differ greatly from $max({\Vert {\varphi }_j\Vert }^2\frac{\mid J\mid }{n})$. This implies that, while a small number of the entries are somewhat problematic and coherent with the elementary basis, the vast majority of terms are perfectly incoherent. This implies that Theorem 2.3 is a nonoptimal lower bound on m. Future work will be to examine the possibility of bounding m below with an average or median coherence, or considering a reweighted nuclear norm sampling similar to [16]. Another possibility is to examine the idea of asymptotic incoherence [1].

3.3. Least squares comparison

One could also attempt to solve for M̃₀ using a least squares algorithm on the observed measurements via the Moore–Penrose pseudoinverse. However, as we shall show, due to noise and ill-conditioning, this is not a viable alternative to the nuclear norm minimization algorithm employed throughout this paper. As an example, we shall again use K₁ and K₂ as described in the beginning of this section. The noise shall range over various signal-to-noise ratios (SNRs).

We will consider a noisy estimate M̃ of the compressed matrix M̃₀, generated through the pseudoinverse, nuclear norm minimization, or simply the projection of a full set of measurements M via $U_1^{\prime }M{U}_2$. Figure 6 shows the relative error of each of these recoveries, defining error to be

Figure 6.

Relative error of least squares approximation compared to nuclear norm minimization versus percentage of measurements kept. Left: SNR = 15dB. Center: SNR = 25dB. Right: SNR = 35dB.

$$\frac{{\Vert {\stackrel{\sim }{M}}_0-\stackrel{\sim }{M}\Vert }_F}{{\Vert {\stackrel{\sim }{M}}_0\Vert }_F}.$$

Clearly, nuclear norm minimization, even for a small fraction of measurements kept, mirrors the full measurement compression almost perfectly, as was shown in Figure 4. However, the least squares minimization error is drastically higher. Even at 20% measurements kept, the difference in error between least squares reconstruction and the full measurement projection error is 4 times higher than the difference between nuclear norm reconstruction and the full measurement projection error.

4. Algorithm

The algorithm for solving for F in (1.1) from partial data consists of three steps. An overview of the original algorithm in [56] is in section 1. Our modification and the specifics of each step are detailed below.

1. Construct M̃ from given measurements. Let y = ℛ_Ω(M̃₀) + e be the set of observed measurements, as in (2.7). Even though section 2 makes guarantees for solving (2.8), we can instead solve the relaxed Lagrangian form

   $$min\mu {\Vert X\Vert }_{\ast }+\frac{1}{2}{\Vert {\mathcal{R}}_{\mathrm{\Omega }}(X)-y\Vert }_2^2.$$ (4.1)
   To solve (4.1), we use the singular value thresholding algorithm from [5, 46]. To do this, we need some notation. Let the matrix derivative of the L₂ norm term be written as

   $$\begin{array}{l}g(X)={\mathcal{R}}_{\mathrm{\Omega }}^{\ast }({\mathcal{R}}_{\mathrm{\Omega }}(X)-y)\\ =U_1^{\prime }({\mathcal{A}}_{\mathrm{\Omega }}^{\ast }({\mathcal{A}}_{\mathrm{\Omega }}(U_{1}X{U}_2^{\prime })-y))U_2.\end{array}$$
   We also need the singular value thresholding operator 𝒮_ν that reduces each singular value of some matrix X by ν. In other words, if the SVD of X = UΣV′, then

   $${\mathcal{S}}_{\nu }(X)=U\mathrm{\sum }^{\sim }{V}^{\prime },{\mathrm{\sum }^{\sim }}_{i,j}=\{\begin{array}{ll}max({\mathrm{\sum }}_{i,i}-\nu ,0),\hfill & \hfill i=j,\\ 0\hfill & \hfill \text{otherwise.}\end{array}$$
   Using this notation, the algorithm can then be written as a simple, two-step iterative process. Choose a τ > 0. Then, for any initial condition, solve the iterative process

   $$\{\begin{array}{l}{Y}^k=X^k-\tau g(X^k),\hfill \\ X^{k+1}={\mathcal{S}}_{\tau \mu }(Y^k).\hfill \end{array}$$ (4.2)

   The choices of τ and μ are detailed in [46], along with adaptations of this method that speed up convergence. However, this method is guaranteed to converge to the correct solution.

   This means that, given partial observations y, the iteration scheme in (4.2) converges to a matrix M̃, which is a good approximation of M̃ +0. Once M̃ has been generated, we recover F by solving

   $$\underset{F\ge 0}{argmin}{\Vert \stackrel{\sim }{M}-(S_1{V}_1^{\prime })F{(S_2{V}_2^{\prime })}^{\prime }\Vert }_F^2+\alpha {\Vert F\Vert }_F^2.$$ (4.3)

2. Optimization. For a given value of α, (4.3) has a unique solution due to the second term being quadratic. This constrained optimization problem is then mapped onto an unconstrained optimization problem for estimating a vector c.

   Let f be the vectorized version of F and m be a vectorized version of M̃. Then we define the vector c from f implicitly by

   $$f=max(0,K^{\prime }c),\text{where}K=(S_1{V}_1^{\prime })\otimes (S_2{V}_2^{\prime }).$$
   Here, ⊗ denotes the Kronecker product of two matrices. This definition of c comes from the constraint that F ≥ 0 in (4.3), which can now be reformed as the unconstrained minimization problem

   $$min\left(\frac{1}{2}{c}^{\prime }[G(c)+\alpha I]c-c^{\prime }m\right),$$ (4.4)
   where

   $$G(c)=K\left[\begin{array}{cccc}H(K_{1,·}^{\prime }c)& 0& \cdots & 0\\ 0& H(K_{2,·}^{\prime }c)& \cdots & 0\\ ⋮& ⋮& & ⋮\\ 0& 0& \cdots & H(K_{N_x\times N_y,·}^{\prime }c)\end{array}\right]K^{\prime }$$

   and H(x) is the Heaviside function. Also, K_i,· denotes the ith row of K. The optimization problem (4.4) is solved using a simple gradient descent algorithm.

3. Choosing α. There are several methods for choosing the optimal value of α.

   • Butler–Reeds–Dawson (BRD) method. Once an iteration of step 2 has been completed, it is shown in [56] that a better value of α can be calculated by

     $${\alpha }_{\mathit{opt}}=\frac{\sqrt{s_1{s}_2}}{\Vert c\Vert }.$$

     If one iterates between step 2 and the BRD method, the value of α converges to an optimal value. This method is very fast; however, it can have convergence issues in the presence of large amounts of noise, as well as on real data [53].

   • S-curve. Let F_α be the value returned from step 2 for a fixed α. The choice of α should be large enough that F_α is not being overfitted and unstable to noise, yet small enough that F_α actually matches reality. This is done by examining the “fit error”

     $$\chi (\alpha )={\Vert M-K_1{F}_{\alpha }{K}_2^{\prime }\Vert }_F.$$

     This is effectively calculating the standard deviation of the resulting reconstruction. Plotting χ(α) for various values of α generates an S-curve, as shown in Figure 7. The interesting value of α occurs at the bottom “heel” of the curve (i.e., $\frac{dlog\chi (\alpha )}{dlog\alpha }\approx .1$). This is because, at α_heel, the fit error is no longer demonstrating overfitting as it is to the left of α_heel, yet is still matching the original data, unlike to the right of α_heel. This method is slower than the BRD method; however, it is usually more stable in the presence of noise.

Figure 7.

Plot of the fit error for various α.

For the rest of this paper, we use the S-curve method of choosing α.

5. Simulation results

In our simulations, we shall use the kernels k₁(τ₁, x) = 1 − e^−τ₁/x and k₂(τ₂, y) = e^−τ₂/y and sample τ₁ logarithmically and τ₂ linearly, as was done in section 3. Our simulations revolve around inverting subsampled simulated data to recover the density function F(x, y). We shall test three models of F(x, y). In Model 1, F(x, y) is a small variance Gaussian. In Model 2, F(x, y) is a positively correlated density function. In Model 3, F(x, y) is a two-peak density, one peak being a small circular Gaussian and the other being a ridge with positive correlation.

The data is generated for a model of F(x, y) by discretizing F and computing

$$M=K_{1}F{K}_2^{\prime }+E,$$

where E is Gaussian noise. That data is then randomly subsampled by keeping only a λ fraction of the entries.

Each true model density F(x, y) is sampled logarithmically in x and y. τ₁ is logarithmically sampled N₁ = 30 times, and τ₂ is linearly sampled N₂ = 4000 times. Each model is examined for various SNRs and values of λ, and α is chosen using the S-curve approach for each trial.

Let us also define the signal-to-noise ratio (SNR) for our data to be

$$\text{SNR}=10{log}_{10}\frac{{\Vert M\Vert }^2}{{\Vert E\Vert }^2}\text{dB}.$$

Note that [56] has extensively examined steps 2 and 3 of this algorithm, including the effects of α and the SNR on the reconstruction of F. Our examination focuses on the differences between the F generated from full knowledge of the data and the F generated from subsampled data. For this reason, F_full refers to the correlation spectrum generated from full knowledge of the data using the algorithm from [56]. F_λ refers to the correlation spectrum generated from only a λ fraction of the measurements using our algorithm.

5.1. Model 1

In this model, F(x, y) is a small variance Gaussian. This is the simplest example of a correlation spectrum, given that the dimensions are uncorrelated. F(x, y) is centered at (x, y) = (.1, .1) and has standard deviation .02. The maximum signal amplitude is normalized to 1. This model of F(x, y) is a base case for any algorithm. In other words, any legitimate algorithm for inverting the 2D Fredholm integral must at a minimum be successful in this case.

Figure 8 shows the quality of reconstruction of a simple spectrum with an SNR of 30dB. Figure 9 shows the same spectrum, but with an SNR of 15dB. Almost nothing is lost in either reconstruction, implying that both the original algorithm and our compressive sensing algorithm are very robust to noise for this simple spectrum.

Figure 8.

Model 1 with SNR of 30dB. Top left: True spectrum. Top right: F_full. Bottom left: Reconstruction from 30% measurements. Bottom right: Reconstruction from 10% measurements.

Figure 9.

Model 1 with SNR of 15dB. Top left: True spectrum. Top right: F_full. Bottom left: Reconstruction from 30% measurements. Bottom right: Reconstruction from 10% measurements.

5.2. Model 2

In this model, F(x, y) is a positively correlated density function. The spectrum has a positive correlation, thus creating a ridge through the space. F(x, y) is centered at (x, y) = (.1, .1), with the variance in the x + y direction being 7 times greater than the variance in the x − y direction. The maximum signal amplitude is normalized to 1. This is an example of a spectrum where it is essential to consider the 2D image. A projection onto one dimension would yield an incomplete understanding of the spectrum, as neither the T1 projection nor the T2 projection would convey that the ridge is very thin. This is a more practical test of our inversion algorithm.

Figure 10 shows the quality of reconstruction of a correlated spectrum with an SNR of 30dB. Figure 11 shows the same spectrum, but with an SNR of 20dB. There is slight degradation in the 10% reconstruction, but the reconstructed spectrum is still incredibly close to F_full. Overall, both of these figures show the quality of our compressive sensing reconstruction relative to using the full data.

Figure 10.

Model 2 with SNR of 30dB. Top left: True spectrum. Top right: F_full. Bottom left: Reconstruction from 30% measurements. Bottom right: Reconstruction from 10% measurements.

Figure 11.

Model 2 with SNR of 20dB. Top left: True spectrum. Top right: F_full. Bottom left: Reconstruction from 30% measurements. Bottom right: Reconstruction from 10% measurements.

5.3. Model 3

In this model, F(x, y) is a two-peak density, with one peak being a small circular Gaussian and the other being a ridge with positive correlation. The ridge is centered at (x, y) = (.1, .1), with the variance in the x + y direction being 7 times greater than the variance in the x − y direction. The circular part is centered at (x, y) = (.05, .4). The maximum signal amplitude is normalized to 1. This is an example of a common, complicated spectrum that occurs during experimentation.

Figure 12 shows the quality of reconstruction of a two-peak spectrum with an SNR of 35dB. In this instance, there is some degradation from F_full to any of the reconstructed data sets. Once again, there is slight degradation in the 10% model, but the compressive sensing reconstructions are still very close matches to F_full.

Figure 12.

Model 3 with SNR of 30dB. Top left: True spectrum. Top right: F_full. Bottom left: Reconstruction from 30% measurements. Bottom right: Reconstruction from 10% measurements.

6. Conclusion

In this paper, we introduce a matrix completion framework for solving 2D Fredholm integrals. This method allows us to invert the discretized transformation via Tikhonov regularization using far fewer measurements than previous algorithms. We proved that the nuclear norm minimization reconstruction of the measurements is stable and computationally efficient, and demonstrated that the resulting estimate of ℱ(x, y) is consistent with using the full set of measurements. This allows us in application to reduce the measurements conducted by a factor of 5 or more.

While the theoretical framework of this paper applies to 2D NMR spectroscopy, the approach is easily generalized to larger-dimensional measurements. This allows for accelerated acquisition of 3D correlation maps [3] that would otherwise take days to collect. This shall be a subject of forthcoming work.

Appendix A. Proof of Lemma 2.7

Let us define

$$U=\{X\in {ℂ}^{s_1\times s_2}\mid {\Vert X\Vert }_{\ast }\le \sqrt{r}{\Vert X\Vert }_F\}.$$

Note that the set of all rank r matrices in ℂ^s₁×s₂ is a subset of U by Hölder’s inequality. For the proof, we need some notation:

$$\begin{array}{r}{U}_2=\{X\in {ℂ}^{s_1\times s_2}\mid {\Vert X\Vert }_F\le 1,{\Vert X\Vert }_{\ast }\le \sqrt{r}{\Vert X\Vert }_F\},\\ {\epsilon }_r(\mathcal{A})=\underset{X\in U_2}{sup}\mid \langle X,({\mathcal{A}}^{\ast }\mathcal{A}-\mathcal{I})X\rangle \mid .\end{array}$$

The RIP can be rewritten as

$${(1-\delta )}^2\langle X,X\rangle \le \langle X,{\mathcal{A}}^{\ast }\mathcal{A}X\rangle \le {(1+\delta )}^2\langle X,X\rangle \forall X\in U,$$

which is implied by

$$\mid \langle X,({\mathcal{A}}^{\ast }\mathcal{A}-\mathcal{I})X\rangle \mid \le 2\delta -{\delta }^2\forall X\in U_2.$$

So we need to show that ε_r(𝒜) ≤ 2δ − δ² ≡ ε.

One can then define a norm on the set of all self-adjoint operators from ℂ^s₁×s₂ to ℂ^s₁×s₂ by

$${\Vert \mathcal{M}\Vert }_{(r)}=\underset{X\in U_2}{sup}\mid \langle X,\mathcal{M}X\rangle \mid .$$

The proof that this is a norm, and that the set of self-adjoint operators is a Banach space with respect to ||·||_(r), is found in [44].

We can now write ε_r(𝒜) = ||𝒜 * 𝒜 − ℐ||_(r). For our purposes, as with most compressive sensing proofs, we first bound 𝔼ε_r(𝒜) and then show that ε_r(𝒜) is concentrated around its mean.

For our problem of dealing with tight frame measurements, let ${\mathcal{A}}^{\ast }\mathcal{A}-\mathcal{I}={\sum }_{i=1}^m{\chi }_i$, where ${\chi }_i=\frac{\mid J\mid }{m}{\varphi }_i^{\ast }{\varphi }_i-\frac{\mathcal{I}}{m}$. Also, let ${\chi }_i^{\prime }$ be independent copies of the random variable χ_i. Finally, let ε_i be a random variable that takes values ±1 with equal probability. Then we have that

$$\begin{array}{l}{\mathbb{E}}_{\mathrm{\Omega }}{\epsilon }_r(\mathcal{A})={\mathbb{E}}_{\mathrm{\Omega }}{\Vert \sum {\chi }_i\Vert }_{(r)}\\ \le {\mathbb{E}}_{\mathrm{\Omega }}{\Vert \sum {\chi }_i-{\chi }_i^{\prime }\Vert }_{(r)}\\ ={\mathbb{E}}_{\mathrm{\Omega }}{\mathbb{E}}_{\epsilon }{\Vert \sum {\epsilon }_i({\chi }_i-{\chi }_i^{\prime })\Vert }_{(r)}\\ ={\mathbb{E}}_{\mathrm{\Omega }}{\mathbb{E}}_{\epsilon }{\Vert \sum {\epsilon }_i({\varphi }_i^{\ast }{\varphi }_i-{({\varphi }_i^{\prime })}^{\ast }{\varphi }_i^{\prime })\frac{\mid J\mid }{m}\Vert }_{(r)}\\ \le 2\frac{n}{m}{\mathbb{E}}_{\mathrm{\Omega }}{\mathbb{E}}_{\epsilon }{\Vert \sum {\epsilon }_i\sqrt{\frac{\mid J\mid }{n}}{\varphi }_i^{\ast }{\varphi }_i\sqrt{\frac{\mid J\mid }{n}}\Vert }_{(r)}.\end{array}$$

Now we cite Lemma 3.1 of [44], which is general enough to remain unchanged in the case of tight frames.

Lemma A.1

Let ${\{V_i\}}_{i=1}^m\subset {ℂ}^{s_1\times s_2}$ have a uniformly bounded norm, ||V_i|| ≤ K. Let n = max(s₁, s₂), and let ${\{{\epsilon }_i\}}_{i=1}^m$ be independent and identically distributed uniform ±1 random variables. Then

$${\mathbb{E}}_{\epsilon }{\Vert \sum _{i=1}^m{\epsilon }_i{V}_i^{\ast }{V}_i\Vert }_{(r)}\le C_1{\Vert \sum _{i=1}^m{V}_i^{\ast }{V}_i\Vert }_{(r)}^{1/2},$$

where $C_1=C_0\sqrt{r}K{log}^{5/2}n{log}^{1/2}m$ and C₀ is a universal constant.

For our purposes, $V_i=\sqrt{\frac{\mid J\mid }{n}}{\varphi }_i$. Then

$$\begin{array}{l}\mathbb{E}{\epsilon }_r(\mathcal{A})\le 2C_1\frac{n}{m}{\mathbb{E}}_{\mathrm{\Omega }}{\Vert \sum \sqrt{\frac{\mid J\mid }{n}}{\varphi }_i^{\ast }{\varphi }_i\sqrt{\frac{\mid J\mid }{n}}\Vert }_{(r)}^{1/2}\\ \le 2C_1\frac{n}{m}{\left({\mathbb{E}}_{\mathrm{\Omega }}{\Vert \sum \sqrt{\frac{\mid J\mid }{n}}{\varphi }_i^{\ast }{\varphi }_i\sqrt{\frac{\mid J\mid }{n}}\Vert }_{(r)}\right)}^{1/2}\\ =2C_1\sqrt{\frac{n}{m}}{(\mathbb{E}\Vert {\mathcal{A}}^{\ast }\mathcal{A}\Vert )}^{1/2}\\ \le 2C_1\sqrt{\frac{n}{m}}{(\mathbb{E}{\epsilon }_r(\mathcal{A})+1)}^{1/2}.\end{array}$$

Here,

$$C_1=C_0\sqrt{r}\sqrt{\mu }{log}^{5/2}n·{log}^{1/2}m.$$ (A.1)

If we take E₀ = 𝔼ε_r(𝒜) and $C=2C_1\sqrt{\frac{n}{m}}$, then (A.1) gives us

$$E_0^2-C^2{E}_0-C^2\le 0.$$

Fix some λ ≥ 1, and choose

$$\begin{array}{l}m\ge C\lambda \mu rn{log}^{5}n·log\mid J\mid \\ \ge \lambda n{(2C_1)}^2.\end{array}$$ (A.2)

This makes $\mathbb{E}{\epsilon }_r(\mathcal{A})\le \frac{1}{\lambda }+\frac{1}{\sqrt{\lambda }}$.

The next step is to show that ε_r(𝒜) does not deviate far from 𝔼ε_r(𝒜). Let 𝒜 * 𝒜 − ℐ = χ be a random variable and χ′ be an independent copy of χ. We now note that

$$Pr({\Vert \chi \Vert }_{(r)}>2\mathbb{E}{\epsilon }_r(\mathcal{A})+u)\le 2Pr({\Vert \chi -{\chi }^{\prime }\Vert }_{(r)}>u).$$

Define ${\mathcal{Y}}_i={\chi }_i-{\chi }_i^{\prime }$, so that $\chi -{\chi }^{\prime }=\mathcal{Y}={\sum }_{i=1}^m{\mathcal{Y}}_i$. Clearly

$${\Vert {\mathcal{Y}}_i\Vert }_{(r)}\le 2{\Vert {\chi }_i\Vert }_{(r)}=2\underset{X\in U_2}{sup}\left|\frac{\mid J\mid }{m}{\mid \langle {\varphi }_i,X\rangle \mid }^2-\frac{1}{m}{\Vert X\Vert }_F^2\right|\le 2\frac{nr\mu +1}{m}\le \frac{1}{2\lambda C_0^2}.$$

We now use the following result by Ledoux and Talagrand in [43].

Theorem A.2

Let ${\{{\mathcal{Y}}_i\}}_{i=1}^m$ be independent symmetric random variables on some Banach space such that ||𝒴_i|| ≤ R. Let $\mathcal{Y}={\sum }_{i=1}^m{\mathcal{Y}}_i$. Then for any integers l ≥ q and any t > 0

$$Pr(\Vert \mathcal{Y}\Vert \ge 8q\mathbb{E}\Vert \mathcal{Y}\Vert +2Rl+t\mathbb{E}\Vert \mathcal{Y}\Vert )\le {(K/q)}^l+2e^{-t^2/256q},$$

where K is a universal constant.

Now for appropriate choices of q, l, and t, and with an appropriate λ such that λ ≥ A/ε² for some constant A, we get that

$$Pr({\Vert \chi \Vert }_{(r)}\ge \epsilon )\le e^{-C{\epsilon }^2\lambda },$$

where C is a constant. Thus, the probability of failure is exponentially small in λ.