Sparse Recovery Beyond Compressed Sensing: Separable Nonlinear Inverse Problems

Abstract

Extracting information from nonlinear measurements is a fundamental challenge in data analysis. In this work, we consider separable inverse problems, where the data are modeled as a linear combination of functions that depend nonlinearly on certain parameters of interest. These parameters may represent neuronal activity in a human brain, frequencies of electromagnetic waves, fluorescent probes in a cell, or magnetic relaxation times of biological tissues. Separable nonlinear inverse problems can be reformulated as underdetermined sparse-recovery problems, and solved using convex programming. This approach has had empirical success in a variety of domains, from geophysics to medical imaging, but lacks a theoretical justification. In particular, compressed-sensing theory does not apply, because the measurement operators are deterministic and violate incoherence conditions such as the restricted-isometry property. Our main contribution is a theory for sparse recovery adapted to deterministic settings. We show that convex programming succeeds in recovering the parameters of interest, as long as their values are sufficiently distinct with respect to the correlation structure of the measurement operator. The theoretical results are illustrated through numerical experiments for two applications: heat-source localization and estimation of brain activity from electroencephalography data.

I. Introduction

A. Separable Nonlinear Inverse Problems

The inverse problem of extracting information from nonlinear measurements is a fundamental challenge in many applied domains, including geophysics, microscopy, astronomy, medical imaging, and signal processing. In this work, we focus on separable nonlinear (SNL) problems [1], [2], where the data are modeled as samples from a linear combination of functions that depend nonlinearly on certain quantities of interest. Depending on the application, these quantities may represent neuronal activity in a human brain, oscillation frequencies of electromagnetic waves, locations of fluorescent probes in a cell, magnetic-resonance relaxation times of biological tissues, or positions of celestial objects in the sky.

Mathematically, the goal in an SNL problem is to estimate k parameters ${\theta }_1,\dots ,{\theta }_k\in {\mathbb{R}}^p$ from samples of a function

$$f(t):=\sum _{i=1}^k{c}_i{\phi }_t({\theta }_i),$$ (I.1)

where $c_1,\dots ,c_k\in \mathbb{R}$ are unknown coefficients. The dependence between each component and the corresponding parameter at a particular value of t is governed by a nonlinear map ${\phi }_t:{\mathbb{R}}^p\to \mathbb{R}$. For simplicity of exposition, we assume that t is one-dimensional and f is a real-valued function, but the framework can be directly extended to multidimensional and complex-valued measurements. The data are samples of f at n locations $s_1,\dots ,s_n\in \mathbb{R}$

$$y:=\left[\begin{array}{c}f(s_1)\\ ⋮\\ f(s_n)\end{array}\right]=\sum _{i=1}^k{c}_i\overrightarrow{\phi }({\theta }_i),$$ (I.2)

where $\overrightarrow{\phi }{({\theta }_i)}_j≔{\phi }_{s_j}({\theta }_i)$, 1 ≤ j ≤ n. Each $\overrightarrow{\phi }({\theta }_i)\in {\mathbb{R}}^n$ is a feature vector associated to one of the parameters θ_i. Without loss of generality, we assume that the feature vectors are normalized, i.e. ${\parallel \overrightarrow{\phi }(\theta )\parallel }_2=1$ for all $\theta \in {\mathbb{R}}^p$. The following examples illustrate the importance of SNL problems in a range of applications.

• Deconvolution of point sources: Deconvolution consists of estimating a signal from samples of its convolution with a fixed kernel K. When the signal is modeled as a superposition of point sources or spikes, representing fluorescent probes in microscopy [4], [5], celestial bodies in astronomy [6] or interfaces between geological layers in seismography [7], this is an SNL problem where θ₁, …, θ_k are the locations of the spikes. In that case, φ_t (θ) is a shifted copy of the convolution kernel K (t − θ), as illustrated in the top row of Figure 1.

• Spectral super-resolution: Super-resolving the spectrum of a multisinusoidal signal from samples taken over a short period of time is an important problem in communications, radar, and signal processing [8]. This is an SNL problem where φ_t (θ) is a complex exponential exp(−i2πθt) with frequency θ (see the second row of Figure 1).

• Heat-source localization: Finding the position of several heat sources in a material with known conductivity from temperature measurements is an SNL problem where φ_t (θ) is the Green’s function of the heat equation parametrized by the location θ of a particular heat source [9]. The bottom row of Figure 1 shows an example (see Section IV-A for more details).

• Estimation of neural activity: Electroencephalography measurements of the electric potential field on the surface of the head can be used to detect regions of focalized activity in the brain [10]. The data are well approximated by an SNL model where the parameters are the locations of these regions [11]. The function φ_t (θ) represents the potential at a specific location t on the scalp, which originates from neural activity at position θ in the brain. This function can be computed by solving the Poisson differential equation taking into account the geometry and electric properties of the head [12]. Figure 2 shows an example. See Section IV-B for more details.

• Quantitative magnetic-resonance imaging: The magnetic-resonance relaxation times T₁ and T₂ of biological tissues govern the local fluctuations of the magnetic field measured by MR imaging systems [13]. MR fingerprinting is a technique to estimate these parameters by fitting an SNL model where each component corresponds to a different tissue [14]–[16]. In this case, the parameter $\theta \in {\mathbb{R}}^2$ encodes the values of T₁ and T₂ and the function φ_t (θ) can be computed by solving the Bloch differential equations [17].

Fig. 1:

Illustration of three inverse problems that can be modeled as SNL problems: deconvolution in reflection seismology (the convolution kernel is a Ricker wavelet [3]), super-resolution of spectral lines, and heat-source localization. The left column shows the continuous measurements φ_t for three parameters θ₁, θ₂, θ₃. The right column shows the data samples corresponding to an example where the coefficients are set to c ≔ (1, 2, 0.5). For super-resolution, only the real part of the data is shown.

Fig. 2:

Localization of brain-activity sources from EEG data is an SNL problem. The image in the top left corner shows the position of three sources in a human brain, situated in the occipital (θ₁), temporal (θ₂) and frontal (θ₃) lobes. The remaining three images show the EEG data corresponding to each of these sources, obtained from 256 sensors located on the surface of the head (see Section IV-B for more details).

B. Reformulation as a Sparse-Recovery Problem

A natural approach to estimate the parameters of an SNL model is to solve the nonlinear least-squares problem,

$$\underset{{\tilde{\theta }}_1,\mathrm{\dots },{\tilde{\theta }}_k\in {\mathbb{R}}^p,\tilde{c}\in {\mathbb{R}}^k}{\text{minimize}}\left|\right|y-\sum _{i=1}^k{\tilde{c}}_i\overrightarrow{\phi }({\tilde{\theta }}_i)|{|}_2^2.$$ (I.3)

Unfortunately, the resulting cost function is typically non-convex and has local minima, as illustrated by the simple example in Figure 3. Consequently, local-descent methods do not necessarily recover the true parameters, even in the absence of noise, and global optimization becomes intractable unless k is very small.

Fig. 3:

Nonlinear least-squares cost functions associated to deterministic SNL problems often have non-optimal local minima. The graph depicts the landscape of the nonlinear least squares cost function $L(\theta )={\min }_{\tilde{c}\in \mathbb{R}}{\parallel \overrightarrow{\phi }({\theta }_1)-\tilde{c}\overrightarrow{\phi }(\theta )\parallel }_2^2$ associated to a deconvolution problem, where the data is generated by convolving a Ricker wavelet [3] with a single spike located θ₁. In addition to the global minimum at θ₁ there are several spurious local minima.

Alternatively, we can reformulate the SNL problem as a sparse-recovery problem and leverage ℓ₁-norm minimization to solve it. This approach was pioneered in the 1970s by geophysicists working on spike deconvolution in the context of reflection seismology [18]–[22]. Since then, it has been applied to many SNL problems such as acoustic sensing [23], [24], radar [25], [26], electroencephalography (EEG) [27], [28], positron emission tomography (PET) [29]–[31], direction of arrival [32], [33], quantitative magnetic resonance imaging [15], [16], and source localization [9], [34], [35]. Our goal is to provide a theory of sparse recovery via convex optimization explaining the empirical success of this approach.

Let us represent the parameters ${\theta }_1,\dots ,{\theta }_k\in {\mathbb{R}}^p$ of an SNL model as a superposition of Dirac measures or spikes in ${\mathbb{R}}^p$, interpreted as a p-dimensional parameter space, (I.2),

$$\mu ≔\sum _{i=1}^k{c}_i{\delta }_{{\theta }_i},$$ (I.4)

where δ_θi denotes a Dirac measure supported at θ_i. Intuitively, the atomic measure μ is a signal that encodes the parameters of interest and their corresponding coefficients. The data described by Equation (I.2) can now be expressed as

$$y={\int }_{{\mathbb{R}}^p}\overrightarrow{\phi }(\theta )\mu (\mathrm{d}\theta ).$$ (I.5)

The SNL problem is equivalent to recovering μ from these linear measurements. The price we pay for linearizing is that the linear inverse problem is extremely underdetermined: y has dimension n, but μ lives in a continuous space of infinite dimensionality! To solve the problem, we need to exploit the assumption that the data only depends on a small number of parameters or, equivalently, that μ is sparse.

For an SNL problem to be well posed, θ₁, …, θ_k should be the only set of k or less parameters such that Equation (I.2) holds. In that case, μ is the solution to the sparse-recovery problem

$$\begin{array}{cc}\underset{\tilde{\mu }}{\text{minimize}}& |\text{support}(\tilde{\mu })|\\ \text{subject to}& {\int }_{{\mathbb{R}}^p}\overrightarrow{\phi }(\theta )(\tilde{\mu })(\mathrm{d}\theta )=y.\end{array}$$ (I.6)

where minimization occurs over the set of measures in ${\mathbb{R}}^p$. The cardinality of the support of an atomic measure is a nonconvex function, which is notoriously challenging to minimize. A fundamental insight underlying many approaches to sparse estimation in high-dimensional statistics and signal processing is that one can bypass this difficulty by replacing the nonconvex function with a convex counterpart. In particular, minimizing the ℓ₁ norm instead of the cardinality function has proven to be very effective in many applications.

In order to analyze the application of ℓ₁-norm minimization to SNL problems, we consider a continuous setting, where the optimization variable is a measure supported on a continuous domain. The goal is to obtain an analysis that is valid for arbitrarily fine discretizations of the domain. This is important because, as we will see below, a fine discretization results in a highly-correlated linear operator, which violates the usual assumptions made in the literature on sparse recovery.

In the case of measures supported on a continuous domain, the continuous counterpart of the ℓ₁ norm is the total-variation (TV) norm [36], [37]¹. Indeed, the TV norm of the atomic measure μ in Equation (I.4) equals the ℓ₁ norm of its coefficients ‖c₁‖₁. Just as the ℓ_∞ norm is the dual norm of the ℓ_∞ norm, the TV norm is defined by

$$\Vert \mu {\Vert }_{\mathrm{TV}}:=\underset{f\in \mathcal{C},\left|\right|f|{|}_{\infty }\le 1}{\sup }\left|\int f\mathrm{d}\mu \right|,$$ (I.7)

where the supremum is taken over all continuous functions in the unit ${\mathcal{L}}_{\infty }$-norm ball. Replacing the cardinality function by this sparsity-promoting norm yields the following convex program

$$\begin{array}{cc}\underset{\tilde{\mu }}{\text{minimize}}\hfill & \Vert \tilde{\mu }{\Vert }_{\mathrm{TV}}\hfill \\ \text{subject to}\hfill & {\int }_{{\mathbb{R}}^p}\overrightarrow{\phi }(\theta )\tilde{\mu }(\mathrm{d}\theta )=y.\hfill \end{array}$$ (I.8)

The goal of this paper is to understand when the solution to Problem (I.8) exactly recovers the parameters of an SNL model.

C. Compressed Sensing

Section I-B shows that solving an SNL problem is equivalent to recovering a sparse signal from linear underdetermined measurements. This is reminiscent of compressed sensing [38]–[40]. In its most basic formulation, the goal of compressed sensing is to estimate a signal $x\in {\mathbb{R}}^m$ with k nonzeros from linear measurements $y\in {\mathbb{R}}^n$ given by y := Ax, where $A\in {\mathbb{R}}^{n\times m}$ and m > n. Remarkably, exact recovery of x is still possible under certain conditions on the matrix A, even though the linear system is underdetermined.

Overdetermined linear inverse problems are said to be ill posed when the measurement matrix is ill conditioned. This occurs when there exist vectors that lie close to the null space of the matrix, or equivalently when a subset of its columns is highly correlated. Analogously, the compressed-sensing problem is ill posed if any sparse subset of columns is highly correlated, because this implies that sparse vectors lie close to the null space. Early works on compressed sensing derive recovery guarantees assuming a bound on the maximum correlation between the columns of the measurement matrix A (sometimes called incoherence). They prove that tractable algorithms such as ℓ₁-norm minimization and greedy techniques achieve exact recovery as long as the maximum correlation is of order n^−1/2 for sparsity levels k of up to order $\sqrt{n}$ [41]–[44], even if the data are corrupted by additive noise [45], [46]. These results were subsequently strengthened to sparsity levels of order n (up to logarithmic factors) [38], [47]–[50] under stricter assumptions on the conditioning of sparse subsets of columns in the measurement matrix, such as the restricted-isometry property [48] or the restricted-eigenvalue condition [49].

The question is whether compressed-sensing theory applies to SNL problems. Let us consider an SNL problem where the parameter space is discretized to yield a finite-dimensional version of the sparse-recovery problem described in Section I-B. The measurement model in Equation (I.5) can then be expressed as

$$y=\sum _{j=1}^m\overrightarrow{\phi }({\eta }_j)x_j$$ (I.9)

$$=\mathrm{\Phi }x.$$ (I.10)

Φ is a measurement matrix whose columns correspond to the feature vectors of ${\eta }_1,\dots ,{\eta }_m\in {\mathbb{R}}^p$, which denote the m points of the discretized parameter space. The signal $x\in {\mathbb{R}}^m$ is the discrete version of μ in Equation (I.4): a sparse vector, such that x_i = c_j when η_j = θ_i for some i ∈ {1, …, k} and x_j = 0 otherwise. For compressed-sensing theory to apply here, the intercolumn correlations of Φ should be very low. Figure 4 shows the intercolumn correlations of a typical compressed-sensing matrix: each column is almost uncorrelated with every other column. Figures 5 and 6 show the correlation function ${\rho }_{\theta }:{\mathbb{R}}^p\to \mathbb{R}$

$${\rho }_{\theta }(\eta )≔\langle \overrightarrow{\phi }(\theta ),\overrightarrow{\phi }(\eta )\rangle ,$$ (I.11)

for the different SNL problems discussed in Section I-A. The intercolumn correlations of the measurement matrix Φ in the corresponding discretized SNL problems are given by samples of ρ_ηi at the locations of the remaining grid points η_j, i ≠ j. The contrast with the intercolumn structure of the compressed-sensing matrix is striking: nearby columns in all SNL measurement matrices are very highly correlated. This occurs under any reasonable discretization of the parameter space; discretizing very coarsely would result in inaccurate parameter estimates, defeating the whole point of solving the SNL problem.

Fig. 4:

Columns and intercolumn correlations for a typical compressed-sensing matrix $A\in {\mathbb{R}}^{100\times 300}$ with i.i.d. standard Gaussian entries. The notation A_i denotes the ith column of A. Note that $\mid A_i^T{A}_j\mid$ is small whenever i ≠ j.

Fig. 5:

Correlation structure of the measurement operators arising in deconvolution, super-resolution, and heat-source localization. The left column shows the discrete data $\overrightarrow{\phi }({\theta }_i)$ for three parameter values. The right column gives the absolute values of the corresponding correlation functions ${\rho }_{{\theta }_i}(\eta )=\overrightarrow{\phi }{({\theta }_i)}^T\overrightarrow{\phi }(\eta )$.

Fig. 6:

Example of correlation functions for the electroencephalography brain-activity localization problem. Each column shows three views of the correlation function ${\rho }_{{\theta }_i}(\eta )=\overrightarrow{\phi }{({\theta }_i)}^T\overrightarrow{\phi }(\eta )$, i = 1, 2, 3, corresponding to one of the neural-activity sources shown in Figure 2.

The difference between the correlation structure of the measurement matrix in compressed-sensing and SNL problems is not surprising. The entries of compressed-sensing matrices are random. As a result, small subsets of columns are almost uncorrelated with high probability. In contrast, matrices in discretized SNL problems arise from a deterministic model tied to an underlying continuous parameter space and to a function φ_t that is typically smooth. Since φ_t (θ) ≈ φ_t (θ′) when θ ≈ θ′, nearby columns are highly correlated. These matrices do not satisfy any of the properties of the conditioning of sparse submatrices commonly assumed in compressed sensing. In conclusion, the answer to our previous question is a resounding no: compressed-sensing theory does not apply to SNL problems.

D. Beyond Sparsity and Randomness: Separation and Correlation Decay

The fact that compressed-sensing theory does not apply to SNL problems involving deterministic measurements is not a theoretical artifact. Sparsity is not a strong enough condition to ensure that such SNL problems are well posed. If φ_t is smooth, which is usually the case in applications, the features $\overrightarrow{\phi }(\theta )$ corresponding to parameters that are clustered in the parameter space are highly correlated. This can be seen in the correlation plots of Figures 5 and 6. As a result, different sparse combinations of features may yield essentially the same data. For a detailed analysis of this issue in the context of super-resolution and deconvolution of point sources we refer the reader to Section 3.2 in [51] (see also [52] and [53]) and Section 2.1 in [54], respectively.

Additional assumptions beyond sparsity are necessary to establish recovery guarantees for SNL problems. At the very least, the features $\overrightarrow{\phi }({\theta }_1),\dots ,\overrightarrow{\phi }({\theta }_k)$ in the data cannot be too correlated. For arbitrary SNL problems it is challenging to define simple conditions to preclude this from happening. However, in most practical situations, SNL problems exhibit correlation decay, meaning that the correlation function ρ_θ defined in Equation (I.11) is bounded by a decaying function away from θ. This is a natural property: the more separated two parameters θ and θ′ are in the parameter space, the less correlated we expect their features $\overrightarrow{\phi }(\theta )$ and $\overrightarrow{\phi }({\theta }^{\prime })$ to be. All the examples in Section I-A have correlation decay (see Figures 5 and 6).

For SNL problems with correlation decay there is a simple way of ensuring that the features corresponding to the true parameters θ₁, …, θ_k are not highly correlated: imposing a minimum separation between them in the parameter space. The main contribution of this paper is showing that this is in fact sufficient to guarantee that TV-norm minimization achieves exact recovery, under some additional conditions on the derivatives of the correlation function.

E. Organization

In Section II we propose a theoretical framework for the analysis of sparse estimation in the context of SNL inverse problems. We focus on the case p = 1 for simplicity, but our results can be extended to higher dimensions, as described in Section III-E. Our main results are Theorems II.4 and II.6, which establish exact-recovery results for SNL problems with correlation decay under a minimum separation on the true parameters. Section III contains the proof of these results, which are based on a novel dual-certificate construction. Section IV illustrates the theoretical results through numerical experiments for two applications: heat-source localization and estimation of brain activity from electroencephalography data.

II. Main Results

A. Correlation decay

In this section we formalize the notion of correlation decay by defining several conditions on the correlation function ρ_θ and on its derivatives. Throughout we assume that the problem is one dimensional (p ≔ 1).

To alleviate notation we define

$${\rho }_{\theta }^{(q,r)}(\eta )≔{\overrightarrow{\phi }}^{(q)}{(\theta )}^T{\overrightarrow{\phi }}^{(r)}(\eta ),$$ (II.1)

for q = 0, 1 and r = 0, 1, 2, where ${\overrightarrow{\phi }}^{(q)}$ is the qth derivative of $\overrightarrow{\phi }$. Recall that we assume ${\parallel \overrightarrow{\phi }(\theta )\parallel }_2$ for all $\theta \in \mathbb{R}$. This implies ρ_θ(θ) = 1 and ${\rho }_{\theta }^{(1,0)}(\theta )={\rho }_{\theta }^{(0,1)}(\theta )=0$ for all $\theta \in \mathbb{R}$. Plots of these derivatives are shown in Figure 8 for the deconvolution, super-resolution, and heat-source localization problems. Our conditions take the form of bounds in different regions of the parameter space: a near region, an intermediate region, and a far region, as depicted in Figure 7.

Fig. 8:

Derivatives of the correlation functions arising in deconvolution, super-resolution, and heat-source localization. In all cases the correlation function is locally concave and all derivatives exhibit decaying tails. In the case of super-resolution the tail decay is not summable, but can be made summable by applying a window function to the measurements.

Fig. 7:

Illustration of the near, intermediate and decay conditions on the correlation function defined in Section II-A. This example shows a correlation function arising in deconvolution.

In the near region the correlation can be arbitrarily close to one, but is locally bounded by a quadratic function.

Condition II.1 (Near Condition). The correlation function ρ_θ satisfies the near condition if

$${\rho }_{\theta }^{(0,2)}(\eta )\le -{\gamma }_0\mathit{and}$$ (II.2)

$$|{\rho }_{\theta }^{(1,2)}(\eta )|\le {\gamma }_1\Vert {\overrightarrow{\phi }}^{(1)}(\theta ){\Vert }_2^2$$ (II.3)

hold for all η in [N⁻, N⁺], where N^± ≔ θ ± N, and γ₀, γ₁ are positive constants.

Equation (II.2) requires correlations to be concave locally, which is natural since the maximum of ρ_θ is attained at θ. Equation (II.3) is a regularity condition that requires ${\rho }_{\theta }^{(0,2)}(\eta )$ to vary smoothly as we change the center θ. The normalization quantity ${\parallel {\overrightarrow{\phi }}^{(1)}(\theta )\parallel }_2^2$ captures how sensitive the features $\overrightarrow{\phi }$ are to perturbations. If this quantity is small for some i then we require more regularity from ρ_θ because θ is harder to distinguish from nearby points using the measurements $\overrightarrow{\phi }$.

In the intermediate region the correlation function ρ_θ is bounded but can otherwise fluctuate arbitrarily. In addition, we require a similar constraint on its derivative with respect to the position of the center θ.

Condition II.2 (Intermediate Condition). The correlation function ρ_θ satisfies the intermediate condition if

$$\Vert {\rho }_{\theta }^{(0,0)}(\eta )|\le {\gamma }_2<1\mathit{and}$$ (II.4)

$$|{\rho }_{\theta }^{(1,0)}(\eta )|\le {\gamma }_3\Vert {\overrightarrow{\phi }}^{(1)}(\theta ){\Vert }_2^2$$ (II.5)

hold for η < N⁻ and η > N⁺, where N^± are defined as in the near condition, and γ₂, γ₃ are positive constants.

In the decay region the correlation and its derivatives are bounded by a decaying function.

Condition II.3 (Decay Condition). The correlation function ρ_θ satisfies the decay condition with decay constant σ > 0 if

$$|{\rho }_{\theta }^{(0,2)}(\eta )|\le C_{0,r^{e-(|n-0|-D)}}/\sigma \text{ and}$$ (II.6)

$$|{\rho }_{\theta }^{(1,r)}(\eta )|\le C_{1,r^{e-(|n-\theta |-D)/\sigma }}\left|\right|{\overrightarrow{\phi }}^{(1)}(\theta )|{|}_2^2$$ (II.7)

hold for η < D⁻ and η > D⁺, where r = 0, 1, 2, D^± ≔ θ ± D, and C_q,r are positive constants.

The choice of exponential decay is for concreteness, and can be replaced by a different summable decay bound². Figure 8 shows the derivatives of the correlation functions for several SNL problems.

The sample locations s₁, …, s_n have a direct effect on the correlation function, and hence on whether an SNL problem satisfies Conditions II.1 to II.3. In Figure 9 we depict the correlation function ρ_θ for different sampling patterns in a Gaussian deconvolution problem with $\overrightarrow{\phi }{(\theta )}_i≔\exp (-{(\theta -s_i)}^2∕2)$. For both regular and irregular sampling patterns, the correlation functions are well behaved as long as there are samples close to the true parameter. In contrast, when there are few samples, or the samples are distant from the true parameter, the conditions in Conditions II.1 to II.3 may be violated. We refer the interested reader to [54] for a more precise analysis of sampling patterns for deconvolution of point sources.

Fig. 9:

Plots of the correlation function ρ_θ(η) for a deconvolution problem with varying sampling patterns (indicated by the x markers on the horizontal axis). Specifically, $\overrightarrow{\phi }{(\theta )}_i≔\exp (-{(\theta -s_i)}^2∕2)$ are samples from a θ-centered Gaussian kernel. In the dense uniform case, the correlation function satisfies Conditions II.1 to II.3. For irregular sampling pattern, the correlation function will in general satisfy the conditions if there are some samples close to θ. For sparse uniform samples, the correlation function is nearly flat at θ since there are too few measurements to distinguish nearby parameters. If the samples are only located to the right of θ, then all parameters on the left side have nearly identical measurements.

B. Exact Recovery for SNL Problems with Uniform Correlation Decay

In this section we focus on SNL problems where the correlation function ρ_θ of the measurement operator is approximately translation invariant, meaning that ρ_θ has similar properties for any value of θ. Examples of such SNL problems include super-resolution, deconvolution, and heat-source localization if the conductivity is approximately uniform. We prove that TV-norm minimization recovers a superposition of Dirac measures exactly as long as the support satisfies a separation condition related to the decay properties of the correlation function and its derivatives.

Theorem II.4. Let Θ ≔ {θ₁, …, θ_k} be the support of the measure μ defined in Equation (I.4). Assume that the correlation functions ρθ_i, θ_i ∈ Θ, satisfy Conditions II.1 to II.3 for fixed constants N, D, σ, $\gamma \in {\mathbb{R}}^4$, and $C\in {\mathbb{R}}^{2\times 3}$. Then μ is the unique solution to Problem (I.8) as long as the minimum separation of Θ satisfies,

$$\begin{gathered} \underset{i\ne j,{\theta }_i,{\theta }_j\in \mathrm{\Theta }}{\min }|{\theta }_i-{\theta }_j|>2D+\mathrm{\Delta }\sigma , \\ \mathrm{\Delta }≔\max (\log (1+2(C_{0,0}+C_{1,1})),{\lambda }_1,{\lambda }_2), \end{gathered}$$ (II.8)

where

$${\lambda }_1=2\log \left(\frac{2A}{-C_{0,0}+\sqrt{C_{0,0}^2+4(1-{\gamma }_2)A}}\right),$$ (II.9)

$${\lambda }_2=2\log \left(\frac{2B}{-C_{0,2}+\sqrt{C_{0,2}^2+4{\gamma }_{0}B}}\right),$$ (II.10)

$$\begin{gathered} A=2(2C_{0,0}+C_{1,1}-C_{1,1}{\gamma }_2+C_{0,1}{\gamma }_3)+1-{\gamma }_2, \\ B=2((2C_{0,0}+C_{1,1}){\gamma }_0+C_{0,2}+C_{0,1}{\gamma }_1+{\gamma }_0, \end{gathered}$$

and the constants C_q,r are chosen so that

$$\begin{gathered} C_{0,1}{C}_{1,0}=C_{0,0}{C}_{1,1,}\mathit{and} \\ {C}_{0,1}{C}_{1,2}=C_{1,1}{C}_{0,2}. \end{gathered}$$ (II.11)

Note that the condition in Equation (II.11) is only needed to simplify the statement and proof of our results.

Theorem II.4 establishes that TV minimization recovers the true parameters of an SNL problem when the support separation is larger than a constant that is proportional to the rate of decay of the correlation function and its derivatives. This separation is measured from the edges of the intermediate regions, as depicted in Figure 10. In stark contrast to compressed-sensing theory, the result holds for correlation functions that are arbitrarily close to one in the near regions, and may have arbitrary bounded fluctuations in the intermediate regions.

Fig. 10:

Illustration of the minimum-separation condition required by Theorem II.4.

Our statement and proof of Theorem II.4 focuses on clarity over sharpness. For example, in the Gaussian deconvolution problem depicted in Figure 9 with dense uniform samples, the theorem requires a minimum separation of approximately 6.6 to guarantee exact parameter recovery. To obtain this bound, we let

$$\begin{array}{l}N=1,D=0,\sigma =1,\\ \gamma \text{=}{[\begin{array}{cccc}0.185& 0.983& 0.788& 0.868\end{array}]}^T,\\ C=\left[\begin{array}{ccc}2.818& 3.348& 4.200\\ 6.786& 8.060& 10.113\end{array}\right],\end{array}$$ (II.12)

and verify the conditions numerically. This is in contrast with the minimum separation of approximately 3 proven in [54]. Part of this discrepancy comes from applying exponential decay bounds to a Gaussian-shaped correlation function.

Our result requires conditions on the correlation functions centered at the true support Θ, and also on their derivatives. The decay conditions on the derivatives constrain the correlation structure of the measurement operator when we perturb the position of the true parameters. For example, they implicitly bound pairwise correlations centered in a small neighborhood of the support. Exploring to what extent these conditions are necessary is an interesting question for future research.

C. Exact Recovery for SNL Problems with Nonuniform Correlation Decay

The measurement operators associated to many SNL problems of practical interest have nonuniform correlations. Figures 5 and 6 show that this is the case for heat-source localization with spatially-varying conductivity, and for estimation of brain-activity from EEG data. Our goal in this section is to establish recovery guarantees for such problems.

The conditions on the correlation structure of the measurement operator required by Theorem II.4 only pertain to the correlation functions centered at the true parameters. In order to generalize the result, we allow the correlation function centered at each parameter to satisfy the conditions in Section II-A with different constants. This makes it possible for the correlation to have near and intermediate regions of varying widths around each element of the support, as well as different decay constants in the far region. Our main result is that TV-norm minimization achieves exact recovery for SNL problems with nonuniform correlation structure, as long as the support satisfies a minimum-separation condition dependent on the corresponding support-centered correlation functions.

Let Θ be the support of our signal of interest, and assume ρ_θi satisfies the decay conditions in Section II-A with parameters σ_i, D_i, and N_i, which are different for all θ_i ∈ Θ. Extending the notation, we let $N_i^{\pm }≔{\theta }_i\pm N_i$ and $D_i^{\pm }≔{\theta }_i\pm D_i$ denote the endpoints of the near and decay regions, respectively. Intuitively, when σ_i and D_i are small, the corresponding correlation function ρ_θi is “narrower” and should require less separation than “wider” correlation functions with large values of σ_i and D_i. This is illustrated in Figure 11, where we depict ρ_θi. for the heat-source localization problem at three different values of i. The decay becomes more pronounced towards the right due to the changing thermal conductivity of the underlying medium. For the problem to be well posed, one would expect θ₁ to require more separation from other active sources than θ₂, which in turn should require more than θ₃. We confirm this intuition through numerical experiments in Section IV-A. To make it mathematically precise, we define the following generalized notion of support separation.

Fig. 11:

Support-centered correlations with varying decay parameters for the heat-source localization problem. The vertical dashed lines indicate the locations of $D_i^{\pm }$ for i = 1, 2, 3, whereas the curved dashed lines indicate the exponential decay bounds. As we move from left to right along the θ-axis the thermal conductivity is decreasing, which causes the correlation functions to become narrower.

Definition II.5 (Generalized support Separation). Suppose for all θ_i ∈ Θ that ρ_θi satisfies Condition II.3 with parameters D_i and σ_i. Define the normalized distance d(θ_i, θ_j) for i ≠ j by

$$d({\theta }_i,{\theta }_j)=\frac{|{\theta }_i-{\theta }_j|-D_i-D_j}{\max ({\sigma }_i,{\sigma }_j)}.$$ (II.13)

Assume that Θ is ordered so that θ₁ < θ₂ < … < θ_k. Θ has separation Δ > 0 if d(θ_i, θ_j) > |i − j|Δ for all θ_i, θ_j ∈ Θ with i ≠ j.

The normalized distance d(θ_i, θ_j) is measured between the edges of the decay regions of θ_i and θ_j, and normalized by the level of decay. This allows sharply decaying correlation functions to be in close proximity with one another. We require d(θ_i, θ_j) > |i − j|Δ to prevent the parameters from becoming too clustered. If we only require the weaker condition d(θ_i, θ_j) > Δ, and if σ_i grows very quickly with i, then we could have d(θ₁, θ_j) ≈ Δ for all j > i. This causes too much overlap between the correlation functions.

Figure 13 gives an example of parameters and correlation functions that satisfy the conditions of Definition II.5. The following theorem establishes exact-recovery guarantees under a condition on the generalized support separation.

Fig. 13:

Non-trivial configuration allowed by Definition II.5. The central parameter θ₄ has weaker decay, and thus requires more separation from the other parameters.

Theorem II.6. Suppose that for all θ_i ∈ Θ that ρ_θi satisfies Conditions II.1 to II.3 and (II.11) with constants N ≔ N_i, D ≔ D_i, σ ≔ σ_i, C, and γ. Note that C and γ are the same for each θ_i. Then the true measure μ defined in (I.4) is the unique solution to Problem (I.8) when Θ has separation Δ (as determined by Definition II.5) satisfying

$$\mathrm{\Delta }>\max (\log (1+2(C_{0,0}+C_{1,1})),{\lambda }_1,{\lambda }_2),$$ (II.14)

(II.9), (II.10), and (II.11).

The proof of Theorem II.6, which implies Theorem II.4, is given in Section III. The theorem establishes that TV minimization recovers the true parameters of SNL problems with nonuniform correlation decays when the generalized support separation is larger than a constant. Equivalently, exact recovery is achieved as long as each true parameter θ_i is separated from the rest by a distance that is proportional to the rate of decay of the correlation function centered at θ_i. The separation is measured from the edges of the intermediate regions, which can also vary in width as depicted in Figure 10. The result matches our intuition about SNL problems: the parameters can be recovered as long as they yield measurements that are not highly correlated. As mentioned previously, the theorem requires decay conditions on the derivatives of the correlation function, which constrain the correlation structure of the measurement operator.

D. Robustness to Noise

In practice, measurements are always corrupted by noisy perturbations. Noise can be taken into account in our measurement model (I.2) by incorporating an additive noise vector $z\in {\mathbb{R}}^n$:

$$y≔\sum _{i=1}^k{c}_i\overrightarrow{\phi }({\theta }_i)+z.$$ (II.15)

To adapt the TV-norm minimization problem (I.8) to such measurements, we relax the data consistency constraint from an equality to an inequality:

$$\begin{array}{cc}\underset{\tilde{\mu }}{\text{minimize}}\hfill & \Vert \tilde{\mu }{\Vert }_{\mathrm{TV}}\hfill \\ \text{subject to}\hfill & \Vert {\int }_{{\mathbb{R}}^p}\overrightarrow{\phi }(\theta )\tilde{\mu }(\mathrm{d}\theta )-y{\Vert }_2\le \xi ,\hfill \end{array}$$ (II.16)

where ξ > 0 is a parameter that must be tuned according to the noise level. Previous works have established robustness guarantees for TV-norm minimization applied to specific SNL problems such as super-resolution [56], [57] and deconvolution [54] at small noise levels. These proofs are based on dual certificates. Combining the arguments in [54], [56] with our dual-certificate construction in Section III yields robustness guarantees for general SNL problems in terms of support recovery at high signal-to-noise regimes. We omit the details, since the proof would essentially mimic the ones in [54], [56]. In Figure 14 we show an application of Equation (II.16) to a noisy deconvolution problem taken from [54].

Fig. 14:

Deconvolution from noisy samples (in red) with a signal-to-noise ratio of 20.7 dB for the Ricker wavelet. The noise is iid Gaussian. The recovered signals are obtained by solving equation (II.16) on a fine grid which contains the location of the original spikes.

E. Discretization

The continuous optimization problem (II.16) can be solved by applying ℓ₁-norm minimization after discretizing the parameter space. This is a very popular approach in practice for a variety of SNL problems [16], [18], [23], [27], [32]. If the true parameters lie on the discretization grid, then our exact-recovery results translate immediately. The following corollary is a discrete version of Theorem II.6.

Corollary II.7. Assume Θ lies on a known discretized grid G ≔ {η₁, …, η_m} so that Θ ⊂ G. Furthermore, suppose the conditions of Theorem II.6 hold so that μ as defined in (I.4) is the unique solution to Problem (I.8). Define the dictionary $\Phi \in {\mathbb{R}}^{n\times m}$ by

$$\mathrm{\Phi }≔[\overrightarrow{\phi }({\eta }_1)\cdots \overrightarrow{\phi }({\eta }_m)].$$ (II.17)

Then the ℓ₁-norm minimization problem

$$\begin{array}{cc}\underset{\tilde{x}}{\text{minimize}}& \left|\right|\tilde{x}|{|}_1\\ \text{subject to}& \mathrm{\Phi }\tilde{x}=y\end{array}$$ (II.18)

has a unique solution $x\in {\mathbb{R}}^m$ satisfying

$$x_j≔\{\begin{array}{cc}{c}_i& if{\eta }_j={\theta }_i,\\ 0& otherwise,\end{array}$$ (II.19)

for j = 1, …, m.

Proof: If the support of $\tilde{\mu }$ in Problem (I.8) is restricted to lie on G, then Problems (I.8) and (II.18) are the same. Thus ‖μ‖_TV must be smaller than ${\parallel \tilde{x}\parallel }_1$ for any $\tilde{x}$ such that $\Phi \tilde{x}=y$. By assumption μ is supported on G, so the result follows. ■

Of course, the true parameters may not lie on the grid used to solve the ℓ₁-norm minimization problem. The proof techniques used to derive robustness guarantees for superresolution and deconvolution in [54], [56] can be leveraged to provide some control over the discretization error. Performing a more accurate analysis of discretization error for SNL problems is an interesting direction for future research.

F. Related Work

1). Sparse Recovery via Convex Programming from Deterministic Measurements:

In [58], Donoho and Elad develop a theory of sparse recovery from generic measurements based on the spark and coherence of the measurement matrix Φ. The spark is defined to be the smallest positive value s such that Φ has s linearly dependent columns. The coherence, which we denote by M(Φ), is the maximum absolute correlation between any two columns of Φ. The authors show that exact recovery is achieved by ℓ₁-minimization when the number of true parameters is less than (1 + 1/M(Φ))/2. As discussed in Section I-C, these arguments are inapplicable to the finely-discretized parameter spaces occurring in SNL problems since neighboring columns of Φ have correlations approaching one. In [59], the authors provide a support-dependent condition for exact recovery of discrete vectors. Using our notation, they require that β(Θ)/(1 − α(Θ)) < 1 where

$$\begin{gathered} \alpha (\mathrm{\Theta })≔\underset{{\theta }_i\in \mathrm{\Theta }}{\max }\underset{{\theta }_k\in \mathrm{\Theta },k\ne i}{\Sigma }|\overrightarrow{\phi }{({\theta }_i)}^T\overrightarrow{\phi }({\theta }_k)|\text{and} \\ \beta (\mathrm{\Theta })≔\underset{{\eta }_j\notin \mathrm{\Theta }}{\max }\underset{{\theta }_k\in \mathrm{\Theta }}{\Sigma }|\overrightarrow{\phi }{({\eta }_j)}^T\overrightarrow{\phi }({\theta }_j)|. \end{gathered}$$ (II.20)

Here $\overrightarrow{\phi }({\eta }_j)$ for η_j ∉ Θ ranges over the columns of Φ that do not correspond to the true parameters. This condition is also inapplicable for matrices arising in our problems of interest because β(Θ) approaches one (or larger) for finely-discretized parameter spaces. Sharper exact-recovery guarantees in subsequent works [38], [47]–[50], [60] require randomized measurements, and therefore do not hold for deterministic SNL problems as explained in Section I-C.

2). Convex Programming Applied to Specific SNL Problems:

In [51], [61], the authors establish exact recovery guarantees for super-resolution via convex optimization by leveraging parameter separation (see Section I-D). Subsequent works build upon these results to study the robustness of this methodology to noise [56], [57], [62], [63], missing data [64], and outliers [65]. A similar analysis is carried out in [54] for deconvolution. The authors establish a sampling theorem for Gaussian and Ricker-wavelet convolution kernels, which characterizes what sampling patterns yield exact recovery under a minimum-separation condition. Other works have analyzed the Gaussian deconvolution problem under nonnegativity constraints [66], [67], and also for randomized measurements [68]. All of these works exploit the properties of specific measurement operators. In contrast, the present paper establishes a general theory that only relies on the correlation structure of the measurements. The works that are closer to this spirit are [69], [70], which analyze deconvolution via convex programming for generic convolution kernels. The results in [69] require quadratically decaying bounds on the first three derivatives of the convolution kernel. In [70], the authors prove exact recovery assuming bounds on the first four derivatives of the autocorrelation function of the convolution kernel. In contrast to these works, our results allow for discrete irregular sampling and for measurement operators that are not convolutional, which is necessary to analyze applications such as heat-source localization or estimation of brain activity.

Algorithms for solving the convex programs arising from SNL problems divide into two categories: grid-based and grid-free (or off-the-grid). In grid-based methods the parameter space is discretized, thus yielding a finite-dimensional ℓ₁ minimization problem that can be solved using standard methods (as we have done in Section IV). Grid-free methods attempt to directly solve the infinite-dimensional, but with weaker guarantees (see [71] and [72] for review articles). Recently, several authors [73]-[75] have proposed grid-free algorithms based on the conditional gradient method [76].

3). Other Methodologies:

SNL parameter recovery can be formulated as a nonlinear least squares problem [2]. For a fixed value of the parameters θ₁, …, θ_k, the optimal coefficients c₁, …, c_k in (I.1) have a closed form solution. This makes it possible to minimize the nonlinear cost function with respect to θ₁, …, θ_k directly, a technique known as variable projection [1]. As shown in Figure 3, a downside to this approach is that it may converge to suboptimal local minima, even in the absence of noise.

Prony’s method [8], [77] and the finite-rate-of-innovation (FRI) framework [78], [79] can be applied to tackle SNL problems, as long one can recast them as spectral superresolution problems. This provides a recovery method that avoids discretizing the parameter space. The FRI framework has also been applied to arbitrary non-bandlimited convolution kernels [80] and nonuniform sampling patterns [81], but without exact-recovery guarantees. These techniques have been recently extended by Dragotti and Murray-Bruce [82] to physics-driven SNL problems. By approximating complex exponentials with weighted sums of Green’s functions, they are able to recast parameter recovery as a related spectral super-resolution problem that approximates the true SNL problem.

III. Proof of Exact-Recovery Results

A. Dual Certificates

To prove Theorem II.6 we construct a certificate that guarantees exact recovery.

Proposition III.1 (Proof in Section A). Let $ϴ=\{{\theta }_1,\dots ,{\theta }_k\}\subset \mathbb{R}$ denote the support of the signed atomic measure μ. Assume that for any sign pattern ξ ∈ {±1}^k there is a $\tilde{q}\in {\mathbb{R}}^p$ such that $\tilde{Q}(\theta )≔{\tilde{q}}^T\overrightarrow{\phi }(\theta )$ satisfies

$$\tilde{Q}({\theta }_i)={\xi }_i,\forall {\theta }_i\in \mathrm{\Theta },$$ (III.1)

$$\left|\tilde{Q}(\theta )\right|<1,\forall \theta \in {\mathrm{\Theta }}^c.$$ (III.2)

Then μ is the unique solution to problem (I.8).

To prove exact recovery of a signal we need to show that it is possible to interpolate the sign pattern of its amplitudes, which we denote by ξ, on its support Θ using an interpolating function $\tilde{Q}(\theta )$ that is expressible as a linear combination of the coordinates of $\overrightarrow{\phi }(\theta )$. The coefficient vector of this linear combination, denoted by $\tilde{q}$, is known as a dual certificate in the literature because it certifies recovery and is a solution to the Lagrange dual of problem (I.8):

$$\begin{array}{l}\underset{\nu }{\max imize}{\nu }^{T}y\\ \text{subject to}{\sup }_{\theta }\left|v^T\overrightarrow{\phi }(\theta )\right|\le 1.\end{array}$$ (III.3)

Dual certificates have been widely used to derive guarantees for inverse problems involving random measurements, including compressed sensing [83], [84], matrix completion [85] and phase retrieval [86]. In such cases, the certificates are usually constructed by leveraging concentration bounds and other tools from probability theory [87]. In contrast, our setting is completely deterministic. More recently, dual certificates have been proposed for specific deterministic problems such as super-resolution [51] and deconvolution [54]. Our goal here is to provide a construction that is valid for generic SNL models with correlation decay.

B. Correlation-Based Dual Certificates

Our main technical contribution is a certificate that only depends on the correlation function of the measurement operator. In contrast, certificate constructions in previous works on SNL problems [51], [54], [69] typically rely on problem-specific structure, with the exception of [70] which proposes a certificate for time-invariant problems based on autocorrelation functions.

In our SNL problems of interest the function $\overrightarrow{\phi }(\theta )$ mapping the parameters of interest to the data is assumed to be continuous and smooth. As a result, Equations (III.1) and (III.2) imply that any valid interpolating function $\tilde{Q}$ reaches a local extremum at each θ_i ∈ Θ. Equivalently, $\tilde{Q}$ satisfies the following 2k interpolation equations for all θ_i ∈ Θ:

$$\tilde{Q}({\theta }_i)={\xi }_i,$$ (III.4)

$${\tilde{Q}}^{(1)}({\theta }_i)=0.$$ (III.5)

Inspired by this observation, we define

$$q≔\sum _{i=1}^k{\alpha }_i\overrightarrow{\phi }({\theta }_i)+{\beta }_i\frac{{\overrightarrow{\phi }}^{(1)}({\theta }_i)}{{\Vert {\overrightarrow{\phi }}^{(1)}({\theta }_i)\Vert }_2^2}$$ (III.6)

where ${\alpha }_i,{\beta }_i\in \mathbb{R}$, i = 1, …, k, are chosen so that $Q(\theta )≔q^T\overrightarrow{\phi }(\theta )$ satisfies the interpolation equations. Crucially, this choice of coefficients yields an interpolation function Q that is a linear combination of correlation functions centered at Θ,

$$Q(\theta )=\sum _{i=1}^k{\alpha }_i\overrightarrow{\phi }{({\theta }_i)}^T\overrightarrow{\phi }(\theta )+{\beta }_i\frac{{\overrightarrow{\phi }}^{(1)}{({\theta }_i)}^T\overrightarrow{\phi }(\theta )}{{\Vert {\overrightarrow{\phi }}^{(1)}({\theta }_i)\Vert }_2^2}$$ (III.7)

$$=\sum _{i=1}^k{\alpha }_i{\rho }_{{\theta }_i}(\theta )+{\beta }_i\frac{{\rho }_{{\theta }_i}^{(1,0)}(\theta )}{{\rho }_{{\theta }_i}^{(1,1)}({\theta }_i)}.$$ (III.8)

In essence, we interpolate the sign pattern of the signal on its support using support-centered correlations. Since ρ_θi(θ_i) = 1 and ${\rho }_{{\theta }_i}^{(1,0)}({\theta }_i)=0$, Equations (III.4) and (III.8) imply α_i ≈ ξ_i when ρ(θ,η) and ρ^(1,0)(θ,η) decay as |θ − η| grows large and the support is sufficiently separated. The term that depends on β can be interpreted as a correction to the derivatives of Q so that (III.5) is satisfied. The normalizing factor used in (III.8) makes this explicit:

$${\frac{\partial }{\partial \theta }|}_{\theta ={\theta }_i}{\beta }_i\frac{{\rho }_{{\theta }_i}^{(1,0)}(\theta )}{{\rho }_{{\theta }_i}^{(1,1)}({\theta }_i)}={\beta }_i$$ (III.9)

for i = 1, …, k. Figure 15 illustrates the construction for the heat-source localization problem. The construction is inspired by previous interpolation-based certificates tailored to superresolution [51] and deconvolution [54].

Fig. 15:

The image shows the interpolating function Q(θ) defined in Equation (III.8) for the heat-source localization problem. The function is a linear combination of ρ_θi (θ) (red curves) and ${\rho }_{{\theta }_i}^{(1,0)}(\theta )$ (orange curves) for θ_i ∈ Θ.

In the remainder of this section we show that our proposed construction yields a valid certificate if the conditions of Theorem II.6 hold. In Section III-C we prove Lemma III.2, which establishes that the interpolation equations have a unique solution and therefore Q satisfies (III.1).

Lemma III.2. Under the assumptions of Theorem II.6 there exist $\alpha ,\beta \in {\mathbb{R}}^k$ which uniquely solve Equations (III.4) and (III.5).

In Section III-D we prove Lemma III.3, which establishes that Q satisfies (III.2). This completes the proof of Theorem II.6.

Lemma III.3. Let $\alpha ,\beta \in {\mathbb{R}}^k$ be the coefficients obtained in Lemma III.2. Under the assumptions of Theorem II.6, the interpolating function Q defined in (III.8) satisfies |Q(θ)| < 1 for θ ∉ Θ.

C. Proof of Lemma III.2

To simplify notation we define the ith normalized correlation and its derivatives by

$${\widehat{\rho }}_{{\theta }_i}^{(q,r)}(\theta )≔\frac{{\rho }_{{\theta }_i}^{(q,r)}(\theta )}{{\Vert {\overrightarrow{\phi }}^{(q)}({\theta }_i)\Vert }_2^2},$$ (III.10)

for q = 0, 1 and r = 0, 1, 2, where ${\rho }_{{\theta }_i}^{(q,r)}$ is defined in Equation (II.1). Using this notation we have

$$Q(\theta )=\sum _{i=1}^k{\alpha }_i{\rho }_{{\theta }_i}(\theta )+{\beta }_i\frac{{\rho }_{{\theta }_i}^{(1,0)}(\theta )}{{\rho }_{{\theta }_i}^{(1,1)}({\theta }_i)}$$ (III.11)

$$=\sum _{i=1}^k{\alpha }_i{\widehat{\rho }}_{{\theta }_i}(\theta )+{\beta }_i{\widehat{\rho }}_{{\theta }_i}^{(1,0)}(\theta ).$$ (III.12)

To prove Lemma III.2 we establish the following stronger result, which also gives useful bounds on α and β. Throughout we assume that ρ satisfies Conditions II.1 to II.3 and (II.11) with parameters γ, C, N, D, and σ, and that Θ has separation Δ.

Lemma III.4. Let s ≔ 2e^−Δ/(1 − e^−Δ). If Δ > log(1 + 2(C_0,0 + C_1,1)) then there are $\alpha ,\beta \in {\mathbb{R}}^k$ which uniquely solve equations (III.4) and (III.5). Furthermore, we have

$$\begin{gathered} {\Vert \alpha \Vert }_{\infty }\le \frac{1-C_{1,1^s}}{1-(C_{0,0}+C_{1,1})s}, \\ {\Vert \beta \Vert }_{\infty }\le \frac{C_{0,1^s}}{1-(C_{0,0}+C_{1,1})s}, \\ 1-{\Vert \alpha -\xi \Vert }_{\infty }\ge \frac{1-(2C_{0,0}+C_{1,1})s}{1-(C_{0,0}+C_{1,1})s}. \end{gathered}$$ (III.13)

To prove Lemma III.4 we begin by rewriting Equations (III.4) and (III.5) in block matrix-vector form for the Q function defined in Equation (III.8):

$$\left[\begin{array}{cc}\mathcal{I}+P^{(0,0)}& P^{(1,0)}\\ P^{(0,1)}& \mathcal{I}+P^{(1,1)}\end{array}\right]\left[\begin{array}{c}\alpha \\ \beta \end{array}\right]=\left[\begin{array}{c}\xi \\ 0\end{array}\right],$$ (III.14)

where $\mathcal{I}\in {\mathbb{R}}^{k\times k}$ is the identity matrix, and $P^{(q,r)}\in {\mathbb{R}}^{k\times k}$ satisfies

$$P_{\mathit{ij}}^{(q,r)}={\widehat{\rho }}_{{\theta }_j}^{(q,r)}({\theta }_i)$$ (III.15)

for i ≠ j and $P_{\mathit{ii}}^{(q,r)}=0$. To see why $P_{\mathit{ii}}^{(1,0)}=0$ agrees with equations (III.4) and (III.5), note that

$${\widehat{\rho }}_{{\theta }_i}^{(1,0)}({\theta }_i)=\frac{{\overrightarrow{\phi }}^{(1)}{({\theta }_i)}^T\overrightarrow{\phi }({\theta }_i)}{{\Vert {\overrightarrow{\phi }}^{(1)}({\theta }_i)\Vert }_2^2}=0.$$ (III.16)

The atoms are normalized– ${\parallel \overrightarrow{\phi }(\theta )\parallel }_2=1$ for all θ– which implies

$$0=\frac{d}{d\theta }{\Vert \overrightarrow{\phi }(\theta )\Vert }_2^2=2{\overrightarrow{\phi }}^{(1)}{(\theta )}^T\overrightarrow{\phi }(\theta ).$$

For the same reason it follows that ${\widehat{\rho }}_{{\theta }_i}^{(0,1)}({\theta }_i)=0$.

Our plan is to bound the norm of each P^(q,r). If these norms are sufficiently small then the matrix in Equation (III.14) is nearly the identity, and the desired result follows from a linear-algebraic argument. Define ${∊}_i^{(q,r)}(\theta )$ by

$${ϵ}_i^{(q,r)}(\theta )≔\sum _{j\ne i}|{\widehat{\rho }}_{{\theta }_j}^{(q,r)}(\theta )|,$$ (III.17)

where q = 0,1 and r = 0,1,2. Here we think of θ as a point close to θ_i, so ${∊}_i^{(q,r)}(\theta )$ captures the cumulative correlation from the other, more distant elements of Θ. We expect ${∊}_i^{(q,r)}(\theta )$ to be small when ρ has sufficient decay and Θ is well separated. For a matrix A let ‖A‖_∞ denote the infinity-norm defined by

$${\Vert A\Vert }_{\infty }≔\underset{{\Vert x\Vert }_{\infty }=1}{\sup }{\Vert Ax\Vert }_{\infty }.$$ (III.18)

‖A‖_∞ equals the maximum sum of absolute values in any row of A. We have

$${\Vert P^{(q,r)}\Vert }_{\infty }={ϵ}^{(q,r)}≔\underset{i\in \{1,\mathrm{\dots }k\}}{\max }{ϵ}_i^{(q,r)}({\theta }_i).$$ (III.19)

The following lemma shows that equation (III.14) is invertible when ε^(q,r) is sufficiently bounded.

Lemma III.5 (Proof in Section B). Suppose

$${ϵ}^{(1,1)}<1and$$ (III.20)

$$c≔{ϵ}^{(0,0)}+\frac{{ϵ}^{(1,0)}{ϵ}^{(0,1)}}{1-{ϵ}^{(1,1)}}<1.$$ (III.21)

Then the matrix in (III.14) is invertible and

$${\Vert \alpha \Vert }_{\infty }\le \frac{1}{1-c}$$ (III.22)

$${\Vert \beta \Vert }_{\infty }\le \frac{{ϵ}^{(0,1)}/(1-{ϵ}^{(1,1)})}{1-c}$$ (III.23)

$${\Vert \alpha -\xi \Vert }_{\infty }\le \frac{c}{1-c}.$$ (III.24)

To apply Lemma III.5 we must first bound ε^(q,r) for q, r ∈ {0, 1}. By the decay and separation conditions (Condition II.3 and Definition II.5), θ_j lies in the exponentially decaying tail of ρ_θi, for i ≠ j. This gives

$$\begin{gathered} {ϵ}_i^{(q,r)}({\theta }_i)=\sum _{j\ne i}|{\widehat{\rho }}_{{\theta }_j}^{(q,r)}({\theta }_i)| \\ \le C_{q,r}\sum _{{\theta }_j>{\theta }_i}\exp (-d({\theta }_i,{\theta }_j)) \end{gathered}$$ (III.25)

$$+C_{q,r}\sum _{{\theta }_j<{\theta }_i}\exp (-d({\theta }_i,{\theta }_j))$$ (III.26)

$$\le 2C_{q,r}\sum _{k=1}^{\infty }\exp (-k\mathrm{\Delta })$$ (III.27)

$$=\frac{2C_{q,r}{e}^{-\mathrm{\Delta }}}{1-e^{-\mathrm{\Delta }}}=C_{q,r}s,$$ (III.28)

where s ≔ 2e^−Δ/(1 − e^−Δ) and the distance d is defined in Definition II.5. As this bound is independent of i, we have

$${ϵ}^{(q,r)}\le C_{q,r}s$$ (III.29)

for q, r ∈ {0, 1}. In terms of the conditions of Lemma III.5, we obtain

$$\begin{gathered} c={ϵ}^{(0,0)}+\frac{{ϵ}^{(1,0)}{ϵ}^{(0.1)}}{1-{ϵ}^{(1,1)}} \\ \le C_{0,0}s+\frac{C_{1,0}{C}_{0,1}{s}^2}{1-C_{1,1}s} \\ =\frac{C_{0,0}s+(C_{1,0}{C}_{0,1}-C_{0,0}{C}_{1,1})s^2}{1-C_{1,1}s} \\ =\frac{C_{0,0}s}{1-C_{1,1}s}, \end{gathered}$$ (III.30)

assuming Equation (II.11) (C_1,0C_0,1 = C_0,0C_1,1) holds. Thus we have c < 1, as required by (III.21), when $s<\frac{1}{C_{0,0}+C_{1,1}}$. Using this, we can find conditions on Δ so that the hypotheses of Lemma III.5 hold. If Δ > log(1 + κ) for some κ > 0 then, using that f(x) = e^−x/(1 − e^−x) is decreasing for x > 0, we have

$$\frac{s}{2}=\frac{e^{-\mathrm{\Delta }}}{1-e^{-\mathrm{\Delta }}}<\frac{1/(1+\kappa )}{1-1/(1+\kappa )}=\frac{1}{\kappa }.$$ (III.31)

Letting κ = 2C_1,1 this shows that Δ > log(1 + 2C_1,1) implies

$${ϵ}^{(1,1)}\le C_{1,1}s=2C_{1,1}\frac{s}{2}<2C_{1,1}\frac{1}{2C_{1,1}}=1,$$ (III.32)

giving (III.20). For condition (III.21) we let κ = 2(C_0,0 + C_1,1). Then (III.31) shows that Δ > log(1 + 2(C_0,0 + C_1,1)) implies

$$s=2\frac{s}{2}<2\frac{1}{2(C_{0,0}+C_{1,1})}=\frac{1}{C_{0,0}+C_{1,1}}$$ (III.33)

as required. Thus when Δ > log(1 + 2(C_0,0 + C_1,1)) both conditions of Lemma III.5 hold. By plugging (III.29) and (III.30) into the bounds of Lemma III.5 we obtain Lemma III.4. This completes the proof of Lemma III.2.

D. Proof of Lemma III.3

Lemma III.2 implies that we can solve (III.4) and (III.5) for α and β and then obtain Q via (III.8). To prove Lemma III.3 we must show that |Q(θ)| < 1 for θ ∈ Θ^c. This is accomplished in two steps. We first show that |Q(θ)| < 1 for θ that are not in the near region of any ρ_θi, i = 1, …, k. Secondly, we show that for any θ in the near region of some ρ_θi we have Q⁽²⁾(θ) < 0, where we assume ξ_i = 1 without loss of generality. This proves that Q has a local maximum at θ_i, and is smaller than one nearby.

To bound Q and Q⁽²⁾ we apply the triangle inequality to (III.8), obtaining the following lemma.

Lemma III.6. Fix θ_i ∈ Θ and let Q be defined as in (III.8). For all $\theta \in \mathbb{R}$

$$\begin{gathered} |Q(\theta )|\le {\Vert \alpha \Vert }_{\infty }|{\widehat{\rho }}_{{\theta }_i}(\theta )|+{\Vert \beta \Vert }_{\infty }|{\widehat{\rho }}_{{\theta }_i}^{(1,0)}(\theta )| \\ +{\Vert \alpha \Vert }_{\infty }{ϵ}_i^{(0,0)}(\theta )+{\Vert \beta \Vert }_{\infty }{ϵ}_i^{(1,0)}(\theta ). \end{gathered}$$ (III.34)

If ${\widehat{\rho }}_{{\theta }_i}^{(0,2)}(\theta )\le 0$ and ξ ∈ {−1, +1}^k is the sign pattern interpolated by Q then

$$\begin{gathered} Q^{(2)}(\theta )\le (1-{\Vert \alpha -\xi \Vert }_{\infty }){\widehat{\rho }}_{{\theta }_i}^{(0,2)}(\theta )+{\Vert \beta \Vert }_{\infty }|{\widehat{\rho }}_{{\theta }_i}^{(1,2)}(\theta )| \\ +{\Vert \alpha \Vert }_{\infty }{ϵ}_i^{(0,2)}(\theta )+{\Vert \beta \Vert }_{\infty }{ϵ}_i^{(1,2)}(\theta ). \end{gathered}$$ (III.35)

In the next lemma we show that the separation conditions in Definition II.5 ensure that the support does not cluster together. We assume that θ₁ < θ₂ < … < θ|_Θ|.

Lemma III.7 (Proof in Section C). Fix θ_i ∈ Θ and let θ > θ_i. If i ≤ k − 1, assume that

$$\frac{\theta -D_i^{+}}{{\sigma }_i}\le \frac{D_{i+1}^{-}-\theta }{{\sigma }_{i+1}}.$$ (III.36)

Then

$$\frac{D_j^{-}-\theta }{{\sigma }_j}\ge \frac{\mathrm{\Delta }}{2}ifj=i+1,$$ (III.37)

$$\frac{D_j^{-}-\theta }{{\sigma }_j}\ge \mathrm{\Delta }(j-(i+1))ifj>i+1,$$ (III.38)

$$\frac{\theta -D_j^{+}}{{\sigma }_j}\ge \mathrm{\Delta }(i-j)ifj<i,$$ (III.39)

where Δ is defined in Definition II.5, as long as Conditions II.1 to II.3 and (II.11) hold.

Inequality (III.36) implies that θ_i is the closest element of Θ to θ, if we use the generalized distance normalized by σ.

Bounding |Q(θ)| Outside the Near Region

Our goal is to bound |Q(θ)| for $\theta \in \mathbb{R}$ that are not in the near region of any ρθ_i. We can assume, flipping the axis if necessary, that the conditions of Lemma III.7 hold for the θ_i closest to θ and that $\theta \ge N_i^{+}$ (recall that $N_i^{+}={\theta }_i+N_i$ is the boundary of the near region of ρθ_i). Then θ lies in the exponentially decaying tail of ρ_θj for j ≠ i, so that

$$\begin{gathered} {ϵ}_i^{(q,r)}(\theta )=\sum _{j\ne i}{\widehat{\rho }}_{{\theta }_j}^{(q,r)}(\theta ) \\ \le C_{q,r}\exp (-(D_{i+1}^{-}-\theta )/{\sigma }_{i+1}) \\ +\sum _{j<i}{C}_{q,r}\exp (-(\theta -D_j^{+})/{\sigma }_j) \end{gathered}$$ (III.40)

$$+\sum _{j>i+1}{C}_{q,r}\exp (-(D_j^{-}-\theta )/{\sigma }_j)$$ (III.41)

$$\le C_{q,r}{e}^{-\mathrm{\Delta }/2}+2C_{q,r}\sum _{k=1}^{\infty }{e}^{-k\mathrm{\Delta }}$$ (III.42)

$$=C_{q,r}(x+s),$$ (III.43)

where x = e^−Δ/2 and s is defined in Lemma III.4. Plugging this into (III.34) and combining with Lemma III.4 yields

$$\begin{gathered} |Q(\theta )|\le \frac{1-C_{1,1}s}{1-(C_{0,0}+C_{1,1})s}(|{\widehat{\rho }}_{{\theta }_i}(\theta )|+C_{0,0}(x+s)) \\ +\frac{C_{0,1}s}{1-(C_{0,0}+C_{1,1})s}\left(|{\widehat{\rho }}_{{\theta }_i}^{(1,0)}(\theta )|+C_{1,0}(x+s)\right) \end{gathered}$$ (III.44)

$$\begin{gathered} \le \frac{1-C_{1,1}s}{1-(C_{0,0}+C_{1,1})s}({\gamma }_2+C_{0,0}(x+s)) \\ +\frac{C_{0,1}s}{1-(C_{0,0}+C_{1,1})s}({\gamma }_3+C_{1,0}(x+s)). \end{gathered}$$ (III.45)

Solving for |Q(θ)| < 1 we obtain

$$\begin{gathered} (1-C_{1,1}s)({\gamma }_2+C_{0,0}(x+s)) \\ +(C_{0,1}s)({\gamma }_3+C_{1,0}(x+s))<1-(C_{0,0}+C_{1,1})s. \end{gathered}$$ (III.46)

Isolating s and x,

$$s(2C_{0,0}+C_{1,1}-C_{1,1}{\gamma }_2+C_{0,1}{\gamma }_3)+x{C}_{0,0}<1-{\gamma }_2,$$ (III.47)

where we apply Equation (II.11) (C_1,0C_0,1 = C_0,0C_1,1) to cancel terms. Since $s=\frac{2x^2}{1-x^2}$ we obtain the inequality

$$\frac{2x^2}{1-x^2}a+xb<c$$ (III.48)

where a ≔ 2C_0,0 + C_1,1 − C_1,1γ₂ + C_0,1γ₃ > 0, b ≔ C_0,0, and c ≔ 1 − γ₂ > 0. The following lemma shows that this inequality is satisfied by our assumptions, completing this part of the proof.

Lemma III.8 (Proof in Section D). Let a, b, c, Δ > 0, and x ≔ e^−Δ/2. Then the inequality

$$\frac{2x^2}{1-x^2}a+xb<c$$ (III.49)

is satisfied when

$$\mathrm{\Delta }>2\log \left(\frac{2(2a+c)}{-b+\sqrt{b^2+4(2a+c)c}}\right).$$ (III.50)

Bounding Q⁽²⁾(θ) in the Near Region

For the final part of the proof we must prove Q⁽²⁾(θ) < 0 for θ in the near region of some θ_i with ξ_i = 1. Since the interpolation equations (III.4) and (III.5) guarantee that Q(θ_i) = 1, strict concavity implies that Q has a unique maximum on $[N_i^{-},N_i^{+}]$ thus establishing that Q(θ) < 1 for $\theta \in [N_i^{-},N_i^{+}]\{{\theta }_i\}$. We cannot have Q(θ) ≤ −1 on the near region since we already showed that |Q(θ)| < 1 outside the near region. We can assume, without loss of generality, that the conditions of Lemma III.7 hold for some i and that $\theta \le N_i^{+}$. By the same argument used to obtain Equation (III.43), we have

$${ϵ}_i^{(q,r)}(\theta )\le C_{q,r}(x+s).$$ (III.51)

Plugging this into (III.34) and applying Lemma III.4 we obtain

$$Q^{(2)}(\theta )\le \frac{1-(2C_{0,0}+C_{1,1})s}{1-(C_{0,0}+C_{1,1})s}{\widehat{\rho }}_{\theta {}_i}^{(0,2)}(\theta )$$ (III.52)

$$\begin{gathered} +\frac{1-C_{1,1}s}{1-(C_{0,0}+C_{1,1})s}(C_{0,2}(x+s)) \\ +\frac{C_{0,1}s}{1-(C_{0,0}+C_{1,1})s}\left(|{\widehat{\rho }}_{{\theta }_i}^{(1,2)}(\theta )|+C_{1,2}(x+s)\right) \end{gathered}$$ (III.53)

$$\begin{gathered} \le \frac{1-(2C_{0,0}+C_{1,1})s}{1-(C_{0,0}+C_{1,1})s}(-{\gamma }_0) \\ +\frac{1-C_{1,1}s}{1-(C_{0,0}+C_{1,1})s}(C_{0,2}(x+s)) \\ +\frac{C_{0,1}s}{1-(C_{0,0}+C_{1,1})s}({\gamma }_1+C_{1,2}(x+s)), \end{gathered}$$ (III.54)

where ${\widehat{\rho }}_{{\theta }_i}^{(0,2)}(\theta )\le 0$ by Condition II.1. Solving for Q⁽²⁾(θ) < 0 yields

$$-(1-(2C_{0,0}+C_{1,1})s){\gamma }_0+$$ (III.55)

$$(1-C_{1,1}s)C_{0,2}(x+s)+C_{0,1}s({\gamma }_1+C_{1,2}(x+s))<0.$$ (III.56)

Grouping into terms involving s and x we obtain

$$s((2C_{0,0}+C_{1,1}){\gamma }_0+C_{0,2}+C_{0,1}{\gamma }_1)+C_{0,2}x<{\gamma }_0,$$ (III.57)

where we apply Equation (II.11) (C_0,1C_1,2 = C_0,2C_1,1) to cancel terms. Letting a ≔ (2C_0,0 + C_1,1)γ₀ + C_0,2 + C_0,1γ₁ > 0, b ≔ C_0,2, and c ≔ γ₀ > 0 we obtain an inequality of the form (III.49). Applying Lemma III.8 completes the proof.

E. Extensions to Higher Dimensions

In this section we briefly describe an extension of our dual certificate construction to settings where the parameter space has dimension p > 1. We leave a more detailed analysis to future work. In higher dimensions, one can build the interpolating function Q by setting

$$Q(\eta )≔\sum _{i=1}^k\left[{\alpha }_i{\rho }_{{\theta }_i}(\eta )+\sum _{l=1}^p{\beta }_{i,l}\frac{{\partial }_{1,l}{\rho }_{{\theta }_i}(\eta )}{{\Vert {\partial }_l\overrightarrow{\phi }({\theta }_i)\Vert }_2^2}\right],$$ (III.58)

where $\alpha \in {\mathbb{R}}^k$, $\beta \in {\mathbb{R}}^{k\times p}$, ∂_l denotes the partial derivative with respect to the 1th coordinate, and ∂_1,lρθ(η) denotes the partial derivative of ρ_θ(η) with respect to the lth coordinate of θ. The coefficients are chosen so that Q satisfies the analog of the interpolation Equations (III.4) and (III.5),

$$Q({\theta }_i)={\xi }_i,$$ (III.59)

$$\nabla Q({\theta }_i)=0.$$ (III.60)

In Figure 16 we show an example of an interpolating function Q for the electroencephalography brain-activity localization problem. The interpolating function is associated to the signal with three active sources of brain activity from Figure 2. To control Q for problems with correlation decay, one can extend the correlation-decay conditions in Section II-A to higher dimensions by requiring analogous bounds on the first and second partial derivatives. For example, the local quadratic bound in Equation (II.2) becomes a bound on the eigenvalues of the Hessian of ρ_θ. Similar conditions have been used in [51], [69] to obtain recovery guarantees for the super-resolution and deconvolution problems in two dimensions.

Fig. 16:

Interpolation function Q associated to a valid dual certificate for the electroencephalography brain-activity localization problem. The dual certificate is associated to a signal consisting of the three sources of brain activity shown in Figure 2. Q interpolates the sign pattern of the signal, which equals +1 on θ₁ and θ₃, and −1 on θ₂. The certificate is built using the multidimensional extension of our proposed interpolation technique described in Section III-E.

IV. Numerical Experiments

A. Heat-source localization

Our theoretical results establish that convex programming yields exact recovery in parameter-estimation problems that have correlation decay. In this section we investigate this phenomenon numerically for a heat-source localization problem in one dimension. The heat sources are modeled as a collection of point sources,

$$\mu ≔\sum _{{\theta }_i\in \mathrm{\Theta }}{c}_i{\delta }_{{\theta }_i},$$ (IV.1)

where Θ is a finite number of points in the unit interval and $c_i\in \mathbb{R}$ for i = 1, …, k.

The heat u(θ,t) at position θ and time t is assumed to evolve following the heat equation with Neumann boundary conditions (see e.g. [9]),

$$\frac{\partial }{\partial t}u(\theta ,t)=\frac{\partial }{\partial \theta }\left(c(\theta )\frac{\partial }{\partial \theta }u(\theta ,t)\right),$$ (IV.2)

$$\frac{\partial }{\partial \theta }u(-0.5,t)=\frac{\partial }{\partial \theta }u(0.5,t)=0$$ (IV.3)

on the unit interval, where c(θ) represents the conductivity of the medium at θ (see Figure 17). The data are heat measurements u(j, T), where j is sampled on a regular grid of 100 points at a fixed time T ≔ 10⁻⁴. Our goal is to recover the initial heat distribution at t = 0 represented by the heat sources μ. This is an SNL problem where $\overrightarrow{\phi }(\theta )$ corresponds to the measurements caused by a source located at θ.

Fig. 17:

The conductivity function c(θ) used in the experiment described in Section IV-A. Heat diffuses faster in the central region, where the conductivity is higher.

If the conductivity is constant, then $\overrightarrow{\phi }(\theta )$ has a Gaussian-like shape with fixed width sampled on the measurement grid, and the correlation function ρ_θ is also shaped like a Gaussian with fixed width. In this example, the conductivity varies (see Figure 17), which results in correlations that are still mostly Gaussian-like but have very different widths (see Figure 18). The measurement operator therefore has nonuniform correlation decay properties. According to the theory presented in Section II-C, we expect convex programming to recover the heat-source location as long as they are separated by a minimum separation that takes into account the structure of the correlation decay. To verify whether this is the case we consider two different separation measures. The first just measures the minimum separation between the sources,

$${\mathrm{\Delta }}_{\mathrm{sep}}≔\underset{i\ne j}{\min }\left|{\theta }_i-{\theta }_j\right|.$$ (IV.4)

Fig. 18:

Examples of the correlation functions ρ_θ1, …, ρ_θ7 for heat sources with different separations and configurations. The sources used in the top row of figures are uniformly spaced on the unit interval. The heat sources in the bottom row have one central source, and two clusters of three sources located at the ends. From left to right the supports are dilated by different factors. This reduces the separation between the heat sources but maintains their relative position.

The second takes into account the correlation function of the SNL problem,

$${\mathrm{\Delta }}_{\mathrm{corr}}≔\underset{i\ne j}{\min }\frac{\left|{\theta }_i-{\theta }_j\right|}{\max ({\sigma }_i,{\sigma }_j)}.$$ (IV.5)

where σ_i, 1 ≤ i ≤ k, is the standard deviation of the best Gaussian upper bound on the correlation function centered at θ_i,

$${\sigma }_i≔\inf \{s>0|{\rho }_{{\theta }_i}(\theta )<e^{-{({\theta }_i-\theta )}^2/(2s^2)}\text{for all}\theta .\}$$ (IV.6)

The question is which of these separations characterizes the performance of the convex-programming approach more accurately. We consider two different heat-source configurations: one where the sources are uniformly spaced and another where they are clustered, as depicted in Figure 18. In the clustered configuration, groups of spikes are placed in regions of low conductivity where each ρ_θi exhibits sharp decay. This creates a noticeable discrepancy between the two measures of separation. We expect exact recovery to occur for much smaller values of Δ_sep in the clustered case than it does in the uniform case since it doesn’t account for the correlation decay. In contrast, we expect Δ_corr to have similar thresholds for exact recovery in both cases, as it captures the inherent difficulty in the problem. To recover these sources we solve the discretized ℓ₁-norm minimization problem described in Section II-E using CVX [88]. The measurement matrix is computed by solving the differential equation on a grid of 10³ points in the parameter space.

Figure 19 shows the results. For both types of support, we observe that exact recovery occurs as long as the minimum separation Δ_sep between sources is larger than a certain value. However, this value is very different for the two types of support. It is much smaller for the clustered support. Intuitively, this is due to the fact that the clustered sources are mostly located in the region where the conductivity is lower, and consequently the correlation decay is much faster. Our theory suggests that exact recovery will occur at smaller separations for such configurations, which is what we observe. Quantifying separation using Δ_corr accounts for the variation in correlation decay, resulting in a similar phase transition for both types of support. This is consistent with the theory for SNL problems with nonuniform correlation decay developed in Section II-C.

Fig. 19:

The figure shows the performance of heat-source localization based on convex programming for the experiment described in Section IV-A. The upper row shows the result for sources with uniformly spaced supports, whereas the bottom row shows results for sources with clustered supports (see Figure 18). The left column plots the results with respect to the minimum separation Δ_sep. The right column plots the results in terms of the correlation-aware separation Δ_corr. Recovery succeeds if the relative ℓ₂-norm recovery error is smaller than 3 · 10⁻⁵. All failures have an absolute error above 9 · 10⁻³, and most have errors above 1. For comparison the ℓ₂ norm of the true signal equals $\sqrt{7}$ (≈ 2.65) in all cases.

B. Estimation of Brain Activity via Electroencephalography

In this section we consider the SNL problem of brain-activity localization from electroencephalography (EEG) data. Areas of focalized brain activity are usually known as sources. In EEG, as well as in magnetoencephalography, sources are usually modeled using electric dipoles, represented mathematically as point sources or Dirac measures at the corresponding locations [89]. EEG measurements are samples of the electric potential on the surface of the head obtained using an array of electrodes. The mapping from the source positions and amplitudes to the EEG data is governed by Poisson’s equation. For a fixed model of the head geometry– obtained for example from 3D images acquired using magnetic-resonance imaging– the mapping can be computed by solving the differential equation numerically. By linearity, this corresponds to an SNL model where θ represents the position of a dipole and φ_t (θ) is the value of the corresponding electrical potential at a location t of the scalp.

To simulate realistic EEG data with associated ground-truth source positions, we use Brainstorm [90], an open-source software for the analysis of EEG and other electrophysiological recordings. We use template ICBM152, which is a nonlinear average of 152 3D magnetic-resonance image scans [91], to model the geometry of the brain and head. The sources are modeled as electrical dipoles situated on the cortical surface, discretized as a tesselated grid with 15,000 points. The orientation of the dipoles is assumed to be perpendicular to the cortex. The data-acquisition sensor array has 256 channels (HydroCel GSN, EGI). The data corresponding to each point on the grid are simulated numerically using the open-source software OpenMEEG [92]. The computation assumes realistic electrical properties for the brain, skull and scalp. As a result, we obtain a 256 × 15,000 measurement operator that can be used to generate data corresponding to different combinations of sources. Figure 2 shows three examples.

The correlation structure of the EEG linear measurement operator is depicted in Figure 6. The complex geometric structure of the folds in cortex induce an intricate correlation structure close to the source location. However, the correlation presents a clear correlation decay; points that are far enough from the source have low correlation values. Our theoretical results consequently suggest that ℓ₁-norm minimization should succeed in recovering superpositions of sources that are sufficiently separated. This is supported by the fact that for such superpositions, we can build valid dual certificates as illustrated in Figure 16.

In order to test our hypothesis, we randomly choose superpositions of three sources located at approximately the same distance from each other, for a range of distances. For each superposition we generate 8 different measurement vectors by assigning every possible combination of positive and negative unit-norm amplitudes to each source. We then estimate the source locations by solving an ℓ₁-norm minimization problem with equality constraints using CVX [88]. Figure 20 shows the results. When the separation is sufficiently large, exact recovery indeed occurs for all possible patterns of positive and negative amplitudes, as expected from our theoretical analysis. As the separation decreases, recovery fails for some of the patterns, as shown in the image on the right.

Fig. 20:

Result of the brain-source localization experiment described in Section IV-B. The horizontal axis indicates the distance between the sources. In the left image success is declared if the three sources are accurately recovered for any pattern of positive and negative amplitudes. In the right image success is declared if the sources are recovered for all patterns of positive and negative amplitudes. Recovery succeeds if the relative ℓ₂-norm recovery error is smaller than 10⁻⁴. All failures have an absolute error above 1.18. For comparison the ℓ₂ norm of the true signal equals $\sqrt{3}$ (≈ 1.73) in all cases.

V. Conclusion and Future Work

In this work, we establish that deterministic SNL problems can be solved via a tractable convex optimization program, as long as the parameters have a minimum separation in the parameter space with respect to the correlation structure of the measurement operator. As mentioned in Sections II-D, our results can be used to establish some robustness guarantees at low-noise levels in terms of support recovery. Deriving more precise stability analysis in the spirit of [93] is an interesting research direction. Another interesting open problem is characterizing the performance of reweighting techniques, which are commonly applied in practice to enhance solutions obtained from ℓ₁-norm minimization [16], [94].

A drawback of the sparse-recovery framework discussed in this paper is computational complexity: for SNL problems in two or three dimensions it is very computationally expensive to solve ℓ₁ -norm minimization problems, even if we discretize the domain. In the last few years, algorithms that minimize nonconvex cost functions directly via gradient descent have been shown to provable succeed in solving underdetermined inverse problems involving randomized measurements [95], [96]. As illustrated in Figure 3, nonlinear least-squares cost functions associated to deterministic SNL problems often have non-optimal local minima. An intriguing question is how to design cost functions for deterministic SNL problems that can be tackled directly by descent methods.

Another computationally efficient alternative is to perform recovery using machine learning. Recent works suggest calibrating a feedforward network to output the model parameters using a training set of examples, with applications in point-source deconvolution [97] and super-resolution of line spectra [98]. Understanding under what conditions such techniques can be expected to yield accurate estimates is a challenging question for future research.

Throughout the appendix, we assume there is some compact interval $I\subset \mathbb{R}$ containing the support Θ of the true measure μ. In problem (I.8), the variable $\tilde{\mu }$ takes values in the set of finite signed Borel measures supported on I.

Fig. 12:

Illustration of the generalized support-separation condition in Definition II.5.

Biographies

Brett Bernstein is a Ph.D. student in Mathematics at the Courant Institute of Mathematical Sciences supervised by Carlos Fernandez-Granda. The goal of his research is to improve the algorithms, techniques, and workflows used in statistical machine learning and the study of inverse problems.

Sheng Liu is a Ph.D. student in Data Science at the NYU Center for Data Science supervised by Carlos Fernandez-Granda. The goal of his research is to improve the methodology used in machine learning and the study of inverse problems as well as their applications in healthcare.

Chrysa Papadaniil received a Ph.D. degree in electrical engineering from the Aristotle University of Thessaloniki in 2016. She was a Postdoctoral Researcher at the Center for Neural Science in New York University from 2016 to 2019. She is currently a Research Scientist at the Center for Brain Imaging in New York University. Her research focuses on brain-imaging modalities such as fMRI, EEG, MEG, and TMS.

Carlos Fernandez-Granda received a Ph.D. degree in electrical engineering from Stanford University in 2014. In 2015 he spent a year at Google as a Postdoctoral Researcher. He is currently an Assistant Professor of Mathematics and Data Science at the Courant Institute of Mathematical Sciences and the Center for Data Science at NYU. The goal of his research is to design new methodology for inverse problems drawing from optimization, machine learning and statistics, with a focus on signal processing and healthcare, as well as to analyze these methods mathematically in order to characterize their potential and limitations.

Appendix A

Proof of Proposition III.1

The following proof is standard, but given here for completeness.

: Let ν be feasible for problem (I.8) and define h = ν − μ. By taking the Lebesgue decomposition of h with respect to |μ| we can write

$$h=h_{\mathrm{\Theta }}+h_{{\mathrm{\Theta }}^c},$$ (A.1)

where h_Θ is absolutely continuous with respect to |μ|, and h_Θ^c, |μ| are mutually singular. In other words, the support of h_Θ is contained in Θ, and h_Θ^c(Θ) = 0. This allows us to write

$$h_{\mathrm{\Theta }}=\sum _{t_j\in T}{b}_j{\delta }_{t_j,}$$ (A.2)

for some $b\in {\mathbb{R}}^{\mid ϴ\mid }$. Set ξ ≔ sign(b), where we arbitrarily choose ξ_j = ±1 if b_j = 0. By assumption there exists a corresponding Q interpolating ξ on Θ. Since μ and ν are feasible for problem (I.8) we have ∫ φ(θ) dh(θ) = 0. This implies

$$0=q^T\int \phi (\theta )dh(\theta )$$ (A.3)

$$=\int Q(\theta )dh(\theta )$$ (A.4)

$$={\Vert h_{\mathrm{\Theta }}\Vert }_{\text{TV}}+\int Q(\theta )d{h}_{{\mathrm{\Theta }}^c}(\theta ).$$ (A.5)

Applying the triangle inequality, we obtain

$$\begin{gathered} {\Vert \nu \Vert }_{\text{TV}}={\Vert \mu +h_{\mathrm{\Theta }}\Vert }_{\text{TV}}+{\Vert h_{{\mathrm{\Theta }}^c}\Vert }_{\text{TV}} \\ (\text{Mutually Singular}) \end{gathered}$$ (A.6)

$$\begin{gathered} \ge {\Vert \mu \Vert }_{\text{TV}}+{\Vert h_{{\mathrm{\Theta }}^c}\Vert }_{\text{TV}}-{\Vert h_{\mathrm{\Theta }}\Vert }_{\text{TV}} \\ (\text{Triangle Inequality}) \end{gathered}$$ (A.7)

$$={\Vert \mu \Vert }_{\text{TV}}+{\Vert h_{{\mathrm{\Theta }}^c}\Vert }_{\text{TV}}+\int Q(\theta )d{h}_{{\mathrm{\Theta }}^c}(\theta )$$ (A.8)

$$(\text{Equation}(\mathrm{A.3}))\ge {\Vert \mu \Vert }_{\text{TV}}(|Q(\theta )|\le 1),$$ (A.9)

where the last inequality is strict if ‖h_Θ^c‖_TV > 0 since |Q(θ)| < 1 for θ ∈ Θ^c. This establishes that μ is optimal for problem (I.8), and that any other optimal solution must be supported on Θ. Equation (A.3) implies that any feasible solution supported on Θ must be equal to μ, (since ‖h_Θ‖_TV = 0), completing the proof of uniqueness. ■

Appendix B

Proof of Lemma III.5

: For any matrix $A\in {\mathbb{R}}^{n\times n}$ such that ‖A‖_∞ < 1 the Neumann series ${\Sigma }_{j=0}^{\infty }{A}^j$ converges to ${(\mathcal{I}-A)}^{-1}$, which implies that $\mathcal{I}-A$ is invertible [99]. By the triangle inequality and the submultiplicativity of the ∞-norm, this gives

$${\Vert {(\mathcal{I}-A)}^{-1}\Vert }_{\infty }\le \sum _{j=0}^{\infty }{\Vert A\Vert }_{\infty }^j=\frac{1}{1-\Vert A_{\infty }\Vert }.$$ (B.1)

Setting A = −P^(1,1) and applying ‖P^(1,1)‖_∞ = ε^(1,1) < 1 proves $\mathcal{I}+P^{(1,1)}$ is invertible. Let $\mathcal{C}$ be the Schur complement of $\mathcal{I}+P^{(1,1)}$ in (III.14) so that

$$\mathcal{C}=\mathcal{I}+P^{(0,0)}-P^{(1,0)}{(\mathcal{I}+P^{(1,1)})}^{-1}{P}^{(0,1)}.$$ (B.2)

By the triangle inequality and (B.1) applied with A = −P^(1,1) we obtain

$${\Vert \mathcal{I}-\mathcal{C}\Vert }_{\infty }\le {ϵ}^{(0,0)}+\frac{{ϵ}^{(1,0)}{ϵ}^{(0,1)}}{1-{ϵ}^{(1,1)}}=c<1,$$ (B.3)

proving $\mathcal{C}$ is invertible. As both $\mathcal{I}+P^{(1,1)}$ and its Schur complement $\mathcal{C}$ are invertible, the matrix in (III.14) is also invertible (see e.g. [100]), which establishes the first claim.

By applying blockwise Gaussian elimination we solve (III.14) in terms of $\mathcal{C}$ to obtain

$$\alpha ={\mathcal{C}}^{-1}\xi$$ (B.4)

$$\beta =-{(\mathcal{I}+P^{(1,1)})}^{-1}{P}^{(0,1)}\alpha .$$ (B.5)

Applying (B.1) and noting that ‖ξ‖_∞ = 1 we obtain the required bounds on ‖α‖_∞ and ‖β‖_∞. Finally,

$$\Vert \mathcal{I}-\mathcal{C}\Vert \alpha =\alpha -\xi ,$$ (B.6)

which implies ‖α − ξ‖∞ ≤ c ‖α‖_∞ and completes the proof.

Appendix C

Proof of Lemma III.7

: Throughout we use the fact that Θ having separation Δ > 0 (Definition II.5) implies $D_i^{+}\le D_j^{-}$ for j > i. When i ≤ k − 1, (III.36) implies that

$$\theta \le \frac{{\sigma }_i{D}_{i+1}^{-}+{\sigma }_{i+1}{D}_i^{+}}{{\sigma }_i+{\sigma }_{i+1}},$$

where the righthand side is the σ-weighted average of $D_i^{+}$ and $D_{i+1}^{-}$. If $\theta \ge D_i^{+}$ this gives

$$\frac{D_{i+1}^{-}-\theta }{{\sigma }_{i+1}}\ge \frac{D_{i+1}^{-}-D_i^{+}}{{\sigma }_i+{\sigma }_{i+1}}\ge \frac{\text{max}({\sigma }_i,{\sigma }_{i+1})\mathrm{\Delta }}{{\sigma }_i+{\sigma }_{i+1}}\ge \frac{\mathrm{\Delta }}{2}.$$

If $\theta <D_i^{+}$ then

$$\frac{D_{i+1}^{-}-\theta }{{\sigma }_{i+1}}>\frac{D_{i+1}^{-}-D_i^{+}}{{\sigma }_{i+1}}\ge \frac{D_{i+1}^{-}-D_i^{+}}{\text{max}({\sigma }_i,{\sigma }_{i+1})}\ge \mathrm{\Delta }>\frac{\mathrm{\Delta }}{2},$$ (C.1)

by the separation conditions, proving (III.37). If i + 1 < j ≤ ‖Θ‖ then d(θ_i+1, θ_j) ≥ Δ(j − (i + 1)), so that

$$\frac{D_j^{-}-\theta }{{\sigma }_j}\ge \frac{D_j^{-}-D_{i+1}^{+}}{{\sigma }_j}\ge \frac{D_j^{-}-D_{i+1}^{+}}{\text{max}({\sigma }_j,{\sigma }_{i+1})}\ge \mathrm{\Delta }(j-(i+1)),$$ (C.2)

by the separation conditions applied to θ_j and θ_i+1, which implies (III.38). Finally, if j < i then

$$\frac{\theta -D_j^{+}}{{\sigma }_j}\ge \frac{D_i^{-}-D_j^{+}}{{\sigma }_j}\ge \frac{D_i^{-}-D_j^{+}}{\text{max}({\sigma }_j,{\sigma }_i)}\ge \mathrm{\Delta }(i-j),$$ (C.3)

by the separation conditions applied to θ_j and θ_i, which establishes (III.39).

Appendix D

Proof of Lemma III.8

: Multiplying through by 1 − x² (which is positive by assumption) in (III.49) we obtain

$$-x^{3}b+(2a+c)x^2+xb-c<0.$$ (D.1)

The above inequality is implied by the simpler quadratic inequality

$$(2a+c)x^2+xb-c<0,$$ (D.2)

where the omitted term −x³b is always negative. Since the inequality is satisfied for x = 0 we obtain the condition

$$0<x<\frac{-b+\sqrt{b^2+4(2a+c)c}}{2(2a+c)}.$$ (D.3)

Translating this to a statement on Δ we obtain

$$\mathrm{\Delta }>2\log \left(\frac{2(2a+c)}{-b+\sqrt{b^2+4(2a+c)c}}\right).$$ (D.4)