Multi-target Detection with an Arbitrary Spacing Distribution

Abstract

Motivated by the structure reconstruction problem in single-particle cryo-electron microscopy, we consider the multi-target detection model, where multiple copies of a target signal occur at unknown locations in a long measurement, further corrupted by additive Gaussian noise. At low noise levels, one can easily detect the signal occurrences and estimate the signal by averaging. However, in the presence of high noise, which is the focus of this paper, detection is impossible. Here, we propose two approaches—autocorrelation analysis and an approximate expectation maximization algorithm—to reconstruct the signal without the need to detect signal occurrences in the measurement. In particular, our methods apply to an arbitrary spacing distribution of signal occurrences. We demonstrate reconstructions with synthetic data and empirically show that the sample complexity of both methods scales as SNR⁻³ in the low SNR regime.

I. Introduction

We consider the multi-target detection (MTD) problem [10] to estimate a signal $x\in {ℝ}^L$ from a long, noisy measurement

$$y=s*x+\epsilon ,\epsilon ~\mathcal{N}\left(0,{\sigma }^2{I}_N\right),$$ (1)

where $y\in {ℝ}^N$ is the linear convolution of an unknown binary sequence s ∈ {0, 1}^N−L+1 with the signal, further corrupted by additive white Gaussian noise with zero mean and variance σ², and we assume L ≪ N. Both x and s are treated as deterministic variables. The signal length L and the noise variance σ² are assumed to be known. The non-zero entries of s indicate the starting positions of the signal occurrences in y. We require the signal occurrences not to overlap, so consecutive non-zero entries of s are separated by at least L positions. Figure 1 gives an example of the measurement y that contains three signal occurrences at different noise levels.

Fig. 1.

An example of a measurement in the MTD model corrupted by additive Gaussian noise with (a) σ =0, (b) σ =0.1 (high SNR regime) and (c) σ =2 (low SNR regime).

MTD belongs to the wider class of blind deconvolution problems [25], which have applications in astronomy [24], [33], microscopy [28], [32], system identification [1], and motion deblurring [17], [26], among others. The main difference is that we are only interested in estimating the signal x: we treat s as a nuisance variable, while most of the literature on blind deconvolution aims to estimate both x and s. This distinction allows us to estimate the signal at higher noise levels, which are usually not addressed in the literature on blind deconvolution, specifically because s cannot be accurately estimated in such regimes. We give a theoretical argument for this claim below, and corroborate it with numerical experiments in Section IV.

This high-noise MTD model has been studied in [9], [10] under the assumption that the signal occurrences either are well separated or follow a Poisson distribution. In applications, however, the signal occurrences in the measurement might be arbitrarily close to each other. In this study, we extend the framework of [9], [10] to allow an arbitrary spacing distribution of signal occurrences by simultaneously estimating the signal and the distance distribution between consecutive signal occurrences.

The solution of the MTD problem is straightforward at high signal-to-noise ratio (SNR), such as the case shown in Figure 1(b). The signal can be estimated by first detecting the signal occurrences and then averaging. In the low SNR regime however, this simple method becomes problematic due to the difficulty of detection, as illustrated by Figure 1(c). More than difficulty, reliable detection becomes impossible beyond a critical noise level. This can be understood from [9, Proposition 3.1], which we reproduce here (the proof is based on the Neyman–Pearson lemma).

Proposition 1:

Consider two known vectors θ₀ = x and θ₁ = 0 in ${ℝ}^L$ and a random variable η taking value 0 or 1 with equal probability. We observe the random vector $X\in {ℝ}^L$ with the following distribution:

$$X~\{\begin{array}{l}\mathcal{N}\left({\theta }_0,{\sigma }^2{I}_L\right)\text{if}\eta =0\\ \mathcal{N}\left({\theta }_0,{\sigma }^2{I}_L\right)\text{if}\eta =1.\end{array}$$

For any deterministic estimator $\widehat{\eta }$ of η,

$$\underset{\sigma \to \infty }{\text{lim}}p\left(\widehat{\eta }=\eta \right)\le \frac{1}{2},$$

that is, the probability of success for the best deterministic estimator of η is no better than a random guess in the limit of σ → ∞.

In our context, this simple fact can be interpreted as follows: even if we know the true signal x and an oracle provides windows of length L in the observation y which, with equal probability, contain either the noisy signal or pure noise, deciding whether a given window contains one or the other cannot be done significantly better than a random chance decision at high noise level. Results similar in spirit (but distinct in mathematical tools) are derived in [3].

In this work, we suggest to circumvent the estimation of s by estimating the signal x directly. Specifically, we propose two different approaches—autocorrelation analysis and an approximate version of the expectation maximization (EM) algorithm [18]. Since these methods do not require estimating s, Proposition 1 does not limit their performance in the low SNR regime.

Autocorrelation analysis relates the autocorrelations calculated from the noisy measurement to the signal. The signal is then estimated by fitting the autocorrelations through least squares. This approach is efficient as it requires only one pass over the data to calculate the autocorrelations; this is of particular importance as the data size grows.

In the second approach, the approximate EM algorithm, the signal is reconstructed by iteratively maximizing the data likelihood, which marginalizes over s; importantly, it does not estimate it explicitly. In contrast to autocorrelation analysis, the approximate EM algorithm scans through the whole dataset in each iteration, and hence requires much longer computational time.

In this study, we demonstrate the reconstruction of the underlying signal from the noisy measurement using the two proposed approaches. Our numerical experiments show that the approximate EM algorithm provides slightly more accurate estimates of the signal in the low SNR regime, whereas autocorrelation analysis is considerably faster, especially at low SNR. It is empirically shown that the sample complexity of both approaches scales as SNR⁻³ at low SNR, with details discussed later in the text. In the high SNR regime, the sample complexity of both methods scales as SNR⁻¹, the same as the sample complexity of the simple method that estimates the signal by first detecting the signal occurrences and then averaging.

The main contributions of this work are as follows.

1. We formalize MTD for an arbitrary signal spacing distribution, making it a more realistic model for applications.

2. We propose two algorithms to solve the MTD problem. In particular, our algorithm based on autocorrelations illustrates why, to recover x, we need not estimate all of s, but rather only a concise summary of it; and why it is possible, in principle at least, to solve this problem for arbitrary noise levels (given a sufficiently long observation). For our second algorithm, we note that the popular EM method is intractable, but we show how to implement an approximation of it, which performs well in practice.

3. For both algorithms, we design a coarse-to-fine multi-resolution scheme to alleviate issues pertaining to non-convexity. This is related to the ideas of frequency marching which are often used in cryo-electron microscopy (cryo-EM) [6], [34].

II. Autocorrelation analysis

In what follows, we discuss autocorrelations of both the signal x (of length L) and the measurement y (of length N). To keep notation general, we here consider a sequence z of length m, and define its autocorrelations of order q = 1, 2, … for any integer shifts l₁, l₂, …, l_q−1 as

$$a_z^q\left[l_1,l_2,\dots ,l_{q-1}\right]=\frac{1}{m}\sum _{i=0}^{m-1}z\left[i\right]z\left[i+l_1\right]\cdots z\left[i+l_{q-1}\right],$$

where z is zero-padded for indices out of the range [0, m−1]. We have m = L when z represents the signal x and m = N when z represents the measurement y. Since the autocorrelations only depend on the differences of the integer shifts and are invariant under any permutation of the shifts, for the second- and third-order autocorrelations we have the symmetries

$$a_z^2\left[l\right]=a_z^2\left[-l\right],a_z^3\left[l_1,l_2\right]=a_z^3\left[l_2,l_1\right]=a_z^3\left[-l_1,l_2-l_1\right].$$ (2)

For applications of higher-order autocorrelations and their symmetries, see for example [4], [22], [31], [35].

In this section, we describe how the autocorrelations of the noisy measurement y are related to the underlying signal x. These relations are later used to estimate x without the need to identify the locations of signal occurrences, which are nuisance variables and difficult, if not impossible, to determine reliably at high noise levels. For completeness, we include a brief discussion of the special case where the signals are well separated and refer the reader to [9], [10] for details. The generalization to an arbitrary signal spacing distribution follows.

A. Well-separated signals

The signals are said to be well-separated when the consecutive non-zero entries of s are separated by at least 2L − 1 positions. Under this condition, the autocorrelations of y with integer shifts l₁, l₂, …, l_q−1 within the range [−(L−1), L−1] are unaffected by the relative positions of signal occurrences. As a result, these autocorrelations of y provide fairly direct information about those of x, and therefore about x itself.

Due to the presence of Gaussian noise, the entries of y are stochastic. Taking the expectations of the first three autocorrelations of y with respect to the distribution of Gaussian noise, we obtain these relations (see Appendix A):

$${\mathbb{E}}_{\epsilon }\left\{a_y^1\right\}={\rho }_0{a}_x^1$$ (3)

$${\mathbb{E}}_{\epsilon }\left\{a_y^2\left[l\right]\right\}={\rho }_0{a}_x^2\left[l\right]+{\sigma }^2\delta \left[l\right]$$ (4)

$$\begin{gathered} {\mathbb{E}}_{\epsilon }\left\{a_y^3\left[l_1,l_2\right]\right\}={\rho }_0{a}_x^3\left[l_1,l_2\right] \\ +{\rho }_0{a}_x^1{\sigma }^2\left(\delta \left[l_1\right]+\delta \left[l_2\right]+\delta \left[l_1-l_2\right]\right), \end{gathered}$$ (5)

where 0 ≤ l < L and 0 ≤ l₁ ≤ l₂ < L. Here, ρ₀ = ML/N denotes the signal density, where M is the number of signal occurrences in y. We assume M grows with N at the constant rate ρ₀/L. The delta functions, defined by δ[0] = 1 and δ[l ≠ 0] = 0, are due to the autocorrelations of the white Gaussian noise. As indicated by the studies in phase retrieval [7], [12], [15], in general a 1D signal x cannot be uniquely determined by its first two autocorrelations. It is thus necessary (and generically sufficient [10]) to include the third-order autocorrelations to uniquely determine x.

The expectations of the autocorrelations of y are estimated by averaging over the given noisy measurement. This data reduction requires only one pass over the data, which is a great computational advantage as the data size grows. As shown in Appendix A, for a given signal x, the average over the noisy entries of y gives the relations:

$$\begin{gathered} a_y^1={\mathbb{E}}_{\epsilon }\left\{a_y^1\right\}+{\eta }^1 \\ {a}_y^2\left[l\right]={\mathbb{E}}_{\epsilon }\left\{a_y^2\left[l\right]\right\}+{\eta }^2\left[l\right] \\ {a}_y^3\left[l_1,l_2\right]={\mathbb{E}}_{\epsilon }\left\{a_y^3\left[l_1,l_2\right]\right\}+{\eta }^3\left[l_1,l_2\right], \end{gathered}$$

where η¹, η²[l] and η³[l₁, l₂] are random variables with zero mean and variances $\mathcal{O}\left({\sigma }^2/N\right)$, $\mathcal{O}\left(({\sigma }^2+{\sigma }^4)/N\right)$, and respectively. For autocorrelations of order q, the standard deviations scale as ${\sigma }^q/\sqrt{N}$ at high noise levels. Therefore, we need $\sqrt{N}/{\sigma }^q\gg 1$ in order for the $a_y^q$ calculated from the noisy measurement to be a good estimator for ${\mathbb{E}}_{\epsilon }\left\{a_y^q\right\}$, and thus to establish a reliable relation with the signal x such as (3)–(5). Since the SNR is proportional to σ⁻², the sample complexity therefore scales as SNR^−q. We also expect the error of the reconstructed signal to depend on the errors of the highest-order autocorrelations used in the analysis at high noise levels.

We estimate the signal density ρ₀ and signal x by fitting the first three autocorrelations of y via non-linear least squares:

$$\begin{gathered} f({\widehat{\rho }}_0,\widehat{x})={(a_y^1-{\widehat{\rho }}_0{a}_{\widehat{x}}^1)}^2+\frac{1}{L}\sum _{l=0}^{L-1}{\left(a_y^2\left[l\right]-{\widehat{\rho }}_0{a}_{\widehat{x}}^2\left[l\right]-{\sigma }^2\delta \left[l\right]\right)}^2 \\ +\frac{2}{L\left(L+1\right)}\sum _{l_2=0}^{L-1}\sum _{l_1=0}^{l_2}(a_y^3\left[l_1,l_2\right]-{\widehat{\rho }}_0{a}_{\widehat{x}}^3\left[l_1,l_2\right] \\ {-{\widehat{\rho }}_0{a}_{\widehat{x}}^1{\sigma }^2(\delta \left[l_1\right]+\delta \left[l_2\right]+\delta \left[l_1-l_2\right]))}^2. \end{gathered}$$ (6)

The weights are chosen as the inverse of the number of terms in the respective sums. Since the autocorrelations have symmetries, as indicated in (2), the summations above are restricted to the non-redundant shifts. Due to the errors in estimating ${\mathbb{E}}_{\epsilon }\left\{a_y^3\right\}$ with $a_y^3$, we expect the root-mean-square error of the reconstructed signal $\widehat{x}$,

$$\text{RMSE}\left(\widehat{x}\right)=\frac{\Vert \widehat{x}-x\Vert 2}{\Vert x\Vert 2},$$

to scale as σ³ in the low SNR regime.

We mention that there exist alternative methods to fit the observed autocorrelations. One possibility is to reformulate the problem as a tensor sensing problem: the autocorrelations are then linear functions of an (L + 1) × (L + 1) × (L + 1) tensor. Such an approach has been proven to be effective for the related problems of multi-reference alignment [5], [29] and phase retrieval [16]. Similar formulations are also found in tensor optimization [2], [30]. Nevertheless, this method requires lifting the original problem to a higher-dimensional space, which increases the computational burden and scales poorly with the dimension of the problem. In contrast, our least-squares formulation operates in the ambient dimension of the problem and thus might be applicable to high-dimensional setups.

B. Arbitrary spacing distribution

The condition of well-separated signals can be further relaxed to allow arbitrary spacing distribution by assuming that the signal occurrences in any subset of y follow the same spatial distribution. To this end, we define the pair separation function as follows.

Definition 1:

For a given binary sequence s identifying M starting positions of signal occurrences (that is, ${\sum }_{i}s[i]=M$), the pair separation function ξ[l] is defined as

$$\xi \left[l\right]=\frac{1}{M-1}\sum _{k=1}^{M-1}\delta \left[s_{k-1}+l-s_k\right],$$ (7)

where s_k indicates the index of the k^th non-zero entry of the sequence s. It is not hard to see that ${\sum }_l\xi [l]=1$.

In particular, we force ξ[0] = ⋯ = ξ[L − 1] = 0 since we exclude overlapping occurrences; then ξ[L] is the fraction of pairs of consecutive signal occurrences that occur right next to each other (no spacing at all), ξ[L + 1] is the fraction of pairs of consecutive signal occurrences that occur with one signal-free entry in between them, etc.

In contrast to the well-separated model, autocorrelations of y may now involve correlating distinct occurrences of x, which may be in various relative positions. The crucial observation is that these autocorrelations depend only indirectly on s, namely, through L − 1 entries of the unknown ξ, which have much smaller dimensions than s. As shown in Appendix A, for a given signal x, the autocorrelations of the given measurement y gives the relations:

$$a_y^1={\mathbb{E}}_{\epsilon }\left\{a_y^1\right\}+{\eta }_1^{\prime }={\rho }_0{a}_x^1+{\eta }_1^{\prime }$$ (8)

$$\begin{gathered} a_y^2\left[l\right]={\mathbb{E}}_{\epsilon }\left\{a_y^2\left[l\right]\right\}+{\eta }_2^{\prime }\left[l\right] \\ ={\rho }_0{a}_y^2\left[l\right]+{\rho }_0\sum _{j=L}^{L+l-1}\xi \left[j\right]a_x^2\left[j-l\right]+{\sigma }^2\delta \left[l\right]+{\eta }_2^{\prime }\left[l\right] \end{gathered}$$ (9)

$$\begin{gathered} a_y^3\left[l_1,l_2\right]={\mathbb{E}}_{\epsilon }\left\{a_y^3\left[l_1,l_2\right]\right\}+{\eta }_3^{\prime }\left[l_1,l_2\right] \\ ={\rho }_0{a}_x^3\left[l_1,l_2\right]+{\rho }_0\sum _{j=L}^{L+l_2-l_1-1}\xi \left[j\right]a_x^3\left[j-l_2,j+l_1-l_2\right] \\ +{\rho }_0\sum _{j=L}^{L+l_1-1}\xi \left[j\right]a_x^3\left[l_2-l_1,j-l_1\right]+{\rho }_0{a}_x^1{\sigma }^2(\delta \left[l_1\right] \\ +\delta \left[l_2\right]+\delta \left[l_1-l_2\right])+{\eta }_3^{\prime }\left[l_1,l_2\right], \end{gathered}$$ (10)

where 0 ≤ l < L and 0 ≤ l₁ ≤ l₂ < L. The random variables ${\eta }_1^{\prime }$, ${\eta }_2^{\prime }\left[l\right]$ and ${\eta }_3^{\prime }\left[l_1,l_2\right]$ have zero mean and variances $\mathcal{O}\left({\sigma }^2/N\right)$, $\mathcal{O}\left(\left({\sigma }^2+{\sigma }^4\right)/N\right)$ and $\mathcal{O}\left(({\sigma }^2+{\sigma }^6)/N\right)$ respectively. Note that (8)–(10) reduce to (3)–(5) when ξ[L] = ξ[L+1] = ··· = ξ[2L−2] = 0, as required by the condition of well-separated signals. Expressions (8)–(10) further simplify upon defining

$${\rho }_1\left[i\right]={\rho }_0\xi \left[i+L\right],i=0,1,\dots ,L-2.$$ (11)

After calculating the first three autocorrelations of y from the noisy measurement, we estimate the signal x and the parameters ρ₀ and ρ₁ by fitting the autocorrelations of y through the non-linear least squares

$$\begin{gathered} f\left(\widehat{x},{\widehat{\rho }}_0,{\widehat{\rho }}_1\right)={\left(a_y^1-{\widehat{\rho }}_0{a}_{\widehat{x}}^1\right)}^2 \\ +\frac{1}{L}{\sum _{l=0}^{L-1}\left(a_y^2\left[l\right]-{\widehat{\rho }}_0{a}_{\widehat{x}}^2\left[l\right]-{\sigma }^2\delta \left[l\right]-\sum _{j=L}^{L+l-1}{\widehat{\rho }}_1\left[j-L\right]a_{\widehat{x}}^2\left[j-L\right]\right)}^2 \\ +\frac{2}{L\left(L+1\right)}\sum _{l_2=0}^{L-1}\sum _{l_1=0}^{l_2}(a_y^3\left[l_1,l_2\right]-{\widehat{\rho }}_0{a}_{\widehat{x}}^3\left[l_1,l_2\right]-{\widehat{\rho }}_0{a}_{\widehat{x}}^1{\sigma }^2(\delta \left[l_1\right] \\ +\delta \left[l_2\right]+\delta \left[l_1-l_2\right])-\sum _{j=L}^{L+l_2-l_1-1}{\widehat{\rho }}_1\left[j-L\right]a_{\widehat{x}}^3\left[j-l_2,j+l_1-l_2\right] \\ {-\sum _{j=L}^{L+l_1-1}{\widehat{\rho }}_1\left[j-L\right]a_{\widehat{x}}^3\left[l_2-l_1,j-l_1\right])}^2. \end{gathered}$$ (12)

Numerically, we find that this problem can often be solved, meaning that, even though s cannot be estimated, we can still estimate x and the summarizing statistics ρ₀ and ρ₁. As discussed in Section II-A, the RMSE of the reconstructed signal is expected to scale as σ³ in the low SNR regime owing to the errors in estimating ${\mathbb{E}}_{\epsilon }\left\{a_y^3\right\}$ with $a_y^3$.

C. Frequency marching

To minimize the least squares in (6) and (12), we use the trust-regions method in Manopt [13] over the product of the Euclidean manifold with the constraints that ${\widehat{\rho }}_0$ and ${\widehat{\rho }}_1$ are positive. However, as the least squares problems are inherently non-convex, we observe that the iterates of the trust-regions method used for minimization are liable to stagnate in local minima. To alleviate this issue, we adopt the frequency marching scheme [6] in our optimization. The idea behind frequency marching is based on the following heuristics:

1. The coarse-grained (low-resolution) version of the original problem has a smoother optimization landscape so that it is empirically more likely for the iterates of the optimization algorithm to reach the global optimum of the coarse-grained problem.

2. Intuitively, the global optimum of the original problem can be reached more easily by following the path along the global optima of a series of coarse-grained problems with incremental resolution.

Our goal is to guide the iterates of the optimization algorithm to reach the global optimum of the original problem by successively solving the coarse-grained problems, which are warm-started with the solution from the previous stage.

The coarse-grained problems are characterized by the order of the Fourier series, $n_{\text{max}}=1,2,\dots ,\lfloor L/2\rfloor$, used to express the low-resolution approximate $x^{\left(n_{\text{max}}\right)}$ by

$$x^{\left(n_{\text{max}}\right)}\left[l\right]=c_0+\sum _{n=1}^{n_{\text{max}}}{c}_n\text{cos}\left(\frac{2\pi nl}{L}\right)+d_n\text{sin}\left(\frac{2\pi nl}{L}\right),$$ (13)

where l = 0, 1, …, L−1. Instead of the entries of x, the least squares are minimized with respect to the Fourier coefficients in our frequency marching scheme. The order n_max is related to the spatial resolution by Nyquist rate:

$$\Delta x=\{\begin{array}{l}1\text{if}{n}_{\text{max}}=\left(L-1\right)/2,\\ L/2n_{\text{max}}\text{otherwise.}\end{array}$$

The spatial resolution subdivides the signal x into L′ = L/Δx units and represents the “step size” of the shifts for the coarse-grained autocorrelations.

We define the coarse-grained autocorrelations of $x^{\left(n_{\text{max}}\right)}$ of order q for integer shifts l₁, l₂, …, l_q−1 as

$$\begin{gathered} b_{x^{\left(n_{\text{max}}\right)}}^q\left[l_1,l_2,\dots ,l_{q-1}\right] \\ =a_{x^{\left(n_{\text{max}}\right)}}^q\left[\lfloor l_1\Delta x\rceil ,\lfloor l_2\Delta x\rceil ,\dots ,\lfloor l_{q-1}\Delta x\rceil \right], \end{gathered}$$

where ⌊·⌉ rounds the argument to the nearest integer. The coarse-grained autocorrelations of y are given by sub-sampling the original autocorrelations calculated from the full measurement. With b[l] denoting the bin centered at lΔx, where l = 0, 1, …, L′−1, we estimate the coarse-grained autocorrelations⌊ of y by

$$\begin{gathered} b_y^1=a_y^1 \\ {b}_y^2\left[l\right]=B_2^{-1}\sum _{i\in b\left[l\right]}\left(a_y^2\left[i\right]-{\sigma }^2\delta \left[i\right]\right) \\ {b}_y^3\left[l_1,l_2\right]=B_3^{-1}\sum _{\underset{\underset{i_1\le i_2}{i_2\in b\left[l_2\right]}}{i_1\in b\left[l_1\right]}}(a_y^3\left[i_1,i_2\right] \\ -a_y^1{\sigma }^2(\delta \left[i_1\right]+\delta \left[i_2\right]+\delta \left[i_1-i_2\right])), \end{gathered}$$

where 0 ≤ l < L′, 0 ≤ l₁ ≤ l₂ < L′, and B₂ and B₃ represent the number of terms in the respective sums.

Following the discussion in Section II-B, we relate the first three autocorrelations of y to those of $x^{\left(n_{\text{max}}\right)}$, as defined in (13), by

$$\begin{gathered} b_y^1\approx {\rho }_0{b}_{x^{\left(n_{\text{max}}\right)}}^1 \\ {b}_y^2\left[l\right]\approx {\rho }_0{b}_{x^{\left(n_{\text{max}}\right)}}^2\left[l\right]+\sum _{j=L\prime }^{L\prime +l-1}{\rho }_1^{\left(n_{\text{max}}\right)}\left[j-L\prime \right]b_{x^{\left(n_{\text{max}}\right)}}^2\left[j-l\right] \\ {b}_y^3\left[l_1,l_2\right]\}\approx {\rho }_0{b}_{x^{\left(n_{\text{max}}\right)}}^3\left[l_1,l_2\right] \\ +\sum _{j=L\prime }^{L\prime +l_2-l_1-1}{\rho }_1^{\left(n_{\text{max}}\right)}\left[j-L\prime \right]b_{x^{\left(n_{\text{max}}\right)}}^3\left[j-l_2,j+l_1-l_2\right] \\ +\sum _{j=L\prime }^{L\prime +l_1-1}{\rho }_1^{\left(n_{\text{max}}\right)}\left[j-L\prime \right]b_{x^{\left(n_{\text{max}}\right)}}^3\left[l_2-l_1,j-l_1\right]. \end{gathered}$$

The autocorrelations are related by approximation instead of equality to reflect the errors due to the low-resolution approximation. Above, we define ${\rho }_1^{\left(n_{\text{max}}\right)}$ as the product of the signal density ρ₀ and the coarse-grained pair separation function:

$${\rho }_1^{\left(n_{\text{max}}\right)}\left[i\right]={\rho }_0{\xi }^{\left(n_{\text{max}}\right)}\left[i+L\prime \right],i=0,1,\dots ,L\prime -2,$$ (14)

where ${\xi }^{\left(n_{\text{max}}\right)}$ is defined as

$${\xi }^{\left(n_{\text{max}}\right)}\left[i\right]=\sum _{l=\text{max}\left\{0,\lfloor \left(i+1/2\right)\Delta x\rceil \right\}}^{\lfloor \left(i+1/2\right)\Delta x\rceil -1}\xi \left[l\right].$$ (15)

In each stage of frequency marching, we estimate the Fourier coefficients, which are related to $x^{\left(n_{\text{max}}\right)}$ through (13), and the parameters ρ₀, ${\rho }_1^{\left(n_{\text{max}}\right)}$ by fitting the coarse-grained autocorrelations of y through the non-linear least squares

$$\begin{gathered} f(\{{\widehat{c}}_0,\dots ,{\widehat{c}}_{n_{\max }}\},\{{\widehat{d}}_1,\dots ,{\widehat{d}}_{n_{\max }}\},{\widehat{\rho }}_0,{\widehat{\rho }}_1^{\left(n_{\max }\right)}) \\ ={(b_y^1-{\widehat{\rho }}_0{b}_{{\widehat{x}}^{\left(n_{\max }\right)}}^1)}^2+\frac{1}{L\prime }\sum _{l=0}^{L\prime -1}{\left(b_y^2\left[l\right] \\ -{\widehat{\rho }}_0{b}_{{\widehat{x}}^{\left(n_{\max }\right)}}^2\left[l\right]-\sum _{j=L\prime }^{L\prime +l-1}{\widehat{\rho }}_1^{{}^{\left(n_{\max }\right)}}\left[j-L\prime \right]b_{{\widehat{x}}^{\left(n_{\max }\right)}}^2\left[j-l\right]\right)}^2 \\ +\frac{2}{L\prime \left(L\prime +1\right)}\sum _{l_2=0}^{L\prime -1}\sum _{l_1=0}^{l_2}{\left(b_y^3\left[l_1,l_2\right] \\ -{\widehat{\rho }}_0{b}_{{\widehat{x}}^{\left(n_{\max }\right)}}^3\left[l_1,l_2\right]-\sum _{j=L\prime }^{L\prime +l_2-l_1-1}{\widehat{\rho }}_1^{\left(n_{\max }\right)}\left[j-L\prime \right]b_{{\widehat{x}}^{\left(n_{\max }\right)}}^3\left[j-l_2,j+l_1-l_2\right] \\ -\sum _{j=L\prime }^{L\prime +l_1-1}{\widehat{\rho }}_1^{\left(n_{\max }\right)}\left[j-L\prime \right]b_{{\widehat{x}}^{\left(n_{\max }\right)}}^3\left[l_2-l_1,j-l_1\right]\right)}^2 \end{gathered}$$

Our frequency marching scheme increments the order n_max from 1 to ⌊L/2⌋, and the computed solution of each stage is used to initialize optimization in the next stage.

III. Expectation maximization

In this section, as an alternative to the autocorrelations approach, we describe an approximate EM algorithm to address both the cases of well-separated signals and arbitrary spacing distribution. A frequency marching scheme is also designed to help the iterates of the EM algorithm converge to the global maximum of the data likelihood.

A. Well-separated signals

Given the measurement y that follows the MTD model (1), the maximum marginal likelihood estimator (MMLE) for the signal x is the maximizer of the likelihood function p(y|x). Within the EM framework [18], the nuisance variable s is treated as a random variable drawn from some distribution under the condition of non-overlapping signal occurrences. The EM algorithm estimates the MMLE by iteratively applying the expectation (E) and maximization (M) steps. Specifically, given the current signal estimate x_k, the E-step constructs the expected log-likelihood function

$$Q\left(x|x_k\right)=\sum _{s}p\left(s|y,x_k\right)\text{log}p\left(y,s|x\right),$$

where the summation runs over all admissible configurations of the binary sequence s. The signal estimate is then updated in the M-step by maximizing Q(x|x_k) with respect to x. The major drawback of this approach is that the number of admissible configurations for s grows exponentially with the problem size. Therefore, the direct application of the EM algorithm is computationally intractable, even for very short measurements.

In our framework of the approximate EM algorithm, we first partition the measurement y into N_d = N/L non-overlapping segments, each of length L, and denote the m^th segment by y_m. Overall, the signal can occur in 2L−1 different ways when it is present in a segment. The signal is estimated by the maximizer of the approximate likelihood function

$$p(y_0,y_1,\dots ,y_{N_d}{}_{-1}|x)\approx \prod _{m=0}^{N_d-1}p(y_m|x),$$ (16)

where we ignore the dependencies between segments. Our approximate EM algorithm works by applying the EM algorithm to estimate the MMLE of (16), without any prior on the signal. As we will see in Section IV, the validity of the approximation is corroborated by the results of our numerical experiments.

Depending on the position of signal occurrences, the segment y_m can be modeled by

$$y_m=C{R}_{l_m}Zx+{\epsilon }_m,{\epsilon }_m~\mathcal{N}\left(0,{\sigma }^2{I}_L\right).$$

Here, Z first zero-pads L entries to the left of x, and $R_{l_m}$ circularly shifts the zero-padded sequence by l_m positions, that is,

$$(R_{l_m}Zx)\left[l\right]=(Zx)\left[(l+l_m)\text{mod}2L\right],$$

where l_m = 0, 1, …, 2L − 1 and is treated as a random variable. The operator C then crops the first L entries of the circularly shifted sequence, which are further corrupted by additive white Gaussian noise. In this generative model, l_m = 0 represents no signal occurrence in y_m, and l_m = 1, …, 2L−1 enumerate the 2L − 1 different ways a signal can appear in a segment.

In the E-step, our algorithm constructs the expected log-likelihood function

$$Q\left(x|x_k\right)=\sum _{m=0}^{N_d-1}\sum _{l=0}^{2L-1}p\left(l|y_m,x_k\right)\text{log}p\left(y_m,l|x\right)$$ (17)

given the current signal estimate x_k, where

$$p\left(y_m|l,x\right)\propto \prod _{i=0}^{L-1}\text{exp}\left(-\frac{{\left(y_m\left[i\right]-\left(C{R}_{l}Zx\right)\left[i\right]\right)}^2}{2{\sigma }^2}\right),$$ (18)

with the normalization ${\sum }_{l=0}^{2L-1}p\left(y_m|l,x\right)=1$. From Bayes’ rule, we have

$$p\left(l|y_m,x_k\right)=\frac{p\left(y_m|l,x_k\right)p\left(l|x_k\right)}{{\sum }_{l=0}^{2L-1}p\left(y_m|l,x_k\right)p\left(l|x_k\right)},$$ (19)

which is the normalized likelihood function p(y_m|l, x_k), weighted by the prior distribution p(l|x_k). In general, the prior distribution p(l|x_k) is independent of the model x_k and can be estimated simultaneously with the signal.

Denoting the prior distribution p(l) by α[l], we rewrite (17) as (up to an irrelevant constant)

$$\begin{gathered} Q(x,\alpha |x_k,{\alpha }_k)=\sum _{m=0}^{N_d-1}\sum _{l=0}^{2L-1}p(l|y_m,x_k) \\ \times \left(\text{log}p(y_m|l,x)+\text{log}\alpha \left[l\right]\right). \end{gathered}$$

We note that the dependence of Q(x, α|x_k, α_k) on the current prior estimates α_k lies in p(l|y_m, x_k) through (19). The M-step updates the signal estimate and the priors by maximizing Q(x, α|x_k, α_k) under the constraint that the priors lie on the simplex Δ_2L:

$$x_{k+1},{\alpha }_{k+1}=\text{arg}\underset{x,\alpha }{\text{max}}Q\left(x,\alpha |x_k,{\alpha }_k\right)\text{s}.\text{t}.\alpha \in {\Delta }_{2L}.$$ (20)

As shown in Appendix B, we obtain the update rules

$$x_{k+1}\left[j\right]=\frac{{\sum }_{m=0}^{N_d-1}{\sum }_{l=j+1}^{j+L}p(l|y_m,x_k)y_m\left[j+L-1\right]}{{\sum }_{m=0}^{N_d-1}{\sum }_{l=j+1}^{j+L}p(l|y_m,x_k)},$$ (21)

where 0 ≤ j < L, and

$${\alpha }_{k+1}\left[l\right]=\frac{1}{N_d}\sum _{m=0}^{N_d-1}p(l|y_m,x_k),$$ (22)

where 0 ≤ l < 2L. We repeat the iterations of the EM algorithm until the estimates stop improving, as judged within some tolerance.

B. Arbitrary spacing distribution

We extend the approximate EM approach to address arbitrary spacing distribution of signal occurrences by reformulating the probability model: each segment y_m can now contain up to two signal occurrences. In this case, the two signals can appear in L(L − 1)/2 different combinations in a segment, which is explicitly modeled by

$$y_m=C{R}_{l_1^m}Zx+C{R}_{l_2^m}Zx+{\epsilon }_m,{\epsilon }_m~\mathcal{N}\left(0,{\sigma }^2{I}_L\right).$$

where $L<l_1^m<2L$ and $0<l_2^m\le l_1^m-L$. Given the signal estimate x_k and the shifts $l_1^m=l_1,l_2^m=l_2$, the likelihood function p(y_m|l₁, l₂, x_k) can be written as

$$\begin{gathered} p(y_m|l_1,l_2,x_k)\propto \prod _{i=0}^{L-1}\text{exp}(-\frac{1}{2{\sigma }^2} \\ \times {\left[y_m[i]-(C{R}_{l_1}Z{x}_k)[i]-(C{R}_{l_2}Z{x}_k)[i]\right]}^2), \end{gathered}$$ (23)

and we have the normalization condition

$$\sum _{l=0}^{2L-1}p(y_m|l,x_k)+\sum _{l_1=L+1}^{2L-1}\sum _{l_2=1}^{l_1-L}p(y_m|l_1,l_2,x_k)=1.$$

Incorporating the terms with two signal occurrences, the E-step constructs the expected log-likelihood function as

$$\begin{gathered} Q\left(x,\alpha |x_k,{\alpha }_k\right) \\ =\sum _{m=0}^{N_d-1}\left[\sum _{l=0}^{2L-1}p(l|y_m,x_k)\left(\text{log}p(y_m|l,x)+\text{log}\alpha \left[l\right]\right) \\ +\sum _{l_1=L+1}^{2L-1}\sum _{l_2=1}^{l_1-L}p(l_1,l_2|y_m,x_k) \\ \times \left(\text{log}p(y_m|l_1,l_2,x)+\text{log}\alpha \left[l_1,l_2\right]\right)\right], \end{gathered}$$ (24)

where the prior p(l₁, l₂) is denoted by α[l₁, l₂].

Under the assumption that the signal occurrences in any subset of y follow the same spatial distribution, we can parametrize the priors with the pair separation function ξ (see Definition 1). Recall that (M − 1)ξ[l] is the number of pairs of consecutive signal occurrences whose starting positions are separated by exactly l positions. The priors α[l₁, l₂] can be related to the probability that two signal occurrences appear in the combination specified by (l₁, l₂) in a segment of length L selected from the measurement y, which is estimated by

$$\alpha \left[l_1,l_2\right]=\frac{\left(M-1\right)\xi \left[l_1-l_2\right]}{N-L+1}\approx \frac{M}{N}\xi \left[l_1-l_2\right],$$ (25)

where L < l₁ < 2L and 0 < l₂ ≤ l₁ −L. Here, (M −1)ξ[l₁ −l₂] is the number of segments that realize the configuration of signal occurrences specified by (l₁, l₂), and N − L + 1 indicates the total number of segment choices. The priors α[l] can similarly be related to the probability that a signal occurs in the way specified by l in a segment of length L:

$$\begin{gathered} \alpha \left[l\right]=\alpha \left[2L-l\right] \\ =\frac{\left(M-1\right)\xi \left[2L-l\right]+\left(M-1\right)\xi \left[2L-l+1\right]+\dots }{N-L+1} \\ \approx \frac{M}{N}\sum _{j=2L-1}^{\infty }\xi \left[j\right], \end{gathered}$$ (26)

where 0 < l ≤ L. An interesting observation is that the number of signal occurrences M can be estimated by

$$\alpha \left[L\right]\approx \frac{M}{N}\sum _{j=L}^{\infty }\xi \left[j\right]=\frac{M}{N},$$

since the signal occurrences are required not to overlap. The value of the prior α[0] is determined by the normalization

$$\sum _{l=0}^{2L-1}\alpha \left[l\right]+\sum _{l_1=L+1}^{2L-1}\sum _{l_2=1}^{l_1-L}\alpha \left[l_1,l_2\right]=1.$$ (27)

From (25), (26) and (27), we see that the priors are uniquely specified by the positive parameters α[0], α[1], ρ₁[0], ρ₁[1], …, ρ₁[L−2], with ρ₁ defined in (11). Therefore, the normalization (27) can be rewritten as

$$\alpha \left[0\right]+\left(2L+1\right)\alpha \left[1\right]+\sum _{i=0}^{L-2}\frac{i+L}{L}{\rho }_1\left[i\right]=1.$$ (28)

In the special case of well-separated signals, where ξ[i] = 0 for L ≤ i ≤ 2L − 2, we have α[1] = α[2] = ⋯ = α[2L − 1] and the normalization α[0] + (2L − 1)α[1] = 1.

In the M-step, we update the signal estimate and the parameters α[0], α[1], ρ₁ by maximizing Q(x, α|x_k, α_k) under the constraint that (28) is satisfied:

$$\begin{gathered} x_{k+1},{\alpha }_{k+1}\left[0\right],{\alpha }_{k+1}\left[1\right],{\left\{{\rho }_1\right\}}_{k+1} \\ =\text{arg}\underset{x,\alpha \left[0\right],\alpha \left[1\right],{\rho }_1}{\text{max}}Q\left(x,\alpha |x_k,{\alpha }_k\right)\text{s.t.}\left(28\right)\text{is}\text{satisfied}. \end{gathered}$$ (29)

As shown in (24), Q(x, α|x_k, α_k) is additively separable for x and α, so the constrained maximization (29) can be reached by maximizing Q(x, α|x_k, α_k) with respect to x and the parameters α[0], α[1], ρ₁ separately. Maximizing Q(x, α|x_k, α_k) with respect to x yields the update rule (see Appendix C for derivation)

$$\begin{gathered} x_{k+1}\left[j\right] \\ =[\sum _{m=0}^{N_d-1}(\sum _{l=j+1}^{j+L}p\left(l|y_m,x_k\right)y_m\left[j+L-l\right] \\ +\sum _{l_1=L+1}^{j+L}\sum _{l_2=1}^{l_1-L}p\left(l_1,l_2|y_m,x_k\right)y_m\left[j+L-l_1\right] \\ +\sum _{l_1=L+j+1}^{2L-1}\sum _{l_2=j+1}^{l_1-L}p\left(l_1,l_2|y_m,x_k\right)y_m\left[j+L-l_2\right])] \\ \times [\sum _{m=0}^{N_d-1}(\sum _{l=j+1}^{j+L}p\left(l|y_m,x_k\right)+\sum _{l_1=L+1}^{j+L}\sum _{l_2=1}^{l_1-L}p\left(l_1,l_2|y_m,x_k\right) \\ {+\sum _{l_1=L+j+1}^{2L-1}\sum _{l_2=j+1}^{l_1-L}p\left(l_1,l_2|y_m,x_k\right))]}^{-1}, \end{gathered}$$ (30)

where 0 ≤ j < L. To update the parameters α[0], α[1], ρ₁, we note that the function Q(x, α|x_k, α_k) is concave with respect to these parameters and the constraint (28) forms a compact convex set. Therefore, the constrained maximization is achieved using the Frank-Wolfe algorithm [20].

C. Frequency marching

Because the iterates of the EM algorithm are not guaranteed to converge to the maximum likelihood, we develop a frequency marching scheme to help the iterates converge to the global maximum. Recall that frequency marching converts the original optimization problem into a series of coarse-grained problems with gradually increasing resolution. The coarse-grained version of the EM algorithmis characterized by the spatial resolution

$$\Delta x=\text{max}\left(1,L/2n_{\text{max}}\right),$$

where n_max = 1, 2, …, ⌊(L + 1)/2⌋. This spatial resolution models the signal shifts in the coarse-grained problem, with the corresponding expected likelihood function given by

$$Q^{\left(n_{\text{max}}\right)}\left(x,{\alpha }^{\left(n_{\text{max}}\right)}|x_k,{\alpha }_k^{\left(n_{\text{max}}\right)}\right)=\sum _{m=0}^{N_d-1}\left[\sum _{l=0}^{2L^{\prime }-1}p\left(\lfloor l{\Delta }_x\rceil |y_m,x_k\right)\left(\text{log}p\left(y_m|\lfloor l{\Delta }_x\rceil ,x\right)+\text{log}{\alpha }^{\left(n_{\text{max}}\right)}[l]\right)+\sum _{l_1=L^{\prime }+1}^{2L^{\prime }-1}\sum _{l_2=1}^{l_1-L^{\prime }}p\left(\lfloor l_1{\Delta }_x\rceil ,\lfloor l_2{\Delta }_x\rceil |y_m,x_k\right)\times \left(\text{log}p\left(y_m|\lfloor l_1{\Delta }_x\rceil ,\lfloor l_2{\Delta }_x\rceil ,x\right)+\text{log}{\alpha }^{\left(n_{\text{max}}\right)}\left[l_1,l_2\right]\right)\right],$$

where L′ = L/Δ_x. This likelihood function is equal to the original one shown in (24) restricted to the rounded shifts, so the signal can be updated with (30) by ignoring the terms with irrelevant shifts.

Mimicking (25) and (26), we construct the expressions for the priors in the coarse-grained problem as

$$\begin{gathered} {\alpha }^{\left(n_{\text{max}}\right)}\left[l_1,l_2\right]\approx \frac{M}{N}{\xi }^{\left(n_{\text{max}}\right)}\left[l_1-l_2\right] \\ {\alpha }^{\left(n_{\text{max}}\right)}\left[l\right]={\alpha }^{\left(n_{\text{max}}\right)}\left[2L^{\prime }-l\right]\approx \frac{M}{N}\sum _{j=2L^{\prime }-1}^{\infty }{\xi }^{\left(n_{\text{max}}\right)}\left[j\right], \end{gathered}$$

where L′ < l₁ < 2L′, 0 < l₂ ≤ l₁ − L′, 0 < l ≤ L′ and ${\xi }^{\left(n_{\text{max}}\right)}$ is defined in (15). Since the priors are uniquely specified by the positive parameters ${\alpha }^{\left(n_{\text{max}}\right)}[0]$, ${\alpha }^{\left(n_{\text{max}}\right)}[1]$, ${\rho }_1^{\left(n_{\text{max}}\right)}[0]$, ${\rho }_1^{\left(n_{\text{max}}\right)}[1]$, …, ${\rho }_1^{\left(n_{\text{max}}\right)}\left[L^{\prime }-2\right]$, with ${\rho }_1^{\left(n_{\text{max}}\right)}$ defined in (14), we can update the priors by maximizing $Q^{\left(n_{\text{max}}\right)}\left(x,{\alpha }^{\left(n_{\text{max}}\right)}|x_k,{\alpha }_k^{\left(n_{\text{max}}\right)}\right)$ with respect to these parameters under the constraint that the parameters lie on the simplex defined by

$${\alpha }^{\left(n_{\text{max}}\right)}\left[0\right]+\left(2L^{\prime }-1\right){\alpha }^{\left(n_{\text{max}}\right)}\left[1\right]+\sum _{i=0}^{L^{\prime }-2}\frac{i+L^{\prime }}{L^{\prime }}{\rho }_1^{\left(n_{\text{max}}\right)}\left[i\right]=1.$$

Incrementing from n_max = 1 to n_max = ⌊(L + 1)/2⌋, the estimated signal and priors in each stage of frequency marching are used to initialize the optimization problem in the next stage.

IV. Numerical Experiments

This section describes the construction of our synthetic data and the performance of the two proposed methods.¹ A measurement is generated by first sampling M integers from [0, N −L], with the constraint that any two samples differ by at least L+W entries, for some integer W. We use W = L−1 for the case of well-separated signals, while W = 0 is chosen to test our methods for arbitrary spacing distribution of signal occurrences. The sampling is done by generating the random integers uniformly one by one, rejecting any integer that would violate the constraint with respect to previously accepted samples. The M integers indicate the starting positions of the signal occurrences, which are recorded as the non-zero entries of the binary sequence s. The noisy measurement y is given by the linear convolution of s with the signal x, with each entry of y further corrupted by additive white Gaussian noise of zero mean and variance σ². Our synthetic data all have length N = 10⁶, and are constructed with the signal shown in Figure 2(a), which has length L = 10. Also shown in Figures 2(b) and 2(c) are examples of the corrupted signals by different levels of noise. The number of signal occurrences M is adjusted so that the signal density ρ₀ = ML/N is equal to 0.3 and 0.5 for the cases of well-separated signals and arbitrary spacing distribution respectively. Figure 3 shows the resulting pair separation function for our test case of arbitrary spacing distribution.

Fig. 2.

(a) The signal used to generate our synthetic data, with the ℓ₂ norm scaled to $\sqrt{L}$. (b), (c) Signals corrupted by additive white Gaussian noise with zero mean and standard deviations σ =1 and σ =2.

Fig. 3.

Pair separation function to test arbitrary signal spacing distributions.

For both cases of signal spacing distribution, we reconstruct the signal from synthetic data using the two proposed methods with the frequency marching scheme over a wide range of σ. When the frequency marching scheme is not implemented, we observe that the algorithms usually suffer from large errors due to the non-convexity of the problems. A total of 20 instances of synthetic data are generated for each value of σ, and each instance is solved by the two methods using the frequency marching schemes. For our autocorrelation analysis methods, we run the optimization with 10 different randoinitializations and retain the one that minimizes the cost function, as defined in (6) or (12). For the EM approach, we also use 10 random initializations and retain the solution with the maximum data likelihood. Given the signal estimate x_k, we approximate the data likelihood for the cases of well-separated signals and arbitrary spacing distribution by

$$p\left(y|x_k\right)\approx \prod _{m=0}^{N_d-1}\sum _{l=0}^{2L-1}p\left(y_m|l,x_k\right)\alpha \left[l\right]$$

and

$$p\left(y|x_k\right)\approx \prod _{m=0}^{N_d-1}\left(\sum _{l=0}^{2L-1}p\left(y_m|l,x_k\right)\alpha \left[l\right]\text{}+\sum _{l_1=L+1}^{2L-1}\sum _{l_2=1}^{l_1-L}p\left(y_m|l_1,l_2,x_k\right)\alpha \left[l_1,l_2\right]\right)$$

respectively. This is the same approximation we use to formulate the EM algorithm.

The RMSEs of the reconstructions for the case of well-separated signals are shown in Figure 4. As predicted in Section II-A, the RMSE for autocorrelation analysis scales in proportion to σ at high SNR, and to σ³ at low SNR. Interestingly, the EM algorithm exhibits the same behavior, which empirically shows that the two approaches share the same scaling of sample complexity in the two extremes of noise levels. Figure 5(a) and 5(b) show an instance of the signals reconstructed from data generated with σ = 10^0.3 and σ = 10^0.5 respectively. The same scaling is not observed for the reconstructed signal density ρ₀ (not shown), although the relative errors are generally well below a few percents even in the noisiest cases.

Fig. 4.

The RMSEs of the reconstructed signal from data generated with well-separated occurrences. We show performance of our autocorrelation analysis (AA) and of expectation-maximization (EM). Also shown are the RMSEs of the signal estimated by the oracle-based deconvolution algorithm ‘deconv’ described in Section IV and the RMSEs of the estimated signal when the binary sequence s is known.

Fig. 5.

(a), (b) An instance of the reconstructed signals from data generated with well-separated occurrences for σ =10^0.3 and σ =10^0.5 respectively. (c), (d) An instance of the reconstructed signals from data generated with an arbitrary spacing distribution for σ =10^0.3 and σ =10^0.5 respectively.

Figure 6 shows the RMSEs of the reconstructed signal for an arbitrary spacing distribution displayed in Figure 3. As a comparison, we also run the estimation algorithms under the (incorrect) assumption of well-separated signals on the same datasets. We first see that autocorrelation analysis assuming well-separated signals obtains poor reconstruction at all noise levels. Although the EM approach that assumes well-separated signals produces more accurate estimates, the nearly constant RMSEs at high SNR indicate the systematic error due to model misspecification. By contrast, the algorithms that assume an arbitrary spacing distribution achieve much better reconstruction, and the resulting RMSEs have the same scaling behaviors at the two extremes of SNR as shown in Figure 4. An instance of the signals reconstructed from data generated with σ = 10^0.3 and σ = 10^0.5 are shown in Figure 5(c) and 5(d) respectively.

Fig. 6.

The RMSEs of the reconstructed signal from data generated with an arbitrary spacing distribution. We show performance of our autocorrelation analysis (AA) and of expectation-maximization (EM), both under the (incorrect) assumptions of well-separated signals (ws) and the (correct) arbitrary spacing distribution model (asd). Also shown are the RMSEs of the signal estimated by the oracle-based deconvolution algorithm ‘deconv’ described in Section IV and the RMSEs of the estimated signal when the binary sequence s is known.

One of the premises of this paper is that, at high noise levels, detection-based methods are destined to fail because detection cannot be done reliably. To support this theoretical argument further, in Figures 4 and 6 we display the performance of the following oracle, named ‘deconv’. This method is given a strictly simpler task to solve: it must estimate x given the observation y, the noise level σ and the number of signal occurrences M, as well as the ℓ₂-distance between the ground truth x and every single window of length L in the observation y. Precisely, the oracle has access to the vector $z\in {ℝ}^{N-L+1}$ defined for i = 0, …, N − L by

$$z\left[i\right]=\sum _{l=0}^{L-1}{\left(x\left[l\right]-y\left[i+l\right]\right)}^2.$$

The vector z provides this estimator the advantage of finding the likely locations of signal occurrences. If the norm of x is known, this oracle advantage is equivalent to knowledge of the cross-correlation between x and y. Then, the oracle-based estimator ‘deconv’ proceeds as follows: it selects the index i corresponding to the lowest value in z, excludes other indices too close to i as prescribed by the signal separation model under consideration, and repeats this procedure M times to produce an estimator of s: the M starting positions of signal occurrences in y. The signal x is then estimated by averaging the M selected segments of y, effectively deconvolving y by the estimator of s. To our point: despite the unfair oracle advantage, this estimator is unable to estimate x to a competitive accuracy when the noise level is large.

Also plotted in Figures 4 and 6 are the RMSEs of the estimated signal when the binary sequence s is known. In this case, the RMSEs scale as σ and serve as the lower bounds of any algorithm. We can see that, at the low noise levels, our approximate EM algorithm performs nearly as well as the case when s is known.

Figure 7 shows the RMSEs of the reconstructed values of ρ₁ for the experiments “AA-asd” and “EM-asd” shown in Figure 6. Although our methods are not able to reconstruct ρ₁ to high precision, the results shown in Figure 6 indicate the necessity to include the pair separation function in the model to achieve good signal reconstruction.

Fig. 7.

The (relative) RMSEs of the reconstructed values of ρ₁ as defined by (11), for the case of arbitrary spacing distribution.

Figure 8 shows the average computation time for the main methods in Figures 4 and 6. EM is slower because it cross-correlates all the observed segments with the signal estimate in each iteration. This issue becomes especially prominent in the case of arbitrary spacing distribution. On the other hand, autocorrelation analysis requires shorter computation time by summarizing the data as autocorrelation statistics with one pass over the data. We note that the distinct computational speeds for autocorrelation analysis and the EM algorithm is also observed in the related problem of multi-reference alignment [8], [14]. The difference in run time and the similar reconstruction quality of the two methods at low SNR make autocorrelation analysis the preferred approach for large datasets.

Fig. 8.

The average run time to solve an instance of synthetic data for the two methods in both cases of signal spacing distribution.

V. Discussion

In this study, we have presented two approaches—autocorrelation analysis and an approximate EM algorithm—to tackle the MTD problem without the need to detect the positions of the underlying signal occurrences. By incorporating the notion of pair separation function, we generalize the solution of the MTD problem from the special case of well-separated signals to allow an arbitrary spacing distribution of signal occurrence. It is empirically shown that the two methods have the same scaling of sample complexity in the two extremes of noise levels, in particular, SNR⁻³ at high noise level. Since the optimizations in both methods are non-convex, computing schemes based on frequency marching are designed to help the iterates converge to the global optimum.

Our study of the MTD model is primarily motivated by the goal to reconstruct the structures of small biomolecules using cryo-EM [11], [21]. In a cryo-EM experiment, individual copies of the target biomolecule are dispersed at unknown 2D locations and 3D orientations in a thin layer of vitreous ice, from which 2D tomographic projection images are produced by an electron microscope. To minimize the irreversible structural damage, it is necessary to keep the electron dose low. As a result, the projection images are considerably noisy, and high-resolution structure estimation requires averaging over a large number of noisy projections. In particular, as the size of the molecule gets smaller, the SNR of the data drops correspondingly.

Currently, the analysis workflow of cryo-EM data is roughly divided into two steps: The first step, known as particle picking, detects the locations of the biomolecules in the noisy projection images. The 3D structure of the biomolecule is reconstructed in the second step from the unoriented particle projections. When the sizes of biomolecules get smaller, however, reliable detection of their positions becomes challenging, which in turn hampers successful particle picking [3], [9]. It is estimated from first principles that, in order to obtain a 3 Å resolution reconstruction, the particle size should be at least 45 Å so that the particle occurrences can be accurately detected [23]. The results of [9], [10] and this work suggest that it is possible to bypass the need to detect particle positions but still reconstruct the structure at high resolution.

A key feature of cryo-EM that the MTD model fails to capture is that the “signals” are actually 2D projections of the underlying biomolecules at random 3D orientations, and the orientation distribution is usually non-uniform and unknown. This distinction makes the direct application of our approximate EM algorithm prohibitive, because each observed window now needs to cross-correlate with model projections with all possible 2D in-plane translations and 3D orientations, let alone the case where multiple copies of biomolecules may appear in a window. Further approximations such as representing the data and model projections in low-dimensional subspaces [19], [27] seem necessary to push this approach forward. We note that the use of frequency marching in the MTD problem is also related to this idea: the signal is mapped to low-dimensional subspaces with growing dimensions specified by n_max.

On the other hand, the autocorrelations calculated from the noisy measurements in cryo-EM involve averages over both the 3D orientations and 2D in-plane translations. It was pointed out that the 3D structure reconstruction problem by fitting the first three autocorrelations of a biomolecule might be ill-conditioned [9], so information from higher-order autocorrelations seems required. We also expect the cross-terms between 2D projections of different copies to further complicate the reconstruction problem. The resolution of these technical challenges may help extend the use of cryo-EM to smaller biomolecules that are currently believed insurmountable, and is the subject of our ongoing studies.

Appendix A

Derivations of Relations (3)–(5) and (8)–(10)

With the noisy measurement y defined in (1), we express its first order autocorrelation as

$$\begin{gathered} a_y^1=\frac{1}{N}\sum _{i=0}^{N-1}\left(s*x\right)\left[i\right]+\frac{1}{N}\sum _{i=0}^{N-1}\epsilon \left[i\right] \\ =\frac{M}{N}\sum _{i=0}^{L-1}x\left[i\right]+\frac{1}{N}\sum _{i=0}^{N-1}\epsilon \left[i\right]={\rho }_0{a}_x^1+\frac{1}{N}\sum _{i=0}^{N-1}\epsilon \left[i\right]. \end{gathered}$$

Taking the expectation of $a_y^1$ with respect to the distribution of Gaussian noise, we obtain

$${\mathbb{E}}_{\epsilon }\left\{a_y^1\right\}={\rho }_0{a}_x^1,$$

with variance $\mathcal{O}\left({\sigma }^2/N\right)$. We note that this expression applies to both the cases of well-separated signals and arbitrary spacing distribution.

The second order autocorrelation of y with shifts l = 0, 1, …, L − 1 can be written as

$$a_y^2[l]=\frac{1}{N}\sum _{i=0}^{N-1}y[i]y[i+l]=\frac{1}{N}\sum _{i=0}^{N-1}\left((s*x)[i](s*x)[i+l]+(s*x)[i]\epsilon [i+l]+\epsilon [i](s*x)[i+l]+\epsilon [i]\epsilon [i+l]\right)=a_{s*x}^2[l]+\frac{1}{N}\sum _{i=0}^{N-1}\epsilon [i]\epsilon [i+l]+\frac{1}{N}\sum _{k=0}^{M-1}\sum _{i=0}^{L-1}x[i]\left(\epsilon \left[s_k-l+i\right]+\epsilon \left[s_k+l+i\right]\right),$$ (31)

where $a_{s*x}^2[l]$ denotes the second order autocorrelation of the linear convolution s * x with shifts l, and s_k indicates the index of the k^th non-zero entry of the sequence s. Taking the expectation of $a_y^2[l]$ with respect to the distribution of Gaussian noise, we have

$${\mathbb{E}}_{\epsilon }\left\{a_y^2[l]\right\}=a_{s*x}^2[l]+{\sigma }^2\delta [l].$$

The term that scales as $\mathcal{O}\left({\epsilon }^2\right)$ in (31) contributes to the variance by $\mathcal{O}\left({\sigma }^4/N\right)$, while the terms that scale as $\mathcal{O}(\epsilon )$ contribute to the variance by $\mathcal{O}\left(M{\sigma }^2/N^2\right)$. Since we assume that M grows with N at a constant rate, the resulting variance scales as $\mathcal{O}\left(\left({\sigma }^2+{\sigma }^4\right)/N\right)$.

For well-separated signals,

$$a_{s*x}^2\left[l\right]=\frac{M}{N}\sum _{i=0}^{L-1}x\left[i\right]x\left[i+l\right]={\rho }_0{a}_x^2\left[l\right],$$

and we obtain the relation in (4). For arbitrary spacing distribution, we need to include the correlation terms between consecutive signal occurrences:

$$\begin{gathered} a_{s*x}^2\left[l\right]={\rho }_0{a}_x^2\left[l\right]+\frac{M}{N}\sum _{j=L}^{L+l-1}\xi \left[j\right]\sum _{i=0}^{L-1}x\left[i\right]x\left[i+j-l\right] \\ ={\rho }_0{a}_x^2\left[l\right]+{\rho }_0\sum _{j=L}^{L+l-1}\xi \left[j\right]a_x^2\left[j-l\right], \end{gathered}$$

which gives the relation in (9). The quantity ξ[j] denotes the number of consecutive 1’s that are separated by exactly j entries in the sequence s, divided by M − 1 so that ∑_j ξ[j]=1.

As for the third order autocorrelation of y with shifts 0 ≤ l₁ ≤ l₂ < L, we have

$$a_y^3\left[l_1,l_2\right]=\frac{1}{N}\sum _{i=0}^{N-1}y[i]y\left[i+l_1\right]y\left[i+l_2\right]=\frac{1}{N}\sum _{i=0}^{N-1}\left((s*x)[i](s*x)\left[i+l_1\right](s*x)\left[i+l_2\right]+(s*x)[i](s*x)\left[i+l_1\right]\epsilon \left[i+l_2\right]+(s*x)[i]\epsilon \left[i+l_1\right]\epsilon \left[i+l_2\right]+\epsilon [i](s*x)\left[i+l_1\right](s*x)\left[i+l_2\right]+\epsilon [i](s*x)\left[i+l_1\right]\epsilon \left[i+l_2\right]+(s*x)[i]\epsilon \left[i+l_1\right](s*x)\left[i+l_2\right]+\epsilon [i]\epsilon \left[i+l_1\right](s*x)\left[i+l_2\right]+\epsilon [i]\epsilon \left[i+l_1\right]\epsilon \left[i+l_2\right]\right)=a_{s*x}^3\left[l_1,l_2\right]+\frac{1}{N}\sum _{i=0}^{N-1}\epsilon [i]\epsilon \left[i+l_1\right]\epsilon \left[i+l_2\right]+\frac{1}{N}\sum _{k=0}^{M-1}\sum _{i=0}^{L-1}\left(x[i]\left(x\left[i+l_1\right]+\epsilon \left[s_k+l_1+i\right]\right)\epsilon \left[s_k+l_2+i\right]+\epsilon \left[s_k-l_1+i\right]x[i]\left(x\left[i+l_2-l_1\right]+\epsilon \left[s_k+l_2-l_1+i\right]\right)+\left(x\left[i-l_2\right]+\epsilon \left[s_k-l_2+i\right]\right)\epsilon \left[s_k-l_2+l_1+i\right]x[i]\right),$$ (32)

where $a_{s*x}^3\left[l_1,l_2\right]$ denotes the third order autocorrelation of the linear convolution s * x with shifts (l₁, l₂), and x is zero-padded for indices out of the range [0, L − 1].

Taking the expectation of $a_y^3\left[l_1,l_2\right]$ with respect to the distribution of Gaussian noise, we obtain

$${\mathbb{E}}_{\epsilon }\left\{a_y^3\left[l_1,l_2\right]\right\}=a_{s*x}^3\left[l_1,l_2\right]+\frac{M}{N}{\sigma }^2\left(\delta \left[l_1\right]+\delta \left[l_2\right]+\delta \left[l_1-l_2\right]\right)\sum _{i=0}^{L-1}x[i]=a_{s*x}^3\left[l_1,l_2\right]+{\rho }_0{a}_x^1{\sigma }^2\left(\delta \left[l_1\right]+\delta \left[l_2\right]+\delta \left[l_1-l_2\right]\right).$$

As for the variance, the term that scales as $\mathcal{O}\left({\epsilon }^3\right)$ in (32) contributes to the variance by $\mathcal{O}\left({\sigma }^6/N\right)$, while the terms that scale as $\mathcal{O}(\epsilon )$ contribute to the variance by $\mathcal{O}\left(M{\sigma }^2/N^2\right)$. Since we assume that M grows with N at a constant rate, the resulting variance scales as $\mathcal{O}\left(\left({\sigma }^2+{\sigma }^6\right)/N\right)$.

For well-separated signals, we have

$$a_{s*x}^3\left[l_1,l_2\right]={\rho }_0{a}_x^3\left[l_1,l_2\right],$$

and we obtain the relation in (5). By including the correlations between consecutive signal occurrences, we expand $a_{s*x}^3\left[l_1,l_2\right]$ for the case of arbitrary spacing distribution as

$$a_{s*x}^3\left[l_1,l_2\right]={\rho }_0{a}_x^3\left[l_1,l_2\right]+\frac{M}{N}\sum _{j=L}^{L+l_2-l_1-1}\xi [j]\sum _{i=0}^{L-1}x\left[i+j-l_2\right]x\left[i+j+l_1-l_2\right]x[i]+\frac{M}{N}\sum _{j=L}^{L+l_1-1}\xi [j]\sum _{i=0}^{L-1}x\left[i+j-l_1\right]x[i]x\left[i+l_2-l_1\right]={\rho }_0{a}_x^3\left[l_1,l_2\right]+{\rho }_0\sum _{j=L}^{L+l_2-l_1-1}\xi [j]a_x^3\left[j-l_2,j+l_1-l_2\right]+{\rho }_0\sum _{j=L}^{L+l_1-1}\xi [j]a_x^3\left[l_2-l_1,j-l_1\right],$$

which gives the relation in (10).

Appendix B

Derivation of (21) and (22)

The constrained maximization in (20) can be achieved with the unconstrained maximization of the Lagrangian

$$\mathcal{L}(x,\alpha ,\lambda )=Q\left(x,\alpha |x_k,{\alpha }_k\right)+\lambda \left(1-\sum _{l=0}^{2L-1}\alpha [l]\right),$$

where λ denotes the Lagrange multiplier. We note that the constraints in (20) involve the inequalities that the priors are non-negative. Such constrained maximization in general cannot be achieved by maximizing the Lagrangian, for the inequalities might be violated. As we will see later, however, these inequalities are automatically satisfied at the computed maximum (or local maximum) of the Lagrangian, which justifies this approach.

Since Q(x, α|x_k, α_k) is additively separable for x and α, we maximize $\mathcal{L}(x,\alpha ,\lambda )$ with respect to x and α separately. At the maximum of $\mathcal{L}(x,\alpha ,\lambda )$, we have

$$0=\frac{\partial \mathcal{L}}{\partial x[j]}=\sum _{m=0}^{N_d-1}\sum _{l=0}^{2L-1}p\left(l|y_m,x_k\right)\frac{\partial \text{log}p\left(y_m|l,x\right)}{\partial x[j]},$$ (33)

where 0 ≤ j < L. With p(y_m|l, x) give by (18), we can write

$$\frac{\partial \text{log}p\left(y_m|l,x\right)}{\partial x[j]}=-\frac{\partial }{\partial x[j]}\sum _{i=0}^{L-1}\frac{{\left(y_m[i]-\left(C{R}_{l}Zx\right)[i]\right)}^2}{2{\sigma }^2}=-\frac{\partial }{\partial x[j]}\sum _{i=0}^{L-1}\frac{{\left(y_m[i]-(Zx)[(i+l)\text{mod}2L]\right)}^2}{2{\sigma }^2}=-\frac{1}{{\sigma }^2}\sum _{i=0}^{L-1}\left(y_m[i]-(Zx)[(i+l)\text{mod}2L]\right)\times \delta [(i+l)-(j+L)].$$ (34)

Substituting this expression into (33), we obtain

$$\sum _{m=0}^{N_d-1}\sum _{l=j+1}^{j+L}p\left(l|y_m,x_k\right)\left(y_m[j+L-l]-x[j]\right)=0.$$

Rearranging the terms gives the update rule shown in (21).

In order to update the priors, we maximize $\mathcal{L}(x,\alpha ,\lambda )$ with respect to α:

$$0=\frac{\partial \mathcal{L}}{\partial \alpha [l]}=\frac{1}{\alpha [l]}\sum _{m=0}^{N_d-1}p\left(l|y_m,x_k\right)-\lambda ,$$

where 0 ≤ l < 2L. We thus obtain the update rule for α as

$$\alpha [l]=\frac{1}{\lambda }\sum _{m=0}^{N_d-1}p\left(l|y_m,x_k\right),$$

and we can immediately solve λ = N_d from the normalization ${\sum }_{l=0}^{2L-1}\alpha [l]=1$.

Appendix C

Derivation of (30)

As shown in (24), Q(x, α|x_k, α_k) is additively separable for x and α. Moreover, the constraint (28) only involves the values of the parameters α[0], α[1], ρ₁, so the maximization of Q(x, α|x_k, α_k) is unconstrained on x. At the maximum of Q(x, α|x_k, α_k), we have

$$\frac{\partial Q\left(x,\alpha |x_k,{\alpha }_k\right)}{\partial x[j]}=0=\sum _{m=0}^{N_d-1}\left[\sum _{l=0}^{2L-1}p\left(l|y_m,x_k\right)\frac{\partial \text{log}p\left(y_m|l,x\right)}{\partial x[j]}+\sum _{l_1=L+1}^{2L-1}\sum _{l_2=1}^{l_1-L}p\left(l_1,l_2|y_m,x_k\right)\frac{\partial \text{log}p\left(y_m|l_1,l_2,x\right)}{\partial x[j]}\right],$$ (35)

where 0 ≤ j < L. With p(y_m|l₁, l₂, x_k) given by (23), we obtain

$$\frac{\partial \text{log}p\left(y_m|l_1,l_2,x\right)}{\partial x[j]}=-\frac{\partial }{\partial x[j]}\sum _{i=0}^{L-1}\frac{{\left(y_m[i]-\left(C{R}_{l_1}Zx\right)[i]-\left(C{R}_{l_2}Zx\right)[i]\right)}^2}{2{\sigma }^2}=-\frac{1}{2{\sigma }^2}\frac{\partial }{\partial x[j]}\sum _{i=0}^{L-1}{\left(y_m[i]-(Zx)\left[\left(i+l_1\right)\text{mod}2L\right]-(Zx)\left[\left(i+l_2\right)\text{mod}2L\right]\right)}^2=-\frac{1}{{\sigma }^2}\sum _{i=0}^{L-1}\left(y_m[i]-(Zx)\left[\left(i+l_1\right)\text{mod}2L\right]-(Zx)\left[\left(i+l_2\right)\text{mod}2L\right]\right)\times \left(\delta \left[\left(i+l_1\right)-(j+L)\right]+\delta \left[\left(i+l_2\right)-(j+L)\right]\right).$$ (36)

Substituting (34) and (36) into (35), we obtain

$$0=\sum _{m=0}^{N_d-1}\left[\sum _{l=j+1}^{j+L}p\left(l|y_m,x_k\right)\left(y_m[j+L-l]-x[j]\right)+\sum _{l_1=L+1}^{j+L}\sum _{l_2=1}^{l_1-L}p\left(l_1,l_2|y_m,x_k\right)\left(y_m\left[j+L-l_1\right]-x[j]\right)+\sum _{l_1=L+j+1}^{2L-1}\sum _{l_2=j+1}^{l_1-L}p\left(l_1,l_2|y_m,x_k\right)\left(y_m\left[j+L-l_2\right]-x[j]\right)\right].$$

Rearranging the terms, we obtain the update rule in (30).