Extreme Compressed Sensing of Poisson Rates from Multiple Measurements

Abstract

Compressed sensing (CS) is a signal processing technique that enables the efficient recovery of a sparse high-dimensional signal from low-dimensional measurements. In the multiple measurement vector (MMV) framework, a set of signals with the same support must be recovered from their corresponding measurements. Here, we present the first exploration of the MMV problem where signals are independently drawn from a sparse, multivariate Poisson distribution. We are primarily motivated by a suite of biosensing applications of microfluidics where analytes (such as whole cells or biomarkers) are captured in small volume partitions according to a Poisson distribution. We recover the sparse parameter vector of Poisson rates through maximum likelihood estimation with our novel Sparse Poisson Recovery (SPoRe) algorithm. SPoRe uses batch stochastic gradient ascent enabled by Monte Carlo approximations of otherwise intractable gradients. By uniquely leveraging the Poisson structure, SPoRe substantially outperforms a comprehensive set of existing and custom baseline CS algorithms. Notably, SPoRe can exhibit high performance even with one-dimensional measurements and high noise levels. This resource efficiency is not only unprecedented in the field of CS but is also particularly potent for applications in microfluidics in which the number of resolvable measurements per partition is often severely limited. We prove the identifiability property of the Poisson model under such lax conditions, analytically develop insights into system performance, and confirm these insights in simulated experiments. Our findings encourage a new approach to biosensing and are generalizable to other applications featuring spatial and temporal Poisson signals.

I. Introduction

As data increasingly informs critical decision-making, efficient signal acquisition frameworks must keep pace. Modern signals of interest are often high-dimensional but can be efficiently recovered by exploiting their underlying structure through signal processing. The field of compressed sensing (CS), reviewed in [1], [2], focuses on the recovery of sparse signals from fewer measurements than the signal dimension. Concretely, an N-dimensional signal x* with at most k nonzero entries (in which case x* is said to be k-sparse) can be recovered from a measurement vector y acquired by M < N sensors. The sensors’ linear responses to entries of x* define a sensing matrix Φ such that, compactly, y = Φx*. Recovering x* from y is known as the single measurement vector (SMV) problem [3]-[6]. In the multiple measurement vector (MMV) problem [7]-[9], D measurements are captured in an M × D matrix Y in order to recover X*, an N × D signal matrix. X* is jointly sparse such that only k rows contain at least some nonzero elements. CS has been applied extensively in imaging [10]-[12] and communications [13]-[15] and only recently in biosensing [16]-[19].

Emerging microfluidics technologies in the field of biosensing motivate a new MMV framework. With microfluidics, a single sample can be split into D small-volume partitions such as droplets or nanowells with D on the order of 10³ to 10⁷ [20]. Microfluidic partitioning captures individual analytes (e.g. cells [21], [22]; genes [23]; proteins [24], [25]; etc.) in partitions, and analyte quantities across partitions are known to follow a Poisson distribution [26], [27]. The common method to detect a library of analytes with large N is to either dilute samples or split samples into more partitions such that the Poisson distributions reduce to either empty or single-analyte capture, i.e., that columns of X* satisfy $\Vert {\mathbf{x}}_d^{\ast }\Vert \in \{0,1\}$ [22], [28]. This assumption motivates a straightforward N-class classification problem for each non-empty partition, but it necessitates clear separation between classes even under noise, some prior knowledge of sample concentration, and the generation of many wasteful, empty partitions. We hypothesize that CS could generalize the signal recovery strategy when samples are sparse, a common characteristic of biological samples. For example, samples may contain only a few microbes or mutations of interest among many possibilities [29], [30].

We propose the following generally applicable framework for the MMV problem with Poisson signals (MMVP). Let each signal ${\mathbf{x}}_d^{\ast }$ be drawn independently from a multivariate Poisson distribution parametrized by the N-dimensional, k-sparse vector λ*. That is, $x_{n,d}^{\ast }\sim \text{Poisson}({\lambda }_n^{\ast })$ are independent. This framework should not be confused with the well-studied “Poisson compressed sensing” problem in imaging where the measurement noise, rather than the signal, follows a Poisson distribution [31], [32]. In contrast to typical MMV problems, our primary goal is to find an estimate $\widehat{\lambda }\approx {\lambda }^{\ast }$ from D observations rather than to estimate $\widehat{\mathbf{X}}$. Each signal and measurement pair (${\mathbf{x}}_d^{\ast }$, y_d) is in one of G different sensor groups, each with its own sensing matrix Φ^(g) such that ${\mathbf{y}}_d={\Phi }^{(g)}{\mathbf{x}}_d^{\ast }$. The group g associated with each index d is known and deterministic. This measurement group multiplexing is similar to that found in the single pixel camera (SPC) [10]; however, in the SPC the ${\mathbf{x}}_d^{\ast }$ are assumed to be equal, whereas in MMVP they are independently sampled. In microfluidics, several sensor groups can be feasibly achieved by forking an input microfluidic channel into G reaction zones each containing its own set of M sensors. Note that ${\mathbf{x}}_d^{\ast }\stackrel{\text{i.i.d.}}{\sim }\text{Poisson}({\lambda }^{\ast })$ regardless of which group it is in. The statement Y = ΦX* is the special case without noise where G = 1 and is illustrated in Fig. 1. For multiple groups with X^(g)* denoting the submatrix of X* in group g, Y is the following concatenation:

$$\mathbf{Y}=\left[{\Phi }^{(1)}{\mathbf{X}}^{(1)\ast }\dots {\Phi }^{(G)}{\mathbf{X}}^{(G)\ast }\right].$$ (1)

Fig. 1.

The multiple measurement vector (MMV) problem with Poisson signals (MMVP) with one sensor group and noiseless measurements. White squares are zeroes and darker colors represent larger values. Each column ${\mathbf{x}}_d^{\ast }$ of X* is drawn from a Poisson distribution governed by the 2-sparse λ (i.e., ${\mathbf{x}}_d^{\ast }\stackrel{\text{i.i.d.}}{\sim }\text{Poisson}({\lambda }^{\ast })$).

A. Contributions and Findings

We present the first exploration of the MMVP problem and develop a novel recovery algorithm and initial theoretical results. We take a maximum likelihood estimation (MLE) approach, treating Y as a set of D observations from which to infer $\widehat{\lambda }$. Our core contributions are 1: the Sparse Poisson Recovery (SPoRe) algorithm that tractably estimates $\widehat{\lambda }$ (Section II); 2: theoretical results on the identifiability of our MMVP model, proving that MLE can asymptotically recover λ* even when M = G = 1 for any k, and insights into MLE performance (Section III); and 3: simulations demonstrating SPoRe’s superior performance over existing and custom baseline algorithms (Section IV). The achievable measurement rate in the MMVP model is unprecedented in CS, and although we are unable to provide theoretical guarantees for our algorithm, we analytically derive insights into the influence of various system parameters and confirm these insights in our simulated experiments. We find that system designers should first maximize M and then increase G as necessary depending on the expected real-world conditions. We found that using G > 1 was helpful for cases with high noise or high ${\sum }_n{\lambda }_n^{\ast }$.

SPoRe’s strong performance even with extremely low M ∈ {1, 2, 3} under very high measurement noise uniquely enables sensor-constrained applications in biosensing. Although microfluidics devices can rapidly generate a large number of partitions D at a tunable rate, most optical and electrochemical sensing modalities that can keep pace are limited in M [20], [33]. Commonly, fluorescently tagged sensors reveal droplets’ contents rapidly as they flow by a detector, but spectral overlap generally limits M to be five or less without highly specialized, system-specific approaches [34]. High M alternatives such as various spectroscopic techniques limit throughput, necessitate additional instrumentation, or complicate workflows [33], [35]. We speculate that these severe restrictions in M may have forestalled research into CS’s potential role in microfluidics.

B. Previous Work

To the best of our knowledge, the MMVP problem has not yet been explored, likely owing to the ongoing maturation of microfluidics and only recent application of CS to biosensing. The Poisson signal model constrains elements of X to be nonnegative integers under a set of defined probability mass functions. Some aspects of this structure have been studied tangentially, but not the MMVP structure directly.

The core MMV problem only imposes joint sparsity. Early greedy algorithms for this generalized scenario extend the classic Orthogonal Matching Pursuit (OMP) algorithm [36] into OMPMMV [8], simultaneously developed as Simultaenous OMP (S-OMP) [9]. Generally, OMP-based algorithms iteratively build a support set of an estimated sparse solution $\widehat{\mathbf{x}}$ (or $\widehat{\mathbf{X}}$) by testing for the correlation between columns of Φ and the residuals between the measurements and previous estimates. A suite of greedy algorithms was recently developed that impose nonnegative constraints to a number of MMV approaches including OMP’s analogues, and the nonnegative extensions outperformed their generalized counterparts [37].

The application of integer constraints to the SMV problem has proven challenging in the literature. Some theory involving sensing matrix design includes [38], [39], but practical algorithms have required additional constraints on the possible integers, e.g., x ∈ {0, 1}^N or other finite-alphabet scenarios [14], [40]-[43]. A recent study verified that these problems, as well as those with unbounded integer signals, are NP-hard [44]. Algorithms for the unconstrained integer SMV problem thus apply greedy heuristics such as OMP-based approaches [45], [46].

Additional structural constraints can also make these problems tractable. The communications problem of multi-user detection (MUD), reviewed in [47], bears some similarity to MMVP. Here, the activity of N users is the signal of interest and generally follows a Bernoulli model where each user is active with the same prior probability p_a [14]. An alternative prior with ${\sum }_{n=1}^N{x}_{n,d}\sim \text{Poisson}(\lambda )$ models the mean number of total active users in any given signal [48] although the authors solely explored an overdetermined system. Applying an MMV framework to MUD enables underdetermined (M < N) applications but has generally assumed that any active user is active for the entire frame of observation (a row of X is entirely zero or nonzero) [49], [50]. Recently, the potency of sensor groups with a G = 2 system was demonstrated in the MUD context [51]. Despite some similarities to MMVP with an MMV framework and discrete signals, MUD most fundamentally differs from MMVP in its utilization of the probabilistic structure of X. In MUD, the model parameters governing user activities are assumed and leveraged in recovery of X, whereas in MMVP, the model parameters in λ* themselves are the target of recovery.

Regardless of the particular structural constraints imposed in the variants of MMV above, M > k is necessary to recover X* [52]. However, in MMV models where the rows of X* are statistically independent, the simpler task of support recovery has been proven to be possible below the M > k regime, as low as $M=\Omega (\sqrt{k})$ [53]-[55]. In particular, multiple measurement sparse Bayesian learning (M-SBL) [53] takes a MLE approach in a setting very similar to MMVP, but where ${\mathbf{x}}_d^{\ast }$ have a Gaussian distribution rather than Poisson, and there $M=\Omega (\sqrt{k})$ is limit for sparse support recovery even for D → ∞. In contrast, we prove that in the MMVP problem, this is improved to M ≥ 1 (i.e., no minimum number of measurements) regardless of k when D → ∞, even when G = 1. Furthermore, the MLE in this setting recovers λ* exactly. We demonstrate empirically that we can recover λ* well for finite D even for extremely low M such as M ∈ {1, 2, 3}.

II. Sparse Poisson Recovery (SPoRe) Algorithm

A. Notation

We denote by P(·) a probability mass function and by p(·) a probability density function. We use ${\mathbb{R}}^N$ and ${\mathbb{Z}}^N$ to represent the N-dimensional Euclidean space and integer lattice, respectively. We denote by ${\mathbb{R}}_{+}^N$ and ${\mathbb{Z}}_{+}^N$ the non-negative restrictions on these spaces. We use script letters ($\mathcal{A}$, $\mathcal{B}$, …) for sets unless otherwise described. We use lowercase and uppercase bold-face letters for vectors and matrices, respectively. We represent their dimensions with uppercase letters (e.g., $\mathbf{X}\in {\mathbb{Z}}_{+}^{N\times D}$) that are indexed by their lowercase counterparts. For example, x_n,d is the element of X in the nth row and dth column, and we use the shorthands x_n and x_d to represent the entire nth row and dth column vectors, respectively. Other lower case letters (a, b, ϵ, etc.) may represent variable or constant scalars depending on context. We use λ* and X* to refer to the true signal values, and we denote estimates $\widehat{\lambda }$ and $\widehat{\mathbf{X}}$ with the source of the estimate (e.g., MLE, SPoRe, baseline algorithm) being implicit from the context. We use ∥ · ∥ to denote a norm, ∥ · ∥₀ for the number of nonzero elements of a vector, ∥ · ∥₁ for the ℓ₁ vector norm, ∥ · ∥₂ for the Euclidean norm, and ∥ · ∥_F for the Frobenius norm. We also use $\Vert \mathbf{X}{\Vert }_{Rx}\triangleq {\sum }_n{\max }_d\mid x_{n,d}\mid$, a relaxation of the row ℓ₀ quasi-norm defined in [56]. We denote the null space of matrix A by $\mathcal{N}(\mathbf{A})$. As one abuse of notation, for densities of the form p(y∣x), we let the corresponding Φ^(g) applied to x and the relevant noise model be implicit. Also, we let the division of two vectors represent element-wise division.

B. Algorithm

If the index d is in sensor group g, we say that linear measurements are corrupted by an additive random noise vector b_d:

$${\mathbf{y}}_d={\Phi }^{(g)}{\mathbf{x}}_d+{\mathbf{b}}_d.$$ (2)

We let b_d be entirely independent (e.g., additive white Gaussian noise (AWGN), as used in our simulations) or dependent on x. With ${\mathbf{x}}_d\stackrel{\text{i.i.d.}}{\sim }\text{Poisson}({\lambda }^{\ast })$, y_d are independent across d as well. The MLE estimate maximizes the average log-likelihood of the measurements:

$${\widehat{\lambda }}_{MLE}=\underset{\lambda }{\text{arg}\max }\prod _{d=1}^{D}p({\mathbf{y}}_d\mid \lambda )$$ (3)

$$=\underset{\lambda }{\text{arg}\max }\frac{1}{D}\sum _{d=1}^D\log \sum _{\mathbf{x}\in {\mathbb{Z}}_{+}^N}p({\mathbf{y}}_d\mid \mathbf{x})P(\mathbf{x}\mid \lambda ).$$ (4)

Because the infinite sum over x yields an intractable posterior distribution, we cannot apply the popular expectation-maximization (EM) algorithm [57] to solve this MLE problem. Instead, our Sparse Poisson Recovery (SPoRe) algorithm (Algorithm 1) optimizes this function with batch stochastic gradient ascent, drawing B elements uniformly with replacement from {1, …, D} to populate a batch set $\mathcal{B}$. First, note that

$${\nabla }_{\lambda }P(\mathbf{x}\mid \lambda )=P(\mathbf{x}\mid \lambda )\left(\frac{\mathbf{x}}{\lambda }-1\right).$$ (5)

Denoting the objective function from the right-hand side of (4) as ℓ, the gradient is

$${\nabla }_{\lambda }\ell =\frac{1}{B}\sum _{d\in \mathcal{B}}\frac{{\sum }_{\mathbf{x}\in {\mathbb{Z}}_{+}^N}p({\mathbf{y}}_d\mid \mathbf{x})P(\mathbf{x}\mid \lambda )\mathbf{x}}{\lambda {\sum }_{\mathbf{x}\in {\mathbb{Z}}_{+}^N}p({\mathbf{y}}_d\mid \mathbf{x})P(\mathbf{x}\mid \lambda )}-1.$$ (6)

With gradient ascent, each iteration updates $\lambda \leftarrow \lambda +\alpha {\nabla }_{\lambda }\ell$ with learning rate α. However, the summations over all of ${\mathbb{Z}}_{+}^N$ are clearly intractable. SPoRe approximates these quantities with a Monte Carlo (MC) integration over S samples of x, newly drawn for each batch gradient step from sampling distribution $Q:{\mathbb{Z}}_{+}^N\to {\mathbb{R}}_{+}$, such that

$$\sum _{\mathbf{x}\in {\mathbb{Z}}_{+}^N}p(\mathbf{y}\mid \mathbf{x})P(\mathbf{x}\mid \lambda )\approx \frac{1}{S}\sum _{s=1}^S\frac{p(\mathbf{y}\mid {\mathbf{x}}_s)P({\mathbf{x}}_s\mid \lambda )}{Q({\mathbf{x}}_s)}.$$ (7)

The optimal choice of Q(x_s) is beyond the scope of this work, but we found that Q(x_s) = P(x_s∣λ) simplifies the expression, is effective in practice, and draws inspiration from the EM algorithm. In other words, the sampling function is updated at each iteration based on the current estimate of λ. The gradient thus simplifies to

$${\nabla }_{\lambda }\ell =\frac{1}{B}\sum _{d\in \mathcal{B}}\frac{{\sum }_{s=1}^{S}p({\mathbf{y}}_d\mid {\mathbf{x}}_s){\mathbf{x}}_s}{\lambda {\sum }_{s=1}^{S}p({\mathbf{y}}_d\mid {\mathbf{x}}_s)}-1.$$ (8)

Note that if only one ${\widehat{\mathbf{x}}}_d\in {\mathbb{Z}}_{+}^N$ satisfied $p({\mathbf{y}}_{c}d\mid {\widehat{\mathbf{x}}}_d)>0$ for every y_d, the objective ℓ would be concave with $\widehat{\lambda }=\frac{1}{D}{\sum }_{d=1}^D{\widehat{\mathbf{x}}}_d$, i.e., the MLE solution if X* were directly observed. Of course, with compressed measurements and noise, multiple signals may vie to “explain” any single measurement, but SPoRe’s key strength is that it jointly considers independent measurements to directly estimate $\widehat{\lambda }$.

We note that for finite samples, since the MC integration occurs inside a logarithm, the stochastic gradient is biased. However, since it converges in probability to the true gradient, we can expect results comparable to SGD with an unbiased gradient for sufficiently large S [58].

Fig. 2 illustrates key concepts of SPoRe and MMVP with a small example where M = G = 1 and λ* = [0.5, 0, 0.5] for which we can numerically compute p(y∣λ) for various λ. The measurements y_d are effectively drawn from an underlying mixture distribution depending on the noise; e.g., under AWGN, y_d follows a Gaussian mixture. The weights on each mixture component are controlled by λ. In simulated recovery, SPoRe assigns weights to the mixture via $\widehat{\lambda }$ according to the distribution of measurements, coming close to the true underlying distribution. In contrast, an ℓ₁-Oracle (Section IV) which represents best-case performance for a standard, convex sparse recovery process fails because M < k as shown by its error in λ and illustrated by the difference in the distributions. Moreover, by using Φ = [1, 2, 3], many x will map to the same y. While CS theory generally focuses on conditions for unique or well-spaced projections of k-sparse signals (e.g., the restricted isometry property, RIP [5]), we demonstrate that such restrictions are unnecessary in MMVP. By accounting for the latent Poisson distribution in the signals, SPoRe succeeds even when M < k.

Fig. 2.

Example of MMVP and Sparse Poisson Recovery (SPoRe) with M < k < N: Φ = [1, 2, 3], λ* = [0.5, 0, 0.5], and D = 1000 measurements under additive white Gaussian noise (AWGN) b ~ N(0, σ²) with σ² = 0.02. SPoRe attempts to fit the distribution of measurements directly and finds $\widehat{\lambda }$ ≈ [0.45, 0.03, 0.44]. For comparison, the ℓ₁-Oracle (see Section IV) minimizes the measurement error $\widehat{\mathbf{X}}=\text{arg}{\text{min}}_{\mathbf{X}}\Vert \mathbf{Y}-\Phi \mathbf{X}{\Vert }_F$ with X ≥ 0 and ${\sum }_{n,d}{x}_{n,d}={\sum }_{n,d}{x}_{n,d}^{\ast }$ as affine constraints. The estimate $\widehat{\lambda }$ for the ℓ₁-Oracle is then set to the average of the columns of $\widehat{\mathbf{X}}$, and in this example, $\widehat{\lambda }$ ≈ [0.33, 0.31, 0.31]. The distributions p(y∣$\widehat{\lambda }$) for each estimation method are compared against the true distribution p(y∣λ*) and the empirical histogram of the D observations.

Algorithm 1 summarizes the implementation details of SPoRe. Even though $\lambda \in {\mathbb{R}}_{+}^N$, we enforce λ ≥ ϵ by clipping (ϵ = 10⁻³ in our simulations) to maintain exploration of the parameter space. Note that in (8), gradients can become very large with finite sampling as some elements of λ approach zero. We found that rescaling gradients to maximum norm γ helps stabilize convergence. For rescaling, we consider only the subvector δ_Γ of the α-scaled gradient δ, defining indices n ∈ Γ ⊆ {1, …, N} if λ_n+δ_n > ϵ. This restriction ensures that rescaling is solely based on the indices still being optimized, excluding those clipping to ϵ.

Algorithm 1 Spare Poisson Recovery (SPoRe)

$$\begin{array}{cc}\hfill & \mathrm{Input}:{\lambda }^{(0)},B,S,\gamma ,\alpha ,ϵ\hfill \\ \hfill & 1:\lambda \leftarrow {\lambda }^{(0)}\hfill \\ \hfill & 2:i=0\hfill \\ \hfill & 3:\mathbf{repeat}\hfill \\ \hfill & 4:\text{Draw}B\text{columns of}\mathbf{Y}\text{uniformly with replacement}\hfill \\ \hfill & 5:\text{Draw}S\text{new samples from}Q({\mathbf{x}}_s)\hfill \\ \hfill & 6:\delta \leftarrow \alpha {\nabla }_{\lambda }\ell (\lambda )▷(8)\hfill \\ \hfill & 7:\mathbf{if}\Vert {\delta }_{\Gamma }{\Vert }_2>\gamma \mathbf{then}\hfill \\ \hfill & 8:\delta \leftarrow \frac{\gamma }{\Vert {\delta }_{\Gamma }{\Vert }_2}\delta ▷\text{Rescale gradient step}\hfill \\ \hfill & 9:\mathbf{end}\mathbf{if}\hfill \\ \hfill & 10:{\lambda }_n\leftarrow \max ({\lambda }_n+{\delta }_n,ϵ)\hfill \\ \hfill & 11:\mathbf{until}\text{stopping criterion met}\hfill \\ \hfill & 12:\mathbf{return}\lambda \hfill \end{array}$$

For our stopping criterion, we evaluate a moving average of $\widehat{\lambda }$ for convergence. We also track the median of p(y_d∣λ) for $d\in \mathcal{B}$, accounting for stochasticity in likelihood approximations and batch selections such that we may reduce α if no improvements in the median have been seen within a patience window and terminate if α is reduced five times. We conducted all experiments on commodity personal computing hardware. Ultimately, recovery of $\widehat{\lambda }$ takes a few minutes on a single core for the problem sizes we consider (M ≤ 15, N ≤ 50), and SPoRe can be easily parallelized in the future for faster performance.

C. Practical Considerations

Within an iteration, we found that using the same S = 1000 samples for all $d\in \mathcal{B}$ helped to vectorize our implementation to dramatically improved speed over sampling S times for each drawn y_d. This simplification had no noticeable influence on performance. While we found random initializations with a small offset λ⁽⁰⁾ ~ Uniform(0, 1) + ν (with ν = 0.1) to be effective in general, we encountered a numerical issue when under low-variance AWGN. Even though AWGN results in nonzero probabilities everywhere, p(y_d∣x_s) may numerically evaluate to zero for all drawn samples in low-noise settings. These zeros across all samples result in undefined terms in the summation over $d\in \mathcal{B}$ in (8). SPoRe simply ignores such undefined terms, but when this numerical issue occurs for all of $\mathcal{B}$, SPoRe takes no gradient step. With very low noise and large N dampening the effectiveness of random sampling, SPoRe may stop prematurely as it appears to have converged. This problem did not arise with larger noise variances where even inexact samples pushed $\widehat{\lambda }$ in the generally appropriate direction until better samples could be drawn (recall that $Q(\mathbf{x})=P(\mathbf{x}\mid \widehat{\lambda })$ at each iteration). Nonetheless, we decided to set λ⁽⁰⁾ = ν for consistency across all simulated experiments. We speculate that setting λ⁽⁰⁾ to a small value helped encourage sampling sparse x’s in early iterations to help find x_s with nonzero p(y_d∣x_s), bypassing the numerical issue altogether.

III. Theory and Analysis

The summation over x inside the logarithm of the objective function complicates the precise analysis of our sPoRe algorithm. However, we can consider the asymptotic MMVP problem as D → ∞ and its MLE solution to gain insight into the superior recovery performance of sPoRe. In this setting, we obtain the powerful result that λ* is exactly recoverable even when M = G = 1 under a simple null space condition on Φ. We then characterize the loss in Fisher Information for MMVP and show how losses accrue with the increase of signals that map to the same measurements, an effect that increases with k. Lastly, we derive insights into the influence of sensor groups through a small-scale analysis. From a system design standpoint, we find that designers should first increase M as much as feasible and then increase G as needed. All proofs can be found in the Appendix.

A. Identifiability of MMVP Models

Our primary theoretical result is that the asymptotic MLE of the MMVP problem is exactly equal to λ* as long as a simple null space condition on Φ is satisfied. This result improves upon the $M=\Omega (\sqrt{k})$ limitations of correlation-aware support recovery methods [53], [55], placing no restrictions on k and M. The reason for this improvement is that Φ transforms an integer lattice rather than a linear subspace such that the resulting model is still identifiable.

Identifiability refers to the uniqueness of the model parameters that can give rise to a distribution of observations. A model $\mathcal{P}=\{p(\cdot \mid \lambda ):\lambda \in {\mathbb{R}}_{+}^N\}$ is a collection of distribution functions which are indexed by the parameter λ; in the MMVP problem, each choice of Φ and noise give rise to a different model $\mathcal{P}$. Through an optimization lens, if our model is identifiable, then λ* is the unique global optimum of the data likelihood as D → ∞. Recall that $p(\mathbf{y}\mid \lambda )={\sum }_{x\in {\mathbb{Z}}_{+}^N}p(\mathbf{y}\mid \mathbf{x})P(\mathbf{x}\mid \lambda )$, meaning that we can interpret this model as each sensor group consisting of a mixture whose elements’ positions are governed by Φ^(g)x, distributions by the noise model, and weights by P(x∣λ). We focus in this analysis on G = 1, since as D → ∞, at least one sensor group contains infinite measurements. If the corresponding Φ^(g) satisfies the conditions we describe here, then the model is identifiable. Formally:

Definition III.1 (Identifiability). The model $\mathcal{P}$ is identifiable if p(y∣λ) = p(y∣λ′) ∀y ⇒ λ = λ′ for all λ, ${\lambda }^{\prime }\in {\mathbb{R}}_{+}^N$.

The identifiability of mixtures is well-studied [59], [60]; if a mixture is identifiable, the mixture weights uniquely parametrize possible distributions. For finite mixtures, a broad set of distributions including multivariate exponential and Gaussian have been proven to be identifiable [61]. A finite case may manifest in realistic MMVP systems where measurements y must eventually saturate; all sensors have a finite dynamic range of values they can capture. In the most general case, p(·∣λ) is a countably infinite mixture. Although less studied, countably infinite mixtures are identifiable under some classes of distributions [62]. The AWGN that we use in our simulations is identifiable for both the finite and countably infinite cases. Characterizing the full family of noise models that are identifiable under countably infinite mixtures is beyond the scope of this work. Our contribution is that given a noise model that yields identifiable mixtures, equal mixture weights induced by λ and λ′ imply λ = λ′. We prove the sufficiency of the following simple conditions on Φ for identifiability:

Theorem III.2 (Identifiability of Mixture Weights). Let b be additive noise drawn from a distribution for which a countably infinite mixture is identifiable. If $\mathcal{N}(\Phi )\cap {\mathbb{R}}_{+}^N=\{0\}$ and ϕ_n ≠ ϕ_n′ ∀n, n′ ∈ {1, …, N} with n ≠ n′, then $\mathcal{P}$ is identifiable.

The null space condition essentially says that any nonzero vector in $\mathcal{N}(\Phi )$ must contain both positive and negative elements. Many practical Φ satisfy this constraint (e.g., any Φ with at least one strictly negative or positive row). The second condition is trivial: no two columns of Φ can be identical. We also obtain a separate sufficient condition, that Φ drawn from any continuous distribution results in identifibiablity.

Corollary III.3 (Identifiability with Random Continuous Φ). Let b be additive noise drawn from a distribution for which a countably infinite mixture is identifiable. If the elements of Φ are independently drawn from any continuous distribution, then $\mathcal{P}$ is identifiable.

We emphasize the general result of Theorem III.2, since discrete sensing is common in biomedical systems. For example, sensors are often designed to bind to an integer number of known target sites and yield “digital” measurements [26], [63]. Discrete Φ can give rise to what we call collisions. Formally:

Definition III.4 (Collisions and Collision Sets). Let $\Phi \in {\mathbb{R}}^{M\times N}$ be a sensing matrix applied to signals $\mathbf{x}\in {\mathbb{Z}}_{+}^N$. A collision occurs between x and x′ when Φx = Φx′. A collision set for an arbitrary $\mathbf{u}\in {\mathbb{Z}}_{+}^N$ is the set ${\mathcal{C}}_{\mathbf{u}}=\{\mathbf{x}:\Phi \mathbf{x}=\Phi \mathbf{u};\mathbf{x}\in {\mathbb{Z}}_{+}^N\}$.

If the distribution from which b is drawn is fixed (e.g., AWGN) or a function of Φx, then the mixture weights are the probability mass of each collision set. Let the set of collision sets be $\mathcal{U}$ with ${\mathcal{C}}_{\mathbf{u}}\in \mathcal{U}$ being an arbitrary collision set.

$$p(\mathbf{y}\mid \lambda )=\sum _{{\mathcal{C}}_{\mathbf{u}}\in \mathcal{U}}p(\mathbf{y}\mid \mathbf{x}\in {\mathcal{C}}_{\mathbf{u}})P({\mathcal{C}}_{\mathbf{u}}\mid \lambda )$$ (9)

$$P({\mathcal{C}}_{\mathbf{u}}\mid \lambda )=\sum _{\mathbf{x}\in {\mathcal{C}}_{\mathbf{u}}}P(\mathbf{x}\mid \lambda ).$$ (10)

The weights of the mixture elements are governed by $P({\mathcal{C}}_{\mathbf{u}}\mid \lambda )$. Given a noise model that yields identifiable mixtures, the same distribution of observations y implies that the mixture weights are identical, i.e. $P({\mathcal{C}}_{\mathbf{u}}\mid \lambda )=P({\mathcal{C}}_{\mathbf{u}}\mid {\lambda }^{\prime })\forall \mathbf{u}$. We prove that $P({\mathcal{C}}_{\mathbf{u}}\mid \lambda )=P({\mathcal{C}}_{\mathbf{u}}\mid {\lambda }^{\prime })$ ∀u implies λ = λ′, which implies the identifiability of $\mathcal{P}$ under both the conditions of Theorem III.2 and Corollary III.3.

Our proofs are based on the existence and implications of single-vector collison sets ${\mathcal{C}}_{\mathbf{x}}=\{\mathbf{x}\}$. When (9) holds, u indexes both the mixture elements and the collision sets. In the general case where b is dependent on x and not simply Φx, signals participating in the same mixture element may have different noise distributions. These differences can only further subdivide collision sets and leaves single-vector collision sets unaffected. Thus, our results also cover the general noise case.

B. Fisher Information of MMVP Measurements

While identifiability confirms that λ* is a unique global optimum of the MLE problem given infinite observations, Fisher Information helps characterize estimation of λ* as D increases. The Fisher Information matrix $\mathcal{I}$ is the (negative) Hessian of the expected log-likelihood function at the optimum λ*, and it is well-known that under a few technical conditions the MLE solution is asymptotically Gaussian with covariance ${\mathcal{I}}^{-1}∕D$. Intuitively, higher Fisher Information implies a “sharper” optimum that needs fewer observations for stable recovery. For direct observations of Poisson signals ${\mathbf{x}}_d^{\ast }$ rather than y_d, $\mathcal{I}$ is diagonal with ${\mathcal{I}}_{n,n}=1∕{\lambda }_n^{\ast }$. In MMVP with observations of noisy projections (y_d), $\mathcal{I}$ and its inverse are difficult to analyze. We can, however, instead characterize the reduction in ${\mathcal{I}}_{n,n}$ in MMVP caused by the noisy measurement of ${\mathbf{x}}_d^{\ast }$ and derive an insight that we empirically confirm in Section IV-D. Concretely, elements of $\mathcal{I}$ follow

$${\mathcal{I}}_{i,j}=\mathbb{E}\left[\left(\frac{\partial }{\partial {\lambda }_i^{\ast }}\log p(\mathbf{y}\mid {\lambda }^{\ast })\right)\left(\frac{\partial }{\partial {\lambda }_j^{\ast }}\log p(\mathbf{y}\mid {\lambda }^{\ast })\right)\right].$$ (11)

We denote the shorthand w_x ≜ p(y∣x)P(x∣λ) and note that ∑_x w_x = p(y∣λ). Following a similar derivation for the partial derivatives in (8), it can be shown that the general expression for diagonal elements ${\mathcal{I}}_{n,n}$ is

$${\mathcal{I}}_{n,n}=\int {\left(\frac{{\sum }_{\mathbf{x}}{w}_{\mathbf{x}}{x}_n}{{\sum }_{\mathbf{x}}{w}_{\mathbf{x}}{\lambda }_n^{\ast }}-1\right)}^2\left(\sum _{\mathbf{x}}{w}_x\right)d\mathbf{y}.$$ (12)

In the ideal scenario, we observe ${\mathbf{x}}_d^{\ast }$ directly such that

$${\mathcal{I}}_{n,n}^{\text{ideal}}=\sum _{\mathbf{x}}P(\mathbf{x}\mid \lambda ){\left(\frac{x_n}{{\lambda }_n}-1\right)}^2\left(\int p(\mathbf{y}\mid \mathbf{x})d\mathbf{y}\right).$$ (13)

It can be easily shown that (13) reduces to the canonical $1∕{\lambda }_n^{\ast }$. The integration of p(y∣x) evaluates to one, but we can manipulate it algebraically to re-express the quantity as

$${\mathcal{I}}_{n,n}^{\text{ideal}}=\int \sum _{\mathbf{x}}\left[w_{\mathbf{x}}{\left(\frac{x_n}{{\lambda }_n}-1\right)}^2\right]d\mathbf{y}.$$ (14)

Let ${\mathcal{I}}_n^{\text{loss}}\triangleq {\mathcal{I}}_{n,n}^{\text{ideal}}-{\mathcal{I}}_{n,n}$ and let ${\sum }_{({\mathbf{x}}^{\prime },\chi )}$ denote the sum over all pairs of signals x′, $\chi \in {\mathbb{Z}}_{+}^N$. Expanding Equations (12) and (14) and simplifying yields

$$\begin{gathered} {\mathcal{I}}_n^{\text{loss}}=\frac{1}{{\lambda }_n^{\ast }{}^2}\int \left(\sum _{\mathbf{x}}{w}_{\mathbf{x}}{x}_n^2-\frac{{\left({\sum }_{\mathbf{x}}{w}_{\mathbf{x}}{x}_n\right)}^2}{{\sum }_{\mathbf{x}}{w}_{\mathbf{x}}}\right)d\mathbf{y} \\ =\frac{1}{{\lambda }_n^{\ast }{}^2}\int \frac{1}{{\sum }_{\mathbf{x}}{w}_{\mathbf{x}}}\left(\sum _{\forall {\mathbf{x}}^{\prime },\chi }{w}_{{\mathbf{x}}^{\prime }}{w}_{\chi }(x_n^{\prime }-{\chi }_n{)}^2\right)d\mathbf{y}. \end{gathered}$$ (15)

Note that ${\mathcal{I}}_n^{\text{loss}}$ is non-negative such that ${\mathcal{I}}_{n,n}\le {\mathcal{I}}_{n,n}^{\text{ideal}}$ and that pairs of signals with $x_n^{\prime }\ne {\chi }_n$ can contribute to ${\mathcal{I}}_n^{\text{loss}}$. Also note that, w_x′w_χ = p(y∣x′)p(y∣χ)P(x′∣λ*)P(χ∣λ*) and that P(x∣λ*) > 0 only when supp(x) ⊆ supp(λ*). Thus, the Fisher Information is only reduced over the direct Poisson observation case when there are pairs of signals that are well-explained by the same y and by the same λ*. Clearly, λ* with higher k will result in more of such pairs, and we empirically confirm this influence of k on Fisher Information in Section IV-D. Although further precise analysis via Fisher Information is challenging, we provide deeper analysis of the special case of the MMVP problem with small λ through a different lens in the next section.

C. Small Scale Analysis

With identifiability, we know that λ* uniquely maximizes the expected log-likelihood. However, because SPoRe uses stochastic gradient ascent to optimize the empirical log-likelihood, it will typically achieve a $\widehat{\lambda }$ that is near but not equal to λ*. We therefore wish to understand how the neighborhood of λ* changes given the parameters of the problem. The natural way to do this for MLE problems is to consider the Fisher Information matrix as in the previous section, but the presence of a sum inside the logarithm makes analysis difficult. Instead, we consider a particular $\tilde{\lambda }$ near λ* that solves an optimization related to the original likelihood maximization problem. To further simplify the setting, we consider the “small scale” case where ${\sum }_n{\lambda }_n^{\ast }$ is small enough that there is almost never a case where ${\sum }_n{x}_n^{\ast }>1$. We emphasize that although this setting is simple, the MLE approach can still drastically outperform a trivial solution such as $\widehat{\lambda }=\mathbb{E}[\widehat{\mathbf{x}}]$, where $\widehat{\mathbf{x}}=\text{arg}{\max }_{\mathbf{x}}p(\mathbf{y}\mid \mathbf{x})$, since with sufficient noise, $\widehat{\mathbf{x}}\ne {\mathbf{x}}^{\ast }$ with arbitrary probability (Section IV-B).

At the small scale, the distribution of each $x_n^{\ast }$ becomes Bernoulli with parameter λ_n, and the probability that $x_n^{\ast }=1$ and $x_{n^{\prime }}^{\ast }=1$ for n ≠ n′ vanishes. Let n* ≜ (the first nonzero index of x*, 0 if none), which has a categorical distribution with parameter λ*. We abuse notation so that ϕ₀ = 0, ${\lambda }_0^{\ast }$ is the probability that n* = 0, and ${\sum }_{n=0}^N{\lambda }_n^{\ast }=1$. Applying Jensen’s inequality to the log-likelihood for the conditional expectation given n*, we obtain

$$\mathbb{E}\left[\log \sum _{n=0}^{n}p(\mathbf{y}\mid n){\lambda }_n\right]\le {\mathbb{E}}_{n^{\ast }}\left[\log \sum _{n=0}^N{\mathbb{E}}_{\mathbf{y}\mid n^{\ast }}[p(\mathbf{y}\mid n)]{\lambda }_n\right].$$ (16)

Call the right-hand side of this inequality the Jensen bound. This Jensen bound via the logarithm has the attractive property of having a gradient that is equal to a first-order Taylor approximation of the gradient of the original likelihood.¹ To see this, consider the partial derivatives for a single λ_n:

$$\mathbb{E}\left[\frac{p(\mathbf{y}\mid n)}{{\sum }_{n^{\prime }=0}^{N}p(\mathbf{y}\mid n^{\prime }){\lambda }_{n^{\prime }}}\right]\approx {\mathbb{E}}_{n^{\ast }}\left[\frac{{\mathbb{E}}_{\mathbf{y}\mid n^{\ast }}[p(\mathbf{y}\mid n)]}{{\sum }_{n^{\prime }=0}^N{\mathbb{E}}_{\mathbf{y}\mid n^{\ast }}[p(\mathbf{y}\mid n^{\prime })]{\lambda }_{n^{\prime }}}\right].$$ (17)

Thus, we can expect the optimizer of the Jensen bound to be close to λ* (this is particularly true as measurement noise vanishes and the bound becomes tight).

In the case where G = 1 under AWGN, we have the following result characterizing the solution of the Jensen bound.

Proposition III.5. If $\mathbf{y}\sim \mathcal{N}({\varphi }_{n^{\ast }},{\sigma }^2\mathbf{I})$ and

$$\mathbf{K}={\left(\text{exp}\left\{-\frac{1}{4{\sigma }^2}\Vert {\varphi }_n-{\varphi }_{n^{\prime }}{\Vert }_2^2\right\}\right)}_{n,n^{\prime }=0}^N$$ (18)

is invertible, then the maximizer $\tilde{\lambda }$ of the Jensen bound satisfies

$$\tilde{\lambda }\propto {\mathbf{K}}^{-1}\left(\frac{{\lambda }^{\ast }}{{\mathbf{K}}^{-1}(\mathbf{s}-\mu )}\right).$$ (19)

where $\mathbf{s}\in \partial \Vert \tilde{\lambda }{\Vert }_1$ and for all n, μ_n ≥ 0 and ${\mu }_n{\tilde{\lambda }}_n=0$.

In the case where all entries of $\tilde{\lambda }$ are positive, s − μ = 1. K has values of one along the diagonal and smaller values off the diagonal, so it mimics the identity matrix. Clearly, as K → I, $\tilde{\lambda }\to {\lambda }^{\ast }$. However, given nonzero σ², K is bounded away from I. Furthermore, since it is impossible to find a set of more than M + 1 equidistant points in ${\mathbb{R}}^M$, the off-diagonal values of K will differ when M < N, introducing distortion in the transformation.

However, even if M < N, if y is a measurement from a random sensor group, then the effect of this distortion can be mitigated such that $\tilde{\lambda }$ is a reliable estimator of λ* from a support recovery perspective:

Theorem III.6. If $\mathbf{y}\sim \mathcal{N}({\varphi }_{n^{\ast }}^{(g)},{\sigma }^2\mathbf{I})$, g is distributed uniformly, and ${\varphi }_n^{(g)}\stackrel{\text{i.i.d.}}{\sim }\mathcal{N}(0,\mathbf{I})$, then if G → ∞ and all elements of the maximizer $\tilde{\lambda }$ of the Jensen bound are stricly positive, there exist c₁ ≥ 0, $c_2\in \mathbb{R}$ such that ${\tilde{\lambda }}_n=c_1{\lambda }_n^{\ast }+c_2$ for 1 ≤ n ≤ N.

If $\widehat{\lambda }$ has the same rank ordering as λ*, the exact support can be recovered. Therefore, we expect an increase in G to improve performance in tasks such as support recovery. From this result, however, we expect gains due to increasing G to be less immediate than those due to increasing M (and indeed, we see this in our simulations in Section IV-C). To see this, contrast the asymptotic nature of Theorem III.6 in G with the fact that for a finite choice of M (specifically M = N) we can select all ϕ_n equidistant (or that for M even smaller we can select Φ satisfying a RIP with some acceptable distortion) and obtain the same reliability result.

IV. Simulations

In this section, we present comparisons of SPoRe against existing and custom baseline algorithms and follow with focused experimentation on SPoRe’s performance and limitations. For most baseline algorithms that find an estimate $\widehat{\mathbf{X}}$, we set their estimates $\widehat{\lambda }=\frac{1}{D}{\sum }_{d=1}^D{\widehat{\mathbf{x}}}_d$, i.e., the canonical Poisson MLE if X* were observed directly. For a performance metric, we chose cosine similarity between $\widehat{\lambda }$ and λ* as it captures the relative distribution of elements of the solution which we believe is most useful to a user in biosensing applications. Although comparisons of cosine similarity mask differences in magnitude, estimates with high cosine similarity also exhibited low mean-squared error in our experience (results not shown). We plot cosine similarity alone for brevity. In all simulations, we use AWGN and set ${\varphi }_{m,n}^{(g)}\stackrel{\text{i.i.d.}}{\sim }\text{Uniform}(0,1)$ since many sensors are restricted to nonnegative measurements. For each parameter combination, we evaluate over 50 trials in which we draw new Φ^(g) and λ* for each trial. Due to high performance variability for some baseline algorithms, all error bars are scaled to $\pm \frac{1}{2}$ standard deviation for consistency and readability.

A. Comparison against existing baselines

With no existing algorithm designed for Poisson signals, we compare against a number of algorithms with various relevant structural assumptions. We compare against both greedy and convex optimization approaches. First, we use DCS-SOMP [64], a generalization of the common baseline Simultaneous Orthogonal Matching Pursuit (S-OMP) [9] that assumes no structure and greedily solves MMV problems for any value of G. Next, we use NNS-SP and NNS-CoSaMP [37], two greedy algorithms for nonnegative MMV CS motivated by subspace pursuit (SP) [65] and compressive sampling matching pursuit (CoSaMP) [66] which exhibited the best empirical performance in [37]. For integer-based recovery, we use PROMP [45], an SMV algorithm for unbounded integer sparse recovery, to recover an estimate for each signal ${\widehat{\mathbf{x}}}_d$. We also use two support recovery algorithms with $M=\Omega (\sqrt{k})$ measurement rates: M-SBL [53] and a convex, ℓ₁-based variance recovery algorithm [54]. Support recovery is generally not quantitative and appears to make cosine similarity an inappropriate metric for comparison. However, these algorithms produce an estimate of the variance of ${\mathbf{x}}_n^{\ast }$, which equals ${\lambda }_n^{\ast }$ for the Poisson distribution, making them coincidentally reasonable baseline approaches for MMVP directly.

For comparison against best-possible performance of the baselines and to avoid hyperparameter search where possible (for regularization weights, stopping criteria, etc.), we arm the baselines with relevant oracle knowledge of λ* or X*. While NNS-SP and NNS-CoSaMP require k as an input, we also give DCS-SOMP and PROMP, algorithms that iteratively and irreversibly select support elements, knowledge of k and have them stop after k elements have been chosen. Additionally, we created three oracle-enabled convex algorithms that minimize ${\sum }_{g=1}^G\Vert \mathbf{Y}-{\Phi }^{(g)}{\mathbf{X}}^{(g)}{\Vert }_F$ under norm constraints with oracle knowledge of the value of the norm. The ℓ₁ norm is commonly used as a penalty for convex solvers to encourage sparsity in sparse recovery. Our ℓ₁-Oracles include SMV and MMV versions, where in the SMV case, Y is collapsed to a single vector by summing ∑_d y_d, and a vector $\widehat{\mathbf{x}}$ is recovered from which $\widehat{\lambda }=\widehat{\mathbf{x}}∕D$. The ℓ₁-Oracle SMV and ℓ₁-Oracle MMV are both given ${\sum }_{n,d}{x}_{n,d}^{\ast }$. In [56], ∥X∥_Rx is suggested as a better alternative for MMV, so that our Rx-Oracle is given ∥X*∥_Rx. In the support recovery algorithm from [54], the vector of sample variances $(\text{Var}({\mathbf{x}}_n){)}_{n=1}^N$ is recovered under a constraint on ∑_n Var(x_n). Because we provide the sum of variances, we call this method the ∑ Var-Oracle. We also enforce non-negativity of the optimization variables in all convex problems, which we solve using the convex optimization package CVX in Matlab [67], [68]. We use no oracle knowledge for M-SBL, but we use the same initialization as for SPoRe and run its fixed point update until convergence.

From Fig. 3a, we see the crucial result that the M < k regime is only feasible with SPoRe, while conventional CS algorithms, both SMV and MMV, fail. Such a result is expected; generally speaking, CS algorithms seek to minimize measurement error (∥Y − ΦX∥_F) while constraining the sparsity of the recovered solution. CS theory focuses on M > k since if M < k, M × k submatrices of Φ yield underdetermined systems in general. In other words, there simply cannot be unique k-sparse minimizers of measurement error alone with M < k, so the conventional CS problem is not well-posed, unlike in the MMVP problem. Support recovery algorithms M-SBL and the ∑ Var-Oracle exhibit improved performance over other CS baselines as M decreases, but SPoRe remains far superior as M → 1. Next, in Fig. 3b, set M = 10, a regime where most baselines performed nearly perfectly according to (Fig. 3a), and we increased the AWGN variance. We see that even in the conventional regime of N > M > k, SPoRe exhibits the highest noise tolerance which reflects the fact that its leverage of the Poisson assumption minimizes its dependence on accurate measurements. Lastly, however, in Fig. 3c, SPoRe has the unique disadvantage of struggling to recover cases with high ${\sum }_n{\lambda }_n^{\ast }$. We observed that as ${\sum }_n{\lambda }_n^{\ast }$ increases, SPoRe’s finite sampling results in few to no gradient steps taken as “good” samples with nonzero (numerically) p(y∣x_s) were drawn increasingly rarely, and SPoRe mistakenly terminates. Under AWGN, larger ${\lambda }_n^{\ast }$ raises the signal-to-noise ratio but can paradoxically compromise SPoRe’s performance. If M ≫ k is a practical design choice, practitioners should consider existing MMV approaches if ${\sum }_n{\lambda }_n^{\ast }$ may be highly variable.

Fig. 3.

Performance of SPoRe vs. compressed sensing baseline algorithms over 50 trials. Common settings unless otherwise specified are M = 10, k = 3, N = 20, D = 100, G = 1, ∑_n ${\lambda }_n^{\ast }$ = 2. (a) Performance as a function of M, with σ² = 10⁻⁶ for comparison in an effectively noiseless setting. (b) Performance as a function of AWGN variance, with M = 10. (c) Performance as a function of ∑_n ${\lambda }_n^{\ast }$, with σ² = 10⁻² and M = 10.

B. Comparison against integer baselines: M < k

In the M < k regime, since with high probability we can bound the elements of X*, we might expect the discrete nature of the problem to admit at the least a brute-force solution for obtaining $\widehat{\mathbf{X}}$ that we can use to obtain $\widehat{\lambda }$. Indeed, if measurement noise is low, then the integer signal that minimizes measurement error for y_d is likely to be ${\mathbf{x}}_d^{\ast }$. But a finite search space alone has not enabled integer-constrained CS research to achieve M < k in general.

One may wonder whether SPoRe is simply taking advantage of this practically finite search space and, by virtue of MC sampling over thousands of iterations, is effectively finding the right solution by brute force. To address this possibility, we compare against an ℓ₀-Oracle that is given k and the maximum value in X* in order to test all $\left(\begin{array}{c}\hfill N\hfill \\ \hfill k\hfill \end{array}\right)$ combinations of X’s support. For each combination, it enumerates the (max(X*) + 1)^k possibilities for each x_d and selects ${\widehat{\mathbf{x}}}_d=\text{arg}{\text{min}}_{\mathbf{x}}\Vert {\mathbf{y}}_d-\Phi \mathbf{x}{\Vert }_2$. Finally, it returns the k-sparse solution with the lowest minimized measurement error. This algorithm is the only Poisson-free approach in this section.

Comparing SPoRe and other Poisson-enabled baselines against the ℓ₀-Oracle characterizes the effect of incorporating the Poisson assumption on recovery performance. An early solution of ours for tackling MMVP, which we now use as a baseline, was an alternating optimization framework to update estimates of $\widehat{\mathbf{X}}=\text{arg}{\max }_{\mathbf{X}}p(\mathbf{X}\mid \mathbf{Y},\widehat{\lambda })$ and $\widehat{\lambda }=\text{arg}{\max }_{\lambda }p(\lambda \mid \mathbf{Y},\widehat{\mathbf{X}})$. Noting that $p(\mathbf{X}\mid \mathbf{Y},\widehat{\lambda })\propto p(\mathbf{Y}\mid \mathbf{X})p(\mathbf{X}\mid \widehat{\lambda })$, this MAP framework for solving for X under AWGN with variance σ² is

$$\widehat{\mathbf{X}}=\underset{\mathbf{X}}{\text{arg}\max }\frac{1}{D}\sum _{d=1}^D\log P({\mathbf{y}}_d\mid {\mathbf{x}}_d)+\log p({\mathbf{x}}_d\mid \widehat{\lambda })$$ (20)

$$\begin{gathered} =\underset{\mathbf{X}}{\text{arg}\max }\frac{1}{D}\sum _{d=1}^D[-\frac{1}{2{\sigma }^2}\Vert {\mathbf{y}}_d-\Phi {\mathbf{x}}_d{\Vert }_2^2\phantom{]} \\ +\phantom{[}\sum _{n=1}^N\left(x_{n,d}\log {\widehat{\lambda }}_n-\log \Gamma (x_{n,d}+1)\right)], \end{gathered}$$ (21)

where the Gamma function Γ(·) is the continuous extension of the factorial (Γ(x_n,d + 1) = x_n,d!) and is log-concave in the space of positive reals ${\mathbb{R}}_{++}^N$. We implemented the classic branch-and-bound (BB) algorithm [69] to find the optimal integer-valued solution $\widehat{\mathbf{X}}$ of the concave objective. Once an estimate $\widehat{\mathbf{X}}$ is available, the update to $\widehat{\lambda }$ is also concave with the closed form solution $\widehat{\lambda }=\frac{1}{D}{\sum }_d{\widehat{\mathbf{x}}}_d$. The biconcavity of this objective function in X and λ makes this approach attractive, but it is unclear how to best initialize $\widehat{\lambda }$. We refer to this alternating baseline algorithm with the prefix “Alt” followed by the method of initialization. For example, for Alt-Random, we use random initialization with a small offset $({\widehat{\lambda }}_n\sim \text{Uniform}(0,1)+0.1)$ to avoid making any particular λ_n irretrievable from the start.

We also explore a few “guided” initialization processes. The quantity ${\sum }_n{\lambda }_n^{\ast }$ can hypothetically be estimated from data if P(x_d = 0∣λ) is significant and easily estimated from Y. In fact, quantification in microfluidics often relies on a clear identification of empty sample partitions (that is, where x_d = 0) [26]. This motivates a strategy of relaxing the problem by optimizing X with a Poisson assumption on the sum of each column ∑_n x_n,d rather than each element of X individually. The ${\sum }_n{\lambda }_n^{\ast }$-Oracle is given ${\sum }_n{\lambda }_n^{\ast }$ and optimizes for $P({\sum }_n{x}_{n,d}\mid {\sum }_n{\lambda }_n^{\ast })$ in place of $P({\mathbf{x}}_d\mid \widehat{\lambda })$ in (20). It is straightforward to show that this objective is also concave. Each estimate ${\widehat{\mathbf{x}}}_d$ is solved via BB from which $\widehat{\lambda }=\frac{1}{D}{\sum }_d{\widehat{\mathbf{x}}}_d$. We use the ${\sum }_n{\lambda }_n^{\ast }$-Oracle as its own baseline and as an initialization to our alternating framework $(\text{Alt-}{\sum }_n{\lambda }_n^{\ast })$. For the next guided initialization, we again use ${\sum }_n{\lambda }_n^{\ast }$ for an unbiased initialization where the first estimate of ${\widehat{\lambda }}_n=({\sum }_{n^{\prime }}{\lambda }_{n^{\prime }}^{\ast })∕N$ for all n (Alt-Unbiased). Finally, we used the output of SPoRe as an initial value for $\widehat{\lambda }$ (Alt-SPoRe). Alt-SPoRe can be understood as a way to use SPoRe to estimate $\widehat{\mathbf{X}}$ if needed.

Fig. 4 illustrates that SPoRe has the greatest tolerance to measurement noise whereas the ℓ₀-Oracle has the least. This comparison illustrates the value of incorporating the Poisson assumption in recovery; specifically, the integer and sparsity structures (perfectly captured by the ℓ₀-Oracle) are not sufficient for recovery under measurement noise. The alternating optimization algorithm’s behavior was unexpected; initialization (other than with Alt-SPoRe) does not appear to have a major influence on performance. Surprisingly, comparing $\text{Alt-}{\sum }_n{\lambda }_n^{\ast }$ versus ${\sum }_n{\lambda }_n^{\ast }$-Oracle and Alt-SPoRe versus SPoRe, alternating seems to worsen the performance under high noise. Our interpretation is that in high noise settings, the ability of SPoRe to not “overcommit” to a particular solution ${\widehat{\mathbf{x}}}_d$ may be especially effective when λ* is the signal of interest rather than X*. Any given measurement y_d may make the specific estimate ${\widehat{\mathbf{x}}}_d$ arbitrarily unreliable. In our alternating framework with ${\widehat{\mathbf{x}}}_d$ recovered separately for each d, errors on individual estimates accumulate. SPoRe instead makes gradient steps based on batches of observations, helping it maintain awareness of the full distribution of measurements.

Fig. 4.

AWGN tolerance of integer-restricted algorithms over 50 trials with M = 2, k = 3, N = 10, D = 100, G = 1, ∑_n ${\lambda }_n^{\ast }=2$.

C. Sparsity and ${\sum }_n{\lambda }_n^{\ast }$

We empirically tested the limitations of SPoRe’s recovery performance under very challenging conditions of M = 2, 3 ≤ k ≤ 7, N = 50, σ² = 10⁻². Here we set D = 1000 to better reflect the typical capabilities of biomedical systems, whereas D = 100 in our baseline comparisons was due to our budget on computational time strained by solving BB for ${\widehat{\mathbf{x}}}_d$. From our analysis and previous simulations, we expect that both k and the magnitudes of ${\lambda }_n^{\ast }$ will influence recovery. Fig. 5 probes when and why SPoRe fails. Fig. 5a illustrates SPoRe’s performance decreases with increasing k and ${\sum }_n{\lambda }_n^{\ast }$. To elucidate the cause of poor performance, Fig. 5b shows SPoRe’s performance under the same conditions when initialized at the optimum. SPoRe’s maintenance of high cosine similarity in this case means that in Fig. 5a, SPoRe is converging to incorrect optima (or terminating before convergence). These two figures depict fundamental limitations of stochastic optimization in a challenging landscape.

Fig. 5.

SPoRe’s performance and behavior as a function of k and ∑_n ${\lambda }_n^{\ast }$ over 50 trials. Common settings unless otherwise specified are M = 2, N = 50, D = 1000, G = 1, σ² = 10⁻². (a) Performance when initialized with standard $\widehat{\lambda }$ = 0.1. (b) Performance when initialized with $\widehat{\lambda }$ ≈ λ*, specifically ${\widehat{\lambda }}_n$ = max{ϵ, ${\lambda }_n^{\ast }$}. (c) Average variance of partial derivatives for indices n ∉ supp(λ*) evaluated at $\widehat{\lambda }$ ≈ λ*.

Moreover, Fig. 5c illustrates that MC gradients decrease in quality with high ${\sum }_n{\lambda }_n^{\ast }$ and k. In SPoRe, recall that we set a minimum ${\widehat{\lambda }}_n\ge ϵ={10}^{-3}$ so that nonzero $x_n^{\ast }$ have a chance of being sampled for all n. We keep S fixed as we increase k and ${\sum }_n{\lambda }_n^{\ast }$, and we see that the variance of the gradient increases at coordinates where ${\widehat{\lambda }}_n=ϵ$ and ${\lambda }_n^{\ast }=0$. Such an effect accounts for some drift from the optimum observed in Fig. 5b that increases with k, and we believe that it helps to explain the inability to converge to the optimum in Fig. 5a. Future work can explore alternative techniques for stochastic optimization and sampling. Practitioners may benefit significantly from reducing ${\sum }_n{\lambda }_n^{\ast }$ if faced with limitations in M. In a biosensing context, these results indicate that given a microfluidics system with fixed D, having fewer total analytes in the sample (i.e., smaller $D{\sum }_n{\lambda }_n^{\ast }$) can counterintuitively improve performance and may be particularly important if both M and G are limited.

However, note in (8) that SPoRe’s gradients are defined by an average of x_s weighted by p(y_d∣x_s). The previous result from Fig. 3c in which SPoRe performed well with ${\sum }_n{\lambda }_n^{\ast }\le 20$ when M = 10 illustrates that limitations of MC sampling may be offset by improving the ability of p(y∣x_s) to guide gradients. In Fig. 6, we explore this notion further for M-constrained systems by increasing G. One may wonder how increasing G compares to a CS problem with GM measurements (i.e., $\overline{\Phi }\in {\mathbb{R}}^{GM\times N}$). Although λ* is fixed across groups, the x_d are random such that there is no reasonable method to directly stack individual measurements y_d from multiple groups. Instead, we created a new baseline ℓ₁-Oracle GM SMV. Denote the average of measurements and signals in each group g as ${\overline{\mathbf{y}}}^{(g)}$ and ${\overline{\mathbf{x}}}^{(g)}$, respectively. Our new baseline stacks all ${\overline{\mathbf{y}}}^{(g)}$ into one measurement vector $\overline{\mathbf{y}}\in {\mathbb{R}}^{GM}$ and minimizes $\Vert \overline{\mathbf{y}}-\overline{\Phi }\lambda {\Vert }_2$ with respect to λ given ${\sum }_{n,d}{x}_{n,d}^{\ast }$ and λ ≥ 0. Stacking measurements and sensing matrices implicitly assumes that for each group, ${\overline{\mathbf{y}}}^{(g)}\approx {\Phi }^{(g)}{\lambda }^{\ast }$, or that ${\overline{\mathbf{x}}}^{(g)}\approx {\lambda }^{\ast }\forall g$. It can be easily shown that the relative errors in these approximations reduce with increasing D or λ*.

Fig. 6.

Performance of SPoRe (solid) vs. ℓ₁-Oracle GM SMV (dashed) as a function of G. Common settings are k = 7, N = 50, D = 1000, σ² = 10⁻². (a) Comparison with ∑_n ${\lambda }_n^{\ast }$ = 10, motivated by Fig. 5a. (b) Comparison with ∑_n ${\lambda }_n^{\ast }$ = 1.

Although increasing D is feasible in microfluidics, it generally corresponds with a reduction in λ* since a sample’s total analyte content is fixed. Therefore, in Fig. 6, we focus on the influence of the magnitude of λ*. In Fig. 6a, used the most challenging settings from Fig. 5 with k = 7 and ${\sum }_n{\lambda }_n^{\ast }=10$. As expected from our analysis in Section III-C, larger choices of M make SPoRe much more effective per sensor group, but near perfect recovery is achievable even with M = 1. However, note that the new oracle baseline performs almost identically to SPoRe, with SPoRe exhibiting a modest advantage only when GM is comparable to or less than k. When we reduce ${\sum }_n{\lambda }_n^{\ast }$ to 1 in Fig. 6b, the assumption that ${\overline{\mathbf{x}}}^{(g)}\approx {\lambda }^{\ast }$ becomes far less valid. As a result, the performance improvement with SPoRe is dramatic. For instance, what SPoRe achieves with M = 1 is only matched by the oracle baseline when M = 3. For applications in which ${\overline{\mathbf{x}}}^{(g)}\approx {\lambda }^{\ast }$ and GM > k, practitioners could consider reformulating the recovery problem as a standard CS problem. However, SPoRe is uniquely suited for systems with GM < k and is the best generalized approach for applications with smaller λ* or D.

D. Efficiency

For system design, it is helpful to know how many observations D are necessary and sufficient for stable estimation of $\widehat{\lambda }$. Such insight is often derived from the analysis of Fisher Information $\mathcal{I}$. Recall that for direct observations of Poisson signals ${\mathbf{x}}_d^{\ast }$, the ideal case, the MLE solution ${\widehat{\lambda }}_n={\sum }_d{x}_{n,d}^{\ast }∕D$ is an efficient estimator and achieves the Cramér–Rao bound such that $\text{var}({\widehat{\lambda }}_n)={\lambda }_n^{\ast }∕D$.

In Section III-B, we derived the reduction in ${\mathcal{I}}_{n,n}$ from MMVP measurements and found reason to expect that the reduction increases with k. Here, we empirically characterize this effect (Fig. 7). The matrix $\mathcal{I}$ is evaluated at λ*, so we only consider n ∈ supp(λ*). In MMVP, the variance of ${\widehat{\lambda }}_n$ will depend on Φ and n, but by redrawing random Φ and λ* over 50 trials, we hope to smooth out these dependencies and capture the broader effect of low dimensional projections. For a concise comparison considering n ∈ supp(λ*), we set all ${\lambda }_n^{\ast }=1$, pool all of ${\widehat{\lambda }}_n$ across all trials, and compute a single average variance for each k and D. Because the Fisher Information describes the optimization landscape near the optimum, we chose parameter settings (M = 2, G = 20) based on our results in Fig. 6a to be confident that SPoRe is arriving near the optimum and the estimation variance is not confounded by poor estimates.

Fig. 7.

Comparison of average variance of ${\widehat{\lambda }}_n$ from SPoRe versus the ideal Cramér-Rao (CR) bound as a function of D over 50 trials with n ∈ supp(λ*), ${\lambda }_n^{\ast }$ = 1, M = 2, N = 50, G = 20, σ² = 10⁻².

Fig. 7 shows a noticeable increase in the estimation variance and verifies that this deviation from the ideal bound is exponentially worsened in k. However, we argue that the variance quickly becomes negligible at reasonable D for practical purposes. Practitioners could consider the necessary precision of estimation and the maximum expected k for an application, increase D as needed, and worry little about the influence of noisy measurements in low dimensions.

V. Discussion

We have found that the structure in the MMVP problem can be easily exploited for substantial improvements in signal recovery. While compressed sensing of arbitrary integer signals has proven challenging in the past, Poisson constraints not only make the recovery problem tractable, but even significantly easier. Most inverse problems necessitate constraints that make the signal-to-measurement transformations nearly isometric: in compressed sensing, these manifest as restrictions on Φ noise, and the relationship between M, N, and k. In MMVP, recovery of λ* is theoretically possible under very lax conditions on Φ (Theorem III.2) and practically achievable as shown in our simulations.

In practice, our new SPoRe algorithm exhibits high performance even under deliberately challenging circumstances of high noise and M < k. Here, recovery with G = 1 appears particularly feasible even with noisy measurements if ${\sum }_n{\lambda }_n^{\ast }$ is small. Because the log-likelihood is not concave, SPoRe’s gradient ascent approach is not theoretically guaranteed to find a global optimum since local optima may exist. However, if they exist, we speculate that SPoRe is naturally poised to evade these traps due to stochasticity in its gradient steps from both batch draws and MC integrations.

We noted a few scenarios in which SPoRe’s MC sampling appears to cause issues with convergence or early termination that are generally associated with increases in k and ${\sum }_n{\lambda }_n^{\ast }$. We anticipate that further increases in N may also contribute to these effects. While k and N are entirely determined by the application, system designers can reduce ${\sum }_n{\lambda }_n^{\ast }$ by increasing the spatial or temporal sampling rate. In microfluidics, this adjustment translates to either generating more (smaller) partitions D given a fixed sample volume or diluting a sample prior to partitioning. Our initial implementation of SPoRe uses S = 1000, can easily run on personal computers, and is sufficient for systems with N < 10². This scale is appropriate for most applications in biosensing, and future work with parallelized or adaptive sampling strategies could improve the reliability of recovery for larger systems. Moreover, we found that increasing M and G appear to mitigate poor performance due to excessive sampling noise.

To date, CS implementations with low M and high G, based on our notation, reformulate the acquired measurements into a canonical CS problem with GM measurements [10]. As we discussed in Section IV-C, this reformulation implicitly assumes that the average signal in every group $({\overline{\mathbf{x}}}^{(g)})$ is approximately λ*. However, this assumption fails in biosensing in two common scenarios: the total number of analytes in a typical sample may be small, or the sensing mechanism may impose restrictions on D. In such cases, SPoRe is the superior approach allowing M < k, achieving high performance even with M = 1 under high noise. In this work, we found practical limits on the sparsity level k that we could accurately solve, hence the “S” in SPoRe, but future research should explore the conditions needed for the reliable recovery of non-sparse signals. High performance in this regime would inspire new methods in biosensing for the analysis of heterogeneous samples with many types of cells.

The ability to recover signals in MMVP, even in the extreme case of M = G = 1, is unprecedented in CS and offers a new paradigm for sensor-constrained applications. The current state-of-the-art method for achieving similar efficiency in microfluidics is to essentially guarantee single-analyte capture for classification by substantially increasing the sampling rate, whereas our MMVP framework is not reliant on such an intervention. Increasing G can make SPoRe reliable under harsh conditions, is straightforward with microfluidics, and offers a potent alternative to increasing the sampling rate. For example, diluting to a tolerable concentration is challenging with samples of unknown content such as in diagnostics applications, and increasing D either delays results or necessitates high throughput measurement acquisition which may not be feasible depending on the nature of the sensors. Our group is continuing research in CS-based microbial diagnostics [18] by working towards an in vitro demonstration of MMVP-based diagnostics with microfluidics.

Our initial theoretical and empirical results show the promise of MMVP, but there are many directions for further research. For instance, theoretical results that precisely relate M, N, D, k, λ* and noise such as in a recovery guarantee would be highly valuable. Moreover, SPoRe can accept any signal-to-measurement model p(y∣x). While we have proven identifiability under linear mappings with common noise models, SPoRe can be easily applied with any application-specific model² even if proving identifiability of the Poisson mixture is difficult. With growing interest in microfluidics, SPoRe’s promising performance in the MMVP problem warrants further research to ensure that the statistical assumptions underlying these new technologies are leveraged to their full potential.

Biographies

Pavan Kota received the B.S.E. in biomedical engineering from Case Western Reserve University (Cleveland, OH) in 2017.

He is currently pursuing his Ph.D. in the Department of Bioengineering at Rice University and an NIH NLM Fellow in Biomedical Informatics and Data Science. Pavan is primarily interested in the application of signal processing and machine learning techniques to biomedical diagnostics.

Daniel LeJeune (S’11) received the B.S. in engineering from McNeese State University (Lake Charles, LA) in 2014 and the M.S. in electrical and computer engineering from the University of Michigan in 2016.

He is currently a PhD candidate in the Department of Electrical and Computer Engineering at Rice University. His research interests include machine learning theory and adaptive algorithms.

Rebekah Drezek received her B.S.E. from Duke University (1996) and her M.Sc. (1998) and Ph.D. (2001) from the University of Texas at Austin, all in electrical engineering.

She is currently a professor and associate chair in the Department of Bioengineering at Rice University and a fellow of AIMBE. Her research interests include optical imaging, nanomedicine, and diagnostics.

Richard Baraniuk (S’85–M’93–SM’98–F’01) received the B.S. from the University of Manitoba, Canada (1987), the M.S. from the University of Wisconsin-Madison (1988), and the Ph.D. from the University of Illinois at Urbana-Champaign (1992), all in electrical engineering.

He is currently the Victor E. Cameron Professor of Electrical and Computer Engineering at Rice University and the Founding Director of OpenStax (openstax.org). His research interests lie in new theory, algorithms, and hardware for sensing, signal processing, and machine learning. He is a Fellow of the American Academy of Arts and Sciences, National Academy of Inventors, American Association for the Advancement of Science, and IEEE. He has received the DOD Vannevar Bush Faculty Fellow Award (National Security Science and Engineering Faculty Fellow), the IEEE James H. Mulligan, Jr. Education Medal, and the IEEE Signal Processing Society Technical Achievement, Education, Best Paper, Best Magazine Paper, and Best Column Awards. He holds 35 US and 6 foreign patents.

Appendix

A. Proof of Theorem III.2

We use a direct proof, assuming $P({\mathcal{C}}_{\mathbf{u}}\mid \lambda )=P({\mathcal{C}}_{\mathbf{u}}\mid {\lambda }^{\prime })\forall \mathbf{u}$ and proving the resulting implication λ = λ′. Let ${\mathbf{z}}^{(\mathbf{x})}\in {\mathbb{Z}}^N$ be such that $\mathbf{x}+{\mathbf{z}}^{(\mathbf{x})}\in {\mathcal{C}}_{\mathbf{x}}$. By Definition III.4, $\mathbf{x}+{\mathbf{z}}^{(\mathbf{x})}\in {\mathbb{Z}}_{+}^N$ and ${\mathbf{z}}^{(\mathbf{x})}\in \mathcal{N}(\Phi )$.

Lemma A.1. If $\mathcal{N}(\Phi )\cap {\mathbb{R}}_{+}^N=\{0\}$, and $P({\mathcal{C}}_0\mid \lambda )=P({\mathcal{C}}_0\mid {\lambda }^{\prime })$, then ${\sum }_n{\lambda }_n={\sum }_n{\lambda }_n^{\prime }$.

Proof. The null space condition on Φ means that ${\mathcal{C}}_0=\{0\}$; there is no vector z⁽⁰⁾ that satisfies $0+z^{(0)}\in {\mathbb{Z}}_{+}^N$ other than z⁽⁰⁾ = 0. Therefore, $P({\mathcal{C}}_0\mid \lambda )=P({\mathcal{C}}_0\mid {\lambda }^{\prime })\Rightarrow e^{-{\sum }_n{\lambda }_n}=e^{-{\sum }_n{\lambda }_n^{\prime }}\Rightarrow {\sum }_n{\lambda }_n={\sum }_n{\lambda }_n^{\prime }$.

We now turn our attention to the one-hot collision sets. Let e_j denote the jth standard basis vector. By Definition III.4, ${\mathcal{C}}_{{\mathbf{e}}_j}=\{\mathbf{x}:\Phi \mathbf{x}={\varphi }_j,\mathbf{x}\in {\mathbb{Z}}_{+}^N\}$. For ${\mathcal{C}}_{{\mathbf{e}}_j}$ that contain only e_j, we have the following result:

Lemma A.2. If $\mathcal{N}(\Phi )\cap {\mathbb{R}}_{+}^N=\{0\}$ and ${\mathcal{C}}_{{\mathbf{e}}_j}=\{{\mathbf{e}}_j\}$, then ${\lambda }_j={\lambda }_j^{\prime }$.

Proof. The restriction on ${\mathcal{C}}_{1j}$ means $P({\mathcal{C}}_{1j}\mid \lambda )=P({\mathcal{C}}_{1j}\mid {\lambda }^{\prime })\Rightarrow {\lambda }_j{e}^{-{\sum }_n{\lambda }_n}={\lambda }_j^{\prime }{e}^{-{\sum }_n{\lambda }_n^{\prime }}$. Applying Lemma A.1 yields ${\lambda }_j={\lambda }_j^{\prime }$.

By similar arguments to Lemmas A.1 and A.2, we can prove Corollary III.3 under the assumption that there are no collisions instead of the null space condition. When the elements of Φ are independently drawn from continuous distributions, the collision of any particular x and x′ occurs with probability zero. Since ${\mathbb{Z}}_{+}^N\times {\mathbb{Z}}_{+}^N$ is countably infinite, there are no collisions almost surely. As such, ${\mathcal{C}}_{{\mathbf{e}}_j}=\{{\mathbf{e}}_j\}\forall j$, and therefore λ = λ′.

For the conditions in Theorem III.2, the following Lemma states the existence of at least one j satisfying ${\mathcal{C}}_{{\mathbf{e}}_j}=\{{\mathbf{e}}_j\}$:

Lemma A.3. If $\mathcal{N}(\Phi )\cap {\mathbb{R}}_{+}^N=\{0\}$ and ϕ_n ≠ ϕ_n′, ∀n, n′ ∈ {1, … , N} with n ≠ n′, then ∃ j such that ${\mathcal{C}}_{{\mathbf{e}}_j}=\{{\mathbf{e}}_j\}$.

Proof. If ${\mathcal{C}}_{{\mathbf{e}}_j}=\{{\mathbf{e}}_j\}$, then ∄ z^(e_j) ≠ 0. Define P as the number of one-hot collision sets that contain more than just e_j, and note that P ≤ N. Without loss of generality, let us say that ${\mathcal{C}}_{{\mathbf{e}}_j}$ for j ∈ {1, … , P} meet this condition. Lemma A.3 effectively says that P < N, such that N – P > 0 one-hot collision sets contain only e_j. We proceed with a proof by contradiction by assuming P = N.

By our null space condition, z^(e_j) must contain both positive and negative integers. There are two additional conditions on nontrivial z^(e_j). First, because ${\mathbf{e}}_j+{\mathbf{z}}^{({\mathbf{e}}_j)}\in {\mathbb{Z}}_{+}^N$, the only negative component of z^(e_j) is $z_j^{({\mathbf{e}}_j)}=-1$. To see this, if $z_i^{({\mathbf{e}}_j)}$ for i ≠ j were negative, then e_j + z^(e_j) would be negative at index i, and if ${\mathbf{z}}_j^{({\mathbf{e}}_j)}$ were less than −1, then e_j + z^(e_j) would be negative at index j. Second, z^(e_j)’s positive elements must total to at least 2. A single positive element of $z_i^{({\mathbf{e}}_j)}=1$ would imply that ϕ_i = ϕ_j, violating a condition on Φ.

With P = N, let us concatenate the z^(e_j) column vectors into a matrix for visualization.

$$\mathbf{Z}=\left[\begin{array}{cccc}\hfill -1\hfill & \hfill z_1^{({\mathbf{e}}_2)}\hfill & \hfill \dots \hfill & \hfill z_1^{({\mathbf{e}}_N)}\hfill \\ \hfill z_2^{({\mathbf{e}}_1)}\hfill & \hfill -1\hfill & \hfill \dots \hfill & \hfill z_2^{({\mathbf{e}}_N)}\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill z_N^{({\mathbf{e}}_1)}\hfill & \hfill z_N^{({\mathbf{e}}_2)}\hfill & \hfill \dots \hfill & \hfill -1\hfill \end{array}\right].$$ (22)

Note that each column z^(e_j) in this matrix is symbolic for any vector that satisfies the conditions we described. All columns of Z are in $\mathcal{N}(\Phi )$. Any linear combination of vectors in $\mathcal{N}(\Phi )$ are in $\mathcal{N}(\Phi )$. Let $\mathcal{S}$ represent a subset of indices of the columns of Z and let ${\mathbf{z}}^{\mathcal{S}}\triangleq {\sum }_{j\in \mathcal{S}}{\mathbf{z}}^{({\mathbf{e}}_j)}$.

First, let $\mathcal{S}=\{1,\dots ,N\}$. Because all off-diagonal components in Z are nonnegative and because ${\mathbf{z}}^{\mathcal{S}}$ must have one negative value, one of the rows of Z must be entirely zero except for the −1 on the diagonal. Note the ordering of the columns in Z is arbitrary, so without loss of generality, let this be the first row. Now, let’s say that $\mathcal{S}=\{2,3,\dots ,N\}$. The same logic holds: at least one row must contain all zeros except for the −1. Without loss of generality, we can set [Z]_2,3, [Z]_2,4, … [Z]_{2, N} = 0. Repeating this process, we get a lower triangular matrix:

$$\mathbf{Z}=\left[\begin{array}{ccccc}\hfill -1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill \dots \hfill & \hfill 0\hfill \\ \hfill z_2^{({\mathbf{e}}_1)}\hfill & \hfill -1\hfill & \hfill 0\hfill & \hfill \dots \hfill & \hfill 0\hfill \\ \hfill z_3^{({\mathbf{e}}_1)}\hfill & \hfill z_3^{({\mathbf{e}}_2)}\hfill & \hfill -1\hfill & \hfill \dots \hfill & \hfill 0\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill z_N^{({\mathbf{e}}_1)}\hfill & \hfill z_N^{({\mathbf{e}}_2)}\hfill & \hfill z_N^{({\mathbf{e}}_3)}\hfill & \hfill \dots \hfill & \hfill -1\hfill \end{array}\right].$$ (23)

However, examining the final column, we see that z^(e_N) is a vector of all zeros and one −1, such that it cannot be in $\mathcal{N}(\Phi )$, proving Lemma A.3 by contradiction. □

Proof of Theorem III.2: Lemma A.3 confirms P < N, meaning that we can form the concatenated matrix of z^e_j vectors:

$$\mathbf{Z}=\left[\begin{array}{cccc}\hfill -1\hfill & \hfill 0\hfill & \hfill \dots \hfill & \hfill 0\hfill \\ \hfill z_2^{({\mathbf{e}}_1)}\hfill & \hfill -1\hfill & \hfill \dots \hfill & \hfill 0\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill z_P^{({\mathbf{e}}_1)}\hfill & \hfill z_P^{({\mathbf{e}}_2)}\hfill & \hfill \dots \hfill & \hfill -1\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill z_N^{({\mathbf{e}}_1)}\hfill & \hfill z_N^{({\mathbf{e}}_2)}\hfill & \hfill \dots \hfill & \hfill z_N^{({\mathbf{e}}_P)}\hfill \end{array}\right].$$ (24)

Let us now apply $P({\mathcal{C}}_{{\mathbf{e}}_P}\mid \lambda )=P({\mathcal{C}}_{{\mathbf{e}}_P}\mid {\lambda }^{\prime })$. For all $\mathbf{x}\in {\mathcal{C}}_{{\mathbf{e}}_P}$,

$$\prod _{n=1}^N\frac{{\lambda }_n^{x_n}}{x_n!}-\prod _{n=1}^N\frac{{\lambda }_n^{\prime }{}^{{}^{x_n}}}{x_n!}=0,$$ (25)

$$\left(\prod _{\forall i>P}\frac{{\lambda }_i^{x_i}}{x_i!}\right)\left(\frac{{\lambda }_P^{x_P}}{x_P!}-\frac{{\lambda }_P^{\prime }{}^{{}^{x_P}}}{x_P!}\right)=0,$$ (26)

where Lemma A.1 (${\sum }_n{\lambda }_n={\sum }_n{\lambda }_n^{\prime }$) yields the first equality,and Lemma A.2 (all ${\lambda }_i={\lambda }_n^{\prime }\forall i>P$) yields the second equality when combined with the fact that x_i = 0 for i < P due to (24). The only $\mathbf{x}\in {\mathcal{C}}_{{\mathbf{e}}_P}$ with x_P ≠ 0 is e_P which simplifies (26) to ${\lambda }_P={\lambda }_P^{\prime }$.

Now we have ${\lambda }_i={\lambda }_i^{\prime }\forall i>P-1$. Following the same arguments, we can start from $P({\mathcal{C}}_{{\mathbf{e}}_{P-1}}\mid \lambda )=P({\mathcal{C}}_{{\mathbf{e}}_{P-1}}\mid {\lambda }^{\prime })$ and arrive at ${\lambda }_{P-1}={\lambda }_{P-1}^{\prime }$. Applying this repeatedly ultimately yields λ = λ′, proving Theorem III.2. □

B. Proof of Proposition III.5

Proof. By straightforward integration,

$${\mathbb{E}}_{\mathbf{y}\mid n^{\ast }}[p(\mathbf{y}\mid n)]\propto \underset{\triangleq \kappa (n,n^{\ast })}{\underbrace{\text{exp}\left\{-\frac{1}{4{\sigma }^2}\Vert {\varphi }_n-{\varphi }_{n^{\ast }}{\Vert }_2^2\right\}}}.$$ (27)

Therefore, given the constraints ∥λ∥₁ ≤ 1 and λ_n ≥ 0, the first-order KKT condition is

$${\mathbb{E}}_{n^{\ast }}\left[\frac{\kappa (n^{\ast })}{\langle \kappa (n^{\ast }),\lambda \rangle }\right]=c\mathbf{s}-\mu ,$$ (28)

where $\kappa (n^{\ast })=(\kappa (n,n^{\ast }){)}_{n=0}^N$, s ∈ ∂∥λ∥₁, c ≥ 0, and μ_n ≥ 0. By complementary slackness, μ_nλ_n = 0 for all n. Because K is symmetric, we can rewrite the above as

$$\mathbf{K}\left(\frac{{\lambda }^{\ast }}{\mathbf{K}\lambda }\right)=c\mathbf{s}-\mu ,$$ (29)

where the fraction represents element-wise division. Solving for λ and rescaling μ, we obtain the desired expression. □

C. Proof of Theorem III.6

Proof. Let κ_g be defined the same as κ from (27) with ${\varphi }_n^{(g)}$. Then again by straightforward integration,

$$\tilde{\kappa }(n,n^{\ast })\triangleq {\mathbb{E}}_g[{\kappa }_g(n,n^{\ast })]$$ (30)

$$=\{\begin{array}{cc}1\hfill & n=n^{\ast },\hfill \\ {\left(\frac{2{\sigma }^2}{2{\sigma }^2+1}\right)}^{M∕2}\hfill & n\ne n^{\ast },0\in \{n,n^{\ast }\},\hfill \\ {\left(\frac{{\sigma }^2}{{\sigma }^2+1}\right)}^{M∕2}\hfill & n\ne n^{\ast },0\notin \{n,n^{\ast }\}.\hfill \end{array}\phantom{\}}$$ (31)

Let $\tilde{\mathbf{K}}=(\tilde{\kappa }(n,n^{\prime }){)}_{n,n^{\prime }=0}^N$, and let $\widehat{\mathbf{K}}=(\tilde{\kappa }(n,n^{\prime }){)}_{n,n^{\prime }=1}^N$ be the sub-matrix of $\tilde{\mathbf{K}}$ excluding n = 0. Then

$$\widehat{\mathbf{K}}=(1-a)\mathbf{I}+a\mathbf{J},$$ (32)

where $a={\left(\frac{{\sigma }^2}{{\sigma }^2+1}\right)}^{M∕2}$ and J is a matrix of all ones.Leveraging the block matrix inverse, we observe that we have the form

$${\tilde{\mathbf{K}}}^{-1}=\left[\begin{array}{cc}\hfill \frac{1}{1-a}\mathbf{I}+b\mathbf{J}\hfill & \hfill c1\hfill \\ \hfill c{1}^T\hfill & \hfill d\hfill \end{array}\right],$$ (33)

assuming the final column corresponds to n = 0, for some scalars b, c, and d. Using the formula from Proposition III.5 and the fact that s – μ = 1 by assumption, we conclude that for n > 0, ${\tilde{\lambda }}_n\propto {\lambda }_n^{\ast }+C$ for some C. Rote algebra verifies that the constant of proportionality is non-negative. □