Toward deterministic compressed sensing

During the past decade, compressed sensing has delivered significant advances in the theory and application of measuring and compressing data. Consider capturing a 10-megapixel image with a digital camera. Emailing an image of this size requires an unnecessary amount of storage space and bandwidth. Instead, users employ a standard digital compression scheme, such as JPEG, to represent the image as a 64-kb file. The compressed image is completely recognizable even though the dimension of the compressed version is a tiny fraction of the original 10 million dimensions. Compressed sensing takes this mathematical phenomenon one step further. Is it possible to capture the pertinent information, such as the 64-kb image, without first measuring the full 10 million pixel values? If so, how should we perform the measurements? If we capture the important information, can we still reconstruct the image from this limited number of observations? Compressed sensing exploded in 2004 when Donoho (1, 2) and Candes and Tao (3) definitively answered these questions by incorporating randomness in the measurement process. Because engineering a truly random process is impossible, a major open problem in compressed sensing is the search for deterministic methods for sparse signal measurement that capture the relevant information in the signal and permit accurate reconstruction. In PNAS, Monajemi et al. (4) provide a major step forward in understanding the potential for deterministic measurement matrices in compressed sensing.

Capturing digital images on a camera is simple; however, there are many applications in which the measurement process has a much greater underlying cost. MRI is a prime example of a high-impact compressed sensing application. For most MRI examinations, a patient is required to lie still in a confined space for approximately 45 min. In some situations, compressed sensing has generated diagnostic-quality magnetic resonance images using only 10% as many measurements (5). MRI is only a single example of compressed sensing applications, which extend well beyond imaging and include computed tomography, electrocardiography, multispectral imaging, seismology, analog-to-digital conversion, radar, X-ray holography, astronomy, DNA sequencing, and genotyping (Rice Compressed Sensing Resources; http://dsp.rice.edu/cs).

Traditional signal processing procedures measure the full signal directly and apply standard compression routines for storage or transmission. When needed, the original signal can be reconstructed by inverting the linear compression procedure. Compressed sensing transfers the workload from the measurement process to the signal reconstruction. Although the measurement process remains linear, the reduced number of measurements forces a highly nonlinear reconstruction process.

Rather than taking point measurements of the entire signal, compressed sensing uses more sophisticated measurement schemes that acquire information throughout the signal and mix the information into relatively few numerical values. Decoding these complicated measurements from the underdetermined system of equations is therefore considerably more challenging than most other signal reconstruction techniques. In fact, because the system is underdetermined and at least one signal could have generated the linear measurements, there exist infinitely many signals that generate the exact same measurements. Compressed sensing relies on the assumption that the original signal has a low information content compared with its physical dimension. Typically, the low information content is interpreted as sparsity whereby a signal is “sparse” when the number of nonzero entries in the signal’s digital representation is dramatically smaller than its ambient dimension. This assumption is justified by the vast literature on signal processing in which, for example, images are known to be sparse in, say, the wavelet domain. That is, images have very few large coefficients when represented in a wavelet basis and can be accurately approximated by using only these large coefficients.

A traditional technique for selecting a particular solution of an underdetermined linear system is to form the least-squares solution by simply multiplying the measurement values by the pseudoinverse of the measurement matrix. Solving the least-squares problem produces the signal with the minimum l₂ norm (the square root of the sum of the squares of the entries) among the infinite set of signals that produce the same measurements. Minimizing the l₂ norm returns a signal whose total energy is distributed throughout the signal and therefore fails to meet the sparsity assumption. In the compressed sensing regime, one wishes to obtain the signal with the fewest number of nonzero entries. This presents a combinatorial optimization problem whose naive solution is as hard as the famous traveling salesman problem. For some time, it has been observed that replacing the least-squares problem with a minimization of the l₁ norm (the sum of the absolute values of the entries) produces a sparse solution. The frenzy surrounding compressed sensing began when Donoho (1, 2, 6) and Candes et al. (3, 7) proved sufficient conditions that ensure the solution to the l₁ minimization problem coincides with the sparsest solution from the combinatorial optimization problem. The beauty of this finding is that the intractable combinatorial optimization is replaced by a tractable convex optimization problem.

The geometric condition of Donoho (1, 2, 6), namely neighborliness of an associated polytope, and the linear algebraic condition of Candes et al. (3, 7), known as the restricted isometry property, are deterministic conditions. However, checking the validity of either of these conditions is also a combinatorial problem. This theoretical obstacle was overcome by analyzing random matrices such as Gaussian matrices whose entries are drawn independently and identically from a normal distribution. They proved that measurement matrices drawn from certain random matrix ensembles captured the pertinent information with very few measurements. Moreover, sparse signals could then be accurately reconstructed by solving the tractable, convex l₁ minimization problem. This combined the measurement and compression processes into a single random measurement process.

Monajemi et al. (4) bring together two major lines of research in compressed sensing, the universality hypothesis and the search for deterministic measurement matrices. The universality hypothesis, formally stated by Donoho and Tanner (8), claims that many families of random matrices exhibit the same signal recovery performance via l₁ minimization as the Gaussian matrices. Donoho and Tanner (1, 2, 9) formally proved that the Gaussian measurement ensemble exhibits a phase transition whereby the probability of successful recovery abruptly changes from 1 to 0 as a function of two parameters defining the compressed sensing problem (Fig. 1). The universality hypothesis was empirically established by Donoho and Tanner (8) for several random matrix ensembles in that their observed success and failure of signal reconstruction under l₁ minimization matches the performance of the Gaussian ensemble. A proof of the universality hypothesis would extend nearly all of the compressed sensing theory established for Gaussian ensembles to a much larger class of random matrix ensembles. In mid-2012, Bayati, Lelarge, and Montanari announced a major advance proving the Universality Hypothesis for a wide class of random matrices (10).

Fig. 1.

Universality hypothesis: random Fourier measurements. Let k be the number of nonzero entries in a signal of length N, and n the number of linear measurements observed through the n × N measurement matrix with k < n < N. Two ratios, ρ = k/n (vertical axis) and δ = n/N (horizontal axis), define the compressed sensing phase space in the unit square 0 < ρ, δ < 1. The black curve is the Gaussian measurement matrix phase transition defined by a function ρ*(δ): if ρ < ρ*(δ), then l₁ minimization successfully reconstructs almost every signal; it almost always fails when ρ > ρ*(δ). Shaded attribute represents the fraction of realizations in which l₁ minimization successfully reconstructs a signal measured by a random subset of n rows of a Fourier matrix. [Reprinted from ref. 8 by permission of the Royal Society.]

An independent line of research seeks to remove randomness from the compressed sensing regime. The main tool in this pursuit is the restricted isometry property of Candes and Tao (3). DeVore presented explicit constructions of matrices satisfying the restricted isometry property, but these constructions require the number of measurements to scale with the square of the sparsity (11). In contrast, the Gaussian matrices require the measurements to scale linearly with the sparsity plus a minor logarithmic penalty. Several other groups have continued this line of work, improving on the requisite number of measurements, for example (12, 13); so far, no theoretical results for deterministic matrices match the optimal measurement rate achieved by Gaussian matrices. Alternative approaches for analyzing deterministic matrices (14–16) established theoretical justification for their successful sparse signal recovery. This important line of research has not yet established deterministic compressed sensing performance on par with the random measurement models.

Monajemi et al. (4) present overwhelming empirical evidence that the universality hypothesis should now also include many deterministic measurement ensembles. Through considerable experimental analysis, they have shown that nine deterministic matrix families exhibit the same phase transition behavior as Gaussian matrices when the signal is reconstructed via l₁ minimization. Most of these deterministic matrices come from the work discussed in the preceding paragraph. The potential advantages of deterministic measurements in applications cannot be understated. Not only does randomness present considerable engineering challenges for implementation, but dense random matrices consume large amounts of memory and require computationally expensive matrix multiplications. Most of the reported deterministic matrix families avoid the need to store the entire matrix and boast fast matrix multiplications akin to the fast Fourier transform, providing massive computational gains. The incorporation of deterministic measurement matrices in the universality hypothesis not only bolsters the importance of this conjecture, it opens a wide range of new possibilities and intriguing questions for compressed sensing.