Complementary compressive imaging for the telescopic system

Abstract

Conventional single-pixel cameras recover images only from the data recorded in one arm of the digital micromirror device, with the light reflected to the other direction not to be collected. Actually, the sampling in these two reflection orientations is correlated with each other, in view of which we propose a sampling concept of complementary compressive imaging, for the first time to our knowledge. We use this method in a telescopic system and acquire images of a target at about 2.0 km range with 20 cm resolution, with the variance of the noise decreasing by half. The influence of the sampling rate and the integration time of photomultiplier tubes on the image quality is also investigated experimentally. It is evident that this technique has advantages of large field of view over a long distance, high-resolution, high imaging speed, high-quality imaging capabilities, and needs fewer measurements in total than any single-arm sampling, thus can be used to improve the performance of all compressive imaging schemes and opens up possibilities for new applications in the remote-sensing area.

Compressed sensing (CS)¹,²,³ is an amazing sampling paradigm that goes against the common wisdom in data acquisition followed by compression. CS theory asserts that one can reconstruct the signals known a priori to be sparse or compressive in some basis from far fewer measurements than traditional approaches. What is remarkable is that it allows a sensor to very efficiently acquire just the important information in a signal without trying to measure full-length signal, so it saves the storage and facilitates the transmission. The general robustness of CS is so-called restricted isometry property (RIP)⁴. In a nutshell, in order to satisfy RIP, we should find sensing matrices with the property that column vectors taken from arbitrary subsets are approximately orthogonal. It is found that random matrices with independent and identically distributed entries, e.g., Gaussian or ±1 binary entries, are largely incoherent with any fixed basis⁵. In truth, the random sampling theorem is a very effective sensing mechanism and has triggered the many related CS applications. The single-pixel camera⁶,⁷,⁸ proposed by R. G. Baraniuk et al. is a good example, which uses a digital micromirror device (DMD) to perform light intensity modulation, and a bucket (single-pixel) detector without spatial resolution to measure merely the total intensity impinging on it. To our knowledge, this system is very similar to computational ghost imaging⁹ proposed by J. H. Shapiro. This single-pixel camera scheme, in particular, has gained considerable attention in the research community because of its highly undersampled measurements, high luminous flux and the fact that it only needs one pixel to acquire a two-dimensional image without raster scanning. Recently, this scheme has been widely used in many areas, including optical encryption¹⁰,¹¹, adaptive computational ghost imaging¹²,¹³, time-of-flight range acquisition¹⁴, and fluorescence microscopy for biological and hyperspectral imaging¹⁵. However, for the practical implementation of a CS-based image acquisition system, it is essential to determine which kind of measurement matrix should be used since the patterns on the DMD constitute a binary matrix taking on the two values 0 or 1. Generally, a measurement matrix can be generated by randomly drawing from a fast transform such as a Walsh, Hadamard, or noiselet transform⁷. And then it will be converted to a binary matrix. Take a Hadamard transform for example, V. Studer et al.¹⁵ use a Hadamard pattern h where each entry is either −1 or +1, and make a shifted and rescaled version of h via ϕ = (h + 1)/2. Actually, the binary matrix doesn't necessarily obey the Bernoulli distribution. R. Berinde and P. Indyk¹⁶ presented sequential sparse matching pursuit, which depends on a scattered binary matrix with the number of 1 much smaller than that of 0. This method not only saves the storage but also gets a better image quality than traditional Bernoulli distributed measurement matrix consisting of 0 or 1. Nevertheless, in the actual optical system, it sacrifices the flux and will fail to work under specific circumstances where the luminous flux is low. These methods all use only a single arm of micro-mirror reflection orientation. Furthermore, to our knowledge, the binary matrix taking on the two values 0 or 1 is not the best choice for compressive imaging.

Recently, the analyzing work of N. D. Hardy and J. H. Shapiro¹⁷ have indicated that computational reflective ghost imaging is feasible for remote sensing. As compressive imaging technique can be transplanted to ghost imaging, C. Q. Zhao et al.¹⁸ have extended pseudo-thermal light ghost imaging to the area of remote-imaging and proposed a high-resolution lidar system based on ghost imaging. The field of view (FOV) of the receiving system as well as the emitting system at 900 m range is about 2 m and 1 m, respectively. G. A. Howland et al.¹⁹ have also experimentally demonstrated CS laser radar for 3D imaging in the laboratory with objects at depths ranging from 0.3 to 2.8 m from the device. All these experimental setups can only detect targets with active pulsed laser illumination, acquiring relatively small FOV over either a long or short distance.

In this paper, we demonstrated a sampling approach for the practical telescopic system named complementary compressive imaging (CCI), based on the traditional single-pixel camera but making full use of ±12 degrees double arm detection. High-resolution large-FOV imaging has been experimentally acquired at a distance about 2.0 km range.

Results

Experimental setup

The schematic diagram of the experimental apparatus is given in Fig. 1. Our optical setup is based on a Cassegrain telescopic unit and a standard single-pixel camera. The object is Tsinghua Tongfang logo on the light-emitting diode (LED) billboard of size 2.6 m × 2.6 m each Chinese character, located on the roof of their working building about 2 kilometers away from our test site. The photons emitting from the target are received by a Cassegrain telescope of aperture 120 mm and focal length 4104 mm, which can be focused by moving the secondary curved mirror of diameter 26 mm. Its maximum FOV on the target plane is of diameter 20 m at 10.0 km range, so it covers large FOV over a long distance, which is very useful for remote huge target imaging. Then the light is imaged onto a DMD. The DMD is a 1024 × 768 array of addressable micro-mirrors of size 13.68 μm × 13.68 μm each, which rotates about a hinge and can be shifted between two positions oriented at +12 degrees (1) or −12 degrees (0) with respect to the surface of the DMD. Therefore, the light can be reflected to two directions depending on the pattern encoded on the DMD. In Fig. 1, we set the left reflection orientation for +12 degrees, the right for −12 degrees. In our experimental apparatus, the highest frame frequency of the DMD is 32,552 Hz. Two photomultiplier tubes (PMTs) (of the same type Hamamatsu H7468-20) are used as the bucket (single-pixel) detectors for ultra-weak light detection, which can be operated at up to 450 Hz. Although our technique is limited by the readout frequency of PMTs, it is still reasonably fast. These PMTs are located in the left and right reflection directions to record the total intensity of two orientations in form of current signals, respectively. In this system, some neutral density filters inserted in front of PMTs attenuate the light intensity in order to avoid PMTs to be light-saturated. To assure perfect focus and make the image match the size of the screen, before starting the sampling, we insert an optical unit consisting of a beam splitter (BS) and a charge-coupled device (CCD) camera in front of DMD, about 50 mm away. After all the measurements have been done, a fine image can be reconstructed by CCI technique.

Figure 1. Experimental apparatus for complementary compressive imaging using a Cassegrain telescope, a DMD, and two photomultiplier tubes.

Experimental results

Figure 2 illustrates the process that creates complementary compressive images. In the experiment, the images of the target often fill the entire screen of DMD, but it will take a long time and huge memory storage for CS to recover such a large image. To simplify the computation but not weaken the total flux, for a 768 × 768 pixels image in the center of the DMD, we set the resolution of the image to be 12 × 12 pixel size, which is also the minimum size of each speckle (white or black). Then we encode M binary speckle patterns (actually 768 × 768 pixels for each pattern but can be easily converted to 64 × 64 pixels) on the DMD, and will obtain a final 64 × 64 pixels image. By reshaping each pattern into a row vector (1 × 4096), the measurement matrix A (M × 4096) is obtained after all the measurements having been performed. Meanwhile, the values recorded by the PMTs constitute a column vector b (M × 1). According to the measurement matrix and recorded values, we can retrieve the image by CS algorithm. As shown in Fig. 2, images in the first row are recovered from the data in the left reflection orientation; the ones in the next row reconstructed from the data in the right. Now we calculate the difference vector of these two data vectors, then divide each element in the difference vector by 2, and the result will be treated as new measured values. In the meantime, we set . After that we acquire the complementary compressive images as shown in the third row of Fig. 2. For comparison, we took a telescopic image of the target by a CCD.

Figure 2. Reconstruction results of a natural target (one Chinese character of Tsinghua Tongfang logo on the LED billboard) versus the sampling rate in the single-arm.

Now we define the sampling rate as the ratio between the number of measurements and the length of the image signal. To demonstrate the effects of the sampling rate on traditional single-pixel cameras and CCI, the experimental reconstruction results for the same target in the identical detection conditions are given in the different columns of Fig. 2. We can see that as the sampling rate increases, the image quality of these methods all becomes better. Although there exists atmospheric turbulence, CCI can still reconstruct the images with sharper edges than those recovered from any single arm. In this experiment, only the center 768 × 768 pixels of the DMD are effective for imaging, but the light from all the 1024 × 768 pixels of the DMD is collected. During each flip time interval of the frames, in order to reduce the influence of the ineffective pixels, the two 768 × 128 pixels margins of the DMD are firstly all set to zeros for the left arm measurements, then all reversed to ones for the right arm measurements, but with the pattern in the center effective region of DMD unchanged. By measuring the left arm and the right arm alternately, we can ensure that each frame pair of pixel size 768 × 768 for the both arms of the system is complementary. Additionally, the noise in CCI may be incurred by the stroboscopic LED with bright and dark change or the observational error before and after reversing micro-mirrors. These factors may also lead to the inconsistent distribution of the noise in both arms, which may be manifested in the mean or variance or both. In the future, if the full-screen micro-mirrors are all used, adopting the synchronous measurements will easily weaken the influence of these factors.

The width of slits in the red circles is 20 cm. We draw a three-dimensional map of the image recovered at the single-arm sampling rate of 14.6% (29.2% in total) by setting pixel coordinates as x-y axis and gray values as z axis, in which the slits can be distinguished well while the other two narrower slits can hardly be seen. Thus high-resolution is also demonstrated, about 2.0 km range with 20 cm resolution. A further experiment for imaging a target at a distance about 10 km and quantitative analysis will be presented in a future paper. In order to get a better image quality in CCI, we have tried methods of increasing the integration time of PMTs, which defines how long the PMTs acquire the input light intensity signal during each flip time interval of the micro-mirrors. As the integration time of PMT increases, the signal will be enlarged, while the noise will be averaged, so the image quality ought to become relatively satisfying. In view of this idea, we performed an experiment to check the validity of this theory, and the results are given in Fig. 3. We can see that the quality is roughly proportional to the integration of PMTs. It is interesting to note that there are some oscillations due to the atmospheric turbulence. Actually, from the CCD, we will see the images shaking all the time, which may also partly because of the wind.

Figure 3. Complementary compressed images of the Chinese character “fang” versus the integration time of PMTs over the duration of each pattern, and with the same sampling rate 40.53% (in total).

Performance evaluation

To get a more quantitative measure of the reconstructed image quality, we calculated the mean square error (MSE)

of recovered image as compared to the original image T_o of pixel size p × q. Naturally, the smaller the MSE value is, the better the quality of the image recovered. We created a head phantom image²⁰ as the original image, which is normalized to a range of 0–1, and did some numerical simulation experiments. The random matrix used here also consisted of two values 0 or 1. Figures 4(a)–(b) show the images, the details of which are all degraded, reconstructed from the data in the left and right reflection directions with 30% sampling rate, respectively. Figure 4(c) gives the result with better visibility restored from a subtracted and rescaled version of these two measured vectors. Actually, the number of measurements has increased to double its original one. In order to compare these three sets of data more intuitively, we treat the sampling rate in the single-arm as the horizontal axis of Figs. 4(d)–(e). As shown in Fig. 4(d), it is clearly visible that the MSE saturates at a pretty high level when using just one arm, while the result is orders of magnitude better when using both arms. This is a general result, whether there is additional random noise added to the imaging process or not. Furthermore, the advantage will be highlighted under conditions that there exists independent and identically distributed noise in both arms. From the curves, we can say that the total number of measurements in CCI is much smaller than that in single-pixel sampling methods in order to get the same MSE value, although CCI needs to measure the signals in both reflection arms of the DMD. Besides, it is evident that as the sampling rate increases, the computation time of CCI gradually becomes shorter than that of either one of these two reflection orientations, as shown in Fig. 4(e).

Figure 4. Simulation results of a head phantom image.

(a) The recovered image using Bernoulli distributed random measurement matrix consisting of 0 or 1, with MSE = 0.0016. (b) The retrieved image using the complementary measurement matrix of (a), with MSE = 0.0029. (c) CCI result with MSE = 1.2416 × 10⁻⁵. (d) and (e) The MSE values and the computation time of all these approaches versus the sampling rate in the single-arm.

Discussion

Considering gaps between adjacent micro-mirrors and gaps that appear when micro-mirrors flip, we calculate that the proportion of the area of 768 × 1024 micro-mirrors to that of total reflection plane of the 0.7 in. DMD is , and the proportion of the projected area of all the micro-mirrors when they are positioned in one of two states to the area of all the micro-mirrors is cos(12°) × 100% = 97.815%, as shown in the right part of Fig. 5, thus the proportion of the area of all the gaps to that of the entire reflection plane of the DMD is about (1 − 0.96986 × 0.97815) × 100% = 5.133%. Namely, the area of all the gaps (the shaded regions as shown in Fig. 5) is a constant no matter what kind of pattern encoded on the DMD. As we know, these gaps show the metal surface under the micro-mirrors. Besides, there is a mirror support post with a little light transmittance in the center of each micro-mirror. The metal surface together with the axis will incur some random noise. Actually, the metal surface is a combination of mirror-like and diffuse reflections. The specular reflections of the metal surface will not affect the detection, and can be ignored. In the latter, it bounces off the light in many directions. Thus, the PMT in each detection arm receives only a very small part of the light reflected diffusely from the metal surface, which is as well associated with each pattern encoded on the DMD to a large extent and is approximately independent identically distributed. In our experiment, as the analog-to-digital (A/D) conversion of PMTs is in 12 bits precision, the saturation value is 4096. To assess the impact of this kind of noise, we set the pattern to be all zero, and record the value reading from the PMT in the left reflection arm, which is turned out to be zero. If we change the pattern to be all one, the other arm shows the same result. It can be seen that the influence of this noise on the measured value is less than 1, thus can be neglected here. However, if the single-pixel detector is replaced by a Geiger-mode avalanche photodiode (APD) or a PMT with higher precision, this kind of noise is worth consideration. Fortunately, complementary measurements can make up for the loss of the image quality caused by this kind of noise.

Figure 5. Schematic diagram of rotating micro-mirrors.

The left is the top view of four micro-mirrors, while the right is the cross-section of one micro-mirror. The micro-mirrors which flip to the 12 degrees positions are depicted in aqua, and the ones turning to −12 degrees in cornflower blue. The exposed surface under the micro-mirrors is represented in dark gray, and the gaps between adjacent micro-mirrors in light gray.

Since CCI regards a subtracted and rescaled vector of these two measured data from different detection arms as the final input values, denoted as b, it not only averages the variance of the noise, but also makes the ratio δ of the variance to the mean of measured sequence b larger, that is equivalent to enlarging the useful signal partially but subtracting the background signal. And this is exactly the reason for its good image quality. We also found that R. Berinde's sparse binary matrix has a mean which tends to be zero, and actually amplifies the variance of b as well.

The setup presented here can be considered as a two-pixel camera (with complementary measurements) or as a single pixel camera scheme with an altered binary measurement matrix using ±0.5 instead of 0 and 1 as the binary values. It may be of interest to discuss the relation to non-binary imaging matrices. Take a grey scale of n levels for example, the dynamical range of the measurement matrix is 0 ~ n. If the measurement matrix for the left arm is A₁, then its complementary matrix for the right arm should be n − A₁. Following the same strategy, we can obtain a distributed altered measurement matrix . Therefore, our complementary sampling protocol can be easily extended to non-binary measurement matrices, with similar or even better effect. Moreover, a grey scale of n levels can be realized by adopting binary pulse width modulation technique (that is, the non-binary imaging matrices encoded on the DMD have a binary relationship to different display periods of pixels) or using different devices instead of DMDs.

In summary, our results represent the first complementary compressive imaging of a target at about 2.0 km range with 20 cm resolution, realizing large FOV imaging over a long distance. Unlike conventional single-pixel cameras which use measured data from only one detection arm, our scheme makes full use of both reflection directions, and recalculates a new observed vector by dividing the difference of the two arms data by two. By solving the mathematical model of CCI, we acquire high-resolution images with the variance of the independent and identically distributed noise decreasing by half. The use of CCI for the telescopic system is a leap forward for improving the image quality of traditional single-pixel cameras where the binary patterns encoded on the DMD generally consist of the two values 0 or 1, opening up possibilities for new applications. Moreover, our protocol can be also easily extended to non-binary measurement matrices, which may be of great interest. We have experimentally investigated the influence of the sampling rate in the single-arm and the integration time of PMTs on the image quality. It is evident that CCI for the telescopic system has the advantages of long remote-sensing distance, high imaging speed and high-resolution imaging capabilities. The challenge arises by the fact that, in practice, the influence of the atmospheric turbulence on the image quality seems to be unavoidable and will lead to the degradation of the image quality, especially when the distance of free-space light propagation increases. As a result, an on-going research is to find some ways to reduce the impact of atmosphere turbulence to some extent.

Methods

Compressed sensing

Generally speaking, many natural signals, such as images, have sparse representations when expressed in a certain basis. Assume we have a two-dimensional image I of pixel size p × q, with can be stretched for a column , where N = p × q. We can expand x in an orthonormal basis (e.g., discrete cosine transform basis, Fourier transform basis, wavelet basis) Ψ = [ψ₁,ψ₂,…ψ_N] as follow: , where is the coefficient sequence of x. Considering that is the coefficient vector with all but the largest K(≤N) elements set to zero, and this vector can be called K-sparse. By taking about O(K·log(N/K)) random projections, one has enough information to recover the signal x with accuracy.

The problem of reconstructing a signal x can be described by:

where (M < N) is a random measurement matrix, which satisfies RIP, is the observation of x via some linear measurements, and e is a stochastic noise term.

For image restoration, recent research has confirmed that the use of total variation (TV) regularization instead of the l₁ term in CS problems makes the recovered image quality sharper by preserving the edges or boundaries more accurately, which is essential to characterize images. Thus, TV regularization is very competent for single-pixel cameras as the core reconstruction solver. A solver (named TVAL3²¹), which adopts alternating minimization scheme, is a good example for TV minimization. C. B. Li has already validated with numerical results that TVAL3 is more efficient and robust in comparison to other TV solvers (such as NESTA, NESTA_UP, TwIST, l₁-Magic, SOCP) and most l₁ minimization algorithms (e.g., FPC, YALL1) and accepts a vast range of measurement matrices (unlike RecPF²², which is restricted to partial Fourier measurements only). The core TV norm minimization of TVAL3 can be written as:

where D_ix denotes the discrete gradient vector of x at position i, and μ > 0 is the penalty parameter.

Mathematical model of CCI

The mathematic model of complementary compressive imaging can be described as:

where A₂ = 1_{M × N} − A₁ (1_{M × N} denotes an array of all ones), e₁ together with e₂ corresponds to the noise on the different sides respectively. Then we can have the following formula:

If we set , , and , we will obtain the same equation as Eq. (2), but obtaining ±0.5 binary measurement matrix instead of the binary matrix consisting of 0 and 1, with the variance of the independent and identically distributed noise decreasing by half. It solves the problem of the single-pixel camera in obtaining high-quality images.