PhlatCam: Designed phase-mask based thin lensless camera

Abstract

We demonstrate a versatile thin lensless camera with a designed phase-mask placed at sub-2 mm from an imaging CMOS sensor. Using wave optics and phase retrieval methods, we present a general-purpose framework to create phase-masks that achieve desired sharp point-spread-functions (PSFs) for desired camera thicknesses. From a single 2D encoded measurement, we show the reconstruction of high-resolution 2D images, computational refocusing, and 3D imaging. This ability is made possible by our proposed high-performance contour-based PSF. The heuristic contour-based PSF is designed using concepts in signal processing to achieve maximal information transfer to a bit-depth limited sensor. Due to the efficient coding, we can use fast linear methods for high-quality image reconstructions and switch to iterative nonlinear methods for higher fidelity reconstructions and 3D imaging.

1. Introduction

A myriad of emerging applications such as wearables, implantables, autonomous cars, robotics, internet of things (IoT), virtual/augmented reality, and human-computer interaction ([1], [2], [3], [4], [5], [6]) are driving the miniaturization of cameras. The use of traditional lenses adds weight and cost, are rigid and occupies volume, and have a stringent requirement of focusing distance that is proportional to the aperture. For these reasons, a radical redesign of camera optics is necessary to meet the miniaturization demands [7].

Lightweight cameras can be created by replacing the lens with a thinner focusing element, like a diffractive lens [8]. In this paper, the terms focusing or lensing optics will be used to describe optical elements that can produce a point-spread-function (PSF) of low support (Fig. 1). In these cases, the sensor measurements resemble an image, albeit some haziness or blurriness due to chromatic and spherical aberrations. With the help of computational methods, the effects of aberrations can be alleviated. However, working in the lensing regime does not reduce the thickness of the camera. This is due to the requirement of the focusing distance that is proportional to the aperture. The use of metalenses [9] are limited for the same reason. Therefore, we need to step in the non-focusing regime to truly reduce the thickness of the camera.

Fig. 1.

[Top] Non-lensing optics provides a way to achieve thin devices at low-cost. Among the various non-lensing optics, phase-masks are veratile in their designs and can produce a larger space of Point-spread-functions (PSF). [Bottom] PSFs from various optics are shown. Lensing optics have a small PSF support while non-lensing optics display large PSFs. The non-lensing optics’ PSFs were experimentally camptured.

Recently, lenless cameras were demonstrated to achieve small form factors by forgoing the need to capture “image” like measurements on the sensor. Instead, what these cameras capture are highly multiplexed measurements, which are computationally demultiplexed into images by incorporating calibrated camera responses. Inevitably, these cameras have point-spread-functions (PSFs) with large support. The design of the point-spread-functions is instrumental in guaranteeing high-quality reconstructions—however, previous lensless designs lack the precise control of point-spread-functions. One of the core contributions of this paper is a framework to precisely realize high-performance PSFs.

A lensless camera consists of an encoding element or a “mask” placed at a close distance from an imaging sensor. Various masks have been considered like amplitude masks [10], [11], [12], phase gratings [13] and diffuser [14]. Amplitude masks were designed to produce binary PSFs, phase gratings were designed to produce robust nulls, and diffusers were used for its pseudorandom caustic pattern. However, each of these masks is limited in their designability (Fig. 1).

While the design of amplitude masks is straightforward, there are two inherent issues: (1) they block a significant amount of light, and (2) diffraction effects cause the PSF to deviate from the original design. On the other extreme, diffusers are inherently statistically while having a minimal light loss. The statistical nature puts the diffuser low on the design flexibility scale.

For our lensless system, we propose to use phase-masks as our optical masks. Among the various diffraction masks, phase-masks have proven to be versatile in realizing a variety of point-spread-functions [15], [16], [17], [18], [19], [20], with and without the assistance of lenses. Additionally, phase-masks are highly light-efficient and hence operationally better suited for a range of illumination scenarios.

The designability of phase-masks allows us to realize high-performance PSFs and hence improve the overall performance of the lensless system. To that end, we propose a PhlatCam design that benefits from our following contributions:

1. We propose a Near-field Phase Retrieval (NfPR) framework to design phase-mask that produces the target PSF at the desired device thickness.

2. We propose a high-performance contour PSF and show it’s superior performance compared to previous lensless PSFs.

3. We show the application of PhlatCam for (a) 2D imaging at any scene depth, (b) refocusing at medium scene depth, and (c) 3D imaging at close scene depth.

2. Related Work

2.1. Previous Lensless Imagers

Without the lens, a bare sensor captures the average light intensity of the entire scene. [21] has shown imaging with a bare sensor by exploiting shadows cast by defects on the sensor’s cover glass, and the anisotropy of pixel response to achieve some level of light modulation. Due to limited control on the light modulation, the possible reconstruction quality is limited to low resolutions. An alternate way to achieve bare sensor imaging is to have an active illumination behind the object to produce light modulation through shadows [22] or interference fringes [23]. This technique was used to create high-resolution wide field-of-view (FoV) microscopy images. However, adding an active illumination behind the object makes the imaging system bulkier and impractical for photography.

To passively achieve a higher control of light modulation, an encoding “mask” is placed in front of an imaging sensor [7]. Lensless imaging has been shown with a variety of diffractive masks [10], [11], [12], [13], [14], [19], [24]. The masks produce a point-spread-function (PSF) on the sensor which can be undone using computational algorithms to produce high-quality images.

2.2. Diffractive Masks

Diffractive masks used for lensless cameras can be broadly categorized into amplitude masks and phase masks. Amplitude masks were used by [10], [11], [12], [24] and phase masks were used by [13], [14], [19].

2.2.1. Amplitude Masks

An amplitude mask modulates the amplitude of incident light by either passing, blocking, or attenuating photons. For ease of fabrication, a binary amplitude is commonly used, and the light modulation by casting shadows. Hence, the PSF of the amplitude mask is its shadow.

A concerning issue in using an amplitude mask is the light throughput. Since the mask modulates light by creating shadows, a significant amount of photons are lost, leading to low signal-to-noise-ratio (SNR) sensor capture. Low SNR is undesirable for low light scenarios and photon-limited imaging like fluorescence or bioluminescence imaging. Additionally, decoding the lensless sensor capture tends to amplify noise leading to poor reconstruction. Amplitude masks also suffer from diffraction effects, which limit the range of achievable PSFs. Diffraction issue is discussed further in Section 2.3.

2.2.2. Phase Masks

A phase-mask modulates the phase of incident light by the principles of wave optics [25]. Phase-masks allow most of the light to pass through, providing high SNR. Hence, they are desirable for low light scenarios and photon-limited imaging.

We propose to use phase-mask for our lensless camera and present a mask design algorithm to achieve desirable PSFs.

2.3. Mask Design

2.3.1. Amplitude Masks

Among all masks, designing an amplitude mask based on a desirable PSF is the most straightforward. The pattern of a binary amplitude mask is merely the PSF itself, where the bright regions of the PSF correspond to the open areas of the mask, and the dark regions correspond to the blocking areas [7].

However, the range of PSFs achievable using the above mentioned, ray-optics based, amplitude mask design is limited due to diffraction. As a rule of thumb, the Fresnel number N_F [26] associated with the amplitude mask can help in determining whether the PSF will be close to the pattern of the mask or different. If the Fresnel number is much greater than 1, then geometrical properties are valid, and the shadow PSF mimics the mask pattern. When the Fresnel number falls below 1, the cast PSF will deviate from the mask pattern. This aspect is elaborated in supplementary Section 2, and the effect of diffraction in amplitude mask PSF is shown in Supp. Fig. 1.

2.3.2. Phase Masks

Odd-symmetry phase gratings were proposed by [27] to achieve robust nulls in the PSF. The wavelength and depth-robust nulls are produced along the axis around which phase gratings have an odd-symmetry. However, this design doesn’t guarantee intensity distributions in the non-null regions of the PSF.

The use of diffuser as a phase mask for lensless imaging was proposed by [14]. Diffusers are low cost and produce caustics patterns. The best performance is achieved from a diffuser when its placed at a distance where it produces the highest contrast caustics. However, since the phase profile of a diffuser is inherently statistical, the optimal distance varies from one to another. Hence, it is harder to design lensless cameras of desired thicknesses with diffusers.

[28] proposed phase mask design using phase retrieval algorithms and subsequently implemented using phase spatial light modulator in [19]. We follow a similar approach in designing our phase mask for a desirable PSF and then fabricate our phase-mask. The camera thickness is a design parameter in our approach and allows us to create lensless cameras of desired thicknesses.

2.4. PSF Engineering

Various PSFs have been used for lensless imaging for their attractive properties. We describe them below.

Separable PSF

A separable PSF is constructed by an outer product of two 1-D vectors. Such construction simplifies the imaging model as convolution along the rows of the image followed by convolution along the columns. In matrix form, this operation can be written as a product of 2-D image with a few small 2-D matrices [10], [11], [24]. An example of separable PSF is shown in Fig. 10, constructed from outer product of two maximum length sequences [10], [29], [30].

Fig. 10.

Simulated reconstruction with previously proposed PSFs, random binary PSF and our Contour PSF Random binary PSF satisfies three of the four desired characteristics of PSF However, random binary PSF doesn’t satisfy the fourth characteristic, that is large contiguous regions of zero intensity. As seen from above, contour PSF consistently produces better results.

Fresnel Zone Aperture

A Fresnel Zone Aperture (FZA) PSF is constructed like a Fresnel zone plate [31] and was used by [12], [32]. Multiplying the sensor capture with a virtual FZA results in overlapping moire fringes. Fast reconstruction is done by applying a 2-D Fourier transform on the moiré fringes. An example of FZA PSF is shown in Fig. 10.

Spiral PSF

A spiral PSF was proposed by [27]. To cover a large sensor area, [13] proposed tessellating the spiral PSF. An example of the tessellated spiral PSF is shown in Fig. 10. A single unit of the tessellation is shown on the top left corner.

We propose a high-performance Contour PSF that provides us the ability to perform high-resolution (a) 2D imaging, (b) refocusing, and (c) 3D imaging at different depth ranges.

3. Diffractive Lensless Imaging

3.1. Imaging Architecture

PhlatCam has a fabricated diffractive element called phase mask placed at a distance d from an imaging sensor (Figure 5(a)). A phase mask modulates the phase of incident light and produces a pattern at the sensor through constructive and destructive interference. In the following sections, we’ll describe how the phase mask produces interference pattern and the consequent diffractive imaging model.

Fig. 5.

Illustration of properties of phase mask in a lensless imaging setup.

3.2. Propagation

When a phase mask, with phase profile ϕ(ξ, η), is illuminated with a coherent collimated light, the intensity pattern p(x, y) captured by the imaging sensor placed at distance d is given by magnitude square of Fresnel propagation [26]:

$$p(x,y)=|{\mathcal{F}}_d(e^{j\varphi (\xi ,\eta )}){|}^2=|\frac{1}{j\lambda d}\int \int e^{j\varphi (\xi ,\eta )}\exp \left[j\frac{\pi }{\lambda d}({(x-\xi )}^2+{(y-\eta )}^2)\right]\text{d}\xi \text{d}\eta {|}^2,$$ (1)

where ${\mathcal{F}}_d(·)$ denotes fresnel propagation by distance d and λ is the wavelength of light. For simplicity, let’s consider a one-dimensional (1D) phase mask with phase map ϕ(¾) and drop the scaling term. Then, the pattern produced from collimated light parallel to optical axis is given by

$$p(x)=\left|\int e^{j\varphi (\xi )}\exp \left[j\frac{\pi }{\lambda d}{(x-\xi )}^2\right]\text{d}\xi {|}^2=\right|\int e^{j\varphi (\xi )}\exp \left[j\frac{\pi }{\lambda d}{({\xi }^2-2x\xi )}^2\right]\text{d}\xi {|}^2,$$ (2)

where the quadratic term was expanded and a constant phase term was removed since we are considering only intensity.

3.3. Point Spread Function

The collimated light or planar waves can be said to be generated from an on-axis point source at a sufficiently large distance from the mask. Then, p(x) (or p(x, y) for 2D) can be called as the point-spread-function (PSF) of the system.

If the point source is off-axis, illuminating the phase mask at an angle θ, it imparts a linear phase $e^{j\frac{2\pi }{\lambda }\sin (\theta )\xi }$ to Eq. 2 and the resultant intensity pattern is

$$I_{\theta }(x)=\left|\int e^{j\frac{2\pi }{\lambda }\sin (\theta )\xi }{e}^{j\varphi (\xi )}\exp \left[j\frac{\pi }{\lambda d}({\xi }^2-2x\xi )\right]\text{d}\xi {|}^2=\right|\int e^{j\varphi (\xi )}\exp \left[j\frac{\pi }{\lambda d}({\xi }^2-2(x-d\sin (\theta ))\xi )\right]\text{d}\xi {|}^2=p(x-d\sin (\theta )).$$

Hence, an off-axis point source causes a lateral shift of the PSF If the point source is at a distance z_∞ and height x_h then by paraxial approximation

$${\delta }_{z_{\infty }}(x-x_h)\to p(x-\frac{d}{z_{\infty }}{x}_h),$$ (3)

where δ_z(x) denotes point source at distance z from the mask. The shift property is illustrated in Figure 5(b) and can be stated as

Property 1. Shift invariance: A lateral shift of point source causes translation of PSF on the sensor plane.

The above property is also called “memory effect” [33], [34] and was recently used to perform non-invasive imaging through scattering media [35], [36], [37] and wavefront sensing [38]. As we will see later, the shift invariance property helps us to write the imaging model as a convolutional model.

For a point source at a finite distance z from the mask, it imparts an additional quadratic phase $e^{j\frac{\pi }{\lambda z}{\xi }^2}$ to Eq. 2 togive an intensity response as:

$$I_z(x)=\left|\int e^{j\frac{\pi }{\lambda z}{\xi }^2}{e}^{j\varphi (\xi )}\exp \left[j\frac{\pi }{\lambda d}({\xi }^2-2x\xi )\right]\text{d}\xi {|}^2=\right|\int e^{j\varphi (\xi )}\exp \left[j\frac{\pi }{\lambda }(\frac{1}{d}+\frac{1}{z}({\xi }^2-2\frac{x}{1+d/z}\xi )\right]\text{d}\xi {|}^2\approx |\int e^{j\varphi (\xi )}\exp \left[j\frac{\pi }{\lambda d}({\xi }^2-2\frac{x}{1+\text{d}/z}\xi )\right]d\xi {|}^2=p(\frac{x}{1+d/z}).$$

Here we assumed d ≪ z which would be the case for our lensless cameras. Therefore, following the same notations as Eq. 3 we have

$${\delta }_z(x)\to p_z(x)=p\left(\frac{x}{1+d/z}\right),$$ (4)

which is a geometrical magnification of the PSF. The magnification property is illustrated in Figure 5(c) and can be stated as

Property 2. An axial shift of point source causes magnification of PSF on the sensor plane.

As we will see later, the PSF variation with scene depth will be exploited for refocusing of images and 3D image reconstruction.

3.4. Convolutional model

A real world 2D scene i (x, y; z) at distance z can be assumed to be made up of incoherent point sources. Each point source will produce a shifted version of PSF p_z(x, y) and since the sources are incoherent to each other, the shifted PSFs will add linearly in intensity [26] at the sensor. By Property 1 of the PSF, we can write the imaging model as following convolution model:

$$\mathbf{b}(x,y)=p_z(x,y)*\mathbf{i}(x,y;z).$$ (5)

Here, b is the sensor’s capture and * denotes 2D convolution over (x, y).

For a 3D scene, the sensor capture is the sum of convolutions at different dephts and the imaging model can be written as:

$$\mathbf{b}(x,y)=\sum _z{p}_z(x,y)*\mathbf{i}(x,y;z).$$ (6)

The above sections established the relation between phase mask and PSF. In the following sections, we propose a high-performance PSF and lay out a method to design phase mask for the desired PSF.

4. Designing PhlatCam

Designing PhlatCam consists of of two parts: (1) PSF engineering and (2) phase-mask optimization. Performance of PhlatCam relies on the PSF while the phas-mask optimization makes the PSF realizable.

4.1. PSF Engineering

A lensless camera encodes an image onto the sensor by convolution of the scene with a PSF. From convolution theorem [39], we can infer that for maximal information transfer, large and almost flat magnitude spectrum is desirable in the PSF. An another way to look at this is that the deconvolution of PSF involves the inversion of the PSF’s frequency spectrum and low values of magnitude spectrum will lead to amplification of noise at those frequencies.

Imaging sensors captures light intensity, which implies that the values in the PSF are always positive. A positive PSF will have larger contribution at DC or the zero-frequency component compared to other frequencies. Hence, efforts need to be taken to minimize the DC component. Additionally, sensors don’t have infinite precision and are usually limited to 8- to 12-bit precision. These two factors also need to be considered when designing the PSF.

Designing the PSF could, potentially, be achieved using many methods such as optimizing over a theoretical metric [8], [40] or using a data-driven approach [17], [41]. In this paper, we take a heuristic approach based on the domain knowledge of signal processing. Considering the previously mentioned factors, we state the desired characteristics of the PSF and the corresponding reasoning as follows:

1. Contain all directional filters to capture textural frequencies at all angles.

2. Spatially sparse to minimize the DC component of PSF’s Fourier transform.

3. High contrast (i.e. binary) to compensate for limited bit depth of sensor pixels.

4. Large regions of contiguous zero intensity to further compensate for limited bit depth of sensor pixels.

Proposed Contour-based PSF

We make the observation that contour lines with sufficient random orientation satisfies all the criterions mentioned above, as shown in Figure 6. There are, however, many possible ways to generate contour PSFs. In our case, we chose to produce contours from Perlin noise [42] due to it’s guarenteed randomness and the ability to control the sparsity.

Fig. 6.

Our Contour PSF is generated by applying canny edge detection on Perlin noise [42].

In graphics, Perlin noise [42] is a popular tool to produce random landscape textures. What we are after is the boundary contours of such landscape textures, which will invariably (a) contain a set of randomly oriented curves that can function as directional filters, (b) be sparse, (c) be binary, and (d) contain large empty regions. To produce the contour PSF, we apply canny edge detection to a generated Perlin noise. Such generation of PSF is a good candidate for lensless imaging and satisfy the above mentioned characteristics. Illustration of generating a contour PSF is shown in Fig. 6. The high performance of our proposed contour-based PSF is validated using the modulation transfer function (MTF) metric in Fig. 7 and using simulated reconstruction in Fig. 10.

Fig. 7. Modulation Transfer Function (MTF) of lensless point-spread-functions (PSFs).

The MTF is computed as the radially averaged magnitude spectrum of the PSFs. The PSFs compared are: Separable MSEQ [10], Fresnel zone apertures (FZA) [32], Tessellated spiral [13], Diffuser [14], Random binary, and our Contour PSF. The PSFs are visualized in Fig. 10. The magnitude spectrum of the proposed Contour PSF remains large for entire frequency range indicating better invertibility characteristic.

4.2. Phase-mask Design

Our goal is to optimize a phase-mask design that produces the target PSF at the target device thickness d (the distance between the sensor and mask). A thing to note is that the phase-mask performs a complex-valued modulation of light wavefront while the target PSF is a real-valued intensity distribution. Hence, obtaining phase-mask profile from the PSF is an undetermined problem. However, there exists computational methods called the phase retrieval algorithms [43] that tries to solve this precise problem of computing complex-valued fields from real-valued intensities.

Usually phase retrieval algorithms are applied in the case of systems involving lens and function under the far-field approximation of Fraunhofer diffraction (implemented with just a Fourier transform) [26], [43]. In our case, we are devoid of lens and are within the regime of near-field Fresnel diffraction (has an additional quadratic phase). . Hence, we call our phase-mask optimization algorithm as Near-field Phase Retrieval (NfPR).

NfPR is motivated by [28] and is similar to the Gerchberg-Saxton (GS) algorithm [44], a popularly used phase retrieval algorithm. The way we differ from the GS algorithm is by replacing the Fourier transforms with near-field Fresnel propagation. NfPR does not guarantee an unique solution. However, what we are after is a phase-mask that can produce the target PSF and not the unique phase-mask profile.

The near-field phase retrieval algorithm for phase mask optimization can be described as follows. The algorithm uses an iterative approach, iterating between the fields at the mask plane and the sensor plane while simultaneously enforcing constraints at the two planes—the amplitude of the field at the mask plane is unity and intensity of the field at the sensor plane is the target engineered PSF. Forward Fresnel propagation is used to go from mask plane to sensor plane, while backward Fresnel propagation (by negating the distance in Eq. 1) is used to go from sensor plane to mask plane. The iterative algorithm is summarized in Alg. 1 and visually illustrated in Fig. 8. The phase mask optimization requires the following inputs: the target PSF, mask to sensor distance, and wavelength of light. The wavelength of light is chosen to be the mid visible wavelength of 532 nm.

Fig. 8.

Visual illustration of the phase mask design.

Phase-mask height map

To physically implement the phase-mask, the phase profile needs to be transformed into the height map of the final mask substrate. Assuming n as the refractive index of the mask substrate, the height map is given by:

$$h(x,y)=\frac{\lambda }{2\pi (n-1)}\varphi (x,y).$$ (7)

Fabrication

Advancements in fabrication techniques have made it possible for physically realizing diffractive masks with quick turnaround times. In this paper, we use a recently developed 2-photon lithography 3D printing system [45] that allows for rapid prototyping of different phase-masks without significant overhead preparation. With an optimized final phase-mask design, fabrication can be scaled through the manufacturing pipeline such as photolithography and reactive-ion-etching processes. The fabricated phase-mask is shown in Fig. 2.

Fig. 2.

Phase-masks are essentially transparent material with different heights at different locations. This causes phase modulation of incoming wavefront and resultant wave interference produces the PSF at the sensor plane. The above image shows the closeup image of the phase-mask using in PhlatCam. The right most image was taken using a scanning electron microscope (SEM).

Algorithm 1.

Phase mask design

Input: Target PSF p(x, y), wavelength λ, mask to sensor distance d and refractive index n of mask substrate.

Output: Mask’s phase profile ϕ(ξ, η) and height map h(ξ, η).

$M_p\leftarrow \sqrt{p}$

repeat

$M_{\varphi }\leftarrow {\mathcal{F}}_{-d}(M_p)$ {Back propagate from sensor to mask}

ϕ → phase(Mϕ)

Mϕ → ejϕ {Constrain amplitude of mask to be unity}

$M_p\leftarrow {\mathcal{F}}_d(M_{\varphi })$ {Forward propagate from mask to sensor}

$M_p\leftarrow \sqrt{p}\odot e^{j\mathrm{phase}(M_p)}$ {Constrain amplitude to be $\sqrt{\mathrm{PSF}}$}

until maximum iterations

4.3. Reconstruction Algorithms

Recovering the scene image from the sensor measurement can be posed as a convex optimization36 problem, where the forward model is the convolution of PSF with the image. Regularization based on image prior is added to the optimization problem to robustify against measurement noise and avoid large amplification of noise in the reconstruction.

For this section, we will be deviating slightly from the notations in the previous sections. The prominent changes are: ‘x’ will denote the scene (instead of ‘i’) and ‘d’ will denote scene depth (instead of device thickness). In effect, we solve the following minimization problem:

$$\widehat{x}=\arg \underset{x\ge 0}{\min }\frac{1}{2}\Vert \mathbf{b}-\sum _d{p}_d*x_d{\Vert }_F^2+\gamma \mathcal{R}(x),$$ (8)

where ∥·∥_F denotes the Frobenius norm, $\mathcal{R}(\eta )$ denotes the regularization function, and γ is the weighting of the regularization.

4.4. 2D Reconstruction

The reconstruction becomes 2D image reconstruction under two contexts. First, if all the scene elements are sufficiently far (i.e. scene depth ⋙ device thickness), the dependance of PSF with depth is almost none (Eq. 4). In such case, P_d ≈ p_∞, and x_d ≈ x_∞. Second, when refocusing to a particular depth. At both these times, the summation in Eq. 8 is removed and we solve the following problem:

$$\widehat{x}=\arg \underset{x\ge 0}{\min }\frac{1}{2}\Vert \mathbf{b}-p*x{\Vert }_F^2+\gamma \mathcal{R}(x),$$ (9)

where p = p_d(or)p_∞, and x = x_d (or) x_∞ according to the context.

4.4.1. Fast Reconstruction

For fast reconstructions, Tikhonov regularization $(\mathcal{R}(\eta )=\parallel ·{\parallel }_F)$ can be used—which has a closed form solution given by Wiener deconvolution. Using the Convolution Theorem [46], the solution can be computed in real time with the Fast Fourier Transform (FFT) algorithm. By differentiating Eq. 9 and setting to zero, we get the following solution for the Tikhonov regularized reconstruction:

$$\widehat{x}={\mathcal{F}}^{-1}\left(\frac{(\mathcal{F}(p))*\odot \mathcal{F}(b)}{|\mathcal{F}(p){|}^2+\gamma }\right),$$ (10)

where $\mathcal{F}(\eta )$ is the fourier transform operator, (·)* is the complex conjugate operator and ‘⊙’ represents hadamard product.

4.4.2. High-fidelity Reconstruction

The reconstruction quality can be further improved by using total-variation (TV) regularization [47], which uses the image prior that natural images have sparse gradients. The TV regularized minimization problem is given by:

$$\widehat{x}=\arg \underset{x\ge 0}{\min }\frac{1}{2}\Vert \mathbf{b}-p*x{\Vert }_F^2+\gamma \Vert \Psi (x){\Vert }_1,$$ (11)

where, Ψ is the 2D gradient operator, and ∥·∥₁ is the l₁ norm. We opt for an iterative ADMM [48] approach to solve the above problem.

Let H be the 2D convolution matrix and we use the following variable splitting:

$$\begin{gathered} \widehat{x}=\arg \underset{x\ge 0,z,x}{\min }\frac{1}{2}\Vert \mathbf{b}-Hx{\Vert }_F^2+\gamma \Vert z{\Vert }_1, \\ \text{s}.\text{t}.z=\Psi x,w=x. \end{gathered}$$

Then the ADMM steps at iteration k are as follows:

$$z^{k+1}\leftarrow {\mathcal{S}}_{\frac{\gamma }{{\mu }_z}}(\Psi x+{\rho }_z^k/{\mu }_z)$$

$$w^{k+1}\leftarrow \max (x+{\rho }_w^k/{\mu }_w,0)$$

$$x^{k+1}\leftarrow {(H^{T}H+{\mu }_z{\Psi }^T\Psi +{\mu }_{w}I)}^{-1}{r}^k,$$

where,

$$r^k=H^T\mathbf{b}+{\Psi }^T({\mu }_z{z}^{k+1}-{\rho }_z^k)+{\mu }_w{w}^{k+1}-{\rho }_w^k.$$

S_μ is the soft-thresholding operator with a threshold value of μ, and ρ_w and p_z are the Langrage multipliers associated with w and z, respectively. The scalars p_w and p_z are the penalty parameters that are computed automatically using a tuning strategy. Operations of H and Ψ can be computed using Fast Fourier Transform (FFT), making each step of the ADMM fast.

4.5. 3D Reconstruction

For 3D imaging, we use both TV and sparsity regularizations and solve using ADMM approach. Let H be the 2D convolution matrix with PSF p_d at depth d. Then the reconstruction problem is posed as:

$$\widehat{x}=\arg \underset{x\ge 0}{\min }\frac{1}{2}\Vert \mathbf{b}-\sum _d{H}_d{x}_d{\Vert }_F^2+\gamma \sum _d\Vert \Psi x_d{\Vert }_1+{\gamma }_1\Vert x{\Vert }_1.$$

We use the following variable splitting:

$$\begin{gathered} \widehat{x}=\arg \underset{w\ge 0,z,x}{\min }\frac{1}{2}\Vert \mathbf{b}-Sv{\Vert }_F^2+\gamma \sum _d\Vert z_d{\Vert }_1+{\gamma }_1\Vert t{\Vert }_1 \\ \text{s}.\text{t}.v_d=H_d{x}_d,z_d=\Psi x_d,w=x,t=x, \end{gathered}$$

where Sv = ∑_dv_d is the sum along depth operator. Then the ADMM stepps at iteration k are as follows:

$$v_d^{k+1}\leftarrow {(S^{T}S+{\mu }_{v}I)}^{-1}(S^T\mathbf{b}+{\mu }_v{H}_d{x}_d^k+{\rho }_v^k)$$

$$z_d^{k+1}\leftarrow {\mathcal{S}}_{\frac{\gamma }{{\mu }_z}}(\Psi x_d^k+{\rho }_z^k/{\mu }_z)$$

$$w^{k+1}\leftarrow \max (x^k+{\rho }_w^k/{\mu }_w,0)$$

$$t^{k+1}\leftarrow {\mathcal{S}}_{\frac{\gamma 1}{{\mu }_t}}(x^k+{\rho }_t^k/{\mu }_t)$$

$$x_d^{k+1}\leftarrow {({\mu }_v{H}_d^T{H}_d+{\mu }_z{\Psi }^T\Psi +{\mu }_{w}I+{\mu }_{t}I)}^{-1}{r}_d^k,$$

where,

$$r_d^k=H_d^T({\mu }_v{v}^{k+1}-{\rho }_v^k)+{\Psi }^T({\mu }_z{z}_d^{k+1}-{\rho }_z^k)+{\mu }_z{w}^{k+1}-{\rho }_w^k+{\mu }_t{t}^{k+1}-{\rho }_t^k.$$

S_μ is the soft-thresholding operator with a threshold value of μ, and ρ_w, ρ_t, and ρ_z are the Langrage multipliers associated with w, t and z, respectively. The scalars μ_w, μ_t, and μ_z are the penalty parameters that are computed automatically using a tuning strategy. Operations of H_d, Ψ, and S can be computed using Fast Fourier Transform (FFT), making each step of the ADMM fast.

The key difference between the above 3D algorithm and the algorithm presented in [14] is that we use the summing operator S instead of the slicing operator in [14]. The use of summing operator makes each iteration of ADMM much more stable and results in high quality reconstruction within a few iterations.

5. Imaging Prototype

5.1. Prototype preparation

We generated a contour-based PSF with 14% sparsity from Perlin noise [42]. The continuous phase mask profiles (in radians) were optimized at 532 nm wavelength of light with 2 pm spatial resolution to produce the PSF at sensor to mask distance of 1.95 mm. The height map of the phase-mask was computed for the mask substrate with refractive index of 1.52. The height map was further discretized into height steps of 200 nm to fit the specifications of fabrication. The phase mask was fabricated using two-photon lithography 3D printer (Photonic Professional GT, Nanoscribe GmbH [45]). The phase mask was printed on a 700 μm thick, 25 mm square fused silica glass substrate using Nanoscribe’s IP-DIP photoresist in a Dip-in Liquid Lithography (DiLL) mode with a 63× microscope objective lens. The IP-DIP has a refractive index of 1.52. The fabricated phase mask is shown in Fig. 2.

We used a FLIR Blackfly S color camera with Sony IMX183 sensor to build our prototype. Without binning, the pixel pitch per color channel is 4.8 μm, and with binning, the pixel pitch per color channel is 9.6 μm. The camera housing was replaced with a 3D printed housing to get unobstructed access to the protective glass on the sensor. The phase mask was affixed (face down) on the protective glass of the sensor using double-sided adhesive carbon tape. Two layers of carbon tape were sufficient to attain the desired distance (1.95 mm) between the phase mask and the sensor, the distance at which the PSFs appear the sharpest. The affixing carbon tape also acted as a square aperture (6.7 mmx 5.3 mm) to restrict the shifts of the PSF to be within the sensor.

The PSF design, optimized phase mask profile and comparison of PSF from 3D printed phase mask is shown in Fig. 9.

Fig. 9.

The proposed PSF, designed phase-mask, and the experimentally realized PSF of PhlatCam are shown. The experimental PSF closely resembles the proposed PSF design, showing the effectiveness of the phase mask design framework.

Calibration

There can be discrepancies between the physically implemented PSFs and the target engineered PSFs due to phase mask height discretization and fabrication inaccuracies. Hence, we experimentally capture the PSFs and use these PSFs for our computation. We approximate a point source by back illuminating a pinhole aperture and capturing sensor data by placing prototypes at different desired depths. For the plot Figure 11, PSFs were captured at distances ranging from 7 in (~178 mm) to 13 in (~330 mm), with steps of 1 in (25.4 mm) using a pinhole aperture of 1 mm diameter. For photography example (Figs. 12,17), PSF was captured at 16 in (~406 mm). The PSF for microscopy example (Fig. 13) was captured at 10 mm from the fluorescence filter using a pinhole aperture of 15 μm diameter. For refocusing (Fig. 15) and 3D imaging (Fig. 16), PSFs were captured at distances ranging from 10 mm to 110 mm, with steps: 1 mm for range 10–30 mm, and 5 mm for range 30–110 mm, using a pinhole aperture of 250 μm diameter.

Fig. 11.

Experimental evaluation of our camera’s resolution using fluorescent USAF target. The inserts are shown for line pairs with contrast close to 20% or higher.

Fig. 12. Experimental results: Photography.

The shortest distance to the scene is about 0.5 m, extending all the way to 3 m in the bottom scene. The bottom scene is a frame from video reconstruction. The video can be found in the supplementary material.

Fig. 17. Experimental comparisons.

We experimentally compare results from three different prototypes — (a) amplitude mask designed for separable PSF (FlatCam [10]), (b) phase mask designed for separable PSF, and (c) proposed phase mask designed for Contour PSF. The camera thicknesses are approximately 2 mm. FlatCam reconstructions are performed using Tikhonov regularization [10], and using deep learning method [49]. Both the phase-mask reconstructions are performed with Eq. 9. The proposed PhlatCam produces cleaner and higher quality images.

Fig. 13. Experimental results: Microscopy.

[Top-left] Fluorescence microscopy setup. [Top-right] Ground truth image of fluorescent sample taken using 2.5× microscope objective lens. The sample is a root cell from lily-of-the-valley (Convallaria majalis) stained with green fluorescent dye. [Bottom] Sample capture (at 10 mm away) and reconstruction.

Fig. 15.

We showcase the refocusing ability of PhlatCam. Three objects at three different distances comes into focus when we use the appropriate depth PSF for the reconstruction.

Fig. 16.

We showcase the 3D image reconstruction ability of PhlatCam at very close distance. The scene is a handwritten text, written using phosphorescent paint. The letter ‘L’ is at the closest distance from the camera, at 10 mm, and the letter ‘T’ is at 38 mm from the camera. Hence, the scene ranges from 0 to 28 mm.

5.2. Resolution characterization

From the lensless camera geometry, theoretical upper-limit resolution of a lensless camera can be derived as:

$$\mathrm{Resolvable\; feature}=\frac{\mathrm{Mask-Scene\; distance}}{\mathrm{Mask-Sensor\; distance}}(\mathrm{Pixel\; pitch})$$ (12)

From experimental testing using fluorescent USAF target, we find that using contour PSF achieves close to theoretical resolution as shown in Fig. 11. The pixel pitch of the camera is 4.8 μm.

5.3. Depth-dependance Characterization

PhlatCam has depth-dependent PSF that can be exploited for refocusing scenes at different depths and also to perform 3D imaging. The effect of depth on PSF is magnification or scaling, where the PSF shrinks at the rate of 1/depth as the scene depth increases (Eq. 4). This effect is a direct outcome of Fresnel propagation. The correlation of PSFs rapidly decreases at closer depths and has broader correlation profiles at farther depths. This property can be used for 2D imaging at far depths, computational depth refocusing of scenes at medium depth range, and 3D imaging of scenes at a closer depth range. Fig. 14 shows the magnification and correlation of the depth-dependant PSFs.

Fig. 14.

PhlatCam has depth dependent PSF that magnifies as the scene gets closer. The magnification falls with inverse depth relation. By looking at the correlation of PSFs, we can broadly categorize scene depth into 3 regimes. At close distances, the correlation falls quickly, enabling us to reconstruct 3D images. At the medium distances, the correlation falls gradually over a wider depth range. In this distance range, we can perform computational refocusing. At much larger depth, the dependence of PSF with depth saturates and all far scene points can be said to be beyond the hyperfocal distance of PhlatCam, thereby allowing only reconstruction of 2D images.

5.4. Imaging Experiments

5.4.1. 2D Imaging

We show 2D imaging using PhlatCam under various scenarios. Photography experiments are shown in Figs. 12, 17, while microscopy experiment is shown in Fig. 13. For all the reconstructions, the camera was 2 × 2 binned to have a pixel pitch of 9.6 pm. Additionally, the biography images are also taken with the PhlatCam.

We, also, experimentally (Fig. 17) compare the proposed PhlatCam with two other prototypes: (a) amplitude mask designed for separable PSF (FlatCam [10]), (b) phase mask designed for separable PSF, and (c) proposed phase mask designed for Contour PSF. FlatCam reconstructions are performed using Tikhonov regularization [10], and using deep learning method [49]. Both the phas-mask reconstructions are performed with Eq. 9.

5.4.2. Image Refocusing

At medium depth ranges, the depth-dependant PSFs uncorrelate slowly. Hence, the exact 3D reconstruction would e ill-conditioned. However, the PSF correlation fall-off can be exploited for performing computational refocusing from single captured measurements. We perform refocusing by reconstructing the image from the single capture by choosing the appropriate depth PSF. Scene points away from the selected depth appear blurry while the scene points at the selected depth plane appear sharp. Our refocusing experiment is shown in Fig. 15.

5.4.3. 3D Imaging

At very close depth range, the depth-dependant PSFs uncorrelate at a fast rate. This property can be exploited to perform 3D imaging. Our 3D imaging experiment is shown in Fig. 16.

6. Discussion and Conclusion

We demonstrated PhlatCam, a designed lensless imaging system that can perform high-fidelity 2D imaging, computational refocusing, and 3D imaging. These abilities are made possible by our proposed Contour PSF and the phase mask design algorithm. We used traditional optimization-based algorithms to reconstruct images in this paper. In the future, we will incorporate data-driven methods (like [49], [50]), to improve the reconstruction quality.

In this work, we used a heuristic approach borne from concepts of signal processing to engineer a high-performance PSF. As a future direction, we aim to optimize the PSF over a theoretical metric or through an end-to-end data-driven approach.

Fig. 3.

Our proposed phase-mask framework takes the input of target PSF and the desired device geometry and outputs an optimized phase-mask design.

Fig. 4. Conventional imaging and PhlatCam.

PhlatCam is 5–10× thinner and can reconstruct high-fidelity images from multiplexed measurements. Additionally, PhlatCam can function in more ways than conventional camera. Specifically PhlatCam can produce 2D images for any scene distance, refocused images at medium distance and 3D imaging at close distance.

Biographies

Vivek Boominathan received the B.Tech degree in Electrical Engineering from the Indian Institute of Technology Hyderabad, Hyderabad, India, in 2012, and the M.S. and Ph.D. degree in 2016 and 2019, from the Department of Electrical and Computer Engineering, Rice University, Houston, TX, USA. He is currently a Postdoctoral Associate with Rice University, Houston, TX. His research interests lie in the areas of computer vision, signal processing, wave optics, and computational imaging.

Jesse K. Adams received the B.Sc. degree in physics from the University of North Florida, Jacksonville, FL, USA, in 2014 and the M.Sc. and Ph.D. degrees in applied physics from Rice University, Houston, TX, USA in 2017 and 2019, respectively. After his Ph.D., he received a Cyclotron Road Fellowship hosted at Lawrence Berkeley National Laboratory designed to provide training, mentorship and networking tailored for transitioning innovative concepts into transformative products. His research interests lie in the areas of nanofabrication, optics, neuroengineering, and computational imaging.

Jacob T. Robinson received the B.S. degree in physics from the University of California, Los Angeles, Los Angeles, CA, USA and the Ph.D. degree in applied physics from Cornell University, Ithaca, NY, USA. After his Ph.D., he worked as a Postdoctoral Fellow with the Chemistry Department, Harvard University. He is currently an Associate Professor with the Department of Electrical & Computer Engineering and Bioengineering, Rice University, Houston, TX, USA, and an Adjunct Assistant Professor in neuroscience, Baylor College of Medicine, Houston, TX, USA. He joined Rice University in 2012, where he currently works on nanotechnologies to manipulate and measure brain activity in humans, rodents, and small invertebrates. Dr. Robinson is currently a Co-Chair of the IEEE Brain Initiative, and the recipient of a Hammill Innovation Award, NSF NeuroNex Innovation Award, Materials Today Rising Star Award, and a DARPA Young Faculty Award.

Ashok Veeraraghavan received the bachelor’s degree in electrical engineering from the Indian Institute of Technology, Madras, Chennai, India, in 2002 and the M.S. and Ph.D. degrees from the Department of Electrical and Computer Engineering, University of Maryland, College Park, MD, USA, in 2004 and 2008, respectively. He is currently an Associate Professor of Electrical and Computer Engineering, Rice University, Houston, TX, USA. Before joining Rice University, he spent three years as a Research Scientist at Mitsubishi Electric Research Labs, Cambridge, MA, USA. His research interests are broadly in the areas of computational imaging, computer vision, machine learning, and robotics. Dr. Veeraraghavan’s thesis received the Doctoral Dissertation Award from the Department of Electrical and Computer Engineering at the University of Maryland. He is the recipient of the National Science Foundation CAREER Award in 2017. At Rice University, he directs the Computational Imaging and Vision Lab.