Snapshot hyperspectral light field tomography

Abstract

We present snapshot hyperspectral light field tomography (Hyper-LIFT), a highly efficient method in recording a 5D (x, y, spatial coordinates; θ, φ, angular coordinates; λ, wavelength) plenoptic function. Using a Dove prism array and a cylindrical lens array, we simultaneously acquire multi-angled 1D en face projections of the object like those in standard sparse-view computed tomography. We further disperse those projections and measure the spectra in parallel using a 2D image sensor. Within a single snapshot, the resultant system can capture a 5D data cube with 270 × 270 × 4 × 4 × 360 voxels. We demonstrated the performance of Hyper-LIFT in imaging spectral volumetric scenes.

1. INTRODUCTION

An optical field can be characterized by a seven-dimensional (7D) plenoptic function, P (x, y, z, θ, φ, λ, t) (x, y, z, spatial coordinates; θ, φ, emittance angles; λ, wavelength; t, time) [1,2]. Recording light rays along all dimensions unveils a complete profile of the input scene, providing volumetric, spectral, and temporal information. However, a conventional image sensor measures only a two-dimensional (x, y) plenoptic function, throwing away much of the information and leading to an inefficient optical measurement.

Measuring a high-dimensional plenoptic function faces two main challenges: dimension reduction and measurement efficiency. On the one hand, because most photon detectors are in two dimensions (2D) (i.e., image sensors), one dimension (1D) (i.e., line sensors), or zero dimensions (0D) (i.e., single pixel sensors), capturing a high-dimensional plenoptic function with a low-dimensional detector usually requires extensive scanning along another dimension. For example, to acquire a plenoptic data cube (x, y, λ), a hyperspectral imager typically scans in either the spatial domain [3,4] or spectral domain [5–7], resulting in a prolonged acquisition. By contrast, snapshot techniques like an image mapping spectrometer (IMS) [8–11], coded aperture snapshot spectral imaging [12–14], and computed tomography imaging spectrometry [15,16] optically remap a 3D (x, y, λ) plenoptic data cube to a 2D detector array, enabling parallel measurement of data cube voxels and maximizing the light throughput [17,18]. To characterize this ability, we define a dimension reduction factor as $ϵ=N_P/N_D$, where N_P and N_D are the dimensionalities of the plenoptic function to be measured and the detector deployed, respectively. Because low-dimensional detectors usually image faster and cost less than their high-dimensional counterparts, the greater the ϵ, the higher the frame rate, and the more economical the system. On the other hand, under the conventional Nyquist sampling condition, measuring a high-dimensional plenoptic function usually requires the detector array to have a large number of elements, posing challenges with the data transfer and storage. An effective approach to break this limitation is compressed sensing [19–23], which allows using much fewer measurements to recover a scene, provided that the object can be considered sparse in a specific domain. To quantify the sampling efficiency, we define a compression ratio as $r=S_N/S_C$, where S_N and S_C are the sampling number determined by the Nyquist–Shannon theorem and compressed sensing, respectively. The higher the r, the more efficient the measurement.

Although techniques have advanced significantly in reducing the dimensionality of a plenoptic function, it is nontrivial to build an imager with a large compression ratio while maintaining high image quality. For example, in hyperspectral light field imaging, to acquire a 5D data cube (x, y, θ, φ, λ), most current imagers [24–28] are built on Nyquist sampling and provide only a unity compression ratio r = 1. For a given detector array, this leads to a trade-off between the samplings along the spatial, spectral, and angular axes. For example, in an IMS-based hyperspectral light field camera [28], the total number of plenoptic data cube voxels is limited to 66 × 66 × 5 × 5 × 40 (x, y, θ, φ, λ), restricting its application in high-resolution imaging. Although this trade-off can be alleviated by using a multiple-camera configuration [24,25,29], it increases the system’s form factor and complexity. Alternatively, Xue et al. utilized compressed sensing to recover a 5D data cube (x, y, θ, φ, λ) of size 1000 × 1000 × 3 × 3 × 31 from undersampled measurements in the spectral domain but with only a modest compression ratio of 3.4 [30]. It is challenging to scale up the compression ratio because the freedom of choosing a low-coherence sensing matrix is limited by the fabrication complexity of filters with an arbitrary transmission spectral curve.

To overcome this problem, we present a new multidimensional imaging architecture and demonstrate it in 5D (x, y, θ, φ, λ) plenoptic imaging. The resultant method, referred to as snapshot hyperspectral light field tomography (Hyper-LIFT), can capture a 270 × 270 × 4 × 4 × 360 (x, y, θ, φ, λ) data cube using a 2D detector array in a single snapshot. Based on the same conceptual thread as light field tomography (LIFT) [31], Hyper-LIFT is highly efficient in acquiring the light field data through simultaneously recording en face parallel beam projections of the input scene along sparsely spaced angles, enabling a compression ratio of 16.8. Moreover, Hyper-LIFT captures additional spectral information by further dispersing the en face beam projections in the spectral domain. By converting the angular information to depths, we demonstrated Hyper-LIFT in hyperspectral volumetric imaging.

2. METHOD

A. Imaging Formation and Optical Setup

The core idea of Hyper-LIFT is to reformulate the light field acquisition as a sparse-view computed tomography (CT) problem. To create multi-angled en face parallel beam projections of an object, we put an array of rotated Dove prisms and cylindrical lenses at the pupil plane of a main objective lens and image the object from different perspectives. For a given perspective, the image formation model through a rotated Dove prism and a cylindrical lens (i.e., a subfield) is shown in Fig. 1. The Dove prism rotates the input perspective image by an angle of 2θ, where θ is the rotation angle of the Dove prism itself. The rotated perspective image is then imaged by a cylindrical lens. The resultant image is essentially a convolution of the geometric image of the rotated object and a line spread function provided by the cylindrical lens. We put a slit at the back focal plane of the cylindrical lens and sample the image along the horizontal axis. The 1D signals obtained is a “projection” of the object at the angle of 2θ, resembling the projection measurement in traditional x-ray CT. The image formation can be formulated as

$$b^{\theta }=\mathit{T}{\mathit{R}}^{\theta }g,$$ (1)

where g is the vectorized two-dimensional (2D) perspective image, R^θ is the rotation operator representing the function of the Dove prism at the angle θ, T denotes the signal integration by the cylindrical lens at the 1D slit, and b^θ is the 1D signal sampled by the slit. After passing through a diffraction grating, the 1D projection is dispersed along the vertical axis. The final image is measured by a 2D detector array, and the spectral channels of the projection are obtained simultaneously.

Fig. 1.

Image formation model.

In Hyper-LIFT, we use an array of Dove prisms with varied rotation angles to acquire the dispersed en face projections in parallel. The forward model with n Dove prisms can be formulated as

$$\mathit{b}(\mathrm{\lambda })=\mathit{T}\left[\begin{array}{c}{R}^{\theta 1}\\ \cdots \\ R^{\theta n}\end{array}\right]g(\mathrm{\lambda })=\mathit{A}g(\mathrm{\lambda }),$$ (2)

where A is the forward operator representing the parallel beam projections at different angles, and b(λ) is the sinogram at wavelength λ. Because each Dove prism observes the same scene from different perspectives, the light field is naturally sampled in the projection data with an angular resolution equal to the number of Dove prisms.

The schematic of the optical setup is shown in Fig. 2. The object locates at the front focal plane of an objective lens (f = 50 mm, f/# = 1.4). The field of view of the system is ~ 3 mm. A Dove prism array that comprises 16 multi-angled Dove prisms (height = 2 mm, length = 8.4 mm, and spacing between adjacent prisms = 2.9 mm) is located at the back focal plane of the objective lens. After the Dove prism array, a 5 × 1 cylindrical lens array (height = 2 mm, length = 12 mm, focal length = 20 mm) focuses the collimated beam onto a 4 × 1 slit array (width = 10 μm), which slices the input image. The resultant 1D projections are reimaged to a CCD sensor (Lumenera, Lt16059H; pixel pitch = 7.4 μm) using a 4 f relay system (f = 100 mm, f/# = 2). A transmission diffraction grating (300 groves/mm) is located at the Fourier plane of the relay system to disperse the projections. The Dove prism array and cylindrical lens array are detailed in Supplement 1.

Fig. 2.

Optical schematic of a Hyper-LIFT camera.

In Hyper-LIFT, the spatial sampling of the projections can be calculated as the height of a Dove prism divided by the camera pixel pitch. The spectral sampling of the reconstructed data cube is limited by the spacing between adjacent Dove prisms. The angular sampling is determined by the number of Dove prisms and their locations at the pupil plane. In our setup, the reconstructed data cube has a dimension of 270 × 270 × 4 × 4 × 360 (x, y, θ, φ, λ). Noteworthily, in Hyper-LIFT, only the voxels along the spectral dimension are directly mapped onto the 2D sensor, while the spatial and angular voxels are multiplexed in projection measurement. Additionally, instead of acquiring the entire set of projections under the Nyquist sampling condition, Hyper-LIFT measures only a sparse subset of the projections based on compressed sensing. By contrast, previous multidimensional imagers [26–28,32] directly map plenoptic data cube voxels to a 2D image sensor. As a result, the size of the reconstructed data cube is limited by the available sensor pixels, leading to a trade-off between spatial, angular, and spectral sampling. Using compressed sensing, Hyper-LIFT alleviates this trade-off, thereby enabling the measurement of a large format plenoptic function.

Like conventional sparse-view CT, Hyper-LIFT requires the object to be sparse in a specific domain. We quantified the resolution under such a condition by imaging a point object (a 10 μm pinhole). The lateral and axial resolution resolutions are measured as the full width half-maximum (FWHM) of the impulse response along the lateral and axial direction, respectively (in Supplement 1, Fig. S3). The measured lateral and axial resolutions are 22 μm and 1 mm, respectively.

B. Image Reconstruction

In Hyper-LIFT, because the number of projections n is smaller than the pixel resolution of the input perspective image, Eq. (2) is under-determined and hence can be considered as a sparse-view CT problem [33]. The image reconstruction of a monochromatic scene can be achieved by iteratively solving the optimization problem:

$$\underset{\widehat{g}}{\mathrm{argmin}}{\Vert \mathit{b}-\mathit{A}g\Vert }_2^2+\rho \phi {(g)}_1,$$ (3)

where φ(g) is a transform function that sparsifies the image, ⋅₁ is the l₁ norm, and ρ is the hyperparameter that weights the regularization term. More details of solving this inverse problem and the sparsity requirements can be found in [31]. A reconstruction example of a planar monochromatic scene is shown in Fig. 3. To limit the illumination bandwidth, a 3 nm bandpass filter centered at 532 nm was placed right after a broadband light source (Amscope, HL250-AS). The ground truth and a raw image captured by the Hyper-LIFT camera are shown in Figs. 3(a) and 3(b), respectively. The sliced projections are along the vertical axis, and the spectra of the projections are dispersed along the horizontal axis. To perform image reconstruction, we first divide the raw image into 16 subfields, each containing a projection. Next, the projections are extracted based on their center coordinates, which can be calibrated by imaging an optical pinhole. Finally, the object is reconstructed using Eq. (3), and the reconstruction result is illustrated in Fig. 3(c).

Fig. 3.

Reconstruction of a planar object illuminated by monochromatic light (wavelength, 532 nm). (a) Ground-truth photographic image. (b) Raw Hyper-LIFT image. (c) Reconstructed image.

To reconstruct images at different wavelengths, we apply Eq. (3) to the corresponding sinogram b(λ). Because the spectrum is dispersed along one axis, the projection at wavelength λ can be directly obtained by extracting the line image at the corresponding position, which can be deduced from the chromatic dispersion curve of the diffraction grating. The calibration of the dispersion curve is detailed in Supplement 1. The spectral range and resolution can be deduced from the dispersion curve, and they are 90 nm and 0.25 nm, respectively, in the current setup. It is worth noting that the spectral range and resolution are scalable by using a different dispersion element or a different focal length relay system.

The whole reconstruction process can be divided into two steps: wavelength selection and CT reconstruction. Because the spectrum is directly dispersed on the detector using a diffraction grating, projections at different wavelength can be extracted using the dispersion curve, where the computational cost is negligible. In the CT reconstruction process, the image reconstruction speed is mainly dependent on the reconstruction algorithm applied. For example, when simply applying the inverse Radon transform through fast Fourier transform, the object can be reconstructed instantaneously. However, when using an iterative algorithm to improve image quality, the reconstruction takes ~5 s for each wavelength on a personal computer (CPU, Intel Core i7-5700HQ).

C. Digital Refocusing

In a conventional light field camera, the acquired light field L can be parameterized by the aperture plane (u, v) and the image plane (x, y) as shown in Fig. 4(a). For a synthetic image plane (x′, y′) defined by a shifting parameter α, the irradiance can be calculated as [34]

$$E(x^{\prime },y^{\prime })=\iint L\left(u,v,\frac{x^{\prime }+(\alpha -1)u}{\alpha },\frac{y^{\prime }+(\alpha -1)v}{\alpha }\right)\mathrm{d}u\mathrm{d}v.$$ (4)

Fig. 4.

Digital refocusing in Hyper-LIFT. (a) Refocusing a synthetic image plane in a conventional light field camera. (b) In Hyper-LIFT, the shifting vector is determined by the pupil coordinate and the rotation angle of the Dove prism. Each blue circle denotes a Dove prism.

Digital refocusing is achieved by shifting and adding up subaperture images. Unlike conventional light field cameras, Hyper-LIFT employs a Dove prism array to rotate the input scene, and their effect must be accounted for when calculating the shifting vector. Here, we denote the original shifting vector as s ⋅ (u, v), where (u, v) is image translation direction vector and s is the translation distance (i.e. shearing parameter). As illustrated in Fig. 4(b), the vector (u, v) is determined by the pupil coordinate and the rotation angle of the Dove prism. If the Dove prism is rotated by θ/2, the resultant shifting vector can be derived as follows:

$$s\cdot \left(\begin{array}{c}{u}^{\prime }\\ v^{\prime }\end{array}\right)=s\cdot \left[\begin{array}{cc}\cos (\theta )& -\sin (\theta )\\ \sin (\theta )& \cos (\theta )\end{array}\right]\cdot \left[\begin{array}{cc}1& 0\\ 0& -1\end{array}\right]\cdot \left(\begin{array}{c}u\\ v\end{array}\right)=s\cdot \left(\begin{array}{c}u\cdot \cos (\theta )+v\cdot \sin (\theta )\\ u\cdot \sin (\theta )-v\cdot \cos (\theta )\end{array}\right).$$ (5)

Because Dove prisms rotate the input scene based on total internal reflection (TIR), an extra flipping matrix is added to reflect the redirection of light rays. The updated shifting vector is then applied to translating subaperture projections to obtain an updated sinogram. The final refocused image can be reconstructed using a classic computational tomography algorithm, such as analytic filtered backprojection [35].

3. RESULTS

A. Hyperspectral Imaging of a Planar Object

We first demonstrated our system in imaging a rainbow planar object [Fig. 3(a)]. The illumination setup is shown in Fig. 5(a). A linear variable visible range bandpass filter (Edmund optics, 88365) is located at the conjugate plane of the planar object, illuminated by a broadband light source. To fit more wavelengths into the field of view, a 3.3:1 lens pair (Thorlabs, MAP1030100-A) is used to demagnify the linear filter. In this way, each lateral coordinate of the planar object is encoded with a unique color. At a given location, the spectral resolution of the linear variable filter is 7–20 nm. The raw image captured by the Hyper-LIFT camera is shown in Fig. 5(b). Compared to the monochromatic data in Fig. 3(b), the spectrum is dispersed along the horizontal axis. The projections at different wavelengths are extracted based on the chromatic dispersion curve. The pseudo-colored reconstructed panchromatic image and four representative color channels are shown in Fig. 5(c). A video shows the sweeping of wavelengths of the captured scene from 510 to 540 nm, with a step size = 1 nm (Visualization 1). This experiment demonstrates the hyperspectral imaging capability of our system.

Fig. 5.

Hyperspectral imaging of a planar object. (a) Illumination setup. (b) Raw Hyper-LIFT image. (c) Pseudo-colored reconstructed panchromatic image and four representative color channels.

B. Hyperspectral Volumetric Imaging of a 3D Object

To demonstrate hyperspectral volumetric imaging enabled by light field capture, we imaged a 3D-printed object illuminated by a broadband light source. To avoid cross talk between adjacent subaperture images, a 40 nm bandpass filter at 550 nm is used to limit the illumination bandwidth. An en face photograph of the object is shown in Fig. 6(a). The letters “U” and “C” were 3D printed at different depths (depth difference = 6 mm). The object is imaged to the front focal plane of the objective lens by a demagnifying relay lens pair (Thorlabs, MAP1030100-A). To render a focal image stack, digital refocusing is performed by calculating the shifting direction vector (u, v) and then changing the shearing parameter s in Eq. (5). In practice, the shearing parameter s can be converted to a real depth. The shearing to depth calibration is detailed in Supplement 1. The reconstructed 3D image and two representative numerically refocused images at the two letters’ depths are shown in Fig. 6(b). The pixel intensities within the boxed area in Fig. 6(b) are averaged, and a spectrum is shown in Fig. 6(c). A video shows the sweeping of focal plane images from −3 to 3 mm at 532 nm (Visualization 2). Finally, we quantified the level of defocus by calculating the sharpness of the object at each refocused depth [Fig. 6(d)]. Here sharpness is defined by $\frac{\sum {(I(x,y))}^2}{{(\sum I(x,y))}^2}$, where I(x, y) is the pixel value at the letter location. The sharpness of letters “U” and “C” reaches a maximum at 0.8 mm and 0 mm, respectively, which gives the physical depth of two letters in the object space of Hyper-LIFT.

Fig. 6.

Hyperspectral volumetric imaging of a 3D object. (a) Ground-truth en face image of the object. “U” and “C” were printed at two different depths. (b) Reconstructed 3D image and two numerically refocused images at the letters’ depths. (c) Spectrum averaged within the dashed boxed area in the low left panel of (c). (d) Sharpness versus depth at two letter locations.

4. DISCUSSION

The application of hyperspectral light field imaging continues to expand, with a variety of technologies sprouting in the past five years. Despite impressive functionality, these techniques have not been discussed in a common framework. Multidimensional imagers are typically evaluated in terms of the snapshot factor [18], which describes the portion of the plenoptic data cube voxels that can be seen by the imager at a time. The greater the snapshot factor, the higher the light throughput. However, the snapshot factor does not reflect the easiness of acquiring high-dimensional data with practical photodetectors. Therefore, we added two additional metrics—dimension reduction factor ϵ and data compression ratio r—to fully characterize the performance of multidimensional imagers. As an example, Table 1 shows a comparison of several state-of-the-art single-camera-based hyperspectral light field imagers, where Hyper-LIFT shows a clear edge over other imagers in the overall performance.

Table 1.

Comparison of Hyperspectral Light Field Imagers in Snapshot Factor ξ, Dimension Reduction Factor ϵ, and Data Compression Ratio r

	ξ	ϵ	r
Tunable filter hyperspectral light field camera [38]	0.09–0.13	2.5	1
Spatial-scanning hyperspectral light field camera [39]	0.04	2.5	1
Light field Fourier transform imaging spectroscopy (FTIS) [26]	1	2.5	1
Light field IMS [28]	1	2.5	1
Spectrally coded catadioptric mirror array imaging [30]	1	2.5	3.4
Hyper-LIFT	1	2.5	16.8

It is noteworthy that, when designing the system, the light efficiency is collectively determined by the desired spectral resolution and the field of view (FOV). On the one hand, a wider slit increases the light throughput, but it also increases the cross talk between adjacent color channels, resulting in a reduced spectral resolution. On the other hand, imaging a large FOV can reduce the light efficiency because an off-axis field point requires the cylindrical lens to provide an elongated line spread function (LSF) in order to extend the signals to the slit location, which, in turn, decreases the irradiance at the slit. Therefore, there is a trade-off between the light throughput and the spectral resolution and the FOV.

Like standard CT, the spatial resolution of Hyper-LIFT is collectively determined by the number of projections and the angle distribution. By contrast, the spectral dimension is directly mapped to the camera without resolution loss. Figure 7(a) shows the peak signal-to-noise ratio (PSNR) versus the number of projections in the reconstructed image for a simulated circular object. The result indicates that the larger the number of the projections, the higher the PSNR. Additionally, Hyper-LIFT provides a uniform projection angle distribution from 0 to π, which minimizes the correlations in the projection data and maximizes information content for reconstruction [31]. This is made possible by using a rotated Dove prism array. By contrast, our previous LIFT system creates the projections using rotated cylindrical lenses, leading to a limited view problem. More specifically, the LIFT system only captures the projections with an angle in $\left[-\frac{\pi }{4},\frac{\pi }{4}\right]$, rather than $\left[-\frac{\pi }{2},\frac{\pi }{2}\right]$. Therefore, the features along vertical dimension cannot be faithfully reconstructed (a more detailed discussion can be found in supplementary materials of Ref. [31]). To demonstrate the capability of Hyper-LIFT in overcoming this problem, we imaged a group of bars of a USAF resolution target (Group 0 element 6) along both horizontal and vertical directions [Fig. 7(b)]. We further plot the intensity along the dashed line in both images. The average FWHM of a bar is 11 and 12 pixels along the vertical and horizontal direction, respectively. This result indicates that our system has the ability to equally resolve the object features along two orthogonal directions.

Fig. 7.

Reconstruction quality in Hyper-LIFT. (a) Peak signal-to-noise ratio (PSNR) versus number of projections. (b) Resolving USAF resolution bars along the vertical and horizontal directions. The light intensities (right panel) were plot along the dashed lines in the reconstructed images.

In conclusion, we developed and experimentally demonstrated a versatile snapshot Hyper-LIFT imager. As shown in Table 1, the system outperforms other state-of-the-art hyperspectral light field imagers in both the snapshot factor and data compression ratio allowed, and it is the only imager that can perform a large-scale hyperspectral light field measurement using a 2D detector array with a moderate format. Because the frame rate of cameras is generally proportional to the reciprocal of the total number of camera pixels in use, our compressed measurement scheme has an advantage in imaging speed. For example, when reading out only the regions of interest (ROIs) that receive light signals from the camera, our system can measure a 5D data cube (x, y, θ, φ, λ) with 270 × 270 × 4 × 4 × 360 voxels at 30 Hz. By contrast, our previous uncompressed (i.e., direct mapping) hyperspectral light field camera [28] can operate at only 5 Hz, even when measuring a data cube of a much smaller size (66 × 66 × 5 × 5 × 40 voxels). This capability is highly desired in imaging applications that simultaneously require a high spatial, spectral, and temporal resolution, such as snapshot spectral-domain optical coherent tomography (SD-OCT) [36,37], where the previous system suffers from the trade-off between the spatial and spectral resolution. Therefore, we believe the Hyper-LIFT approach will be uniquely positioned in addressing the leading challenges in multidimensional optical imaging [2].

Data availability.

Data and codes underlying this work may be obtained from the corresponding author upon reasonable request.