Computational spectrometers enabled by nanophotonics and deep learning

Li Gao; Yurui Qu; Lianhui Wang; Zongfu Yu

doi:10.1515/nanoph-2021-0636

. 2022 Jan 24;11(11):2507–2529. doi: 10.1515/nanoph-2021-0636

Computational spectrometers enabled by nanophotonics and deep learning

Li Gao ^1,^✉, Yurui Qu ², Lianhui Wang ^1,^✉, Zongfu Yu ^2,^✉

PMCID: PMC11502016 PMID: 39635673

Abstract

A new type of spectrometer that heavily relies on computational technique to recover spectral information is introduced. They are different from conventional optical spectrometers in many important aspects. Traditional spectrometers offer high spectral resolution and wide spectral range, but they are so bulky and expensive as to be difficult to deploy broadly in the field. Emerging applications in machine sensing and imaging require low-cost miniaturized spectrometers that are specifically designed for certain applications. Computational spectrometers are well suited for these applications. They are generally low in cost and offer single-shot operation, with adequate spectral and spatial resolution. The new type of spectrometer combines recent progress in nanophotonics, advanced signal processing and machine learning. Here we review the recent progress in computational spectrometers, identify key challenges, and note new directions likely to develop in the near future.

Keywords: compressive sensing, deep learning, nanophotonics, spectral imaging, spectral sensing, spectroscopy

1. Introduction and overview

Optical spectrometers serve as one of the most important instruments in fields such as astronomy, biology, and chemical analysis [1, 2]. Conventional optical spectrometers typically consist of bulky dispersive elements such as gratings and prisms or interferometric elements. In conventional spectrometers like monochromators or Fourier-transform spectrometers, light only passes through the instrument once or twice, which means the optical path needs to be long enough to separate different wavelengths and achieve high spectral resolution. The long optical path, which is typically on the order of tens of centimeters, limits the size of the instrument. Recent demand for wearable electronics and portable instruments has stimulated enormous interest in smaller, cheaper spectrometers that can also provide real-time, accurate readouts [3], [4], [5], [6], [7], [8], [9], [10].

In contrast to conventional geometric optics, nanophotonic structures can increase the length of the optical path to millions of times larger than their physical size [11, 12]. Different materials and geometric designs of nanophotonic structures can induce completely different transmission, reflection, scattering, and absorption characteristics at every wavelength. Such strong light–matter interaction is referred to as the response function of nanophotonic structures. The response function of Fabry–Perot resonator is ideally a narrow-band delta function, while many periodic nanophotonic structures are broadband, with randomly distributed spectral features. The term filter response functions is commonly used since the transmission mode of operation is adopted more often. The micrometer-sized nanophotonic filters can be integrated directly onto pixels of photodetectors so that the spectral information can be encoded by precalibrated response functions and recorded in a snapshot by detector arrays. With the aid of appropriate computational algorithms based on compressive sensing (CS) theory, the spectral information can be accurately decoded and reconstructed [13–25].

Such emerging and novel spectroscopic technique is called computational spectroscopy. As no moving elements or bulky components are involved, such spectrometers can be extremely compact in size – with a footprint as small as sub-millimeters – and robust enough for portable or wearable application. Furthermore, the data acquisition in a snapshot and the data processing assisted by computation allows for simple, automated operations. Exciting innovations in both hardware and software solutions are merged in such systems, which are fundamentally different from conventional spectroscopy.

The typical workflow of computational spectrometer consists of three steps: calibration, measurement, and the spectrum reconstruction process. The calibration process measures the response function of the optical filter array using a monochromator or a tunable laser. The measurement process is usually to capture a single-shot image of the unknown sample. The third step is to reconstruct the unknown spectrum based on the filter matrix and the single-shot image data from the first two steps, usually through CS theory and reconstruction algorithms based on minimizing regularized squares errors with non-negativity constraints [26–29]. The typical operation scheme for spectral sensing and imaging are sketched in Figure 1. A single spectrometer is often used for a single-point measurement without much spatial resolution. Therefore, for spectral sensing, a single spectrometer can perform a single spectrum $I (λ)$ reconstruction.

Figure 1: — The major components and common operation mechanisms of computational spectrometers. Transmitted, reflected, or absorbed spectra are filtered by multiple nanophotonic structures, each with distinct response functions. The wavelength-dependent light signal is collected by sensor pixels and used for computational spectrum reconstruction.

In contrast, hyperspectral imaging technology allows both spectral and spatial information to be obtained simultaneously. In hyperspectral imaging, a 3D data cube is generated from a 2D spatial image plus a third dimension of spectral information, such as an RGB color image. Hyperspectral data cubes can contain absorption, reflectance, or fluorescence spectrum data for each image pixel. Therefore, for spectral imaging, a data cube $I (λ, x, y)$ with both spectral and spatial information can be reconstructed.

Although the hardware set-up and data acquisition operations are straightforward and simple, the decoding process for interpreting the multiplexed spectral data is challenging. In the encoding process (the above-mentioned measurement process), the complex interaction between incident spectrum and nanophotonic filters forms the detected multiplexed signal data. For decoding, we must solve inverse problems using computational algorithms to solve a set of linear equations, provided that the parameters are carefully chosen and sufficient iterative computation time is available [30–35]. Conventional reconstruction methods use regularizations based on smoothness and sparsity to improve the reconstruction quality, but these methods often result in severe distortions in the reconstructed spectrum due to the ill-posedness of the problems under very noisy environments. As this research area evolves, the solution to the most challenging spectrum reconstruction problems is ideally found in integration with data-driven deep learning (DL) techniques [36–39]. With this approach, the specific filter response functions do not need to be precalibrated. Instead, a large, labeled dataset is fed in to the neural network for training and mapping the relationship between the spectrum and detected signal. Such a one-time investment in training procedure would enable numerous and instantaneous spectrum reconstructions at the subsequent application stage. The reconstruction accuracy and efficiency are no longer experience-dependent and can also be greatly enhanced with better noise tolerance. Therefore, machine learning tools can potentially dissolve the limitation of computational techniques and deal with big data problem, which can enable truly intelligent spectroscopic sensing and imaging systems with unprecedented performance.

In this review, we introduce computational techniques in spectroscopy where nanophotonics are involved as the hardware component in the encoding process and relevant computational algorithms as the software component used for decoding process. Later, we focus more on the fusion of DL techniques for spectrum reconstruction tasks. Please note that DL has been used intensively in nanophotonic inverse design problems; i.e., with desired resonant spectral features in mind, deep neural networks (DNN) can output the best set of nanophotonic design to produce the desired spectrum. This topic has previously been discussed in multiple comprehensive reviews [40–44] and is significantly different from our scope. In our current topic, we care about the application of computational and DL techniques in spectroscopic reconstructive processes, rather than using DL to design photonic components. With the implementation of DL tools, the spectral and spatial data can be well reconstructed for next-generation sensing and imaging technologies.

2. The encoding and decoding process in computational spectrometers

2.1. Nanophotonic filters as spectral encoders

Powered by the manufacturing ability to precisely sculpt the metallic and dielectric nanostructure geometry, light can easily be controlled to great distance with tailored light–matter interactions below the optical wavelength. It is fascinating that an ultra-thin, flat nanophotonic device composed of arrays of metallic or dielectric resonators can alter light propagation so effectively [45]. The field of nanophotonics has attracted significant attention from those working with light manipulation and photonic information technology. Examples include photonic crystal slabs fabricated with high-refractive index materials [13, 46–48]; plasmonics that confine light into an extremely small scale, resulting in huge near-field enhancement [49]; and metasurfaces that induce anomalous refraction and reflection by engineering the spatial and spectral response of the impinging wavefront [50, 51]. Therefore, careful selection of nanophotonic geometry, materials, and unit cell arrangements can excite the desired resonance with incident light, resulting in impactful applications such as high-quality structural colors [52], ultraflat metalens [53, 54], high-density data storage units [55], etc. Moreover, nanophotonics have been used as color filters and polarizers in image sensors [56].

The origin of intense light–matter interaction in periodic nanophotonics is due to the light bouncing back and forth with very enhanced optical paths at interfaces that modify the collective amplitude, phase and polarization properties of light propagation. The diverse design of nanophotonic filters, such as cavity and resonator filters, can theoretically result in a Q factor of ∼10⁹ [12] and an experimental Q factor of more than 2 × 10⁶ [11] at specific resonant frequencies. However, the range of application wavelengths is limited to narrow bands of resonance. Such highly resonant nanophotonic designs are desired in narrow-band filters, spectrometers and those spectral-sensing applications which require narrow, sharp resonance peaks with high Q factors; thus precise spectrum shifts caused by small traces of analytes are easy to detect [29, 57, 58]. In earlier experiments, high-Q resonance Fabry–Perot and etalon-array types of structures have also been used as band-pass spectral filters for computational spectrometers [20, 31]. In general, these resonant filters have simple Lorentz line shapes and lack the spectral diversity to provide high-spectral resolution. In order to pursue high-spectral resolution, a number of filters with sharp passing bands are usually required, which is practically challenging due to the design and fabrication complexity [32, 47, 59–62]. To resolve this dilemma, researchers turn to arrays of flat nanophotonic filters that can induce broadband, wavelength-dependent and diverse resonant features to encode the unknown spectrum. Such nanophotonic filters are advantageous when compared with conventional narrow-pass band filters, because they can expand the operational range with high-spectral resolution. The production of these filters is also compatible with nearly all photonic materials and structures [63]; some typical examples are discussed in Figure 2.

As illustrated in Figure 2a, a diverse range of random spectral features can be created by absorption in colloidal quantum dots (CQD) filters [19]. The absorption spectra can be tuned continuously and finely over wavelengths ranging from deep ultraviolet to mid-infrared simply by changing the CQD’s size, shape and composition. In contrast to quantum dots, where the fabrication could be complicated, photonic crystal (PC) slabs can be defined via single-exposure photolithography and only require standard CMOS materials [22, 23]. In PC slabs, the light can bounce in a lateral direction parallel to the slab into the interfaces of air holes, forming guided resonances with high-quality factors. The light can also bounce along interfaces in a normal vertical direction to the slab and form Fabry–Perot resonances with low-quality factors. The combination of multiple resonances can enhance the diversity of spectral features through Fano interference [66, 67]. Three representative examples of different PC slabs and their corresponding spectra are shown in Figure 2b, where multiple random resonance features are present. As the spectral response functions are entirely extrinsic and enabled by structures instead of materials’ properties, the concept can be applied to other wavelength ranges by scaling the dimension of PC properly. Other than perfect periodic photonic structures, filters can also be made based on the semi-regular Voronoi structure, which forms multilayered photonic crystals with random wave-like structures in each of the Voronoi cells. The transmission characteristics of the multilayered photonic crystal can be controlled simply by changing the cell’s microstructure, and there is no need to change the manufacturing process and materials. Furthermore, such Voronoi cells can be randomly distributed, so the filter can be positioned at arbitrary positions and angles, followed by filter-response characterization after mounting [68].

Other than photonic crystals, metallic plasmonic structures are another important type of broadband filters. An array of plasmonic metasurface filters was fabricated by e-beam lithography and used for short- and long-wave infrared spectra reconstruction [21]. One can also utilize a flat spectral encoding chip with an array of nanostructured plasmonic tiles to filter the incident light, where each tile has a specific transmission spectrum, as shown in Figure 2c. The broad maxima and resonance features result from the plasmonic modes supported by the metal nanostructures. Such filter chip can be fabricated through a scalable and low-cost imprint lithography process with metal deposition [38].

During the past few years, dielectric metasurfaces have been examined intensively. Here, meta-atoms can effectively alter the light phase and polarization with much less energy loss than metallic structures. Ultrathin and high-performance metasurfaces are compatible with CMOS fabrication [69] and can replace bulky geometric optics [51]. The thin, compact form of dielectric metasurfaces is also ideally suited for transmission filter responses. Changing the geometry of dielectric metasurfaces can result in sensitive optical resonance features. For example, a planar, array-like geometry of silicon nanostructure shows unique scattering spectrum depending on whether a silicon block is present. Such scattered spectrum response can be used for optical data information storage [55]. A C4 symmetric metasurface design that is formed on silicon-on-insulator chips and later transferred onto CMOS image-sensor chips is proposed for polarization-independent performance with expanded spectrometer applications. A total of 400 different filters are fabricated; three representative examples are shown in Figure 2d.

On the contrary to manmade devices, where randomness and diversity are introduced on purpose, a natural or cultured pearl has a large number of alternating layers of crystalline aragonite and organic macromolecules with an irregular brick-and-mortar structure, as seen in Figure 2e. The low-dimensional irregular nanostructures found in pearls can be ideal for constructing highly uncorrelated spectral resonance via strong light localization to demonstrate spectral compressive sampling. Different areas of the pearl shows distinct intensity speckle maps and transmission spectrum [64]. Pearls thus provide a clue for simple, scalable, man-made filter components by carefully introducing the degree of disorder. With the recent advancement in materials growth and optoelectronic device design, it has also been demonstrated that a passive nanophotonic filter component and an active photodetector pixel can be integrated into one single nanowire device (Figure 2f), which means the spectral multiplexing and detection can be completed simultaneously [65]. In this pioneering work, an array of single nanowires can be engineered with a compositionally graded semiconducting CdS_xSe_1−x nanowire, which corresponds to a continuous gradient of bandgaps spanning from 1.74 to 2.42 eV along their length, and results in different spectrum response functions. The photodetector units are defined as units between two neighboring electrodes, which can measure the photocurrents at different wavelengths. Such nanowire schemes serve both as spectral filter and photodetector, which allows further miniaturization of compact spectrometers. Similarly, a black phosphorous single spectrometer has leveraged the wavelength and bias-dependent responsivity matrix learned from the spectra of a tunable blackbody source. Unknown monochromatic and broadband spectra can be reconstructed from their corresponding photoresponse vectors in the mid-infrared range [70].

Reconstructing accurate spectrum by CS theory relies on the randomness of the measuring bases. Therefore, each filter response function should have diverse spectral features with both broad and narrow line shapes, as discussed. In addition, different response functions in the same spectrometer should have minimal correlation. To quantify such effect, a correlation coefficient $r$ value measures the similarity between each pair of the nanostructure transmission spectra at different wavelengths. A value smaller than 0.5 indicates a moderate or weak correlation between different filter structures, and thus good designs. It is calculated by $r_{i j} = E {[t_{i} - E (t_{i})] \cdot [t_{j} - E (t_{j})]} / (δ_{i} δ_{j})$ for each pair of rows in the transmission matrix in different wavelength ranges, where $t_{i}$ is the transmission of the $i$ th filter, $E$ indicates the average, and $δ$ indicates the standard deviation [23]. Both material property and nanophotonic geometry determine the response functions and their correlation. For example, different PC slab filter designs have been investigated for their response function correlation for the entire 400–700 nm spectral range, as shown in Figure 3a. The mean values of the correlation coefficient are 0.84, 0.49, and 0.35 for PC slabs fabricated using SiC, SiO₂, and SiN_x. These results indicate the transmission spectra of SiC PC slabs are highly similar over a broad wavelength band, while SiN_x PC slabs show minimum similarity and most randomness in their spectra features. The undesirable high correlation in SiC is due to the strong absorption in the shorter wavelength range, while the correlation in the SiO₂ PC slabs increases significantly in the longer wavelength range because due to a smaller refractive index, they can support fewer number of optical modes in the long wavelength range. It is found that the SiN_x PC slabs maintain low mutual correlation values in all spectral ranges and is most promising for spectral reconstruction. In the pearl spectrometer case [64], spectral similarity with high correlation coefficients (>0.5) is also avoided at adjacent transmission spots that couple to each other. The pairwise comparison of different transmission spectra are generally uncorrelated, with an average correlation coefficient of 0.161, which proves such naturally occurring irregular and random multilayers are desirable and may inspire similar biomimetic filter structures.

Figure 3: — The correlation map between different nanophotonic filters and data acquisition set-up for spectral reconstruction and spectral imaging based on photonic crystal slabs. (a) Mapping of correlation coefficients between each pair of the 42 simulated transmission spectra in different wavelength ranges. Blue indicates weak or negative correlation; red indicates strong positive correlation. (b) Sensor operating principle and schematics (top). The spectral sensor consists of an array of PC slabs atop a CMOS image sensor. The transmission spectra of the PC slabs differ significantly from each other and form a sampling basis, T. For an incident light, the pixel values of the sensor and the sampling basis T are used to infer the unknown spectrum of the light. Device photograph (bottom) of an array of PC slabs fabricated on a SiN_x/quartz substrate and placed on a CMOS image sensor. (c) Hyperspectral imaging using the PC spectrometer. The imaging target is a mix of the numbers “5” and “9”, with the “5” comprising light incident on the screen at 610 ± 5 nm and the “9” comprising light at 670 ± 0.7 nm. The spectral information of this target is obtained via a single snapshot. The optical and captured images are shown on the left and right of the device, respectively. (a) and (b) are reprinted with permission from Ref. [23], copyright 2019 Optical Society of America; (c) is reprinted with permission from Ref. [22], copyright 2019 Nature Publishing Group.

2.2. Spectral data acquisition

In a computational spectroscopic system, the spectral data acquisition and processing is mainly based on the concept of compressive sensing. Compressive sensing is digital-signal processing that offers the potential of high-resolution readout of sparse physical signals from fewer measurements than the Nyquist–Shannon sampling theorem requires [71–74]. The unknown spectrum is measured by projecting it to a set of random bases. Typically, the number of the measurements is smaller than the dimension of the unknown spectrum. The spectrum can be recovered by finding solutions to an underdetermined linear system. It is particularly powerful for recovering sparse signals using a fewer number of measurements. L₁ norm regularization is used to force the unknown signal to be sparse [75]. In practice, the unknown spectrum signal is projected to random bases through the broadband nanophotonic filters. Random response functions are preferred to improve the spectral resolution, as the correlation between the functions is minimized. In general, whether a system is over- or under-sampling is determined by whether the number of detected signals is higher or lower than the number of data points in the reconstructed spectrum, respectively. As seen in many previous studies [22, 23, 27, 61], by just sampling tens of filtered detector responses, a band of a few hundred nanometers can be reconstructed with approximately 1 nm resolution: a compression ratio better than 10:1. This reduced sampling is obviously superior to a monochromator-based conventional spectrometer, where high spectral resolution must rely on high-precision gratings or tunable narrow-filter bands that sample hundreds of times.

In a typical setup of CS-based computational spectrometer (shown in Figure 1), a set of nanophotonic filters are coupled or integrated onto detector arrays made of charge-coupled devices (CCD) or CMOS technology. After a spectrum shines on the nanophotonic filters, transmitted, absorbed or reflected light intensity is recorded on the detector array underneath each filter. All these filters define the size of a single spectrometer, which may be considered as “spectrometer pixel” or “super pixel.” On the spectrometer pixel, there are multiple filters containing different nanostructures with distinct transmission, absorption or reflection response functions, and the size of a single filter is defined as “filter pixel.” Under each filter pixel, there are a few photodetector pixels that detect the signal; each one of them is called “detector pixel” or “subpixel.” In a single snapshot, multiple spectral measurements can be acquired without scanning or switching filters, and all spectral intensity data are recoded and used to reconstruct the incident spectrum.

S_{i} = \int_{λ_{1}}^{λ_{n}} I (λ) T_{i} (λ) η (λ) dλ, i = 1,2, \dots, M .

(1)

The above equation describes the mathematical operation mechanism of the spectra encoding process. Here, $I (λ)$ is the spectrum of incident light source or the unknown spectrum signal to be probed. $η (λ)$ is the spectral responsivity of the photodetector, which can be taken into account and measured in experiments. The signal $S_{i}$ , is the measured signal by the detectors underneath the $i$ th filter, among the total M filters. Since the detector pixel size is usually smaller than the filter size, the signals from all the photodetector pixels are summed for each filter and normalized by the pixel sensor signal without filter. The total M filters and all associated detectors form one spectrometer pixel to collect all necessary signal $S_{i}$ . Most importantly, before the reconstruction of the unknown spectrum, the spectral response functions of the nanophotonic filters need to be calibrated first. This is usually done by using a wavelength-tunable light source incident from the normal direction with a spectral resolution of 1 nm or less. Thus $T_{i} (λ)$ is the different transmission spectra or response function produced by the $i$ th filters, where λ is the free-space wavelength with the sampling number of $N$ .

To apply the CS algorithm, we need to digitize all continuous spectral functions and transform Eq. (1) to a set of linear equations.

S_{i} \approx \sum_{λ = λ_{1}}^{λ_{n}} I_{λ} T_{i, λ} η_{λ} = I_{1 \times n} T_{m \times n}

(2)

The transmission matrix $T_{m \times n}$ ( $M$ different nanophotonic filters at $N$ different wavelengths) is a collection of different response functions of optical filters including the factor of η(λ). The $i$ th row of the filter matrix $T_{m \times n}$ is the transmission spectrum of the $i$ th filter. The filter matrix needs to be calibrated first, and is then used in the spectrum reconstruction process in formula $T I = S$ , where $S$ is the measurement and $I$ is the target spectrum signal. The measured signal $S_{i}$ is collected by the photodetector array. The unknown $I_{1 \times n}$ can be solved numerically. It is always desirable to digitalize the unknown signal with a large $N$ for fine resolution. Therefore, CS focuses on the case N > M, and the linear equations are underdetermined. The unknown $I (λ)$ can be recovered by multiple computation algorithms (discussed in the next section).

As shown in Figure 1, if spatio-spectral information such as the “T” letter is to be probed, multiple spectrometer pixels such as 3×3 spectrometer array can be integrated on the photodetector surface so the signal captured in a snapshot can reconstruct the spectral information at multiple locations that forms a spatial image. Therefore, the snapshot hyperspectral imaging applications are realized if multiple spectrometers are placed on the detector surface to record spatio-spectral signal $S_{i} (λ, x, y)$ . Hyperspectral imaging is more challenging due to the large data cube sampled and processed with both spectral and spatial variation. The size of each super pixel defines the spatial resolution of the hyperspectral imaging. A high spectral resolution requires more filters in a single super pixel, but increasing filter numbers means a large super pixel size and low spatial imaging resolution.

Let’s look at a typical example of such a spectrometer system [22]. In the PC slab work shown in Figure 3b and c, the signal light of interest is in the spectral range from 550 to 750 nm. The spectrum is digitalized with 1 nm resolution, represented by $N = 201$ unknowns to be determined. Each photonic crystal slab forms a single-filter pixel size of $32 \times 32$ μm. $M = 36$ different PC slab filters defining a super pixel size of $210 \times 210$ μm. A similar work completed by the same group [23] has also demonstrated that a super pixel can be defined by just $3 \times 3$ filters and successfully recovers 301 spectral bands from 400 to 700 nm. The demonstrated PC slab hyperspectral imaging uses $10 \times 10$ super pixels, each composed of 36 filters with ∼200 μm size on a side, which successfully identifies the spectral and spatial information of simple digits at different wavelengths. Although this spatial resolution is very coarse (∼200 μm), they could be improved theoretically to be better than 10 μm if the filter size is reduced to a few micrometers and only 9 filters are used. A more recent work reported that the metasurface-based spectrometer can reconstruct spectrum with 25 filter units with a size of $80 μm \times 80 μm$ , covering a 300 nm broad visible spectrum with an ultra-high center-wavelength accuracy of 0.004 nm and spectral resolution of 0.8 nm [27]. Most importantly, in this study, a one-shot ultraspectral imaging device fitting 6336 pixels of micro-spectrometers on a chip no larger than 0.5 cm² has been demonstrated.

2.3. Computational algorithms as spectral decoder

The nanophotonic filters encode the input spectrum $I (λ)$ using distinct filter functions while reconstruction algorithms are then tasked to recover the incident spectrum from the raw data sampled by detectors. As discussed in the previous section, the most common approach to algorithmic reconstruction is to use a priori information on the encoding operation (or the response function) $T_{i} (λ)$ and spectral sensitivity of the photodetectors $η (λ)$ to define a transformation between the target spectrum $I (λ)$ and raw measurements $S_{i}$ , that is, $S_{i} = T_{i} (λ) η (λ) I (λ)$ for each $i$ th encoding operation. The reconstructed spectrum should minimize the sum of the squares of the differences between measured light intensities and numerically computed intensities, which are the products of the reconstructed incident spectrum and the filter response function, which are measured for each filter and integrated with respect to wavelength. By expressing this transformation operation over all the encoding elements, a least squares linear regression problem can be defined, and a solution for $I (λ)$ can be obtained by minimizing ∥ $S - T η I$ ∥².

However, the inversion problems presented above are typically ill-posed with associated measurement error and noise. Standard least squares solutions are limited and insufficient to solve the underdetermined problem. Therefore, regularization techniques and constrained optimization methods are necessary to overcome the limitations and solve the inverse problem properly. Some more popular reconstruction algorithms are regularization-based algorithms, such as non-negative least squares [30], Tikhonov regularization (TR) [32], and sparse recovery based algorithm [15, 33–35]. There are several widely used regularization terms in CS spectrometers. One regularization term is L₁ norm $∥ I ∥_{1}$ of the signal $I$ . Minimizing L₁ norm forces the signal $I$ to be sparse and is well-suited to signals that can be efficiently represented in a relatively small number of measurements in spectrometers. It is also a convex optimization problem that can be reduced to a linear program known as basis pursuit [16, 20, 74]. The other is the second-order derivative of the spectrum signal $∥ D I ∥_{2}$ , where $D$ is the second-order derivative of the spectrum signal $I$ . Minimizing $∥ D I ∥_{2}$ forces the spectrum signal $I$ to be smooth. The method of TR based on L₂ norm minimization was also introduced to solve the spectrum reconstruction problem by careful regularization parameter selection. Different regularization parameters, such as the L-curve criterion and generalized cross validation criterion, have been examined [26]. This method can be used across different spectrometric devices of different filtering characteristics without any manual calibration of parameters. It was also reported that TR is able to account for experimental noise by suppressing divergences in the singular values below a critical noise parameter. CS based on L₁ norm minimization produces relatively low computational reconstruction noise and also allows the recovery of a more complex spectrum than TR reconstruction due to its ability to suppress background computational noise.

The reconstruction of spectral snapshot compressive imaging can be more complex, as it involves large spatio-spectral data cubes. Generally used regularizers or priors for spectral snapshot compressive imaging include the sparsity [76], total variation [77], patch-based methods such as dictionary learning [78, 79] and Gaussian mixture models [80]. By utilizing the nonlocal similarity in the spectral data cube, group sparsity [81], and low-rank models [82] have also been developed to achieve state-of-the-art results in snapshot compressive imaging. Recently, a new encoding and decoding scheme has taken advantage of both spectral filters and lensless imaging to achieve spatio-spectral high-resolution snapshot hyperspectral imaging [28]. This system consists of a tiled spectral filter array placed directly onto the sensor and a diffuser placed a small distance away from the sensor, which is termed the DiffuserCam architecture [83] shown in Figure 4a. The system consists of a diffuser and spectral filter array bonded to an image sensor. A one-time calibration procedure measures the point spread function (PSF) and filter function. The diffuser spatially multiplexes the incoming light, and each spatial point maps to corresponding pixels on the camera. The spectral filter array then spectrally encodes the incoming light such that the exposure on each pixel is the sum of point-wise multiplications with the discrete filter function. The multiplexing effect of the diffuser allows recovery of scene information from a subset of sensor pixels. Figure 4b shows the image formation model for a scene with two point sources of different colors, each with narrow-band irradiance centered at y (yellow) and r (red). The final measurement is the sum of the contributions from each individual spectral filter band in the array. Due to the spatial multiplexing of the lensless architecture, all scenes project information to multiple spectral filters; that is why a high-resolution hyperspectral cube can be recovered from a single image after solving an inverse problem. To recover the hyperspectral data cube from the 2D measurement, the authors also use L₁ minimization within the framework of CS to solve an underdetermined inverse problem with a prior to sparsity [84, 85]. The result is a 3D hyperspectral cube with 64 channels of spectral information for each of 448,320 spatial points, generated from a 2D sensor measurement that is 448,320 pixels. In the experiment, the authors built a prototype and demonstrated reconstructions of complex spatio-spectral scenes, achieving up to 0.19 super pixel spatial resolution across 64 spectral bands, which performs better compared to previous studies based on only spectral CS.

Figure 4: — The working mechanism of combining a diffuser and spectral filter array to improve the spatial resolution in spectral imaging. (a) Overview of the Spectral DiffuserCam imaging pipeline, which reconstructs a hyperspectral data cube from a single-shot 2D measurement. (b) Image formation model for a scene with two point sources of different colors, each with narrow-band irradiance centered at y (yellow) and r (red). The final measurement is the sum of the contributions from each individual spectral filter band in the array. (a) and (b) are reprinted with permission from Ref. [28], copyright 2019 Optical Society of America.

3. Spectrum reconstruction enabled by deep learning

3.1. Introduction of deep neural networks

The successful adoption of computational algorithms to reconstruct spectrum requires a priori calibration of filter response functions and careful selection of algorithm operations. Thus, the reconstruction process is generally iterative and time consuming. DL algorithms are part of artificial intelligence (AI), which can perform supervised learning tasks through data-driven approaches. With the invention of new training and regularization techniques, such as activation and loss functions, dropout and batch normalization, DL can now exploit large and complex datasets with unprecedented capability [86–88]. In general, DNN consists of a large series of interconnected processing nodes or neurons that connect the inputs to the outputs through a series of intermediate layers. The large number of internal parameters that can be optimized during the training offers high plasticity to learn complex and nonlinear relationships within the training data. Another distinct advantage of DNN is that a wide variety of architectures can be employed to model the specific data more efficiently and accurately [89–92].

The rise of DL tools has reformed multiple interdisciplinary research topics and enabled manifold applications which formerly seemed strictly impossible. A prominent example is a data-driven approach to solve various inverse problems, whereas conventional methods are computationally expensive and slow. The photonics community benefited from such rapid advances in DL techniques, especially in nanophotonic inverse designs [93–95]. Unlike conventional optimization that explores the design space iteratively and consumes lots of computing resources and time, DNN can produce fast and accurate forward predictions that can replace numerical simulations and realize direct inverse design of photonic devices. For example, by leveraging inverse design capability of DL techniques, sensor hardware can be fundamentally redesigned to obtain the optimal sensing data with respect to a user-defined cost function or design constraint. Thus a new generation of computational sensing systems with improved sensing capabilities are also emerging [96]. Well-trained DNN models can directly map the complex structure-response relationships in various scenarios so that the cost of one-time training can allow orders-of-magnitude efficiency improvement for problem solving. DL technique has previously proved promising in decoding complex data, as it enables rapid and accurate analysis of optical spectra and images [97]. Moreover, it has also become evident that DL architectures can be effective for computational image formation from as well as interpretation. The DL-based imaging approach has proven to be especially useful when the image measurement is noisy and the measurement operator is ill-posed or uncertain [98].

There is growing interests in applying DNN to computational spectrometers. Since DNNs can operate based on data-driven and “black box” learning processes, they do not require priori knowledge of the filter response functions to reconstruct the spectrum. Instead, they can learn the response functions from large spectra-signal datasets and directly map the relationship between spectrum and measured signal. Although the early work shown in Figure 5 was not targeted for spectrum reconstruction, it demonstrated that DNN can effectively learn spectrum data features and map the complex resonant spectrum to corresponding bit sequence readouts. In this work, a 1D convolutional neural network followed by a fully connected network is used. The spectra are fed into the network input layer, which consists of one or two parallel channels depending on whether one or both polarization cases are used. At the output layer, each neuron with the highest activation indicates the encoded bit sequence. Therefore, DL algorithms can read far-field spectrum encoded information accurately. Intuitively, by feeding sufficient training data to DNN with proper training, we can also directly map the relationship between incident spectrum and filtered snapshot signals without knowing the exact filter response functions. After a one-time training, the unknown spectra can be reconstructed instantly by inputting the measured photodetector signal. The performance of the DNN-based computational spectrometer is often evaluated by the mean square error (MSE) between the predicted spectra and the ground-truth. Since the spectral intensity value is in the range of 0–1, an MSE value on the order of 10⁻⁴ to 10⁻⁵ indicates a good reconstruction performance [35, 99–105].

Figure 5: — The application of DNN to decode nanophotonic spectral encoded optical information. (a) Geometry of a silicon nanostructure encoding 3 × 3 = 9 bits. Each silicon block occupies an area of 105 × 105 nm². The L-shaped sidewall is 45 nm wide, and the height is 90 nm. An SEM image of a fabricated structure is shown in the inset. The example shown represents the decimal number ‘321.’ (b) Selected examples illustrating training data generation by a numerical expansion of the experimental spectra. SEM images show areas of 550 × 550 nm². A large set of spectra is generated via random superposition of the experimental scattering spectra of these four copies to allow training and testing the performance of the binary information readout ANN. (c) Sketch of the 1D convolutional neural network used for the classification task, measured spectra and corresponding digital information are fed into the network. The error is back-propagated using a variant of the stochastic gradient descent (SGD) algorithm. (d) Scheme of the fully connected ANN used for the RGB classification task. (a–d) are reprinted with permission from Ref. [55], copyright 2019 Nature Publishing Group.

3.2. DNN-assisted algorithm for spectrum reconstruction

DNN opens up new opportunities for designing accurate and robust algorithms for spectrum reconstruction and hyperspectral imaging. A natural and intuitive idea to utilize DL for spectrum reconstruction is to train a multilayer perceptron just like those in Figure 5, which takes pairs of raw measurements and the true spectrum as its input and output. Unlike the regularized reconstruction methods in which an explicit mathematical formulation of the underlying model is needed, DNN uses a black box to map the relationship between the raw measurements and the true spectrum. The application and training of CNN can predict spectra from the feature extracted by applying a direct inversion operator for CS computational spectrometers [104]. A residual CNN is proposed to deliver better reconstruction performance than CNN and sparse recovery [105]. These research findings demonstrate the potential of DL to facilitate spectrum reconstruction in ultracompact spectral imaging and sensing systems.

Instead of using the direct inversion operator, Bao et al. [36] employed an existing spectrum reconstruction solver to extract spectral features from the raw measurements of CQD filters and then use the extracted features to train the DNN to learn a map from the features to the ground-truth spectrum (which is referred as solver informed NN). The architecture is compared with the plain NN based on raw measurements, as shown in Figure 6. The plain NN is a typical architecture composed of an input layer, several hidden layers and an output layer together with a nonlinear activation function. An appropriate loss function such as MSE is chosen to train a NN model. Such standard DNN training method is straightforward, and the specific training instructions can be found in relevant works [36, 93–95]. The plain NN outperforms the regularized non-negative least squares (Reg NNLS) and the sparse recovery-based solver with roughly 80% fewer average reconstruction errors and 50% fewer errors in the standard deviation.

Figure 6: — The proposed solver-informed NN and denoising autoencoder architecture used for spectrum reconstruction. (a) The architecture of the plain NN (top): taking pairs of raw measurements and the true spectrum as input and output, the flowchart of the solver-informed NN (bottom) includes applying a specific spectrum reconstruction solver to extract spectral features from the raw measurements and using the spectral features as an input of the NN to learn a map from the features to the true spectrum. (b) The denoising autoencoder architecture with one pair of input and output. The input and output layers match the corrupted and ideal measurements of the CQD spectrometer. (a) is reprinted with permission from Ref. [36], copyright 2021 Optical Society of America; (b) is reprinted with permission from Ref. [37], copyright 2021 IEEE.

The solver-informed NN applies an existing spectrum reconstruction solver, such as the Reg NNLS solver, to the raw measurements to extract relevant features by training an NN to map the extracted features to the true spectrum. When compared with the plain NN, the average reconstruction error can be further reduced by leveraging the merits of the existing reconstruction solver and the neural network. The combined approach of solver-informed NN can significantly improve the reconstruction performance to achieve higher reconstruction accuracy and stability on synthetic dataset and real datasets collected by the CQD spectrometer. Qualitatively, the average reconstruction error of both Reg NNLS and sparse-recovery-based solver-informed NNs is further reduced by 40% as compared with the plain NN.

The same group also proposed to utilize a denoising autoencoder (originally developed for imaging processing) as one pre-processing technique to the corrupted measurements, and then reconstruct the spectra with the denoised data via a sparse-recovery algorithm [37]. The experiments demonstrate that compared with its non-denoised counterpart, the reconstruction quality can be greatly enhanced by the proposed approach when the raw measurement is noisy under different levels. The results demonstrate that the proposed combination of DNN and reconstruction algorithms holds great promise to deliver a more accurate and stable reconstruction performance for filter-based miniature spectrometers under noisy measurements.

3.3. All data-driven spectrum reconstruction

The working principle of a pure data-driven computational spectrometer based on plasmonic encoders is shown in Figure 7 [38]. A trained DNN is used to reconstruct the unknown input spectra from the lens-free diffraction images in a feed-forward manner without the need for a priori information on the encoding operation or the input illumination. The training data set consisted of 50,352 spectra, whereas the validation data set consisted of 8824 spectra. When blindly tested on new input spectra of varying complexity captured after the training phase, the DL based spectrometer correctly identified 96.86% of the spectral peaks with a peak localization error of 0.19 nm, a peak height error of 7.60%, and a peak bandwidth error of 0.18 nm. When compared to a linear regression model with L₂-norm regularization trained on the same data set, the trained neural network achieved a reduction in average MSE from $3.85 \times 10^{- 4}$ to $7.77 \times 10^{- 5}$ . These results prove that pure data-driven DNN can be used for highly accurate spectrum reconstruction. However, the authors also discovered that new spectra captured ∼5.8 days after the last training/validation spectrum degraded in performance. To tackle this issue, the authors implemented a transfer learning approach, where the weights of the previously trained neural network were adjusted through further training on a small fraction of the spectra captured at the start of the new measurement period. All performance metrics were significantly improved after the transfer learning step. In addition to these considerable improvements in spectral inference metrics, background spectral noise and erroneous peaks were also suppressed well after the transfer learning step. Therefore, transfer learning can be an effective software-based calibration tool for future data-driven computational on-chip spectrometer. By leveraging such batch computation, the network generates spectral reconstructions in ∼28 μs per spectrum, which is orders-of-magnitude faster than other computational spectroscopy methods. These performance metrics demonstrate significant improvements compared to earlier generations of computational spectrometers despite visible fabrication defects in the plasmonic encoder chip, illustrating the robustness of the DNN-based spectral reconstruction method.

3.4. DNN co-design of encoder and decoder

So far, nanophotonic encoding filters in computational spectrometers discussed here are built on a conventional filter design and selection strategy, where sufficient filters are produced to ensure randomness. A filter set is selected via permutation and combination to ensure a relatively low correlation coefficient and allow efficient encoding operation. The encoding potential of optimally designed filters have not been explored. It would be ideal if fewer spectral filters are used to improve spatial resolution for hyperspectral imaging while ensuring the diversity and low correlation of filters for good spectral resolution. The profound inverse-design capability of DL tools should effortlessly empower better intelligent filter design. A novel DNN-based co-design of nanophotonic filters and decoders in computational spectrometers was proposed [106] and is shown in Figure 8. Through the innovative algorithm, metasurface and thin-film types of filters form an encoding network that contains a single fully connected layer. By attaching a decoder behide it, the whole network is named the ‘spectral encoder and decoder’ (SED) it captures and reconstructs the input spectrum. Training the SED grants access to the optimized spectral response functions. By adding regularizations to limit the desired filter parameter range, the design can also be tolerant to fabrication errors. The complete algorithm is referred to as a parameter constraining the spectral encoder and decoder (PCSED) neural network.

Figure 8: — Schematic of parameter-constrained spectral encoder and decoder design framework. The random optical filters act as an encoding neural network whose connection weights and filter responses are constrained by the filters’ structure parameters through a pre-trained forward-modeling network. During each training epoch, a batch of spectrums is fed into the encoder and the decoder gives the corresponding output, such as the reconstructed spectrums. The loss function, such as the mean square error for the reconstructed spectrum, is evaluated, and the errors are propagated back to the structure parameters. Under this framework, the encoding filters and the decoder neural network are jointly designed. Figure is reprinted with permission from Ref. [106], copyright 2021 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim.

This co-design framework allows the structure parameters of filters to be directly trained and leverages back-propagation to obtain the learned structure parameters. Thus it is more convergent than the sequential-inverse design methods for the same task. To compare it with a sequential-design paradigm used for the same task, an SED is built with an L₂-norm-based smoothness regularization to the encoder [99]. After training the SED, the trained target spectral responses are fed into an inverse DNN to derive the structures of metasurface filters. This comparison shows the co-designed DNN framework can provide more effective spectral responsivity for each random spectral filter than decomposed sequential-design methods. Spectral cameras that use metasurface- and interference-thin-film-based filters present higher reconstruction accuracy with up to 30 times enhancement and protect against fabrications errors more effectively.

Based on this strategy, a simplified schematic of the proposed spectral camera was built using 16 random spectral filters to use in reconstructive hyperspectral imaging tasks. The operation scheme of this novel camera, which has both active and passive modes, is illustrated in Figure 9. The passive mode is the more common mode of operation: the spectrum reflected by the sample is encoded by the filters and the device detects the spectral radiance of the reflected light. In the passive mode, the authors used the camera to acquire a hyperspectral image from a standard color calibration card. The MSE between the measured and ground truth spectra ranged from 5 × 10⁻⁶ to 0.0032, with an average value of 0.0008 for different colors. For natural objects such as plants, the hyperspectral camera could also reconstruct the spatio-spectral information accurately, with an average reconstruction MSE of 0.0016. In the active mode, the illumination spectrum is filtered first and then encodes the sample; thus, the device detects the spectral reflectance of the sample. Compared with passive mode, active mode excels at measuring spectral reflectance regardless of the ambient illumination. The active hyperspectral camera is set up by placing a random spectral filter array in front of white light-emitting-diodes to modulate the illumination light. By feeding all captured images into the pretrained DNN, its hyperspectral image – a 640 × 480 × 301 3D matrix – is captured. The results cover the entire visible range (400–700 nm) at a step size of 1 nm. The MSE between the measured and ground-truth spectra was quantified at an average MSE of 0.0010. As a comparison, the spectra reconstructed by the CS algorithm have an average MSE of 0.0061, which is worse than the DNN results.

Figure 9: — Principle and performance of broadband encoding stochastic camera. (a) Simplified schematic. Depending on where the light spectrum is encoded, the camera can work either in the active (upper) or passive (lower) modes. (b) Principle of DNN-based spectral reconstruction algorithm. The initial data captured by the monochrome camera is fed into the DNN and outputs the reconstructed 3D hyperspectral data cube. (c) and (d) Spectral profiles of laser beams with narrow bandwidth. In (c), the DNN is trained by the “precise” dataset, whereas (d) is for the results from the DNN trained by “general” datasets. (e) Spectral profile of two peaks corresponding to 598.0 and 603.2 nm. The peak-to-peak distance is highlighted in black. In (c–e), the ground truths and the DNN-reconstructed results are represented by dashed (ground truth) and solid (reconstructed) curves, respectively. The graphs are normalized to their peak intensity. (a–e) are reprinted with permission from Ref. [39], copyright 2021 Nature Publishing Group.

3.5. Deep plug-and-play priors for spectral snapshot compressive imaging

As discussed earlier, the reconstruction algorithm plays a critical role in reconstruction spectral imaging as it outputs the desired data cube. The main bottleneck of previously developed iterative optimization-based algorithms is the slow reconstruction speed. The spectral data cube is usually large-scale, requiring hours to reconstruct from a snapshot measurement. To address the low efficiency of reconstructive algorithms, CNN approaches have been used to improve the reconstruction speed [107–111]. These networks can lead to better results than their computational counterparts if sufficient training data and time are given. In general, these networks are usually trained and used for specific problems. More recently, the plug-and-play (PnP) framework [112, 113] has been proposed for inverse problems [114, 115], employing advanced deep denoisers [116–118] in the iterative optimization algorithm to speed up the reconstruction process. In such a framework, no training is required for different tasks, and the same denoising network can be used directly in different systems. This strategy inspired a new PnP method that uses DL-based denoisers as regularization priors for reconstructive spectral imaging, as shown in Figure 10 [119]. This work proposed a CNN-based deep spectral denoising network as the spatio-spectral prior, which is flexible in terms of data size and input noise levels. The proposed denoising network is trained on the CAVE data set, and PyTorh and Adam are used for implementation and optimizer, respectively. Different spectral imaging systems have different settings and parameters for CNN methods, such as λ-net. A different network needs to be trained for each system with large training data and computing resources. However, the PnP method offers flexibility here and can easily be applied to different systems. The actual application of PnP algorithm in three coded aperture snapshot spectral imaging systems, one multispectral endomicroscopy, and a ghost spectral compressive imaging system, shows finer detail and a clean background compared to other algorithms.

Figure 10: — Image formation process of a typical spectral snapshot compressive imaging system, i.e., single disperser coded aperture snapshot spectral imaging and the reconstruction process using the proposed deep PnP prior algorithm. Figure is reprinted with permission from Ref. [119], copyright 2021 Optical Society of America.

3.6. Optical diffractive network for spectral encoding

There is growing interest in optical computing to benefit from the low power, low latency, and scalability of passive optical systems. Specially designed light–matter interaction can lead to all-optical processers and perform specific computation tasks through wave propagation. The optical diffractive layers are composed of passive pixels, where the complex-values transmission or reflection coefficients of each pixel are optimized though DL algorithms. It has been demonstrated that DL-based diffractive networks can encode the spatial information of objects into the optical spectrum through learnable diffractive layers that collectively process the information contained at multiple wavelengths [120]. Such systems can perform optical classification of objects and image reconstruction using a single-pixel detector, as shown in Figure 11. A broadband diffractive network composed of layers is trained to transform the spatial information of the objects into the spectral domain through a preselected set of class-specific wavelengths measured by a single-pixel spectroscopic detector at the output plane. The resulting spectral class scores are denoted by the vector s = [s0, s1, …, s9]. After its training and design phase for a given input/test image, it learns to channel relatively more power to the spectral component assigned to the correct class, e.g., digit “8” in Figure 11a; therefore, max(s) reveals the correct data class. As demonstrated in Figure 11b, the same class score vector, s, can also be used as an input to an NN with two hidden layers to reconstruct an image of the input object, decoding the spectral encoding performed by the broadband diffractive network. Even if the input objects are incorrectly classified by the trained diffractive network, by feeding back the incorrectly reconstructed images into the same network as new inputs, their optical classification is corrected with significantly improved inference accuracy. This framework demonstrates broadband diffractive optical networks that operate with pulsed illumination at terahertz wavelengths to achieve >96% blind testing accuracy for optical classification of handwritten digits based on the spectral power at 10 distinct wavelengths, each assigned to one digit. The authors also experimentally verified the results by using a terahertz time domain spectroscopy system and 3D-printed diffractive models. These results emphasized the vital role of collaboration between a trainable optical front end and an all-electronic NN-based back end.

Figure 11: — Schematics of spectral encoding of spatial information for object classification and image reconstruction. (a) Optical layout of the single-detector machine vision concept for spectrally encoded classification of objects. As an example, a handwritten digit eight is illuminated with a broadband pulsed light, and the subsequent diffractive optical network transforms the object information into the power spectrum of the diffracted light collected by a single detector. The object class is determined by the maximum of the spectral class scores, s, defined over a set of discrete wavelengths, each representing a data class. (b) Schematic of task-specific image reconstruction using the diffractive network’s spectral class scores as input. A separately trained shallow artificial neural network recovers the images of handwritten digits from the spectral information encoded in s. Each reconstructed image is composed of >780 pixels, whereas the input vector, s, has 10 values. (a) and (b) are reprinted with permission from Ref. [120], copyright 2021 AAAS.

4. Discussion

4.1. Spectrum reconstruction quality

Previous iterative reconstruction methods rely on precise measurements of the encoding operation and the spectral response of the underlying photodetectors, which are both used as a priori information. This presents additional challenges, as the reconstruction performance relies on how precisely one can characterize the underlying hardware when system noises are present. In addition, the number of filters also plays a critical role in successful spectral reconstruction. In earlier FB-resonator-based works [20, 61], M = 23 or 100 filter response functions are used. The total number of filters sets a limit on the spectra that can be successfully reconstructed, and this limit is related to the sparsity of the measured spectrum. Laser spectrum that is sparse in wavelength is well-suited for compressive sensing, while a larger number of filters are needed to reconstruct spectra that are less sparse. The adoption of filter response functions with broadband and random resonance features have indicated that an array of 42 PC filters can reconstruct more accurately than an array of 9 PC filters [23]. The investigation on the effect of the number of metasurface units M in a single spectrometer also reveals that increasing M can decrease reconstruction error, as it reduces the randomness of the solution to the matrix equation. Based on the noise level, M can be dynamically adjusted based on the reconfigurable metasurface supercell to achieve an optimal trade-off between density and reconstruction quality. In other words, a small M and large spectral pixel density could be used at low noise levels, whereas a large M, with degraded spectral pixel density could be used at high noise levels [27].

In addition, some earlier iterative reconstruction algorithms employ the smoothness constraint in their optimization process. The selection of a singular weighting parameter on this constraint usually introduces a trade-off in performance between narrow-band and broad-band spectral reconstructions. However, this is not a challenge if one uses DL-based spectroscopy approach. Therefore, the reconstructive spectrum can obtain ultrahigh spectral resolution, as seen in summarized Table 1. The resolution of spectral reconstruction networks is primarily limited by the spectral resolution of the instrument used for ground truth measurements of the training data. The best resolution a computational spectrometer can achieve is below 1 nm. The reconstructed spectral resolution is comparable for both CS computational algorithms and DL based approaches. In direct comparisons, all methods of TR, CS, and DL perform well in the oversampling regime, while in the undersampling case, DL yields a very good performance and even clearly outperforms CS for dense spectra [103]. One major weakness of utilizing such spectrometers for hyperspectral imaging is that the spatial imaging resolution is limited by the spectrometer pixel size. However, this issue can be continuously improved with better filter design with fewer numbers of filters and by scaling down the single filter size to match the single detector pixel size. A recent study of snapshot ultraspectral imaging has proved that 6336 microspectrometers containing a total of 154,800 metasurface units can be integrated with CMOS image sensors [27]. The adoption of diffuser can spatially multiplex the information and thus improve the spatial resolution as well [83].

Table 1:

The performance comparison of some recent computational spectrometers.

Filter structure	Reconstruction method	Detector type	Spectral range (μm)	Spectral resolution (nm)	Spatial resolution (μm)	Ref.
Plasmonic structure	TR	CMOS	0.350–1.05	10	–	[32]
Random photonic chip	CS	–	1.5–1.525	0.75	–	[13]
Colloidal quantum dot	CS	CCD	0.39–0.69	3.2	–	[19]
Elaton array	CS	CCD	0.4–0.75	8	–	[20]
Fabry–Perot mirror	CS	–	0.5–0.9	1	–	[62]
Nanowire mat	TR and CS	CMOS	0.61–0.67	1	2.4	[26]
Plasmonic metasurface	RLS	IR microscope	1.5–1.9	–	–	[21]
Photonic crystal slab	CS	CMOS	0.55–0.75	1	200	[22]
Photonic crystal slab	CS	CMOS	0.4–0.7	1	–	[23]
Single nanowire	Adaptive TR	Paired nanowire	0.5–0.63	15	50	[65]
Voronoi photonic crystal	Demosaicking	–	0.42–0.72	10	232.5	[68]
Colloidal quantum dot	NN	CCD	0.4–0.75	5	–	[36]
Metasurface	CS	CMOS	0.45–0.75	0.8	80	[27]
Pearl	CS	–	0.45–0.7	7.4	–	[64]
Plasmonic nanohole array	NN	CMOS	0.48–0.750	0.229	100	[38]
Colloidal quantum dot	NN	CCD	0.4–0.75	5	–	[37]
Multilayer film	NN	CCD	0.4–0.7	5.2	–	[39]
Black phosphorus	TR	Black phosphorus	2–9	420	–	[70]

Open in a new tab

4.2. Spectrum reconstruction speed

The key advantage of the data-driven approach is its speed compared to common spectral reconstruction algorithms based on least-squares minimization. Previous study observes that although solver-informed NN requires additional computational cost to run existing spectrum reconstruction solvers for each spectrum to generate training input, it is still advantageous in terms of overall speed, as the time-consuming fine selection process for regularization parameters is eliminated [36]. The major burden for all data-driven reconstruction processes is the training data collection. Although the DNN training process requires a large number of measured data sets, this is a one-time effort. After the training is complete, the trained model can blindly recover unknown spectra from raw sensing signals in microseconds – several orders of magnitude shorter than the time required to solve iterative minimization problems employed in earlier spectral reconstruction methods [38]. This efficiency leap is consequential because DNN operates matrix multiplexing and parallel computing while the iterative algorithm solves optimization problem sequentially for each pixel. This is especially true for computational algorithms, as the reconstruction time of a spectral image increases with increasing numbers of pixels. However, the reconstruction speed of the DL is almost pixel-independent, and is more advantageous for images with higher resolution.

For applications such as hyperspectral imaging, which may demand a spectral recovery across a large sequence of images, each with a large number of individual pixels, DL could be the only viable choice. In general, the speed of DNN is almost irrelative to the spatial resolution, while the consumption time of the CS algorithm and the number of pixels follow a roughly linear relationship. Running iterative CS algorithms is inefficient on GPU because frequent data exchange is needed for iteration. The comparison study between DL and CS algorithm in Ref [39] shows that to reconstruct a hyperspectral image (480 × 640 pixels, 1 nm spectral step size, 400–700 nm spectral scale) on an Intel Core i9-10900x CPU, Nvidia GeForce 2080Ti GPU platform, the DNN could generate such an image in 0.48 s. Reconstructing the same image using the basis pursuit denoising algorithm requires 3307.3 s on the same computer platform – nearly 7000 times slower than that of the proposed DNN. Although further optimization is possible, the current speed of DNN is sufficient to perform real-time data reconstruction, as the proposed recording frame rate is below 2 fps. DL-based hyperspectral reconstruction is suitable for real-time image processing. DL reconstruction speed may still be increased via performance optimization of the code, the implementation of multi-GPU platforms, and training networks that reconstruct speckles in parallel.

4.3. Spectrum reconstruction denoising

In CS algorithms, however, denoising is more experience-dependent, as the selection of the non-negative parameter influences the reconstruction accuracy under noisy conditions. In general, a variable parameter is manually set during the iteration to cancel the noise-induced bias. This operation works to some extent, but does not provide convincing results when the noise level changes. As examined, the DNN enhances noise tolerance by 8.14 times under different noise levels [39], even compared to the CS algorithm with optimal parameter settings. In a real-life scenario for spectrum reconstruction, there is shot noise, electronic camera noise or other non-specific backgrounds present. An imaging system may also experience some drift caused by environmental effects such as vibrations and thermal variations [103]. On the other hand, the use of a denoising autodecoder shows the flexibility of DNN to deal with spectrum data measured under noisy environments [37, 117, 118]. More importantly, well-trained DNN has better denoising abilities, which can be readily obtained by adding more samples with noise at different levels during training. Moreover, adding noise has a regularization effect and further improves the robustness of the model. To assess the robustness of the different approaches against typical perturbations, performance robustness of DL, CS, and TR is compared in speckle reconstruction. As CS and TR are based on analytic methods that are intrinsically inflexible to variation, DL has the advantage in adapting to perturbed data and noise through the choice of proper training data. Noise-adapted DL shows a distinct performance boost at reconstruction from non-perfect speckle images. The training set might not only contain data accounting for shifts of the CCD, but also for shifts of fiber or variations in the incident angle of light. All these effects affect the speckle patterns in a deterministic way and could be corrected with properly trained DNN, while this is impossible with an analytical method [103].

4.4. Potential applications for computational sensing

Since the computational spectrometer can capture spectral information accurately, it can potentially be used as a novel computational sensor [121–128]. Figure 12a introduces a metasurface-based multi-resonance sensor design and a new concept: a hyperspectral decoder that associates spectral information from each CMOS pixel to its spatial index [58]. By employing a pixel-based thresholding method, the detection limit can be improved by three orders of magnitude when compared to ensemble averaging (∼3 versus 1500 molecules per μm²). This encoder-decoder scheme eliminates the need for bulky and expensive instrumentation and can be used to realize field-deployable sensors. Similarly, a mid-IR nanophotonic sensor based on all-dielectric high-Q metasurface elements demonstrates its capability for enhancing, detecting, and differentiating the absorption fingerprints of various molecules [57]. The high-Q resonance design is spectrally clean without additional resonance background, which allows for highly spectral-selective enhancement of spectroscopically rich molecular fingerprint information. As the resonance positions of individual metapixels are linearly varied over a target mid-IR fingerprint range, so each resonance position is assigned to a specific pixel of the metasurface, establishing one-to-one mapping between spectral and spatial information. By comparing the imaging-based readout of this spatially encoded vibrational information before and after the coating of target analyte molecules, chemically specific molecular barcodes are proved to be suitable for chemical identification and compositional analysis, as shown in Figure 12b. Although the abovementioned cases are not meant for spectrum reconstruction, they share similar hardware set-up and potential decoding algorithms. In conventional biological or chemical spectral sensing systems, the small resonance shifts caused by analytes are characterized by conventional spectrometers. Computational spectrometer systems have the potential to be used for computational sensing as well, since the system can decode spectral changes with high resolution and thus probe tiny spectral signal shifts caused by analytes on nanophotonic filters. In this case, high Q- and narrow-band nanophotonic filters may be still preferred for sensing as opposed to spectrum reconstruction applications where broadband and random resonant structures are desired.

Figure 12: — Hyperspectral imaging-based biomolecule detection using all-dielectric metasurfaces and molecular fingerprint retrieval based on spatial absorption mapping. (a) Sketch of the hyperspectral imaging principle showing a representative dielectric metasurface sensor array illuminated with a narrow-band tunable laser source. At each wavelength illumination, images are recorded by a CMOS camera to create a hyperspectral data cube where each CMOS pixel captures high-resolution spectral information. The hyperspectral data cube is processed to extract [57] spatial resonance maps of each individual sensor. For biomolecule detection, the sensing and reference resonance maps are combined to create the resonance shift map, which conveys spectral shift information across the whole sensor. (b) Normalized metapixel reflectance spectra before and after physisorption of a protein A/G monolayer. R0 indicates the envelope of peak reflectance amplitudes and RS is reflectance envelope RS. Protein absorption fingerprint calculated from the reflectance envelopes R0 and RS compared to an independent IRRAS measurement. Broadband spectrometerless operation of the metasurface can be emulated by integrating the reflectance signal of all pixels. Spectral integration translates the absorption signature into a 2D spatial absorption map that represents the molecular barcode of the protein. (a) is reprinted with permission from Ref. [58], copyright 2019 Nature Publishing Group; (b) is reprinted with permission from Ref. [57], copyright 2018 AAAS.

5. Summary and outlook

Although conventional spectrometers can be further miniaturized by scaling down the dispersive optical components through advanced fabrication, computational spectrometer may be a novel and revolutionary technology shift. It has rapidly developed proof-of-concept applications over the past decade. It shows new possibilities of ultra-compact, low-cost spectrometers, and spectral imagers with potential to be integrated with smartphones and lab-on-a-chip systems. Computational techniques possess great advantages compared with conventional optical spectroscopy technology, but are still in their infancy, with significant space for improvement and application expansion. The implementation of various random, broadband nanophotonic filters and DL algorithms have expediated development. DL-based computational spectrometers show prominent advantages compared to CS methods in terms of operation speed, denoising capability, and lack of required precalibration. However, emerging technology like DL-based spectroscopy also has inherent drawbacks and challenges. Copious well-characterized and well-labeled training data needs to be collected, and this could be a great burden for experimental data generation. The data can be easily biased or noised due to system instability over a period of time, and this problem may require better network architecture and training methods.

Nonetheless, efforts will continue as the co-design of hardware and computational algorithm will lead to wider use of spectral information. Maturation of DL algorithms could ease the burden for denoising and speed up the reconstructive process. Moreover, with flexible fabrication and packaging of filters and detectors, the miniature computational spectrometer could be made suitable for wearable devices, as proved in related flexible electronic technologies. Furthermore, bandgap engineering of semiconductor nanowires and two-dimensional materials could offer a route toward printable, conformable nanoscale spectrometers that combine filter and detector functions in a single device. The simultaneous capture of spectral and spatial information in large data cubes for reconstructive spectral imaging could potentially revolutionize existing hyperspectral imaging applications, and most importantly reduce the cost of spectral image cameras. Beyond smartphones and lab-on-a-chip sensing and imaging applications, such high-performance computational spectrometers may also find applications in a vast range of research fields and industrial technologies, including satellites and drones, the food industry, chemical and biological imaging, etc.

Footnotes

Author contribution: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.

Research funding: The authors acknowledge support from National Natural Science Foundation of China (62022043, 61974069); National Key Research and Development Program of China (2017YFA0205302); Natural Science Foundation of Jiangsu Province (BK20191379); Leading-edge Technology Program of Jiangsu Natural Science Foundation (BK20212012); Jiangsu Provincial Key Research and Development Program (BE2018732); NUPTSF NY219008, NY221044, 1311 Talent Program.

Conflict of interest statement: The authors declare no conflicts of interest regarding this article.

Contributor Information

Li Gao, Email: iamlgao@njupt.edu.cn.

Lianhui Wang, Email: iamlhwang@njupt.edu.cn.

Zongfu Yu, Email: zyu54@wisc.edu.

References

[1].Bacon C. P., Mattley Y., DeFrece R. Miniature spectroscopic instrumentation: applications to biology and chemistry. Rev. Sci. Instrum. . 2004;75:1–16. doi: 10.1063/1.1633025. [DOI] [Google Scholar]
[2].Pavia D. L., Lampman G. M., Kriz G. S., Vyvyan J. A. Introduction to Spectroscopy . Belmont: Cengage Learning; 2014. [Google Scholar]
[3].Malinen J., Rissanen A., Saari H., et al. Conference on Next-Generation Spectroscopic Technologies VII . Baltimore, MD: SPIE; 2014. Advances in miniature spectrometer and sensor development; p. 91010C. [Google Scholar]
[4].Crocombe R. A. Portable spectroscopy. Appl. Spectrosc. . 2018;72:1701–1751. doi: 10.1177/0003702818809719. [DOI] [PubMed] [Google Scholar]
[5].McGonigle A. J. S., Wilkes T. C., Pering T. D., et al. Smartphone spectrometers. Sensors . 2018;18:233. doi: 10.3390/s18010223. [DOI] [PMC free article] [PubMed] [Google Scholar]
[6].Faraji-Dana M., Arbabi E., Arbabi A., Kamali S. M., Kwon H., Faraon A. Compact folded metasurface spectrometer. Nat. Commun. . 2018;9:4196. doi: 10.1038/s41467-018-06495-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
[7].Cheng R., Zou C.-L., Guo X., Wang S., Han X., Tang H. X. Broadband on-chip single-photon spectrometer. Nat. Commun. . 2019;10:4104. doi: 10.1038/s41467-019-12149-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
[8].Nezhadbadeh S., Neumann A., Zarkesh-Ha P., Brueck S. R. J. Chirped-grating spectrometer-on-a-chip. Opt Express . 2020;28:24501–24510. doi: 10.1364/oe.398072. [DOI] [PubMed] [Google Scholar]
[9].Li A., Fainman Y. On-chip spectrometers using stratified waveguide filters. Nat. Commun. . 2021;12:2704. doi: 10.1038/s41467-021-23001-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
[10].Yang Z., Albrow-Owen T., Cai W., Hasan T. Miniaturization of optical spectrometers. Science . 2021;371:480. doi: 10.1126/science.abe0722. [DOI] [PubMed] [Google Scholar]
[11].Noda S., Fujita M., Asano T. Spontaneous-emission control by photonic crystals and nanocavities. Nat. Photonics . 2007;1:449–458. doi: 10.1038/nphoton.2007.141. [DOI] [Google Scholar]
[12].Tanaka Y., Asano T., Noda S. Design of photonic crystal nanocavity with Q-factor of ∼109 . J. Lightwave Technol. . 2008;26:1532–1539. doi: 10.1109/jlt.2008.923648. [DOI] [Google Scholar]
[13].Redding B., Liew S. F., Sarma R., Cao H. Compact spectrometer based on a disordered photonic chip. Nat. Photonics . 2013;7:746–751. doi: 10.1038/nphoton.2013.190. [DOI] [Google Scholar]
[14].August Y., Stern A. Compressive sensing spectrometry based on liquid crystal devices. Opt Lett. . 2013;38:4996–4999. doi: 10.1364/ol.38.004996. [DOI] [PubMed] [Google Scholar]
[15].Oliver J., Lee W.-B., Lee H.-N. Filters with random transmittance for improving resolution in filter-array-based spectrometers. Opt. Express . 2013;21:3969–3989. doi: 10.1364/oe.21.003969. [DOI] [PubMed] [Google Scholar]
[16].Wang Z., Yu Z. Spectral analysis based on compressive sensing in nanophotonic structures. Opt. Express . 2014;22:25608–25614. doi: 10.1364/oe.22.025608. [DOI] [PubMed] [Google Scholar]
[17].Wang P., Menon R. Computational spectrometer based on a broadband diffractive optic. Opt. Express . 2014;22:14575–14587. doi: 10.1364/oe.22.014575. [DOI] [PubMed] [Google Scholar]
[18].Yang T., Xu C., Ho H.-p., et al. Miniature spectrometer based on diffraction in a dispersive hole array. Opt Lett. . 2015;40:3217–3220. doi: 10.1364/ol.40.003217. [DOI] [PubMed] [Google Scholar]
[19].Bao J., Bawendi M. G. A colloidal quantum dot spectrometer. Nature . 2015;523:67–70. doi: 10.1038/nature14576. [DOI] [PubMed] [Google Scholar]
[20].Huang E., Ma Q., Liu Z. Etalon array reconstructive spectrometry. Sci. Rep. . 2017;7:40693. doi: 10.1038/srep40693. [DOI] [PMC free article] [PubMed] [Google Scholar]
[21].Craig B., Shrestha V. R., Meng J., Cadusch J. J., Crozier K. B. Experimental demonstration of infrared spectral reconstruction using plasmonic metasurfaces. Opt. Lett. . 2018;43:4481–4484. doi: 10.1364/ol.43.004481. [DOI] [PubMed] [Google Scholar]
[22].Wang Z., Yi S., Chen A., et al. Single-shot on-chip spectral sensors based on photonic crystal slabs. Nat. Commun. . 2019;10:1020. doi: 10.1038/s41467-019-08994-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
[23].Zhu Y., Lei X., Wang K. X., Yu Z. Compact CMOS spectral sensor for the visible spectrum. Photon. Res. . 2019;7:961–966. doi: 10.1364/prj.7.000961. [DOI] [Google Scholar]
[24].Hartmann W., Varytis P., Gehring H., et al. Waveguide-integrated broadband spectrometer based on tailored disorder. Adv. Opt. Mater. . 2020;8:1901602. doi: 10.1002/adom.201901602. [DOI] [Google Scholar]
[25].Zhu X., Bian L., Fu H., et al. Broadband perovskite quantum dot spectrometer beyond human visual resolution. Light Sci. Appl. . 2020;9:73. doi: 10.1038/s41377-020-0301-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
[26].French R., Gigan S., Muskens O. L. Speckle-based hyperspectral imaging combining multiple scattering and compressive sensing in nanowire mats. Opt. Lett. . 2017;42:1820–1823. doi: 10.1364/ol.42.001820. [DOI] [PubMed] [Google Scholar]
[27].Cai X., Xiong J., Cui K., et al. One-shot Ultraspectral Imaging with Reconfigurable Metasurfaces . 2020. preprint arXiv:2005.02689. [Google Scholar]
[28].Monakhova K., Yanny K., Aggarwal N., Waller L. Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array. Optica . 2020;7:1298–1307. doi: 10.1364/optica.397214. [DOI] [Google Scholar]
[29].McClung A., Samudrala S., Torfeh M., Mansouree M., Arbabi A. Snapshot spectral imaging with parallel metasystems. Sci. Adv. . 2020;6:eabc7646. doi: 10.1126/sciadv.abc7646. [DOI] [PMC free article] [PubMed] [Google Scholar]
[30].Chang C.-C., Lee H.-N. On the estimation of target spectrum for filter-array based spectrometers. Opt. Express . 2008;16:1056–1061. doi: 10.1364/oe.16.001056. [DOI] [PubMed] [Google Scholar]
[31].Carmo J. P., Rocha R. P., Bartek M., de Graaf G., Wolffenbuttel R. F., Correia J. H. A review of visible-range Fabry-Perot microspectrometers in silicon for the industry. Opt. Laser. Technol. . 2012;44:2312–2320. doi: 10.1016/j.optlastec.2012.03.036. [DOI] [Google Scholar]
[32].Kurokawa U., Choi B. I., Chang C.-C. Filter-based miniature spectrometers: spectrum reconstruction using adaptive regularization. IEEE Sensor. J. . 2011;11:1556–1563. doi: 10.1109/jsen.2010.2103054. [DOI] [Google Scholar]
[33].Chang C.-C., Lin N.-T., Kurokawa U., Choi B. I. Spectrum reconstruction for filter-array spectrum sensor from sparse template selection. Opt. Eng. . 2011;50:114402. doi: 10.1117/1.3645086. [DOI] [Google Scholar]
[34].Oliver J., Lee W., Park S., Lee H.-N. Improving resolution of miniature spectrometers by exploiting sparse nature of signals. Opt. Express . 2012;20:2613–2625. doi: 10.1364/oe.20.002613. [DOI] [PubMed] [Google Scholar]
[35].Zhang S., Dong Y., Fu H., Huang S.-L., Zhang L. A spectral reconstruction algorithm of miniature spectrometer based on sparse optimization and dictionary learning. Sensors . 2018;18:644. doi: 10.3390/s18020644. [DOI] [PMC free article] [PubMed] [Google Scholar]
[36].Zhang J., Zhu X., Bao J. Solver-informed neural networks for spectrum reconstruction of colloidal quantum dot spectrometers. Opt. Express . 2020;28:33656–33672. doi: 10.1364/oe.402149. [DOI] [PubMed] [Google Scholar]
[37].Zhang J., Zhu X., Bao J. Denoising autoencoder aided spectrum reconstruction for colloidal quantum dot spectrometers. IEEE Sensor. J. . 2021;21:6450–6458. doi: 10.1109/jsen.2020.3039973. [DOI] [Google Scholar]
[38].Brown C., Goncharov A., Ballard Z. S., et al. Neural network-based on-chip spectroscopy using a scalable plasmonic encoder. ACS Nano . 2021;15:6305–6315. doi: 10.1021/acsnano.1c00079. [DOI] [PubMed] [Google Scholar]
[39].Zhang W., Song H., He X., et al. Deeply learned broadband encoding stochastic hyperspectral imaging. Light Sci. Appl. . 2021;10:108. doi: 10.1038/s41377-021-00545-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
[40].So S., Badloe T., Noh J., Rho J., Bravo-Abad J. Deep learning enabled inverse design in nanophotonics. Nanophotonics . 2020;9:1041–1057. doi: 10.1515/nanoph-2019-0474. [DOI] [Google Scholar]
[41].Jiang J., Chen M., Fan J. A. Deep neural networks for the evaluation and design of photonic devices. Nat. Rev. Mater. . 2021;6:679–700. doi: 10.1038/s41578-020-00260-1. [DOI] [Google Scholar]
[42].Wiecha P. R., Arbouet A., Girard C., Muskens O. L. Deep learning in nano-photonics: inverse design and beyond. Photon. Res. . 2021;9:B182–B200. doi: 10.1364/prj.415960. [DOI] [Google Scholar]
[43].Xu Y., Zhang X., Fu Y., Liu Y. Interfacing photonics with artificial intelligence: an innovative design strategy for photonic structures and devices based on artificial neural networks. Photon. Res. . 2021;9:B135–B52. doi: 10.1364/prj.417693. [DOI] [Google Scholar]
[44].Ma W., Liu Z., Kudyshev Z. A., Boltasseva A., Cai W., Liu Y. Deep learning for the design of photonic structures. Nat. Photonics . 2021;15:77–90. doi: 10.1038/s41566-020-0685-y. [DOI] [Google Scholar]
[45].Koenderink A. F., Alu A., Polman A. Nanophotonics: shrinking light-based technology. Science . 2015;348:516–521. doi: 10.1126/science.1261243. [DOI] [PubMed] [Google Scholar]
[46].Joannopoulos J. D., Villeneuve P. R., Fan S. H. Photonic crystals: putting a new twist on light. Nature . 1997;386:143–149. doi: 10.1038/386143a0. [DOI] [Google Scholar]
[47].Pervez N. K., Cheng W., Jia Z., Cox M. P., Edrees H. M., Kymissis I. Photonic crystal spectrometer. Opt. Express . 2010;18:8277–8285. doi: 10.1364/oe.18.008277. [DOI] [PubMed] [Google Scholar]
[48].Gan X., Pervez N., Kymissis I., Hatami F., Englund D. A high-resolution spectrometer based on a compact planar two dimensional photonic crystal cavity array. Appl. Phys. Lett. . 2012;100:231104. doi: 10.1063/1.4724177. [DOI] [Google Scholar]
[49].Yang K., Yao X., Liu B., Ren B. Metallic plasmonic array structures: principles, fabrications, properties, and applications. Adv. Mater. . 2021:2007988. doi: 10.1002/adma.202007988. [DOI] [PubMed] [Google Scholar]
[50].Yu N., Genevet P., Kats M. A., et al. Light propagation with phase discontinuities: generalized laws of reflection and refraction. Science . 2011;334:333–337. doi: 10.1126/science.1210713. [DOI] [PubMed] [Google Scholar]
[51].Chen W. T., Zhu A. Y., Capasso F. Flat optics with dispersion-engineered metasurfaces. Nat. Rev. Mater. . 2020;5:604–620. doi: 10.1038/s41578-020-0203-3. [DOI] [Google Scholar]
[52].Rezaei S. D., Dong Z., Chan J. Y. E., et al. Nanophotonic structural colors. ACS Photonics . 2021;8:18–33. doi: 10.1021/acsphotonics.0c00947. [DOI] [Google Scholar]
[53].Chen W. T., Zhu A. Y., Sanjeev V., et al. A broadband achromatic metalens for focusing and imaging in the visible. Nat. Nanotechnol. . 2018;13:220. doi: 10.1038/s41565-017-0034-6. [DOI] [PubMed] [Google Scholar]
[54].Chen W. T., Zhu A. Y., Sisler J., Bharwani Z., Capasso F. A broadband achromatic polarization-insensitive metalens consisting of anisotropic nanostructures. Nat. Commun. . 2019;10 doi: 10.1038/s41467-019-08305-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
[55].Wiecha P. R., Lecestre A., Mallet N., Larrieu G. Pushing the limits of optical information storage using deep learning. Nat. Nanotechnol. . 2019;14:237–244. doi: 10.1038/s41565-018-0346-1. [DOI] [PubMed] [Google Scholar]
[56].Chen Q., Hu X., Wen L., Yu Y., Cumming D. R. S. Nanophotonic image sensors. Small . 2016;12:4922–4935. doi: 10.1002/smll.201600528. [DOI] [PMC free article] [PubMed] [Google Scholar]
[57].Tittl A., Leitis A., Liu M., et al. Imaging-based molecular barcoding with pixelated dielectric metasurfaces. Science . 2018;360:1105–1109. doi: 10.1126/science.aas9768. [DOI] [PubMed] [Google Scholar]
[58].Yesilkoy F., Arvelo E. R., Jahani Y., et al. Ultrasensitive hyperspectral imaging and biodetection enabled by dielectric metasurfaces. Nat. Photonics . 2019;13:390–396. doi: 10.1038/s41566-019-0394-6. [DOI] [Google Scholar]
[59].Emadi A., Wu H., de Graaf G., Wolffenbuttel R. Design and implementation of a sub-nm resolution microspectrometer based on a Linear-Variable Optical Filter. Opt. Express . 2012;20:489–507. doi: 10.1364/oe.20.000489. [DOI] [PubMed] [Google Scholar]
[60].Wang S.-W., Xia C., Chen X., et al. Concept of a high-resolution miniature spectrometer using an integrated filter array. Opt Lett. . 2007;32:632–634. doi: 10.1364/ol.32.000632. [DOI] [PubMed] [Google Scholar]
[61].Oiknine Y., August I., Blumberg D. G., Stern A. Compressive sensing resonator spectroscopy. Opt Lett. . 2017;42:25–28. doi: 10.1364/ol.42.000025. [DOI] [PubMed] [Google Scholar]
[62].Degraux K., Cambareri V., Geelen B., Jacques L., Lafruit G. Multispectral compressive imaging strategies using fabry-perot filtered sensors. IEEE Trans. Comput. Imag. . 2018;4:661–673. doi: 10.1109/tci.2018.2864659. [DOI] [Google Scholar]
[63].Deng W., You C., Zhang Y. Spectral discrimination sensors based on nanomaterials and nanostructures: a review. IEEE Sensor. J. . 2021;21:4044–4060. doi: 10.1109/jsen.2020.3032720. [DOI] [Google Scholar]
[64].Kwak Y., Park S. M., Ku Z., Urbas A., Kim Y. L. A pearl spectrometer. Nano Lett. . 2021;21:921–930. doi: 10.1021/acs.nanolett.0c03618. [DOI] [PubMed] [Google Scholar]
[65].Yang Z., Albrow-Owen T., Cui H., et al. Single-nanowire spectrometers. Science . 2019;365:1017–1020. doi: 10.1126/science.aax8814. [DOI] [PubMed] [Google Scholar]
[66].Fan S. H., Joannopoulos J. D. Analysis of guided resonances in photonic crystal slabs. Phys. Rev. B . 2002;65:235112. doi: 10.1103/physrevb.65.235112. [DOI] [Google Scholar]
[67].Fan S. H., Suh W., Joannopoulos J. D. Temporal coupled-mode theory for the Fano resonance in optical resonators. J. Opt. Soc. Am. A . 2003;20:569–572. doi: 10.1364/josaa.20.000569. [DOI] [PubMed] [Google Scholar]
[68].Shinoda K., Ohtera Y. Alignment-free filter array: snapshot multispectral polarization imaging based on a Voronoi-like random photonic crystal filter. Opt Express . 2020;28:38867–38882. doi: 10.1364/oe.411488. [DOI] [PubMed] [Google Scholar]
[69].Li N., Xu Z., Dong Y., et al. Large-area metasurface on CMOS-compatible fabrication platform: driving flat optics from lab to fab. Nanophotonics . 2020;9:3071–3087. doi: 10.1515/nanoph-2020-0063. [DOI] [Google Scholar]
[70].Yuan S., Naveh D., Watanabe K., Taniguchi T., Xia F. A wavelength-scale black phosphorus spectrometer. Nat. Photonics . 2021;15:601–607. doi: 10.1038/s41566-021-00787-x. [DOI] [Google Scholar]
[71].Donoho D. L. Compressed sensing. IEEE Trans. Inf. Theor. . 2006;52:1289–1306. doi: 10.1109/tit.2006.871582. [DOI] [Google Scholar]
[72].Baraniuk R. G. Compressive sensing. IEEE Signal Process. Mag. . 2007;24:118. doi: 10.1109/msp.2007.4286571. [DOI] [Google Scholar]
[73].Candes E., Romberg J. Sparsity and incoherence in compressive sampling. Inverse Probl. . 2007;23:969–985. doi: 10.1088/0266-5611/23/3/008. [DOI] [Google Scholar]
[74].Peyre G. Best basis compressed sensing. IEEE Trans. Signal Process. . 2010;58:2613–2622. doi: 10.1109/tsp.2010.2042490. [DOI] [Google Scholar]
[75].Candes E. J., Tao T. Decoding by linear programming. IEEE Trans. Inf. Theor. . 2005;51:4203–4215. doi: 10.1109/tit.2005.858979. [DOI] [Google Scholar]
[76].Figueiredo M. A. T., Nowak R. D., Wright S. J. Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Topics Sig. Proc. . 2007;1:586–597. doi: 10.1109/jstsp.2007.910281. [DOI] [Google Scholar]
[77].Bioucas-Dias J. M., Figueiredo M. A. T. A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process. . 2007;16:2992–3004. doi: 10.1109/tip.2007.909319. [DOI] [PubMed] [Google Scholar]
[78].Aharon M., Elad M., Bruckstein A. K.-S. V. D. An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. . 2006;54:4311–4322. doi: 10.1109/tsp.2006.881199. [DOI] [Google Scholar]
[79].Yuan X., Tsai T.-H., Zhu R., Llull P., Brady D., Carin L. Compressive hyperspectral imaging with side information. IEEE J. Sel. Topics Sig. Proc. . 2015;9:964–976. doi: 10.1109/jstsp.2015.2411575. [DOI] [Google Scholar]
[80].Yang J., Liao X., Yuan X., et al. Compressive sensing by learning a Gaussian mixture model from measurements. IEEE Trans. Image Process. . 2015;24:106–119. doi: 10.1109/tip.2014.2365720. [DOI] [PubMed] [Google Scholar]
[81].Wang L., Xiong Z., Shi G., Wu F., Zeng W. Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging. IEEE Trans. Pattern Anal. Mach. Intell. . 2017;39:2104–2111. doi: 10.1109/tpami.2016.2621050. [DOI] [PubMed] [Google Scholar]
[82].Liu Y., Yuan X., Suo J., Brady D. J., Dai Q. Rank minimization for snapshot compressive imaging. IEEE Trans. Pattern Anal. Mach. Intell. . 2019;41:2990–3006. doi: 10.1109/tpami.2018.2873587. [DOI] [PubMed] [Google Scholar]
[83].Antipa N., Kuo G., Heckel R., et al. DiffuserCam: lensless single-exposure 3D imaging. Optica . 2018;5:1–9. doi: 10.1364/optica.5.000001. [DOI] [Google Scholar]
[84].Beck A., Teboulle M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imag. Sci. . 2009;2:183–202. doi: 10.1137/080716542. [DOI] [Google Scholar]
[85].Kamilov U. S. A parallel proximal algorithm for anisotropic total variation minimization. IEEE Trans. Image Process. . 2017;26:539–548. doi: 10.1109/tip.2016.2629449. [DOI] [PubMed] [Google Scholar]
[86].Nair V., Hinton G. E. Rectified linear units improve restricted Boltzmann machines. Proc. 27th Int. Conf. Machine Learning . 2010:807–814. [Google Scholar]
[87].Srivastava N., Hinton G., Krizhevsky A., Sutskever I., Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. . 2014;15:1929–1958. [Google Scholar]
[88].Ioffe S., Szegedy C. 32nd International Conference on Machine Learning . France: Lille; 2015. Batch normalization: accelerating deep network training by reducing internal covariate shift; pp. 448–456. [Google Scholar]
[89].Krizhevsky A., Sutskever I., Hinton G. E. 25th International Conference on Neural Information Processing Systems . 2012. ImageNet classification with deep convolutional neural networks; pp. 1097–1105. [Google Scholar]
[90].Hochreiter S., Schmidhuber J. Long short-term memory. Neural Comput. . 1997;9:1735–1780. doi: 10.1162/neco.1997.9.8.1735. [DOI] [PubMed] [Google Scholar]
[91].Goodfellow I. J., Pouget-Abadie J., Mirza M., et al. 28th Conference on Neural Information Processing Systems (NIPS) Montreal, Canada: 2014. Generative adversarial nets; pp. 2672–2680. [Google Scholar]
[92].Kingma D. P., Welling M. Proc. 2nd Int. Conf. Learning Representations . Auto-encoding variational bayes. preprint at https://arxiv.org/abs/1312.6114 . [Google Scholar]
[93].Liu D., Tan Y., Khoram E., Yu Z. Training deep neural networks for the inverse design of nanophotonic structures. ACS Photonics . 2018;5:1365–1369. doi: 10.1021/acsphotonics.7b01377. [DOI] [Google Scholar]
[94].Peurifoy J., Shen Y., Jing L., et al. Nanophotonic particle simulation and inverse design using artificial neural networks. Sci. Adv. . 2018;4:eaar4206. doi: 10.1126/sciadv.aar4206. [DOI] [PMC free article] [PubMed] [Google Scholar]
[95].Gao L., Li X., Liu D., Wang L., Yu Z. A bidirectional deep neural network for accurate silicon color design. Adv. Mater. . 2019;31:1905467. doi: 10.1002/adma.201905467. [DOI] [PubMed] [Google Scholar]
[96].Ballard Z., Brown C., Madni A. M., Ozcan A. Machine learning and computation-enabled intelligent sensor design. Nat. Mac. Intell. . 2021;3:556–565. doi: 10.1038/s42256-021-00360-9. [DOI] [Google Scholar]
[97].Fang J., Swain A., Unni R., Zheng Y. Decoding optical data with machine learning. Laser Photon. Rev. . 2021;15:2000422. doi: 10.1002/lpor.202000422. [DOI] [PMC free article] [PubMed] [Google Scholar]
[98].Barbastathis G., Ozcan A., Situ G. On the use of deep learning for computational imaging. Optica . 2019;6:921–943. doi: 10.1364/optica.6.000921. [DOI] [Google Scholar]
[99].Nie S., Gu L., Zheng Y., et al. 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Salt Lake City, UT: I. C. I. C. Soc; 2018. Deeply learned filter response functions for hyperspectral reconstruction; pp. 4767–4776. [Google Scholar]
[100].Gedalin D., Oiknine Y., Stern A. DeepCubeNet: reconstruction of spectrally compressive sensed hyperspectral images with deep neural networks. Opt. Express . 2019;27:35811–35822. doi: 10.1364/oe.27.035811. [DOI] [PubMed] [Google Scholar]
[101].Li S., Zhu X., Liu Y., Bao J. Adaptive spatial-spectral feature learning for hyperspectral image classification. IEEE Access . 2019;7:61534–61547. doi: 10.1109/access.2019.2916095. [DOI] [Google Scholar]
[102].Li S., Zhu X., Bao J. Hierarchical multi-scale convolutional neural networks for hyperspectral image classification. Sensors . 2019;19:1714. doi: 10.3390/s19071714. [DOI] [PMC free article] [PubMed] [Google Scholar]
[103].Kurum U., Wiecha P. R., French R., Muskens O. L. Deep learning enabled real time speckle recognition and hyperspectral imaging using a multimode fiber array. Opt. Express . 2019;27:20965–20979. doi: 10.1364/oe.27.020965. [DOI] [PubMed] [Google Scholar]
[104].Kim C., Park D., Lee H.-N. Conference on Optical Data Science II . San Francisco, CA: Spie; 2019. Convolutional neural networks for the reconstruction of spectra in compressive sensing spectrometers; p. 109370L. [Google Scholar]
[105].Kim C., Park D., Lee H.-N. Compressive sensing spectroscopy using a residual convolutional neural network. Sensors . 2020;20:594. doi: 10.3390/s20030594. [DOI] [PMC free article] [PubMed] [Google Scholar]
[106].Song H., Ma Y., Han Y., et al. Deep-learned broadband encoding stochastic filters for computational spectroscopic instruments. Adv. Theory Simul. . 2021;4:2000299. doi: 10.1002/adts.202000299. [DOI] [Google Scholar]
[107].Choi I., Jeon D. S., Nam G., Gutierrez D., Kim M. H. High-quality hyperspectral reconstruction using a spectral prior. ACM Trans. Graph. . 2017;36:218. doi: 10.1145/3130800.3130810. [DOI] [Google Scholar]
[108].Meng Z., Qiao M., Ma J., Yu Z., Xu K., Yuan X. Snapshot multispectral endomicroscopy. Opt. Lett. . 2020;45:3893–3900. doi: 10.1364/OL.393213. [DOI] [PubMed] [Google Scholar]
[109].Miao X., Yuan X., Pu Y., Athitsos V. IEEE/CVF International Conference on Computer Vision (ICCV) Seoul, South Korea: I. I. C. S. CVF; 2019. Lambda-net: reconstruct hyperspectral images from a snapshot measurement; pp. 4058–4068. [Google Scholar]
[110].Wang L., Sun C., Fu Y., Kim M. H., Huang H., Soc I. C. 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Long Beach, CA: I. C. I. C. Soc; 2019. Hyperspectral image reconstruction using a deep spatial-spectral prior; pp. 8024–8033. [Google Scholar]
[111].Meng Z., Ma J., Yuan X. European Conference on Computer Vision . 2020. End-to-end low cost compressive spectral imaging with spatial-spectral self-attention; pp. 187–204. [Google Scholar]
[112].Venkatakrishnan S., Bouman C. A., Wohlberg B. 1st IEEE Global Conference on Signal and Information Processing (GlobalSIP) Austin, TX: Elect Engineers; 2013. Plug-and-Play priors for model based reconstruction; pp. 945–948. [Google Scholar]
[113].Sreehari S., Venkatakrishnan S. V., Wohlberg B., et al. Plug-and-Play priors for bright field electron tomography and sparse interpolation. IEEE Trans. Comput. Imag. . 2016;2:408–423. doi: 10.1109/tci.2016.2599778. [DOI] [Google Scholar]
[114].Chan S. H., Wang X., Elgendy O. A. Plug-and-Play ADMM for image restoration: fixed-point convergence and applications. IEEE Trans. Comput. Imag. . 2017;3:84–98. doi: 10.1109/tci.2016.2629286. [DOI] [Google Scholar]
[115].Ryu E., Liu J., Wang S., Chen X., Wang Z., Yin W. International Conference on Machine Learning . 2019. Plug-andplay methods provably converge with properly trained denoisers; pp. 5546–5557. [Google Scholar]
[116].Zhang L., Zuo W. Image restoration: from sparse and low-rank priors to deep priors. IEEE Signal Process. Mag. . 2017;34:172–179. doi: 10.1109/msp.2017.2717489. [DOI] [Google Scholar]
[117].Zhang K., Zuo W., Chen Y., Meng D., Zhang L. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. . 2017;26:3142–3155. doi: 10.1109/tip.2017.2662206. [DOI] [PubMed] [Google Scholar]
[118].Zhang K., Zuo W., Zhang L. FFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. . 2018;27:4608–4622. doi: 10.1109/tip.2018.2839891. [DOI] [PubMed] [Google Scholar]
[119].Zheng S., Liu Y., Meng Z., et al. Deep plug-and-play priors for spectral snapshot compressive imaging. Photon. Res. . 2021;9:B18–B29. doi: 10.1364/prj.411745. [DOI] [Google Scholar]
[120].Li J., Mengu D., Yardimci N. T., et al. Spectrally encoded single-pixel machine vision using diffractive networks. Sci. Adv. . 2021;7 doi: 10.1126/sciadv.abd7690. [DOI] [PMC free article] [PubMed] [Google Scholar]
[121].Ballard Z. S., Shir D., Bhardwaj A., Bazargan S., Sathianathan S., Ozcan A. Computational sensing using low-cost and mobile plasmonic readers designed by machine learning. ACS Nano . 2017;11:2266–2274. doi: 10.1021/acsnano.7b00105. [DOI] [PMC free article] [PubMed] [Google Scholar]
[122].Ho C.-S., Jean N., Hogan C. A., et al. Rapid identification of pathogenic bacteria using Raman spectroscopy and deep learning. Nat. Commun. . 2019;10:4927. doi: 10.1038/s41467-019-12898-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
[123].Cho S.-Y., Lee Y., Lee S., et al. Finding hidden signals in chemical sensors using deep learning. Anal. Chem. . 2020;92:6529–6537. doi: 10.1021/acs.analchem.0c00137. [DOI] [PubMed] [Google Scholar]
[124].Brown C., Tseng D., Larkin P. M. K., et al. Automated, cost-effective optical system for accelerated antimicrobial susceptibility testing (AST) using deep learning. ACS Photonics . 2020;7:2527–2538. doi: 10.1021/acsphotonics.0c00841. [DOI] [Google Scholar]
[125].Meng J., Cadusch J. J., Crozier K. B. Plasmonic mid-infrared filter array-detector array chemical classifier based on machine learning. ACS Photonics . 2021;8:648–657. doi: 10.1021/acsphotonics.0c01786. [DOI] [Google Scholar]
[126].John-Herpin A., Kavungal D., von Mucke L., Altug H. Infrared metasurface augmented by deep learning for monitoring dynamics between all major classes of biomolecules. Adv. Mater. . 2021;33:e2006054. doi: 10.1002/adma.202006054. [DOI] [PMC free article] [PubMed] [Google Scholar]
[127].Song M. K., Chen S. X., Hu P. P., Huang C. Z., Zhou J. Automated plasmonic resonance scattering imaging analysis via deep learning. Anal. Chem. . 2021;93:2619–2626. doi: 10.1021/acs.analchem.0c04763. [DOI] [PubMed] [Google Scholar]
[128].Smith J. D., Reza M. A., Smith N. L., et al. Plasmonic anticounterfeit tags with high encoding capacity rapidly authenticated with deep machine learning. ACS Nano . 2021;15:2901–2910. doi: 10.1021/acsnano.0c08974. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_001] [1].Bacon C. P., Mattley Y., DeFrece R. Miniature spectroscopic instrumentation: applications to biology and chemistry. Rev. Sci. Instrum. . 2004;75:1–16. doi: 10.1063/1.1633025. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_002] [2].Pavia D. L., Lampman G. M., Kriz G. S., Vyvyan J. A. Introduction to Spectroscopy . Belmont: Cengage Learning; 2014. [Google Scholar]

[j_nanoph-2021-0636_ref_003] [3].Malinen J., Rissanen A., Saari H., et al. Conference on Next-Generation Spectroscopic Technologies VII . Baltimore, MD: SPIE; 2014. Advances in miniature spectrometer and sensor development; p. 91010C. [Google Scholar]

[j_nanoph-2021-0636_ref_004] [4].Crocombe R. A. Portable spectroscopy. Appl. Spectrosc. . 2018;72:1701–1751. doi: 10.1177/0003702818809719. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_005] [5].McGonigle A. J. S., Wilkes T. C., Pering T. D., et al. Smartphone spectrometers. Sensors . 2018;18:233. doi: 10.3390/s18010223. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_006] [6].Faraji-Dana M., Arbabi E., Arbabi A., Kamali S. M., Kwon H., Faraon A. Compact folded metasurface spectrometer. Nat. Commun. . 2018;9:4196. doi: 10.1038/s41467-018-06495-5. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_007] [7].Cheng R., Zou C.-L., Guo X., Wang S., Han X., Tang H. X. Broadband on-chip single-photon spectrometer. Nat. Commun. . 2019;10:4104. doi: 10.1038/s41467-019-12149-x. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_008] [8].Nezhadbadeh S., Neumann A., Zarkesh-Ha P., Brueck S. R. J. Chirped-grating spectrometer-on-a-chip. Opt Express . 2020;28:24501–24510. doi: 10.1364/oe.398072. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_009] [9].Li A., Fainman Y. On-chip spectrometers using stratified waveguide filters. Nat. Commun. . 2021;12:2704. doi: 10.1038/s41467-021-23001-6. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_010] [10].Yang Z., Albrow-Owen T., Cai W., Hasan T. Miniaturization of optical spectrometers. Science . 2021;371:480. doi: 10.1126/science.abe0722. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_011] [11].Noda S., Fujita M., Asano T. Spontaneous-emission control by photonic crystals and nanocavities. Nat. Photonics . 2007;1:449–458. doi: 10.1038/nphoton.2007.141. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_012] [12].Tanaka Y., Asano T., Noda S. Design of photonic crystal nanocavity with Q-factor of ∼109 . J. Lightwave Technol. . 2008;26:1532–1539. doi: 10.1109/jlt.2008.923648. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_015] [13].Redding B., Liew S. F., Sarma R., Cao H. Compact spectrometer based on a disordered photonic chip. Nat. Photonics . 2013;7:746–751. doi: 10.1038/nphoton.2013.190. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_016] [14].August Y., Stern A. Compressive sensing spectrometry based on liquid crystal devices. Opt Lett. . 2013;38:4996–4999. doi: 10.1364/ol.38.004996. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_017] [15].Oliver J., Lee W.-B., Lee H.-N. Filters with random transmittance for improving resolution in filter-array-based spectrometers. Opt. Express . 2013;21:3969–3989. doi: 10.1364/oe.21.003969. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_018] [16].Wang Z., Yu Z. Spectral analysis based on compressive sensing in nanophotonic structures. Opt. Express . 2014;22:25608–25614. doi: 10.1364/oe.22.025608. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_019] [17].Wang P., Menon R. Computational spectrometer based on a broadband diffractive optic. Opt. Express . 2014;22:14575–14587. doi: 10.1364/oe.22.014575. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_020] [18].Yang T., Xu C., Ho H.-p., et al. Miniature spectrometer based on diffraction in a dispersive hole array. Opt Lett. . 2015;40:3217–3220. doi: 10.1364/ol.40.003217. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_021] [19].Bao J., Bawendi M. G. A colloidal quantum dot spectrometer. Nature . 2015;523:67–70. doi: 10.1038/nature14576. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_022] [20].Huang E., Ma Q., Liu Z. Etalon array reconstructive spectrometry. Sci. Rep. . 2017;7:40693. doi: 10.1038/srep40693. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_023] [21].Craig B., Shrestha V. R., Meng J., Cadusch J. J., Crozier K. B. Experimental demonstration of infrared spectral reconstruction using plasmonic metasurfaces. Opt. Lett. . 2018;43:4481–4484. doi: 10.1364/ol.43.004481. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_024] [22].Wang Z., Yi S., Chen A., et al. Single-shot on-chip spectral sensors based on photonic crystal slabs. Nat. Commun. . 2019;10:1020. doi: 10.1038/s41467-019-08994-5. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_025] [23].Zhu Y., Lei X., Wang K. X., Yu Z. Compact CMOS spectral sensor for the visible spectrum. Photon. Res. . 2019;7:961–966. doi: 10.1364/prj.7.000961. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_026] [24].Hartmann W., Varytis P., Gehring H., et al. Waveguide-integrated broadband spectrometer based on tailored disorder. Adv. Opt. Mater. . 2020;8:1901602. doi: 10.1002/adom.201901602. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_027] [25].Zhu X., Bian L., Fu H., et al. Broadband perovskite quantum dot spectrometer beyond human visual resolution. Light Sci. Appl. . 2020;9:73. doi: 10.1038/s41377-020-0301-4. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_028] [26].French R., Gigan S., Muskens O. L. Speckle-based hyperspectral imaging combining multiple scattering and compressive sensing in nanowire mats. Opt. Lett. . 2017;42:1820–1823. doi: 10.1364/ol.42.001820. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_029] [27].Cai X., Xiong J., Cui K., et al. One-shot Ultraspectral Imaging with Reconfigurable Metasurfaces . 2020. preprint arXiv:2005.02689. [Google Scholar]

[j_nanoph-2021-0636_ref_030] [28].Monakhova K., Yanny K., Aggarwal N., Waller L. Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array. Optica . 2020;7:1298–1307. doi: 10.1364/optica.397214. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_031] [29].McClung A., Samudrala S., Torfeh M., Mansouree M., Arbabi A. Snapshot spectral imaging with parallel metasystems. Sci. Adv. . 2020;6:eabc7646. doi: 10.1126/sciadv.abc7646. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_032] [30].Chang C.-C., Lee H.-N. On the estimation of target spectrum for filter-array based spectrometers. Opt. Express . 2008;16:1056–1061. doi: 10.1364/oe.16.001056. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_033] [31].Carmo J. P., Rocha R. P., Bartek M., de Graaf G., Wolffenbuttel R. F., Correia J. H. A review of visible-range Fabry-Perot microspectrometers in silicon for the industry. Opt. Laser. Technol. . 2012;44:2312–2320. doi: 10.1016/j.optlastec.2012.03.036. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_034] [32].Kurokawa U., Choi B. I., Chang C.-C. Filter-based miniature spectrometers: spectrum reconstruction using adaptive regularization. IEEE Sensor. J. . 2011;11:1556–1563. doi: 10.1109/jsen.2010.2103054. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_035] [33].Chang C.-C., Lin N.-T., Kurokawa U., Choi B. I. Spectrum reconstruction for filter-array spectrum sensor from sparse template selection. Opt. Eng. . 2011;50:114402. doi: 10.1117/1.3645086. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_036] [34].Oliver J., Lee W., Park S., Lee H.-N. Improving resolution of miniature spectrometers by exploiting sparse nature of signals. Opt. Express . 2012;20:2613–2625. doi: 10.1364/oe.20.002613. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_037] [35].Zhang S., Dong Y., Fu H., Huang S.-L., Zhang L. A spectral reconstruction algorithm of miniature spectrometer based on sparse optimization and dictionary learning. Sensors . 2018;18:644. doi: 10.3390/s18020644. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_038] [36].Zhang J., Zhu X., Bao J. Solver-informed neural networks for spectrum reconstruction of colloidal quantum dot spectrometers. Opt. Express . 2020;28:33656–33672. doi: 10.1364/oe.402149. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_039] [37].Zhang J., Zhu X., Bao J. Denoising autoencoder aided spectrum reconstruction for colloidal quantum dot spectrometers. IEEE Sensor. J. . 2021;21:6450–6458. doi: 10.1109/jsen.2020.3039973. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_040] [38].Brown C., Goncharov A., Ballard Z. S., et al. Neural network-based on-chip spectroscopy using a scalable plasmonic encoder. ACS Nano . 2021;15:6305–6315. doi: 10.1021/acsnano.1c00079. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_041] [39].Zhang W., Song H., He X., et al. Deeply learned broadband encoding stochastic hyperspectral imaging. Light Sci. Appl. . 2021;10:108. doi: 10.1038/s41377-021-00545-2. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_042] [40].So S., Badloe T., Noh J., Rho J., Bravo-Abad J. Deep learning enabled inverse design in nanophotonics. Nanophotonics . 2020;9:1041–1057. doi: 10.1515/nanoph-2019-0474. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_043] [41].Jiang J., Chen M., Fan J. A. Deep neural networks for the evaluation and design of photonic devices. Nat. Rev. Mater. . 2021;6:679–700. doi: 10.1038/s41578-020-00260-1. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_044] [42].Wiecha P. R., Arbouet A., Girard C., Muskens O. L. Deep learning in nano-photonics: inverse design and beyond. Photon. Res. . 2021;9:B182–B200. doi: 10.1364/prj.415960. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_045] [43].Xu Y., Zhang X., Fu Y., Liu Y. Interfacing photonics with artificial intelligence: an innovative design strategy for photonic structures and devices based on artificial neural networks. Photon. Res. . 2021;9:B135–B52. doi: 10.1364/prj.417693. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_046] [44].Ma W., Liu Z., Kudyshev Z. A., Boltasseva A., Cai W., Liu Y. Deep learning for the design of photonic structures. Nat. Photonics . 2021;15:77–90. doi: 10.1038/s41566-020-0685-y. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_047] [45].Koenderink A. F., Alu A., Polman A. Nanophotonics: shrinking light-based technology. Science . 2015;348:516–521. doi: 10.1126/science.1261243. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_048] [46].Joannopoulos J. D., Villeneuve P. R., Fan S. H. Photonic crystals: putting a new twist on light. Nature . 1997;386:143–149. doi: 10.1038/386143a0. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_049] [47].Pervez N. K., Cheng W., Jia Z., Cox M. P., Edrees H. M., Kymissis I. Photonic crystal spectrometer. Opt. Express . 2010;18:8277–8285. doi: 10.1364/oe.18.008277. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_050] [48].Gan X., Pervez N., Kymissis I., Hatami F., Englund D. A high-resolution spectrometer based on a compact planar two dimensional photonic crystal cavity array. Appl. Phys. Lett. . 2012;100:231104. doi: 10.1063/1.4724177. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_051] [49].Yang K., Yao X., Liu B., Ren B. Metallic plasmonic array structures: principles, fabrications, properties, and applications. Adv. Mater. . 2021:2007988. doi: 10.1002/adma.202007988. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_052] [50].Yu N., Genevet P., Kats M. A., et al. Light propagation with phase discontinuities: generalized laws of reflection and refraction. Science . 2011;334:333–337. doi: 10.1126/science.1210713. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_053] [51].Chen W. T., Zhu A. Y., Capasso F. Flat optics with dispersion-engineered metasurfaces. Nat. Rev. Mater. . 2020;5:604–620. doi: 10.1038/s41578-020-0203-3. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_054] [52].Rezaei S. D., Dong Z., Chan J. Y. E., et al. Nanophotonic structural colors. ACS Photonics . 2021;8:18–33. doi: 10.1021/acsphotonics.0c00947. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_055] [53].Chen W. T., Zhu A. Y., Sanjeev V., et al. A broadband achromatic metalens for focusing and imaging in the visible. Nat. Nanotechnol. . 2018;13:220. doi: 10.1038/s41565-017-0034-6. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_056] [54].Chen W. T., Zhu A. Y., Sisler J., Bharwani Z., Capasso F. A broadband achromatic polarization-insensitive metalens consisting of anisotropic nanostructures. Nat. Commun. . 2019;10 doi: 10.1038/s41467-019-08305-y. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_057] [55].Wiecha P. R., Lecestre A., Mallet N., Larrieu G. Pushing the limits of optical information storage using deep learning. Nat. Nanotechnol. . 2019;14:237–244. doi: 10.1038/s41565-018-0346-1. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_058] [56].Chen Q., Hu X., Wen L., Yu Y., Cumming D. R. S. Nanophotonic image sensors. Small . 2016;12:4922–4935. doi: 10.1002/smll.201600528. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_059] [57].Tittl A., Leitis A., Liu M., et al. Imaging-based molecular barcoding with pixelated dielectric metasurfaces. Science . 2018;360:1105–1109. doi: 10.1126/science.aas9768. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_060] [58].Yesilkoy F., Arvelo E. R., Jahani Y., et al. Ultrasensitive hyperspectral imaging and biodetection enabled by dielectric metasurfaces. Nat. Photonics . 2019;13:390–396. doi: 10.1038/s41566-019-0394-6. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_061] [59].Emadi A., Wu H., de Graaf G., Wolffenbuttel R. Design and implementation of a sub-nm resolution microspectrometer based on a Linear-Variable Optical Filter. Opt. Express . 2012;20:489–507. doi: 10.1364/oe.20.000489. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_062] [60].Wang S.-W., Xia C., Chen X., et al. Concept of a high-resolution miniature spectrometer using an integrated filter array. Opt Lett. . 2007;32:632–634. doi: 10.1364/ol.32.000632. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_063] [61].Oiknine Y., August I., Blumberg D. G., Stern A. Compressive sensing resonator spectroscopy. Opt Lett. . 2017;42:25–28. doi: 10.1364/ol.42.000025. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_064] [62].Degraux K., Cambareri V., Geelen B., Jacques L., Lafruit G. Multispectral compressive imaging strategies using fabry-perot filtered sensors. IEEE Trans. Comput. Imag. . 2018;4:661–673. doi: 10.1109/tci.2018.2864659. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_065] [63].Deng W., You C., Zhang Y. Spectral discrimination sensors based on nanomaterials and nanostructures: a review. IEEE Sensor. J. . 2021;21:4044–4060. doi: 10.1109/jsen.2020.3032720. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_070] [64].Kwak Y., Park S. M., Ku Z., Urbas A., Kim Y. L. A pearl spectrometer. Nano Lett. . 2021;21:921–930. doi: 10.1021/acs.nanolett.0c03618. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_071] [65].Yang Z., Albrow-Owen T., Cui H., et al. Single-nanowire spectrometers. Science . 2019;365:1017–1020. doi: 10.1126/science.aax8814. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_066] [66].Fan S. H., Joannopoulos J. D. Analysis of guided resonances in photonic crystal slabs. Phys. Rev. B . 2002;65:235112. doi: 10.1103/physrevb.65.235112. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_067] [67].Fan S. H., Suh W., Joannopoulos J. D. Temporal coupled-mode theory for the Fano resonance in optical resonators. J. Opt. Soc. Am. A . 2003;20:569–572. doi: 10.1364/josaa.20.000569. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_068] [68].Shinoda K., Ohtera Y. Alignment-free filter array: snapshot multispectral polarization imaging based on a Voronoi-like random photonic crystal filter. Opt Express . 2020;28:38867–38882. doi: 10.1364/oe.411488. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_069] [69].Li N., Xu Z., Dong Y., et al. Large-area metasurface on CMOS-compatible fabrication platform: driving flat optics from lab to fab. Nanophotonics . 2020;9:3071–3087. doi: 10.1515/nanoph-2020-0063. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_072] [70].Yuan S., Naveh D., Watanabe K., Taniguchi T., Xia F. A wavelength-scale black phosphorus spectrometer. Nat. Photonics . 2021;15:601–607. doi: 10.1038/s41566-021-00787-x. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_073] [71].Donoho D. L. Compressed sensing. IEEE Trans. Inf. Theor. . 2006;52:1289–1306. doi: 10.1109/tit.2006.871582. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_074] [72].Baraniuk R. G. Compressive sensing. IEEE Signal Process. Mag. . 2007;24:118. doi: 10.1109/msp.2007.4286571. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_075] [73].Candes E., Romberg J. Sparsity and incoherence in compressive sampling. Inverse Probl. . 2007;23:969–985. doi: 10.1088/0266-5611/23/3/008. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_076] [74].Peyre G. Best basis compressed sensing. IEEE Trans. Signal Process. . 2010;58:2613–2622. doi: 10.1109/tsp.2010.2042490. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_077] [75].Candes E. J., Tao T. Decoding by linear programming. IEEE Trans. Inf. Theor. . 2005;51:4203–4215. doi: 10.1109/tit.2005.858979. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_078] [76].Figueiredo M. A. T., Nowak R. D., Wright S. J. Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Topics Sig. Proc. . 2007;1:586–597. doi: 10.1109/jstsp.2007.910281. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_079] [77].Bioucas-Dias J. M., Figueiredo M. A. T. A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process. . 2007;16:2992–3004. doi: 10.1109/tip.2007.909319. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_080] [78].Aharon M., Elad M., Bruckstein A. K.-S. V. D. An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. . 2006;54:4311–4322. doi: 10.1109/tsp.2006.881199. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_081] [79].Yuan X., Tsai T.-H., Zhu R., Llull P., Brady D., Carin L. Compressive hyperspectral imaging with side information. IEEE J. Sel. Topics Sig. Proc. . 2015;9:964–976. doi: 10.1109/jstsp.2015.2411575. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_082] [80].Yang J., Liao X., Yuan X., et al. Compressive sensing by learning a Gaussian mixture model from measurements. IEEE Trans. Image Process. . 2015;24:106–119. doi: 10.1109/tip.2014.2365720. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_083] [81].Wang L., Xiong Z., Shi G., Wu F., Zeng W. Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging. IEEE Trans. Pattern Anal. Mach. Intell. . 2017;39:2104–2111. doi: 10.1109/tpami.2016.2621050. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_084] [82].Liu Y., Yuan X., Suo J., Brady D. J., Dai Q. Rank minimization for snapshot compressive imaging. IEEE Trans. Pattern Anal. Mach. Intell. . 2019;41:2990–3006. doi: 10.1109/tpami.2018.2873587. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_085] [83].Antipa N., Kuo G., Heckel R., et al. DiffuserCam: lensless single-exposure 3D imaging. Optica . 2018;5:1–9. doi: 10.1364/optica.5.000001. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_086] [84].Beck A., Teboulle M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imag. Sci. . 2009;2:183–202. doi: 10.1137/080716542. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_087] [85].Kamilov U. S. A parallel proximal algorithm for anisotropic total variation minimization. IEEE Trans. Image Process. . 2017;26:539–548. doi: 10.1109/tip.2016.2629449. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_088] [86].Nair V., Hinton G. E. Rectified linear units improve restricted Boltzmann machines. Proc. 27th Int. Conf. Machine Learning . 2010:807–814. [Google Scholar]

[j_nanoph-2021-0636_ref_089] [87].Srivastava N., Hinton G., Krizhevsky A., Sutskever I., Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. . 2014;15:1929–1958. [Google Scholar]

[j_nanoph-2021-0636_ref_090] [88].Ioffe S., Szegedy C. 32nd International Conference on Machine Learning . France: Lille; 2015. Batch normalization: accelerating deep network training by reducing internal covariate shift; pp. 448–456. [Google Scholar]

[j_nanoph-2021-0636_ref_091] [89].Krizhevsky A., Sutskever I., Hinton G. E. 25th International Conference on Neural Information Processing Systems . 2012. ImageNet classification with deep convolutional neural networks; pp. 1097–1105. [Google Scholar]

[j_nanoph-2021-0636_ref_092] [90].Hochreiter S., Schmidhuber J. Long short-term memory. Neural Comput. . 1997;9:1735–1780. doi: 10.1162/neco.1997.9.8.1735. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_093] [91].Goodfellow I. J., Pouget-Abadie J., Mirza M., et al. 28th Conference on Neural Information Processing Systems (NIPS) Montreal, Canada: 2014. Generative adversarial nets; pp. 2672–2680. [Google Scholar]

[j_nanoph-2021-0636_ref_094] [92].Kingma D. P., Welling M. Proc. 2nd Int. Conf. Learning Representations . Auto-encoding variational bayes. preprint at https://arxiv.org/abs/1312.6114 . [Google Scholar]

[j_nanoph-2021-0636_ref_095] [93].Liu D., Tan Y., Khoram E., Yu Z. Training deep neural networks for the inverse design of nanophotonic structures. ACS Photonics . 2018;5:1365–1369. doi: 10.1021/acsphotonics.7b01377. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_096] [94].Peurifoy J., Shen Y., Jing L., et al. Nanophotonic particle simulation and inverse design using artificial neural networks. Sci. Adv. . 2018;4:eaar4206. doi: 10.1126/sciadv.aar4206. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_097] [95].Gao L., Li X., Liu D., Wang L., Yu Z. A bidirectional deep neural network for accurate silicon color design. Adv. Mater. . 2019;31:1905467. doi: 10.1002/adma.201905467. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_098] [96].Ballard Z., Brown C., Madni A. M., Ozcan A. Machine learning and computation-enabled intelligent sensor design. Nat. Mac. Intell. . 2021;3:556–565. doi: 10.1038/s42256-021-00360-9. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_099] [97].Fang J., Swain A., Unni R., Zheng Y. Decoding optical data with machine learning. Laser Photon. Rev. . 2021;15:2000422. doi: 10.1002/lpor.202000422. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_100] [98].Barbastathis G., Ozcan A., Situ G. On the use of deep learning for computational imaging. Optica . 2019;6:921–943. doi: 10.1364/optica.6.000921. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_101] [99].Nie S., Gu L., Zheng Y., et al. 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Salt Lake City, UT: I. C. I. C. Soc; 2018. Deeply learned filter response functions for hyperspectral reconstruction; pp. 4767–4776. [Google Scholar]

[j_nanoph-2021-0636_ref_102] [100].Gedalin D., Oiknine Y., Stern A. DeepCubeNet: reconstruction of spectrally compressive sensed hyperspectral images with deep neural networks. Opt. Express . 2019;27:35811–35822. doi: 10.1364/oe.27.035811. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_103] [101].Li S., Zhu X., Liu Y., Bao J. Adaptive spatial-spectral feature learning for hyperspectral image classification. IEEE Access . 2019;7:61534–61547. doi: 10.1109/access.2019.2916095. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_104] [102].Li S., Zhu X., Bao J. Hierarchical multi-scale convolutional neural networks for hyperspectral image classification. Sensors . 2019;19:1714. doi: 10.3390/s19071714. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_105] [103].Kurum U., Wiecha P. R., French R., Muskens O. L. Deep learning enabled real time speckle recognition and hyperspectral imaging using a multimode fiber array. Opt. Express . 2019;27:20965–20979. doi: 10.1364/oe.27.020965. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_106] [104].Kim C., Park D., Lee H.-N. Conference on Optical Data Science II . San Francisco, CA: Spie; 2019. Convolutional neural networks for the reconstruction of spectra in compressive sensing spectrometers; p. 109370L. [Google Scholar]

[j_nanoph-2021-0636_ref_107] [105].Kim C., Park D., Lee H.-N. Compressive sensing spectroscopy using a residual convolutional neural network. Sensors . 2020;20:594. doi: 10.3390/s20030594. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_108] [106].Song H., Ma Y., Han Y., et al. Deep-learned broadband encoding stochastic filters for computational spectroscopic instruments. Adv. Theory Simul. . 2021;4:2000299. doi: 10.1002/adts.202000299. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_109] [107].Choi I., Jeon D. S., Nam G., Gutierrez D., Kim M. H. High-quality hyperspectral reconstruction using a spectral prior. ACM Trans. Graph. . 2017;36:218. doi: 10.1145/3130800.3130810. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_110] [108].Meng Z., Qiao M., Ma J., Yu Z., Xu K., Yuan X. Snapshot multispectral endomicroscopy. Opt. Lett. . 2020;45:3893–3900. doi: 10.1364/OL.393213. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_111] [109].Miao X., Yuan X., Pu Y., Athitsos V. IEEE/CVF International Conference on Computer Vision (ICCV) Seoul, South Korea: I. I. C. S. CVF; 2019. Lambda-net: reconstruct hyperspectral images from a snapshot measurement; pp. 4058–4068. [Google Scholar]

[j_nanoph-2021-0636_ref_112] [110].Wang L., Sun C., Fu Y., Kim M. H., Huang H., Soc I. C. 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Long Beach, CA: I. C. I. C. Soc; 2019. Hyperspectral image reconstruction using a deep spatial-spectral prior; pp. 8024–8033. [Google Scholar]

[j_nanoph-2021-0636_ref_113] [111].Meng Z., Ma J., Yuan X. European Conference on Computer Vision . 2020. End-to-end low cost compressive spectral imaging with spatial-spectral self-attention; pp. 187–204. [Google Scholar]

[j_nanoph-2021-0636_ref_114] [112].Venkatakrishnan S., Bouman C. A., Wohlberg B. 1st IEEE Global Conference on Signal and Information Processing (GlobalSIP) Austin, TX: Elect Engineers; 2013. Plug-and-Play priors for model based reconstruction; pp. 945–948. [Google Scholar]

[j_nanoph-2021-0636_ref_115] [113].Sreehari S., Venkatakrishnan S. V., Wohlberg B., et al. Plug-and-Play priors for bright field electron tomography and sparse interpolation. IEEE Trans. Comput. Imag. . 2016;2:408–423. doi: 10.1109/tci.2016.2599778. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_116] [114].Chan S. H., Wang X., Elgendy O. A. Plug-and-Play ADMM for image restoration: fixed-point convergence and applications. IEEE Trans. Comput. Imag. . 2017;3:84–98. doi: 10.1109/tci.2016.2629286. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_117] [115].Ryu E., Liu J., Wang S., Chen X., Wang Z., Yin W. International Conference on Machine Learning . 2019. Plug-andplay methods provably converge with properly trained denoisers; pp. 5546–5557. [Google Scholar]

[j_nanoph-2021-0636_ref_118] [116].Zhang L., Zuo W. Image restoration: from sparse and low-rank priors to deep priors. IEEE Signal Process. Mag. . 2017;34:172–179. doi: 10.1109/msp.2017.2717489. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_119] [117].Zhang K., Zuo W., Chen Y., Meng D., Zhang L. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. . 2017;26:3142–3155. doi: 10.1109/tip.2017.2662206. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_120] [118].Zhang K., Zuo W., Zhang L. FFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. . 2018;27:4608–4622. doi: 10.1109/tip.2018.2839891. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_121] [119].Zheng S., Liu Y., Meng Z., et al. Deep plug-and-play priors for spectral snapshot compressive imaging. Photon. Res. . 2021;9:B18–B29. doi: 10.1364/prj.411745. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_122] [120].Li J., Mengu D., Yardimci N. T., et al. Spectrally encoded single-pixel machine vision using diffractive networks. Sci. Adv. . 2021;7 doi: 10.1126/sciadv.abd7690. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_123] [121].Ballard Z. S., Shir D., Bhardwaj A., Bazargan S., Sathianathan S., Ozcan A. Computational sensing using low-cost and mobile plasmonic readers designed by machine learning. ACS Nano . 2017;11:2266–2274. doi: 10.1021/acsnano.7b00105. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_124] [122].Ho C.-S., Jean N., Hogan C. A., et al. Rapid identification of pathogenic bacteria using Raman spectroscopy and deep learning. Nat. Commun. . 2019;10:4927. doi: 10.1038/s41467-019-12898-9. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_125] [123].Cho S.-Y., Lee Y., Lee S., et al. Finding hidden signals in chemical sensors using deep learning. Anal. Chem. . 2020;92:6529–6537. doi: 10.1021/acs.analchem.0c00137. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_126] [124].Brown C., Tseng D., Larkin P. M. K., et al. Automated, cost-effective optical system for accelerated antimicrobial susceptibility testing (AST) using deep learning. ACS Photonics . 2020;7:2527–2538. doi: 10.1021/acsphotonics.0c00841. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_127] [125].Meng J., Cadusch J. J., Crozier K. B. Plasmonic mid-infrared filter array-detector array chemical classifier based on machine learning. ACS Photonics . 2021;8:648–657. doi: 10.1021/acsphotonics.0c01786. [DOI] [Google Scholar]

[j_nanoph-2021-0636_ref_128] [126].John-Herpin A., Kavungal D., von Mucke L., Altug H. Infrared metasurface augmented by deep learning for monitoring dynamics between all major classes of biomolecules. Adv. Mater. . 2021;33:e2006054. doi: 10.1002/adma.202006054. [DOI] [PMC free article] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_129] [127].Song M. K., Chen S. X., Hu P. P., Huang C. Z., Zhou J. Automated plasmonic resonance scattering imaging analysis via deep learning. Anal. Chem. . 2021;93:2619–2626. doi: 10.1021/acs.analchem.0c04763. [DOI] [PubMed] [Google Scholar]

[j_nanoph-2021-0636_ref_130] [128].Smith J. D., Reza M. A., Smith N. L., et al. Plasmonic anticounterfeit tags with high encoding capacity rapidly authenticated with deep machine learning. ACS Nano . 2021;15:2901–2910. doi: 10.1021/acsnano.0c08974. [DOI] [PubMed] [Google Scholar]

PERMALINK

Computational spectrometers enabled by nanophotonics and deep learning

Li Gao

Yurui Qu

Lianhui Wang

Zongfu Yu

Abstract

1. Introduction and overview

Figure 1:

2. The encoding and decoding process in computational spectrometers

2.1. Nanophotonic filters as spectral encoders

Figure 2:

Figure 3:

2.2. Spectral data acquisition

2.3. Computational algorithms as spectral decoder

Figure 4:

3. Spectrum reconstruction enabled by deep learning

3.1. Introduction of deep neural networks

Figure 5:

3.2. DNN-assisted algorithm for spectrum reconstruction

Figure 6:

3.3. All data-driven spectrum reconstruction

Figure 7:

3.4. DNN co-design of encoder and decoder

Figure 8:

Figure 9:

3.5. Deep plug-and-play priors for spectral snapshot compressive imaging

Figure 10:

3.6. Optical diffractive network for spectral encoding

Figure 11:

4. Discussion

4.1. Spectrum reconstruction quality

Table 1:

4.2. Spectrum reconstruction speed

4.3. Spectrum reconstruction denoising

4.4. Potential applications for computational sensing

Figure 12:

5. Summary and outlook

Footnotes

Contributor Information

References

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases