Optical hyperspectral imaging in microscopy and spectroscopy – a review of data acquisition

Abstract

Rather than simply acting as a photographic camera capturing two-dimensional (x, y) intensity images or a spectrometer acquiring spectra (λ), a hyperspectral imager measures entire three-dimensional (x, y, λ) datacubes for multivariate analysis, providing structural, molecular, and functional information about biological cells or tissue with unprecedented detail. Such data also gives clinical insights for disease diagnosis and treatment. We summarize the principles underpinning this technology, highlight its practical implementation, and discuss its recent applications at microscopic to macroscopic scales.

1. Introduction

Spectral imaging collects information from three dimensions – two spatial (x, y) and one spectral (λ), resulting in a (x, y, λ) dataset which is typically referred to as a datacube. Depending on spectral resolution, the number of spectral bands, and the continuousness of the collected spectrum, spectral imaging is generally divided into multispectral imaging and hyperspectral imaging (HSI) (Table 1). It is noteworthy that there are two different criteria differentiating HSI from multispectral imaging. On the one hand, the criterion can be the number of spectral bands. HSI generally captures tens to hundreds of spectral bands while multispectral imaging has much less bands. On the other hand, the criterion can also be spectral band continuousness. HSI continuously measures the spectrum while multispectral imaging normally acquires non-continuous, spaced spectral bands. Compared to multispectral imaging, HSI collects more spectral information and thereby exhibits more sensitivity to subtle spectral variations. Although HSI was initially developed for astronomy [1-4] , widespread applications have been recently found in biomedicine, ranging from fundamental research [5] to clinical diagnostics [6, 7].

Table 1.

Comparison between multispectral imaging and hyperspectral imaging

	Multispectral imaging	Hyperspectral imaging
Spectral resolution	Low (normally > 10 nm)	High (normally <10 nm)
Criterion 1: Number of spectral bands	From three to tens	From tens to hundreds
Criterion 2: Spectral band continuousness	Normally spaced	continuity

The prominent advantage of HSI is its superior capability for discriminating multiple chemical species, particularly when their emission or reflectance spectra are partially overlapped. In biomedicine, the major impetus towards the widespread application of HSI is the ongoing development of exogenous biosensors, such as fluorescent proteins [8], quantum dots [9], and organic fluorophores [10], and the revelation of various endogenous chromophores, such as hemoglobin, lipid, and melanin. To fully utilize the information of these exogenous and endogenous contrast agents, detection systems must simultaneously monitor the spectroscopic variations of a combination of chromophores. This requirement comes from the fact that most cellular responses do not occur in isolation: rather, there is a complex sequence of events that occurs in response to cellular effectors. In order to determine the time sequence of such events in vivo, a biomedical HSI system must exhibit an appropriate combination of high spatial, spectral, and temporal resolution.

In this review, we review the fundamental principles of HSI, and discuss state-of-the-art HSI modalities in biomedicine, considering their pros and cons in light throughput, sectioning capability, computation complexity, and suitability for use in specific applications. In particular, we focus on the role of quantitative HSI in three typical biomedical applications: imaging Förster resonance energy transfer (FRET), hemodynamics, and ocular autofluorescence.

2. Fundamentals

2.1 Hyperspectral datacube acquisition strategies

Depending on the datacube acquisition mode, HSI is generally based on one of four distinct strategies (Fig. 1). The first strategy, referred to as point-scanning spectrometry, employs a linear array of detectors to measure spectral information (λ) at an instant, followed by scanning across all spatial locations (x, y) to fill out the datacube. The most common technique using this strategy is hyperspectral confocal microscopy [11, 12]. The second strategy, often referred to as pushbroom spectrometry, utilizes a 2D detector array to collect one (y, λ) slice of the datacube at once, so that only one spatial (x) dimension subsequently needs to be scanned. A representative modality is hyperspectral line-scanning microscopy [13, 14]. The third strategy, referred to as wavelength-scanning spectrometry, captures one (x, y) slice of the datacube at a time, and then scans across all wavelengths (λ). Representative methods are digital-light-processing-based imaging spectrometry [15], and acousto-optic [16] or liquid crystal tunable filter-based imaging spectroscopy [17]. The fourth strategy, referred to as snapshot imaging spectrometry, acquires the entire 3D datacube in a single exposure, leading to a dramatic improvement in light throughput compared to scanning-based systems. Within this category, representative technologies include image mapping spectrometry (IMS) [18-20], computed tomography imaging spectrometry (CTIS) [21], and coded aperture snapshot spectral imaging (CASSI) [22].

Fig. 1.

Datacube acquisition strategies in hyperspectral imaging. x, y, spatial coordinates; λ, wavelength.

2.2 Hyperspectral imaging advantages and linear spectral unmixing basics

Linear spectral unmixing is one of the basic methods used for HSI datacube analysis [23]. Given the (M × L) hyperspectral measurement matrix X, where L is the number of image pixels and M is the number of spectral wavelengths, linear spectral unmixing assumes that X represents a linear mixture of underlying constituents, i.e.,

$$X=SA+N.$$ (1)

Here S is the spectral component matrix and has dimensions (M × K), consisting of the absorption extinction coefficients of K postulated components at the M measurement wavelengths. A is the chromophore concentration matrix and has dimensions (K × L). And N is the matrix representing additive noise. Depending on whether the matrix S is known or not, linear spectral unmixing is based on one of two strategies: supervised or unsupervised. Supervised linear spectral unmixing requires a priori knowledge of the matrix S. Under this condition, the chromophore concentration matrix A can be estimated as [24]

$$\widehat{A}+S^{+}X.$$ (2)

Here  represents an estimate of matrix A and S⁺ is the Moore-Penrose pseudo-inverse of the matrix S. Obtaining a reasonable estimate for the concentration matrix A typically requires that the rank of the spectral component matrix S must be equal to or larger than the number of the postulated chromophores K. If any two or more of the chromophores are too spectrally similar, however, A will have a low condition number, and a regularized form of Eq. 2 will be more useful.

Compared to multispectral imaging systems with relatively low spectral resolution [25], a HSI system offers two advantages. First, HSI allows a generic experimental setting applicable to imaging a variety of chromophore combinations without the need to change filter settings. Provided that the detectors are shot noise limited, the accuracy of linear spectral unmixing is dependent on the condition number of the spectral component matrix S – the smaller the value, the lower the magnitude of the unmixing errors. Unlike with HSI, the condition number of the matrix S in multispectral imaging is sensitive to the positioning of spectral channels. For example, in a typical FRET imaging experiment, two spectral channels are used to unmix the constituent donor fluorophore, cyan fluorescent protein (CFP), and the acceptor fluorophore, green fluorescent protein (GFP). The entire fluorescence emission spectrum from this FRET pair is separated into two channels by a dichroic filter with a cut-on wavelength λ₀ (Fig. 2a). The condition number of the matrix S was calculated for different λ₀ (Fig. 2b). Assuming we require a condition number ≤ 5 for optimal spectral unmixing, the results shown in Fig. 2b indicate that this condition exists only when λ₀ falls within the spectral range 493 – 503 nm. For different chromophore combinations, this spectral range varies. For instance, the optimal dichroic cut-on wavelength for imaging a GFP/YFP (yellow fluorescent protein) FRET pair is within the spectral range 510 – 550 nm (condition number ≤ 5) (Fig. 2c-d). Achieving an optimized condition number for each fluorophore combination necessitates adjusting the positioning of spectral channels, thereby complicating the experimental procedure. In contrast, if these fluorophores are imaged by a HSI system continuously sampling the same spectral range with a 5 nm spectral channel bandwidth, the condition number of the matrix S remains optimized (e.g., condition number = 3.3 for CFP/GFP, and condition number = 2.0 for GFP/YFP), allowing the same system to image various FRET pairs without hardware adjustment.

Fig. 2.

Spectral unmixing of two fluorophores by a two-channel spectral imaging system. The entire fluorescence emission spectrum is separated into two channels by a dichroic filter with a cut-on wavelength λ₀. (a) Fluorescent emission spectra of CFP and GFP. (b) The condition number of the spectral component matrix for the CFP/GFP fluorophore combination at different λ₀. (c) Fluorescent emission spectra of GFP and YFP. (d) The condition number of the spectral component matrix for the GFP/YFP fluorophore combination at different λ₀.

A second advantage of HSI is that it facilitates the unsupervised linear spectral unmixing. To construct the spectral component matrix S, the supervised linear spectral unmixing needs accurate information about the number of chromophores in the mixture and their emission or reflectance spectra. However, if such knowledge is either difficult to obtain, as when the mixture contains unknown sources, e.g., for in vivo tissue measurements, or the knowledge is unreliable to use, as when the chromophores’ spectra are subject to experimental and biological variability, the supervised spectral unmixing falters. In these circumstances, HSI's continuous spectral sampling becomes necessary because optimizing the positions of spectral channels as described earlier is impracticable. In addition, to estimate both the spectral component matrix S and the chromophore concentration matrix A, unsupervised linear spectral unmixing algorithms must be employed. Unlike its supervised counterpart, the unsupervised linear spectral unmixing does not rely on exact knowledge about the spectral component matrix S; rather, S is unknown or only approximately known. Representative unsupervised spectral unmixing algorithms are principal components analysis (PCA) [26], independent component analysis (ICA) [27], and non-negative matrix factorization (NMF) [28, 29].

Among these algorithms, NMF attracts the most attention in biomedical applications because it provides the most easily interpretable results which are consistent with physical reality [30, 31]. Although NMF was initially developed for remote sensing, the extraordinary versatility of the method lends itself to applications far from the imaging realm. More recently, for example, NMF has been exploited in various applications in computational biology and bioinformatics [32-34].

Briefly, in Eq. 1, since the matrix A represents concentrations, we can assume A to be non-negative. Additionally, because the spectral component matrix S denotes signal intensities, the values of S should also be non-negative. Therefore the basic NMF algorithm imposes two constraints,

$$A\ge 0\text{and}S\ge 0,$$ (3)

on the matrix factorization of the hyperspectral measurement X. However, in practice, the nonnegative constraint alone may not be sufficient to generate an adequate result [35]. In these cases, additional constraints, such as sparsity [36], piecewise smoothness of spectral data [30], and forcing the minimum amplitude of the spectral data to be zero [37], are also needed as regularization. The NMF paradigms employing these additional constraints are generally referred to as constrained non-negative matrix factorization (cNMF).

Moreover, traditionally the NMF algorithm is initialized with random spectra and then converges to a solution by minimizing an error function which describes the similarity between the hyperspectral measurement X and the matrix product SA. Thus no information about the structure of the spectral shape of the components is typically employed in the basic NMF algorithm. However, it is important to remember that, in contrast to the supervised spectral unmixing described earlier, the solution of NMF might not converge to a global minimum, and hence might not provide a physiologically reasonable estimate about chromophore concentrations. In fact, even the optimum number of postulated chromophores to use in the matrix S might not be obvious, and often it must be user-defined and evaluated. Therefore, to get a more reliable estimate of constituents’ spectra and concentration, the NMF technique may include a supervision step that models a subset of the measured spectra [31, 38]. Multiple applications of the algorithm, with a variety of random or partially supervised initial conditions, and with statistical and empirical evaluation of the results, are sometimes required to achieve a useful solution. As an example, a real-world biomedical application of the supervised NMF algorithm is discussed in Section 4.3. Additionally, interested readers can also refer [33] and references therein for more details.

3. Major implementations of hyperspectral imaging

In this section, we review state-of-the-art biomedical HSI modalities, classified by their datacube acquisition strategy as discussed in Section 2.1. For each modality we compare advantages and limitations regarding light throughput, sectioning capability, computation complexity, and applicability.

3.1 Hyperspectral confocal microscopy

A hyperspectral confocal microscope is a point-scanning spectrometer with a typical optical setup as shown in Fig. 3. The microscope is based on a standard laser scanning confocal microscope. However, rather than directly collecting the fluorescence by a photodetector, a hyperspectral confocal microscope first disperses the light with a spectral dispersion unit, such as a prism or grating, and then images the spectrum onto a linear detector array. Due to its diffraction-limited spatial resolution and high spectral resolution (3 nm [11]), hyperspectral confocal microscopy has been widely used in live cell multiplexed imaging [5, 39]. The number of spectral channels varies from 32 in commercial microscopes (LSM 780 microscope from Carl Zeiss or C2+ microscope from Nikon) to 512 in a custom-built microscope [11], measuring photons with wavelengths in the visible light range.

Fig. 3.

Typical optical setup of a hyperspectral confocal microscope. At the detection side, the photons are dispersed by a spectral dispersion unit, such as a prism, and imaged by a linear detector array.

The major advantage that hyperspectral confocal microscopy provides is its intrinsic depth-resolved spectral imaging capability, reducing the crosstalk among adjacent depth layers and generating a four-dimensional datacube (x, y, z, λ). However, since the acquisition of this datacube is mostly achieved via raster scanning, hyperspectral confocal microscopy suffers from the trade-off between signal-to-noise ratio (SNR) and microscope's temporal resolution, particularly when monitoring low light level cellular dynamic processes [40]. Although, to some extent, confocal microscopy can compensate for this trade-off by using a high intensity excitation source, this increases photobleaching and photo-toxicity to the sample [41-43]. Moreover, once the fluorophores are boosted to their saturation excitation state, a situation that is commonly reached by current confocal microscopes, even the method of using high power excitation sources fails because one could not further increase the fluorescence emission rate [44].

3.2 Hyperspectral line-scanning microscopy

A hyperspectral line-scanning microscope is a pushbroom spectrometer with a typical optical setup as shown in Fig. 4. By using a line-focusing lens, such as a Powell lens [13], the excitation light is focused into a uniform line at the sample. The excited fluorescence is then collected by the same objective and imaged onto the entrance slit of a spectrometer. The line image is dispersed, and its spectral components are imaged onto a 2D detector array, such as a CCD camera. By scanning the sample along the other spatial axis (x axis in Fig. 4), an entire (x, y, λ) datacube is acquired. Compared to hyperspectral point-scanning confocal microscopy, hyperspectral line-scanning microscopy scans only along one spatial dimension (x axis in Fig. 4), thereby accelerating the datacube acquisition speed by a factor of N_y with no loss in SNR. However, since the microscope operates with slits rather than pinholes, the gain in datacube acquisition speed is accomplished at the cost of a decreased spatial resolution and image contrast [45]. The spectral range of a hyperspectral line-scanning microscope varies from visible light [13, 46-49] to near infrared [50, 51].

Fig. 4.

Typical optical setup of a hyperspectral line-scanning microscope. The Powell lens generates a uniform excitation line. Figure adapted from [13].

3.3 Tunable-filter-based imaging spectrometry (TF-IS) and digital-light-processing-based imaging spectrometry (DLP-IS)

A tunable-filter-based imaging spectrometer (TF-IS) is a wavelength-scanning spectrometer. The most important tunable filters used in this category are acousto-optic tunable filters (AOTF) [16, 52], liquid-crystal tunable filters (LCTF) [17, 53-56], filter array [57], and linear variable filters (LVF) [58]. There are two types of tunable-filter-based imaging spectrometers: one type, referred to as excitation-side TF-IS, mounts tunable filters in front of a light source, and sequentially illuminates the sample with different wavelengths [57, 59, 60]. The corresponding monochromatic scenes are then captured by a 2D detector array. The other type, referred to as emission-side TF-IS, mounts tunable filters in front of a detector, followed by scanning the spectral window to image selected wavelengths [53, 55]. Representative commercial HSI tunable filters are the Gooch & Housego HSi-300 and the PerkinElmer VariSpec LC Tunable Filter, with typical wavelength tuning response times of 100 μs and 50 ms, respectively. The measured spectra vary from visible light [16, 56] to near infra-red [54]. It is worth mentioning that, in reflectance imaging spectrometry, the excitation-side filtering techniques are advantageous over the emission-side counterparts. This is because, to achieve a given SNR, the excitation-side filtering requires less dose of illumination, thereby presenting the sample from being altered by the measurement.

Another important embodiment of excitation-side wavelength-scanning spectrometry in biomedicine is the digital-light-processing-based imaging spectrometer (DLP-IS). Compared to the tunable-filter-based imaging spectrometry, DLP-IS enables higher wavelength tuning speed (up to 12.5 kHz, Gooch & Housego OL 490 Agile Light) and generates arbitrary illumination spectra [15, 61]. DLP-IS has been demonstrated in a variety of clinical and surgical applications, such as non-invasive measuring of chemical contents in tissue and monitoring wound healing [62-64]. Commercial HSI systems utilizing DLP-IS include the OneLight Spectra and the Gooch & Housego OL 490 Agile Light Source.

Compared to hyperspectral point-scanning confocal microscopy, the datacube acquisition speed of TF-IS and DLP-IS is faster by a factor of N_x × N_y, because no scanning is required along the spatial dimensions, but loses a factor of N_λ due to the need for wavelength scanning. However, the drawback of these techniques is a lack of intrinsic optical sectioning. To achieve depth-resolved imaging, TF-IS and DLP-IS must be combined with auxiliary techniques, such as structured illumination [65].

3.5 Snapshot imaging spectrometry

Snapshot imaging spectrometry captures the entire datacube within a single integration event, without employing any scanning mechanism. The major advantage of a snapshot imaging spectrometer is that it allows full optical throughput, an improvement of N_xN_yN_λ over that of a point-scanning imaging spectrometer [66]. Although a number of snapshot imaging spectrometry techniques have been developed, many of these have difficulties in adapting to high resolution imaging [67]. Only a few have been implemented in biomedical imaging applications: Computed Tomography Imaging Spectrometers (CTIS) [21], Coded Aperture Snapshot Spectral imagers (CASSI), and Image Mapping Spectrometers (IMS).

Rather than the one-dimensional linear disperser used in conventional dispersive spectrometers, a CTIS uses a 2D computer-generated-hologram disperser, projecting the mixture of the spatial and spectral data at different view angles onto the detector (Fig. 5a). Tomographic reconstruction algorithms are then used to estimate the object datacube. A state-of-the-art CTIS can measure a datacube with 203×203×55 (x, y, λ) voxels and cover the entire visible light range [68]. A major advantage of CTIS is its relative compactness, allowing it to couple onto a variety of biomedical imaging instruments, such as a microscope [21, 68, 69], and a fundus camera [31]. However, CTIS has been plagued by computational complexity, calibration difficulty, and measurement artifacts [70].

Fig. 5.

System schematics of (a) Computed Tomography Imaging Spectrometers (CTIS), Coded Aperture Snapshot Spectral imagers (CASSI), and (c) Image Mapping Spectrometers (IMS).

CASSI takes advantage of compressed sensing to accomplish datacube acquisition in a snapshot format. Analogous to CTIS, CASSI can also be considered as a generalization of slit spectrometer, however with the entrance slit replaced by a 2D field stop (Fig. 5b). Inside this field stop, CASSI inserts a random binary mask to modulate the spatial-spectral projection at the detector so that each wavelength of the datacube experiences a shifted modulation code. If this system matrix and object datacube can be made to satisfy the conditions of compressed sensing, the datacube can be recovered through compressed sensing algorithms [71]. Advantages of CASSI are its compactness and the use of a modest size detector array, allowing a high datacube acquisition speed. For example, a state-of-the-art CASSI can acquire data at 30 fps for datacubes of size 248×248×33 (x, y, λ) and measure spectra in the visible light range [72]. The disadvantages of CASSI include its computation complexity, making it difficult to recover accurate datacubes in real time, and the requirement that the object must be sparse in the gradient domain [72]. In biomedicine, the integration of CASSI with microscopy has been demonstrated in high-resolution fluorescence imaging [73, 74].

Rather than dealing with mixed spatial and spectral data as in CTIS and CASSI, an IMS avoids spatial-spectral crosstalk at the detector by utilizing a field division unit, referred to as an image mapper [75]. An image mapper consists of hundreds of mirror facets, each with a 2D tilt angle. These mirror facets cut the field into slices and reflect them towards different locations at the detector to create void spaces between adjacent sliced images (Fig. 5c). The sliced images are dispersed by a field dispersion unit, such as a prism. The resulting spectra then occupy previously created void spaces at the detector. In this way, each camera's pixel is unambiguously encoded with the spatial and spectral information of the sample. The datacube size that a state-of-the-art IMS measures is 350×350×46 (x, y, λ) with a frame rate up to 7.6 fps [40]. The measured spectral range can be either visible [40] or near infrared [76]. Compared to CTIS and CASSI, the IMS has two advantages: First, the IMS involves little computational cost, allowing the real-time display of acquired datacubes; second, the IMS does not pose any limitations on the imaged objects, thereby maximizing its compatibility with different biomedical imaging modalities, such as microscopy [40], endoscopy [77], ophthalmoscopy [78], and macroscopy [6]. The primary drawback of the IMS lies in the need for high precision fabrication of its core component, the image mapper, and the need for large-format cameras, which normally have a low frame rate. However, the potential marriage between the IMS and scientific CMOS cameras may provide a solution for demanding dynamic imaging applications.

3.6 Comparison of hyperspectral imaging modalities

We compare hyperspectral imaging modalities in Table. 2. Here the throughput is defined as the time that a voxel can be continuously seen by the instrument during datacube acquisition and scaled to the time seen by snapshot imaging spectrometry. In shot-noise-limited systems, higher throughput results in higher SNR at a given imaging speed; equivalently speaking, higher throughput leads to faster datacube acquisition for a given SNR. This advantage, referred to as the snapshot advantage [66], becomes dominated when measuring datacubes with a large number of voxels. It is worth nothing that although hyperspectral confocal microscopy and line-scanning microscopy have relatively low throughput, they provide an intrinsic optical sectioning capability, enabling 4D (x, y, z, λ) imaging. By contrast, to achieve optical sectioning with the other three modalities, one must rely on auxiliary techniques, such as structured illumination [44].

Table 2.

Comparison of hyperspectral imaging modalities

	Hyperspectral confocal microscopy	Hyperspectral line-scanning microscopy	Tunable-filter-based imaging spectrometry	Digital-light- processing-based imaging spectrometry	Snapshot imaging spectrometry
Hyperspectral imaging strategy	Point-scanning	Pushbroom	Wavelength-scanning (excitation & emission sides)	Wavelength-scanning (excitation side)	Snapshot
Throughput*	1/(NxNy)	1/Ny	1/Nλ	1/Nλ	1
Optical sectioning capability	High	Medium	Not without auxiliary techniques	Not without auxiliary techniques	Not without auxiliary techniques
Computation complexity	Low	Low	Low	Low	High for CTIS and CASSI;
Limitation on applicable imaging objects	None	None	None	None	Low for IMS Missing-cone limitation for CTIS [79]; Gradient- domain sparsity limitation for CASSI [72];

^*Herein the throughput is defined as the time that a voxel can be continuously seen by the instrument during datacube acquisition and is scaled to the time seen by snapshot imaging spectrometry.

4. Quantitative hyperspectral imaging, from organelles to organs

4.1 Hyperspectral imaging of FRET in living cells

The expanding library of fluorescent probes available for fluorescence microscopy has led to an ever-increasing range of applications accessible to live cell microscopy. Advanced fluorescent labeling techniques allow simultaneously staining cells with up to 17 different fluorophores [80]. Given the limited range of visible and NIR fluorophores, experiments that utilize multiple probes often suffer from significant emission spectral overlap. In multicolor cellular imaging, HSI in combination with linear spectral unmixing becomes an indispensable tool because it enables mapping a variety of cells or cellular organelles with high sensitivity. As representative images, Fig. 6 shows hyperspectral imaging results of a triple-labeled HeLa cell expressing ECFP in the mitochondria, EGFP on the plasma membrane, and SYFP in the nucleus. In the conventional RGB color image, all three fluorophores appear green and thus cannot be distinguished (Fig. 6a). However, by using a HSI system, an image mapping spectrometer (IMS), these fluorophores can be spectrally unmixed, with their sub-cellular locations unambiguously revealed in separate channels (Fig. 6c-e).

Fig. 6.

Hyperspectral imaging of ECFP, EGFP, and SYFP in a triple-labeled HeLa cell. (a) Reference image captured by a color camera. (b) Emission spectra of ECFP, EGFP, and SYFP (Life Technologies). (c-e) Unmixed images showing the sub-cellular locations of ECFP, EGFP, and SYFP. (f) Merged image of c-e. ECFP, enhanced cyan fluorescence protein; EGFP, enhanced green fluorescence protein; SYFP, super yellow fluorescence protein.

FRET microscopy, an important application of HSI in live cell imaging, serves as a molecular ruler in measuring intermolecular distances or determining the formation of molecular complexes [81]. An efficient FRET signal requires considerable spectral overlap between the emission spectrum of the donor and the excitation spectrum of the acceptor, a requirement that generally results in a significant spectral overlap between the emission spectra of the donor and the acceptor. As discussed in Section 2.2, the most basic FRET imaging requires two spectral channels. However, the accuracy of subsequent spectral unmixing is sensitive to the spectral positioning of detection channels. Furthermore, two channel FRET imaging assumes that there are only two fluorophores within the imaging FOV, a hypothesis that is problematic when the autofluorescence intensity becomes comparable to that of the FRET signals. HSI can overcome these limitations by continuously sampling the emission spectra with a fine spectral resolution. Combined with blind source spectral unmixing algorithms, such as non-negative matrix factorization, HSI can effectively remove the autofluorescence background, resulting in a significant increase in unmixing accuracy and image contrast [82].

However, for scanning-based HSI modalities, the cost of achieving these advantages in FRET-involved imaging is a reduced temporal resolution at a given SNR. The trade-off among the spatial, temporal, and spectral resolutions normally hinders these instruments from assaying dynamic cellular processes. In these cases, snapshot HSI imagers have an intrinsic advantage because all the voxels in the datacube are acquired simultaneously (Section 3.6), yielding an ideal scenario that allows one to monitor the interplay of multiple biosensors in real time. For example, a recent study utilizes an image mapping spectrometer (IMS) to study the roles of cellular second messengers [Ca²⁺]_i and cAMP in regulating glucose-stimulated insulin secretion in pancreatic β-cells [40]. To monitor the temporal dynamics of [Ca²⁺]_i and cAMP, the β-cells were transfected with a fluorescent protein, GCaMP5G, and a FRET-based cAMP sensor, ^T-Epac-^VV, a fusion protein containing the donor fluorophore, mTurquoise, and the acceptor fluorophore, cpVenus-Venus. As shown in Fig. 7a, the excitation and emission spectra of these fluorescent sensors overlap significantly. By using the hyperspectral imager IMS, the spatial-temporal interaction between glucose-induced [Ca²⁺]_i and cAMP oscillations was revealed with sub-micron spatial resolution and sub-second temporal resolution (Fig. 7b-d).

Fig. 7.

Simultaneous imaging of [Ca²⁺]_i and cAMP signaling in β-cells by a snapshot hyperspectral imager IMS. (a) Emission spectra of fluorophore GCaMP5G, mTurquoise, and cpVenus-Venus. (b) Unmixed spectral component images. (c) [Ca²⁺]_i (green) and cAMP (blue) oscillations after 5 minutes of stimulation with 20 mM glucose and 20 mM TEA stimulation collected at 2 fps. (d) Time-resolved components of ^T-Epac-^VV (mTurquoise in blue, cpVenus-Venus in yellow) from the cells shown in c. Reprinted with permission from [40].

4.2. Hyperspectral imaging of tissue hemodynamics

The ability to visualize hemodynamics in vivo is important in preclinical and clinical studies because it aids our understanding of the way that oxygen is provided and consumed in the tissue, for both healthy and diseased conditions. The key measure of hemodynamics is the oxygen saturation of hemoglobin (sO₂), which is defined as the ratio of oxy-hemoglobin concentration to the total hemoglobin concentration:

$$s{O}_2=\frac{C_{Hb{O}_2}}{C_{Hb{O}_2}+C_{HHb}}.$$ (4)

where C_HbO2 and C_HHb are the concentrations of oxy-hemoglobin and deoxy-hemoglobin, respectively. Since oxy-hemoglobin and deoxy-hemoglobin have different absorption spectra, HSI can discriminate these two chromophores and quantify their concentration in vivo without labeling.

Most optical in vivo sO₂ measurements are carried out in reflectance mode, i.e., the sample is illuminated by a broadband light source and the reflected light is collected and analyzed. The measured spectra are a mixture of the unaffected photons which experience direct reflection from the sample surface, and the affected photons which have interacted with the sample via absorption and scattering. While the specular reflection normally carries the original spectrum of the light source, both absorption and scattering processes are strong functions of wavelength [83]. The modified Beer-Lambert law can be used to describe the absorption [84, 85]:

$$\frac{I}{I_0}=\exp (-{\mu }_{a}d+G),$$ (5)

where I is the detected light intensity, I₀ is the incident light intensity, d is the mean pathlength traveled by light in tissue (a function of wavelength and tissue optical properties), G is a geometry-dependent factor, and μ_a is the absorption coefficient of the tissue as given by:

$${\mu }_a=\sum _i{\xi }_i{\mathcal{C}}_i.$$ (6)

Here ξ_i is the wavelength-dependent molar extinction coefficient of a particular chromophore in the mixture, and c_i is its concentration. For in vivo measurements, both I₀ and G are difficult to quantify. The common approach to circumvent this problem is to measure the changes in the detected signal at time t relative to that at initial time t₀ or a reference state, using

$$\begin{array}{c}\hfill \ln \left[I\left(t\right)\right]-\ln \left[I\left(t_0\right)\right]=-[{\mu }_a\left(t\right)-{\mu }_a\left(t_0\right)]d\hfill \\ \hfill \Delta \ln \left(I\right)=-\sum _{i}d{\xi }_i\Delta {\mathcal{C}}_i.\hfill \end{array}$$ (7)

Under the assumptions that all chromophores are known and the mean pathlength d is known for each wavelength, the changes in chromophores’ concentrations can be calculated by utilizing the supervised linear spectral unmixing model (Eq. 2) [64, 86]. Furthermore, given the estimated initial concentrations for oxy- and deoxy-hemoglobin, the change in absolute oxygen saturation can be derived by

$$\Delta s{O}_2=\frac{C_{Hb{O}_2}\left(t\right)}{C_{Hb{O}_2}\left(t\right)+C_{HHb}\left(t\right)}-\frac{C_{Hb{O}_2}\left(t_0\right)}{C_{Hb{O}_2}\left(t_0\right)+C_{HHb}\left(t_0\right)}.$$ (8)

However, the estimation of wavelength-dependent pathlength d in Eq. 7 is not trivial. In practice, it is normally calculated by a Monte-Carlo model of light propagation in tissue [85, 87, 88]. However, assumptions have to be made on the reduced scattering parameter (μ′_s), a fact which can result in crosstalk and errors in deriving ΔC_HbO2 and ΔC_HHb [87, 89-91].

To eliminate the need to approximate d in HSI, there are two general approaches. The first method is to utilize unsupervised spectral unmixing algorithms, such as non-negative matrix factorization (NMF) and principle component analysis (PCA), which allows the scattering-modified components’ spectra (d(λ)ξ_i(λ) in Eq. 7) to vary to obtain the best fit and most realistic spectrum [31, 92]. Utilizing this strategy, the quantification of hemoglobin has been demonstrated in imaging human skin [93]. The second method is to employ diffuse optical imaging techniques to separate optical absorption from optical scattering effects by characterizing the remitted or transmitted light field either in the time or spatial domain [94]. A representative technique within this category is spatial frequency-domain imaging (SFDI), which modulates the light source and projects different spatial frequency patterns onto tissue, followed by comparing the measurements to a mathematical model of light propagation in that tissue [94-97]. The underlying principle of SFDI is that lower spatial frequencies have longer optical pathlengths and tend to be more attenuated by absorption, while the higher spatial frequencies have shorter optical pathlengths and are more sensitive to scattering. Combined with HSI, the ability of SFDI to quantify sO₂ has been demonstrated in imaging the rat cortex [69, 95], and representative images are shown in Fig. 8.

Fig. 8.

Chromophore maps for a mouse cortex at and surrounding the site of an inflicted injury. The mouse cortex was imaged using spatial frequency domain imaging (SFDI) and a computed-tomography imaging spectrometer (CTIS). (a) Mouse cortex reflectance image at 650 nm. The top of the image is the anterior side of the brain. (b) Oxy-hemoglobin concentration. (c) Deoxy-hemoglobin concentration. (d) Water percentage. (e) Total hemoglobin concentration. (f) Oxygen saturation percentage. Reprinted with permission from [69].

4.3. Hyperspectral imaging of autofluorescence in ocular tissue

Hyperspectral autofluorescence imaging has been applied to a fundamental problem in the cell biology and biochemistry of an important ocular tissue, the retinal pigment epithelium (RPE). The fluorophores of the RPE, a species of vitamin A adducts known as bis-retinoids, have long been intensively studied in the laboratory as byproducts of the visual cycle [98-100]. They aggregate in granules of material known as lipofuscin. Autofluorescence attributable to lipofuscin was imaged in the living human eye by Delori et al. using spectrophotometry and by Fitzke et al. using scanning laser ophthalmoscopy [101, 102]. The fundamental observation that RPE lipofuscin increases with age [98] raises the possibility of RPE toxicity playing a role in biogenesis of age-related macular degeneration (AMD). The extensive literature on both sides attests to the importance of resolving this question, and unraveling RPE bis-retinoid biology [103, 104].

In the last decade, there has been a rapid expansion in biochemical knowledge about these bis-retinoid compounds [105, 106]. Clinically, a wealth of novel descriptive and diagnostic information has been harvested from autofluorescence imaging of AMD and inherited retinal degenerations. In particular, abnormal patterns of increased and decreased autofluorescence can be diagnostic in these disorders, and can serve as markers for disease progression. Multimodal imaging with autofluorescence and spectral-domain optical coherence tomography (SD-OCT) leverages the diagnostic and prognostic power of these images even further. Recently, the ability to standardize autofluorescence across patient populations via quantitative autofluorescence offers even better understanding of lipofuscin biology in the clinic [107, 108].

What has been lacking until very recently has been a merger of laboratory biochemistry and the identification, quantification, and localization of the important fluorophore species in human RPE tissue itself. Hyperspectral autofluorescence imaging is now being applied to this task, guided by fundamental new tissue techniques [109], using mathematical modeling of RPE autofluorescence with Gaussian mixture models and signal recovery with NMF. The analysis pipeline for extracting individual spectral components and their spatial localizations from hyperspectral autofluorescence emission data proceeds in three steps. Each step guides the next, and each provides successively more precise spectral recovery.

Step 1: Recovery of RPE autofluorescence signal from tissue flatmount

As an example, Fig. 9a shows an RPE autofluorescence image (exc. 436 nm) from a 49-year-old female donor. The sample's mean spectral data were acquired in the marked regions using a Nuance FX camera (Caliper Life Sciences, US) between 420-720 nm in 10 nm intervals. The measured BrM and the RPE and BrM combined spectra are shown in Fig. 9b. The isolated mean RPE signal was calculated by subtracting the measured BrM spectrum from the RPE and BrM combined spectrum.

Fig. 9.

Recovery of RPE auto-fluorescence from ocular tissue flatmount. (a) RGB composite autofluorescence image from 40× field (49-year-old female donor, exc. 436 nm). (b) Autofluorescence emission curves. (c) Gaussian component fitting to RPE autofluorescence signal.

Step 2: Gaussian component fitting to RPE autofluorescence signal

As shown in Fig. 9c, we fitted the RPE spectrum to a Gaussian model. Individual Gaussians of the model are centered at approximately 575, 600, 640, and 700 nm, respectively. The sum of the Gaussians (solid pink) closely follows the contours of the RPE signal (black dots).

Step 3: Non-negative matrix factorization (NMF)

The standard NMF algorithm is initialized with random spectra and then converges to an approximate solution by minimizing an error criterion. These solutions may or may not be physiologically reasonable, even though they satisfy the mathematical equations. In this case, however, we had already measured the Bruch's membrane (BrM) spectrum in Step 1 and had also recovered four reasonable Gaussian candidate spectra from each RPE sample in Step 2. We therefore chose to initialize the algorithm with these five explicit spectra and then let the algorithm converge to a solution. A typical solution by NMF factorization of a hyperspectral data cube with BrM and RPE Gaussian initializations is shown in Fig. 10a. In addition to the five major spectra recovered, the other part of the data factorization is a series of five chromophore concentration images (Fig. 10b-f), which display the spatial localization of each recovered signal. The recovered spectra themselves are interpretable as single major peaks, but other details are present that may represent error, noise, or more complex component signals.

Fig. 10.

Hyperspectral imaging of ocular tissue flatmount. (a) NMF solution with Gaussian initial conditions. (b-f) Recovered five chromophore concentration maps. Four localize adjacent to nuclei, where lipofuscin is expected. These four signals peak at approximately 560 nm, 600 nm, 640 nm, and 700 nm. The signal at 600 nm, C5, appears to be the weakest. Signals C3 and C4 have secondary shorter wavelength minor peaks, suggesting additional compounds. C2 localizes mostly to bare BrM, with the shortest wavelength peak. The amplitudes of the spectra are the mean emissions from that source over the 40× field.

Hyperspectral autofluorescence imaging analysis of flatmounts of pure human RPE/BrM ex vivo with a Gaussian mixture/NMF model thus enabled consistent recovery of candidate spectra for at least four individual RPE and one BrM fluorophore emission signals, with well-defined emission maxima and with histologically plausible localizations across age, gender, retinal locations, and excitation wavelengths. The biochemical sources of these signals have yet to be determined. However, one of the four had an emission maximum at 600 nm, and thus could be a well-known bisretinoid, A2E. These spectra should now be matched to their corresponding biochemical components with techniques such as imaging mass spectroscopy. Determining the actual abundant source molecules that produce these signals will be important in understanding RPE physiology.

5. Conclusions and outlook

In summary, HSI has broad applications ranging from laboratory research to clinical studies, yet future endeavors are still needed to further mature this technology. In this review, both fundamental principles of HSI and its state-of-art biomedical implementation were detailed, and three biomedical applications of HSI were discussed.

We envision several trends for HSI in biomedicine. First, the integration of HSI with conventional bio-imaging devices, such as microscopes [40], endoscopes [77], fundus cameras [78], and surgical laparoscopes [110], will produce a wealth of hidden information about biological samples, opening up new areas of investigation both for in vitro and in vivo imaging. In particular, the marriage between HSI and ophthalmoscopy will help us identify the distribution and spectral signature of various chromophores and fluorophores associated with ocular diseases, such as AMD and glaucoma. For example, drusen, the hallmark lesions of AMD, are biochemically heterogeneous and are key to understanding this disease [111]. However, their composition cannot be determined in vivo with even the current highest resolution imaging techniques [92, 112]. By contrast, HSI data can potentially offer a vision of spectral biopsy of retina by dissecting the spectral reflectance and autofluorescence signatures form drusen and other AMD lesions. The insights so obtained will be of high value in clinical diagnosis and treatment. In addition, such a system will accelerate translations research with sensitive and early outcome testing of prospective therapeutic agents, saving sight and thereby providing benefits to society. Second, although most biomedical HSI needs are currently fullfilled by scanning-based devices, snapshot hyperspectral imagers will become the mainstream. Snapshot instruments not only promise to improve measurement SNR for most situations but also add capabilities beyond those of current devices. For instance, in brain functional imaging, many questions remain about the neurovascular coupling [113], especially regarding the spatiotemporal features of blood oxygen level dependent (BOLD) signals and their relevance to neuronal activity. Scanning-based HSI systems are not suitable for such a demanding application because the time scale of neuron actions can be as short as a millisecond [114]. Significant progress in elucidating the relation between BOLD signals and neuronal activities will be possible with the simultaneous monitoring provided by snapshot HSI systems. Last but not least, advances in developing new un-supervised spectral unmixing algorithms [37, 115, 116] will make the optimization tasks more accurate and more efficient, thereby facilitating a full utilization of the rich spatio-spectral information measured by HSI. Among these analysis techniques, NMF-based algorithms emerge as spotlights in biomedical applications because they provide the most easily interpretable results that agree with physiological reality. Interested readers may refer to a more specific article for the cutting-edge NMF algorithms and their implementation as a software package [34].