mHealth hyperspectral learning for instantaneous spatiospectral imaging of hemodynamics

Abstract

Hyperspectral imaging acquires data in both the spatial and frequency domains to offer abundant physical or biological information. However, conventional hyperspectral imaging has intrinsic limitations of bulky instruments, slow data acquisition rate, and spatiospectral trade-off. Here we introduce hyperspectral learning for snapshot hyperspectral imaging in which sampled hyperspectral data in a small subarea are incorporated into a learning algorithm to recover the hypercube. Hyperspectral learning exploits the idea that a photograph is more than merely a picture and contains detailed spectral information. A small sampling of hyperspectral data enables spectrally informed learning to recover a hypercube from a red–green–blue (RGB) image without complete hyperspectral measurements. Hyperspectral learning is capable of recovering full spectroscopic resolution in the hypercube, comparable to high spectral resolutions of scientific spectrometers. Hyperspectral learning also enables ultrafast dynamic imaging, leveraging ultraslow video recording in an off-the-shelf smartphone, given that a video comprises a time series of multiple RGB images. To demonstrate its versatility, an experimental model of vascular development is used to extract hemodynamic parameters via statistical and deep learning approaches. Subsequently, the hemodynamics of peripheral microcirculation is assessed at an ultrafast temporal resolution up to a millisecond, using a conventional smartphone camera. This spectrally informed learning method is analogous to compressed sensing; however, it further allows for reliable hypercube recovery and key feature extractions with a transparent learning algorithm. This learning-powered snapshot hyperspectral imaging method yields high spectral and temporal resolutions and eliminates the spatiospectral trade-off, offering simple hardware requirements and potential applications of various machine learning techniques.

Significance Statement.

The development of ultrafast hyperspectral imaging systems is an active area of research for a variety of applications from biomedical to defense domains. However, such hardware-focused approaches have been challenging mainly because of the intrinsic trade-off between spatial and spectral resolutions and the slow rates of data acquisition. Hyperspectral learning takes inspiration from the idea that a photograph is more than merely a picture and indeed contains rich spectral information. Hyperspectral learning enables the recovery of spectral information with high spectral and temporal resolutions from red–green–blue color values acquired using a conventional camera. The main advantages of hyperspectral learning include hardware simplicity, no trade-off between spatial and spectral resolutions, high temporal resolution, and machine learning augmentation for snapshot imaging.

Introduction

Hyperspectral (with a high spectral resolution of ∼10 nm) or multispectral (with several spectral bands of ∼50 nm) imaging systems acquire a hyperspectral image data set (hypercube)—a 3D data set of spectral intensity in spatial coordinates. Subsequently, both the spatial and spectral data are processed (1–3). Hyperspectral imaging technologies offer extensive physical and biological information in stationary or dynamic samples, ranging from microscopic settings to airborne remote sensing environments, for a variety of applications in geology, mineralogy, agriculture, environmental science, astronomy, forensic medicine, defense, security, and biomedicine (4–9). Notably, hyperspectral imaging technologies have been reinvigorated through recent advances in data-driven machine learning (2, 10–14). For example, deep learning approaches have enabled the effective processing of extremely large hypercube data for classical imaging tasks (11, 15–17) and allowed for the optimization of hypercube acquisition to achieve specific tasks and objectives (18–22). Data fusion of complementary images with high spectral or high spatial resolutions (23, 24) and neural networks of improving spatial resolutions (25) can overcome the intrinsic trade-off between spatial and spectral resolutions. However, conventional hyperspectral imaging systems still face the intrinsic limitations: bulky instruments, slow data acquisition rates, low detection efficacy (i.e. low signal-to-noise ratio), and motion artifacts (1, 2).

Typically, hyperspectral imaging systems rely on mechanical scanning elements either in the spectral or spatial domains. In particular, spectral scanning systems employ a number of narrow bandpass spectral filters or dispersive optical components, whereas point scanning and line scanning systems rely on mechanical translational components that require high precision (1, 2, 6, 7, 9). Thus, these scanning elements result in bulky instruments and yield suboptimal temporal resolutions. In particular, prolonged time of data acquisition time fundamentally limits dynamic imaging with a high temporal resolution. In this respect, the development of snapshot imaging technologies capable of acquiring a hypercube in a single-shot manner has been an active area of research (summarized in Table S1). The most common configuration used for snapshot imaging involves simultaneously capturing multiple images with different spectral bands using a large-area image sensor (26–32). In particular, large-area image sensor-based snapshot imaging is beneficial for reducing the acquisition time (2, 33, 34). Other snapshot imaging technologies employ dispersion patterns or coded apertures projecting irradiance mixed with spatial and spectral information to further enhance the light collection efficiency and readout rate (26, 27, 35–37). Subsequently, the modulated projection comprising spatial and spectral information is reconstructed into a hypercube by utilizing computational algorithms such as compressed (or compressive) sensing (32, 38, 39), Fourier transformation (40, 41), or deep learning (19, 20, 42–48).

However, previously developed hyperspectral imaging technologies with a snapshot ability face several limitations (1, 2, 9, 46, 49). First, typical snapshot systems are limited by the intrinsic trade-off that must be made between the spectral and spatial resolutions; an improvement in spatial resolution causes a deterioration in the number of spectral bands, thereby compromising the spectral resolution or the spatial resolution (or imaging area). Second, snapshot imaging systems are sensitive to light conditions and imaging configurations, thereby introducing significant errors in field applications. Third, the hyperspectral filter arrays, dispersion patterns, and coded apertures require high-precision fabrication or nanofabrication, including precision alignment of array components, optimized miniaturization, integration with pixel-level filters, and customized calibrations, all of which inhibit manufacturability. Consequently, the previous studies have generally been performed under laboratory settings (50, 51) or with stationary biological samples (7, 9, 52), thereby hampering the practical and widespread utilization.

To develop an alternative to instantaneous hyperspectral imaging, we take advantage of spectral learning (also known as spectral superresolution), which enables the recovery of spectral information from red–green–blue (RGB) values (tristimulus) acquired by a conventional trichromatic camera; a full reflectance spectrum in the visible range is computationally reconstructed from an RGB image. Several scientific communities, especially in the fields of color science, machine vision, and biomedical imaging, have addressed this ill-posed problem using different but related methods including compressed sensing, machine learning, and deep learning (26, 30, 53–67). Owing to its hardware simplicity, spectral learning can be performed by using a smartphone camera. In practice, dispersive optical components, such as spectrometers and bulky optical filters, are usually attached to mobile health (mHealth) sensing applications, potentially weakening user acceptance and hampering the practical translation from research to clinical practice (68–74). Conversely, spectral learning requires only a built-in camera, potentially offering hardware-independent mHealth tools using unmodified smartphones.

In this paper, we introduce a learning-based spatiospectral imaging method offering high spectral and temporal resolutions. The proposed spectral learning involves mapping from a sparse spectral space (i.e. RGB values) to a dense spectral space. Specifically, the spectral resolution is in a range of 0.5–1 nm, comparable to those of scientific spectrometers and spectrographs for biomedical or biochemical applications (thereby referred to as hyperspectral learning, compared with spectral learning). Hyperspectral learning also allows us to use a video frame rate with a high temporal resolution. First, we construct a customized dual-channel imaging setup coupled with a trichromatic camera (e.g. smartphone camera) and a spectrograph to simultaneously acquire an RGB image and subarea hyperspectral data. Second, we establish a simple statistical assumption to infer the entire field of view from a sampled subarea and recover a hypercube from incomplete measurements. Third, we establish two complementary machine learning frameworks based on statistical and deep learning, incorporating the domain knowledge of tissue optics into learning algorithms. Finally, we demonstrate reliable extractions of hemodynamic parameters from several different samples of tissue phantoms, chick embryos, and human conjunctiva; the results are validated through conventional hyperspectral imaging and functional near-infrared spectroscopy (fNIRS). Moreover, this hyperspectral learning method is applied to smartphone video recording to demonstrate the dynamic imaging of peripheral microcirculation and ultrafast imaging of oxygen depletion in tissue phantoms.

Results

Figure 1A illustrates the concept of hyperspectral learning for instantaneous spatiospectral imaging by significantly minimizing the number of necessary hyperspectral measurements. If hyperspectral data in a small yet representative subarea are available, a hyperspectral learning algorithm can be trained using the RGB and hyperspectral data in the subarea. This hyperspectral learning algorithm trained by the sampled (RGB and hyperspectral) data is applied to the entire image area, generating a hypercube without the need for a complete spectral or spatial scan. Thus, the key advantages of hyperspectral learning and hypercube recovery include the hardware simplicity offered by the use of conventional cameras, high temporal resolution if a video is used (e.g. slow-motion video recording on a smartphone), independence (no trade-off) between the spatial and spectral resolutions, and abundant spectral information for a variety of machine learning techniques. Locally sampled hyperspectral data can serve as prior information or physical constraints for incorporating domain-specific modeling into the learning algorithm, extracting critical features and parameters, and resulting in explainable and interpretable neural networks (64, 75–79).

Fig. 1.

Schematic illustration of hyperspectral learning for instantaneous imaging using a conventional trichromatic camera (i.e. three-color image sensor). A sampling of detailed spectral (hyperspectral) information in a small yet representative subarea enables hyperspectral learning to recover a hypercube in the entire image as well as to perform spectrally driven machine learning for physical or biological parameter extractions. A) Hyperspectral learning for hypercube recovery. Hyperspectral data in a small subarea are used to train a learning algorithm that takes the RGB values in each pixel as the input and returns a spectrum with a high spectral resolution (also known as spectral superresolution). Hyperspectral learning is not affected by an intrinsic trade-off between spatial and spectral resolutions, which often limits conventional snapshot hyperspectral imaging. B) Subarea sampling of hyperspectral data. A dual-channel spectrograph with a photometric slit (Materials and methods) simultaneously acquires an RGB image in the entire field of view and a hyperspectral line scan in a small subarea (i.e. the central line) in a single-shot manner. The sampled hyperspectral data serve as prior information or physical constraints for training the hyperspectral learning algorithm. The spectrally driven informed machine learning algorithm trained by the sampled hyperspectral data transforms the RGB image into a hyperspectral image data set (also known as a hypercube), which can further be used to extract physical or biological parameters.

To instantaneously sample hyperspectral data in a small subarea, a trichromatic camera (e.g. smartphone camera) is combined with a line-scan spectrograph (Fig. 1B). This configuration is commonly used in amateur telescopes that are equipped with astronomical spectroscopy systems (80). Specifically, a dual-channel spectrograph with a photometric slit simultaneously acquires an RGB image in the entire area and the hyperspectral data of a subarea (i.e. central line) in a single-shot manner (Fig. 1B and Materials and methods). The field of view is as small as 2.5 mm × 2 mm with a spatial resolution of 55 µm (Fig. S1). The sampled hyperspectral data have a spectral range of λ = 380–720 nm with a spectral resolution Δλ = 0.5 nm (Fig. S2 and Materials and methods). The dual-channel imaging setup provides sufficient training data (750–1,500 data points). This data set is randomly split into training (80%) and testing (20%) data sets for effectively training the hyperspectral learning algorithm to be applied to the entire area eventually. In addition, this imaging setup allows us to use a smartphone camera that can acquire videos at different frame rates. In particular, highly dynamic imaging is even possible with a high temporal resolution even up to 0.0005 s for 1,920 frames per second (fps), using commercially available smartphone models (at the time of publication).

Hyperspectral learning addresses an ill-posed problem, which is also known as spectral superresolution and hyperspectral reconstruction. The mathematical relationship between the RGB and hyperspectral intensity is described:

$${\mathit{x}}_{3\times 1}={\mathit{S}}_{3\times k}{\mathit{y}}_{k\times 1}+{\mathit{e}}_{3\times 1},$$ (1)

where $\mathit{x}$ denotes a 3 × 1 vector corresponding to three color values in the R, G, and B channels ($\mathit{x}=[\mathrm{R},\mathrm{G},\mathrm{B}{]}^T$), $\mathit{S}$ represents a 3 × k matrix of the RGB spectral response of the three-color sensor the spectral response functions in the R, G, and B channels of the smartphone camera (also known as the sensitivity function of the camera) (81, 82), $\mathit{y}$ is a k × 1 vector that has the spectral intensity ($\mathit{y}=[I({\lambda }_1),I({\lambda }_2),\dots ,I({\lambda }_k){]}^T$) where λ is discretized in the visible range with a spectral interval of 1 nm (Materials and methods), and $\mathit{e}$ symbolizes a 3 × 1 vector of the system noise. Basically, hyperspectral learning is to obtain a pseudoinverse of ${\mathit{S}}_{3\times k}$. Depending on the availability of training data and the desired spectral resolution, three machine learning approaches can be used for solving this underdetermined problem: l₁ norm minimization (i.e. compressed sensing or sparsity regularization), l₂ norm minimization (i.e. least squares regression), and deep learning (26, 30, 53–65). Among these approaches, statistical learning using fixed-design linear regression with polynomial expansions offers a highly stable inverse calculation that can transform RGB data to high-resolution spectral data owing to the nature of l₂ norm minimization (Materials and methods) (55, 62, 81, 83, 84).

The key assumption for reliable hyperspectral learning is that a sampling distribution (i.e. RGB values of the sampled subarea) should follow the parent distribution (i.e. RGB values of the entire image area); the intensity distributions between the sampled subarea and the entire field of view of interest are statistically identical. Specifically, the probability distribution of the R, G, and B values in the subarea needs to conform to those in the entire area in terms of variability and shape. In addition, to reliably predict unknown hyperspectral output responses from RGB values outside the subarea, the hyperspectral learning algorithm should be applied within the same (minimum and maximum) range of sampled RGB values used to train the algorithm. In a similar manner to nonparametric tests with non-Gaussian distributions, quantile–quantile (Q–Q) plots can conveniently be used to assess if the two sets of data plausibly follow the same distribution within the same range. This assumption of interpolation offers an important advantage over conventional snapshot hyperspectral imaging. If these assumptions are valid, then the hyperspectral learning is not limited by the intrinsic trade-off between spatial and spectral resolutions.

Spectrally informed learning allows for the incorporation of physical and biological understanding of domain knowledge into learning algorithms. Typically, purely data-driven learning requires a large volume of training data and lacks explainable and interpretable learning (76, 78). Among the various snapshot imaging applications, we focus on extracting biological parameters or spectral signatures from a hypercube using the domain knowledge of tissue optics. In this perspective, light propagation in tissue can be explained by the theory of radiative transport and robust approximations (e.g. diffusion, Born, and empirical modeling) (85–90). Specifically, taking advantage of tissue reflectance spectral modeling (Materials and methods), we extract the key hemodynamic parameters: oxygenated hemoglobin (HbO₂), deoxygenated hemoglobin (Hb), and oxygen saturation (sPO₂), which are the fundamental determinants of oxygen transport to tissue associated with a variety of physiological changes, diseases, and disorders.

$$\mathrm{sP}{\mathrm{O}}_2={\displaystyle \frac{\mathrm{Hb}{\mathrm{O}}_2}{\mathrm{Total}\mathrm{hemoglobin}}={\displaystyle \frac{\mathrm{Hb}{\mathrm{O}}_2}{\mathrm{Hb}{\mathrm{O}}_2+\mathrm{Hb}}.}}$$ (2)

Notably, tissue optics serves as the cornerstone of biophotonics and biomedical optics to deepen our knowledge of light–tissue interactions and develop noninvasive optical diagnostic methods and devices (89, 91).

To demonstrate the versatility of hyperspectral learning and hypercube recovery, we formulate two complementary machine learning frameworks. In statistical learning (Fig. 2A), sampling of hyperspectral data in a subarea serves as a training data set for hyperspectral learning. The hypercube of the entire area is reconstructed using a statistical learning algorithm (Materials and methods). Fixed-design linear regression, featuring polynomial expansions, is utilized as the method of least squares (l₂ norm minimization) (55, 62, 81, 83). A full spectroscopic resolution is achieved in a range of Δλ = 0.5–1 nm, highly comparable to those of scientific spectrometers or spectrographs. This level of spectral resolution is 10 times better than the typical resolution of hyperspectral imaging of Δλ = 10 nm. Subsequently, the recovered hypercube is fitted using a tissue reflectance spectral model to extract the hemodynamic parameters (Materials and methods) (85–90). In deep learning (Fig. 2B), hyperspectral data in a subarea are directly fed into the tissue reflectance spectral model to compute the hemodynamic parameters within the same subarea. The obtained data set serves to train a deep neural network that computes the hemodynamic parameters with RGB values (tristimulus) as an input. Specifically, the deep neural network is directly trained by using the hemodynamic parameters extracted from the tissue reflectance spectral model fed with the sampled data, thereby reducing the computational load. In both cases, separate training and validation data sets are employed to strengthen the learning algorithm for training: 80% of the data points among the sampled data (i.e. 600 data points out of 750) are randomly selected as a training data set and the remaining 20% (i.e. 150 data points) are blindly tested as a testing data set.

Fig. 2.

Flowcharts of two complementary machine learning frameworks for hypercube recovery and hemodynamic imaging and schematic illustration of deep learning networks. A) Statistical learning involving hyperspectral learning and tissue reflectance spectral modeling. Hyperspectral and RGB data in a subarea are used to train a statistical learning algorithm. The trained statistical learning algorithm is subsequently applied to the RGB data in the entire area to reconstruct the hypercube. Using tissue reflectance spectral modeling, the key hemodynamic parameters are extracted from the hypercube to generate the hemodynamic maps. B) Deep learning informed by the tissue reflectance spectral model. The hemodynamic parameter values (output) from the hyperspectral data in the subarea extracted via tissue reflectance spectral modeling of the corresponding RGB values (input) are used to train the deep learning network. The trained deep learning network is thereafter applied to the RGB data for the entire area to generate the hemodynamic maps. C) Illustration of the deep neural network that takes RGB values and returns key hemodynamic parameters (i.e. HbO₂ and Hb). This network is trained by the RGB values and hemodynamic parameters in a subarea; HbO₂ and Hb values are computed from the ground-truth hyperspectral data obtained with conventional pushbroom-type hyperspectral imaging system in the subarea via the tissue reflectance spectral model (Materials and methods). D) The first hidden layer is fully connected to 18 nodes (or neurons) to mimic the polynomial expansions of statistical learning. E) Representative spectral intensity differences in the ground-truth hyperspectral data obtained from a tissue phantom with oxygenated and deoxygenated hemoglobin. Inset: The ground-truth hyperspectral data when the sample is oxygenated and deoxygenated. F) Representative difference in the computed output values of the first hidden layer between two different RGB values of HbO₂ and Hb from the same tissue phantom in E). The order of the nodes is assigned such that the rank of the output differences is the same as that of the wavelengths in the spectral intensity differences.

Importantly, deep learning informed by hyperspectral information is advantageous for designing explainable and interpretable neural networks. Among similar yet distinct terms, such as understandability and comprehensibility (77), spectrally informed deep learning enables transparency in the learning algorithm as it is understandable in a manner similar to statistical regression. The deep neural network model is designed to mimic the concept of polynomial expansions in statistical hyperspectral learning (Figs. 2C and S3 and Materials and methods). In particular, the first hidden layer, which is one of the important hyperparameters for building a neural network, is fully connected with a relatively large number of nodes (18 nodes) to mimic hyperspectral learning, which transforms RGB values to a spectral intensity profile with a high spectral resolution. After the network is trained, each node in the first hidden layer possesses a distinct weight representing hyperspectral information, such that the RGB values of a certain hemodynamic parameter (e.g. sPO₂) generate the corresponding spectral feature, which is further propagated throughout the network (Fig. 2D). Figure 2E illustrates two representative cases of hyperspectral data measured from a tissue phantom by varying sPO₂ between HbO₂ and Hb (Fig. S3 and Supplementary Methods). The output values at different nodes in the first hidden layer can be understood based on the spectral intensity differences as a function of λ (Fig. 2E and F) that a scientific spectrometer or spectrograph can quantify. This direct spectral understanding should be differentiated from other conventional heat maps or saliency maps employed for visualizing or explaining features extracted through typical convolutional neural networks (92, 93). In particular, the differences in the computed output values of the first hidden layer between the two different RGB values of HbO₂ and Hb in a tissue phantom (Supplementary Methods) resemble the spectral intensity differences between HbO₂ and Hb measured from the same tissue phantom (Fig. 2E and F).

We comprehensively validate the hyperspectral learning-based instantaneous imaging method using an experimental model of vascular development. Chicken embryos serve as an excellent model system for imaging the development of blood vessel formation and hemodynamics (Fig. 3A) (94, 95). The key statistical assumptions between the sampled area (i.e. the central line) and the entire image area are tested by evaluating the probability distribution profiles of each RGB channel (Fig. 3B) and the Q–Q plots (Fig. 3C). The sampled data set (both hyperspectral and RGB data along the central line) is used to train a learning algorithm that has the input of RGB values and the output of hemodynamic parameters (i.e. HbO₂ and Hb). The residuals between the ground-truth (acquired by a pushbroom-type hyperspectral imaging system in Materials and methods) and reconstructed hyperspectra are minimal, as shown with 95% CIs as a function of λ (Fig. 3D, E, and G and Materials and methods). The hemodynamic parameter extractions are further supported by the 95% CIs of residuals between the ground-truth and fitted hyperspectra obtained using the tissue reflectance spectral model (Fig. 3D, F, and H and Materials and methods). Figure 3G and H show that the reconstructed and fitted spectra are in excellent agreement with the ground-truth spectra. Moreover, the spectral angle mapper (Materials and methods) between the ground-truth and reconstructed hypercubes captures a high degree of similarity with an average value of 0.035 rad (Fig. 3I). The direct comparison (i.e. spectral angle mapper) between the ground-truth and fitted hypercubes also returns an average value of 0.046 rad (Fig. 3J), supporting the performance of hyperspectral learning and tissue reflectance spectral modeling.

Fig. 3.

Comprehensive evaluations of hyperspectral learning, hypercube recovery, and hemodynamic parameter extractions in an experimental vascular developmental model. A) Photograph (RGB image) of the chicken embryo on day 8. The vertical gray line represents a subarea for hyperspectral sampling. B) Probability distributions and histograms of R, G, and B values of the entire image area and the subarea. The black bins show the distributions of R, G, and B values of the entire image area. The black solid lines are displayed together with the black bins to visualize the trends of the data distributions of the entire image area. The colored bins depict the distributions of R, G, and B values of the subarea [i.e. the center line in A)] according to the RGB colors. The colored lines are illustrated together with the colored bins to visualize the trends in the data distribution of the subarea. C) Q–Q plots for distribution comparisons between the entire field of view and the subarea. The subarea [the center line in A)] ensures a similar intensity distribution as the intensity distributions of the entire area A) in each RGB channel. The black dotted straight diagonal line indicates that the two distributions are identical. D, F) Visualization of the ground-truth D), reconstructed E), and fitted F) spectra from the testing data (data size = 150) of the sampled data (data size = 750). A logarithmic scale of intensity is used for enhanced visualization. G) Comparison between the ground-truth and reconstructed spectra in the testing data set. The differences (residuals) between the ground-truth and reconstructed spectra are plotted with 95% CIs for each λ. H) Comparison between the ground-truth and fitted spectra in the testing data set. Hemodynamic parameters are extracted using the tissue reflectance spectral model (Materials and methods). The differences (residuals) between the ground-truth and fitted spectra are plotted with 95% CIs. I) Spectral angle mapping image between the ground-truth and reconstructed hypercubes. J) Spectral angle mapping image between the ground-truth and fitted hypercubes.

Figure 4 depicts the hemodynamic maps of Hb, HbO₂, and sPO₂ corresponding to a white leghorn chicken (Gallus gallus domesticus, Hy-Line W-36) embryo on day 8 (Fig. 4A–C). In this statistical learning framework, after recovering the hypercube for the entire area via a hyperspectral learning algorithm, the recovered hypercube is fitted by using the tissue reflectance spectral model (Fig. 2A and Materials and methods), thereby computing the hemodynamic parameters in a pixel-by-pixel manner. Figure 4D clearly demonstrates the high levels of sPO₂ in the major blood vessel compared with capillary vessels. As a deep learning framework (Fig. 2B), we design a fully connected deep neural network that takes RGB values as the input and returns the hemodynamic parameters as the output (Figs. 2C and S3 and Materials and methods). As stated previously, the two unique design aspects are that the neural network encompasses the concept of polynomial expansions in hyperspectral learning and incorporates the output parameters extracted from the tissue reflectance spectral modeling (Fig. 2D and Materials and methods). As illustrated in Fig. 4E, the hemodynamic parameters are directly computed using the deep neural network. In particular, the hemodynamic maps obtained from both frameworks are in excellent agreement with the ground-truth hemodynamic maps generated via conventional pushbroom-type hyperspectral imaging, which are supported by high values of structural similarity index (Fig. 4F and Table S2). Interestingly, the conventional hemodynamic maps are noisier due to the motion artifact of the live sample, which results from the slow rate of data acquisition (data acquisition time = 45 min) and the mechanical scanning, as shown by the horizontal lines in Fig. 4F.

Fig. 4.

Hemodynamic maps of HbO₂, Hb, and sPO₂ in the experimental model of vascular development. A) Chick embryo in a petri dish as an experimental vascular developmental model. The image area for learning-based snapshot imaging with a high spatial resolution is marked by a box. B) RGB image in field of view [the box in A)] for hyperspectral learning-based snapshot imaging. C) RGB image generated by the conventional pushbroom-type hyperspectral imaging system. D) Statistically learning-based hemodynamic maps. Hyperspectral learning generates a hypercube that is processed to compute hemodynamic maps using tissue reflectance spectral modeling. E) Deep learning-based hemodynamic maps. Hemodynamic maps are directly generated using a deep neural network that takes the RGB values as the input and returns HbO₂ and Hb values as the output. F) Reference hemodynamic maps generated by a conventional pushbroom-type hyperspectral imaging system for validation. The two learning-based hemodynamic maps D, E) are in excellent agreement with the ground-truth maps F). Statistical learning and deep learning-based hemodynamic maps are further assessed using the structural similarity index (Supplementary Methods) to show that they are qualitatively identical to the reference hemodynamic maps (Table S2).

Moreover, another advantage offered by learning-based snapshot imaging is that a video can be transformed into parameter maps after the learning algorithm is trained. Essentially, a video comprises a time series of multiple RGB images and each frame represents an RGB image. A video captured using a common trichromatic camera (e.g. a smartphone camera) can conveniently be processed using hyperspectral learning. Recently, smartphone models are capable of slow-motion video mode, featuring as high frame rates as 960–1,920 fps. Thus, hyperspectral learning enables unprecedentedly fast hyperspectral imaging with high temporal resolution (e.g. 0.0005–0.0010 s). To demonstrate ultrafast dynamic imaging, we take advantage of an off-the-shelf smartphone (Samsung Galaxy S21) to record a video at 960 fps (temporal resolution = 0.0010 s). Each frame representing an RGB image of the video is fed into the hyperspectral learning algorithm to compute the corresponding hemodynamic images. Then the hemodynamic images are restacked into a video format (Materials and methods and Supplementary Methods). Figure 5 shows time series hemodynamic maps of sPO₂ changes during a selected short period (Videos S1 and S2 for the entire duration of 0.5 s) in a tissue phantom that has a rapid oxygen exchange process (Supplementary Methods and Fig. S3). The hemodynamic video with the millisecond resolution reveals rapid hemoglobin dispersion and oxygen diffusion, varying sPO₂ between HbO₂ and Hb. It should be noted that hyperspectral learning enables ultrafast dynamic imaging merely using a commercially available unmodified smartphone.

Fig. 5.

Ultrafast dynamic imaging of sPO₂ changes via hyperspectral learning of smartphone's ultraslow video recording. A) Selected 15 frames (RGB images) from a video of the dynamic tissue phantom recorded by the smartphone's built-in ultraslow-motion mode (960 fps). B) Hemodynamic change images of sPO₂ (ΔsPO₂) compared with the initial image, processed using the corresponding RGB images with the deep learning algorithm. Hyperspectral learning allows an ultrafast data acquisition rate at a temporal resolution of 0.0010 s (Video S1 for the entire duration) using the off-the-shelf smartphone.

To further demonstrate dynamic imaging of peripheral microcirculation at a particular video frame rate, we record a video at 60 fps for a duration of 180 s (temporal resolution = 0.0167 s) (Video S3 and Figs. S5 and S6). As a model system for peripheral microcirculation in humans, we visualize spatiotemporal hemodynamic changes in the microvessels of the inner eyelid (i.e. the palpebral conjunctiva) (Fig. 6A and B). The inner eyelid is an easily accessible and highly vascularized peripheral tissue site that receives blood from the ophthalmic artery. Thus, the inner eyelid serves as a feasible sensing site for various diseases and disorders (96–98). Figure 6C and D show the peripheral hemodynamic maps of Hb, HbO₂, and sPO₂ obtained during the resting state of a healthy adult volunteer. Videos of hemodynamic maps computed from the smartphone video are generated using the two complementary machine learning frameworks of statistical learning and informed deep learning (Materials and methods and Videos S1–S3). In Fig. 6, the sPO₂ maps reveal spatially complex patterns of perfusion in the inner eyelid, which are not evident in the photo (i.e. the RGB image). The statistical learning and deep learning-based hemodynamic maps of HbO₂ and Hb are in excellent agreement with each other, thereby indicating the reliable extraction of hemodynamic parameters.

Fig. 6.

Representative peripheral hemodynamic maps of HbO₂, Hb, and sPO₂ extracted from a healthy adult in the resting state. The inner eyelid (i.e. palpebral conjunctiva) has high vasculature connected to the ophthalmic artery, serving as an easily accessible peripheral tissue site. A) Photograph of a healthy adult volunteer taking a picture with a smartphone while the inner eyelid pooled down. B) RGB image of the inner eyelid with high spatial resolution showing the field of view for hyperspectral learning-based snapshot imaging. Microvessels in the inner eyelid are clearly visible without the effects of skin pigments, which are easily accessible for imaging. C) Statistical learning-based snapshot hemodynamic maps of the inner eyelid. Hemodynamic maps are generated using tissue reflectance spectral modeling based on spectral information from the recovered hypercube. D) Deep learning-based snapshot hemodynamic maps of the inner eyelids. Hemodynamic maps are directly generated using the deep neural network, which returns HbO₂ and Hb values as the output when the RGB values are employed as the input.

Furthermore, we validate the computed hemodynamic maps obtained via the smartphone video (Video S3) against the functional imaging signals obtained using a fNIRS system (Materials and methods and Supplementary Methods). In this in vivo imaging setting, simultaneously acquiring the concurrent hypercube from the same sensing site (i.e. inner eyelid in Fig. 7A) is practically unachievable using a conventional hyperspectral imaging system. In this study, the peripheral hemodynamic signals obtained from the smartphone video are compared with the concurrent fNIRS signals averaged from all brain regions (Fig. 7D) and the fingertip (Fig. 7F) of the same subject measured using an fNIRS neuroimaging system (Materials and methods and Supplementary Methods). We focus on the existence of the same low-frequency oscillations of HbO₂ and Hb associated with nonneuronal circulatory signals. In particular, low-frequency oscillations of HbO₂ and Hb in a frequency range of 0.01–0.1 Hz are commonly present in functional imaging signals (e.g. blood oxygen level-dependent functional magnetic resonance imaging and fNIRS) (99, 100). Specifically, a hemodynamic phase of approximately 180° between HbO₂ and Hb changes (ΔHbO₂ and ΔHb) during the resting state of healthy adults conveys autonomic physiological information, potentially serving as a biomarker of circulatory dysfunction.

Fig. 7.

Temporal changes in the eyelid's hemodynamic signals extracted from the smartphone video capture and concurrent resting-state fNIRS signals from the brain and another peripheral site (fingertip). A) Representative photographs of data acquired from the inner eyelid of a healthy participant. A smartphone camera can easily record a video of the inner eyelid. B, C) Temporal traces of statistical learning-based B) and deep learning-based C) hemodynamic signals of changes in HbO₂ (ΔHbO₂) and Hb (ΔHb) in the inner eyelid. A relatively large blood vessel is selected (Fig. S7) to analyze the hemodynamic parameters. A temporal resolution of 0.016 s is set by the smartphone video recording setting (Video S3). A vector representation of the time-averaged phase difference between the corresponding ΔHbO₂ and ΔHb signals in the eyelid is displayed on the right-hand side of the temporal trace. D) Representative photographs of the data acquisition from the brain of the subject using fNIRS. The fNIRS signals from the brain of the same subject are also measured at the same time to be compared with the signal from the smartphone video. E) Temporal traces of concurrent fNIRS signals averaged from all brain regions of the same subject with a vector representation of the time-averaged phase difference between ΔHbO₂ and ΔHb from all brain regions. F) Representative photographs of the data acquisition from the fingertip of the subject using fNIRS. The fNIRS signals from the fingertip of the same subject are also measured at the same time to be compared with the signal from the smartphone video. G) Temporal traces of concurrent fNIRS signals averaged from the fingertip of the same subject with a vector representation of the time-averaged phase difference between the corresponding ΔHbO₂ and ΔHb in the fingertip. The phase difference extracted from the smartphone video B, C) is in excellent agreement with the functional imaging modality E, G).

We track hemodynamic signals in a relatively large vessel (Fig. S7) in terms of the changes in HbO₂ and Hb (ΔHbO₂ and ΔHb), which are computed via statistical learning as well as deep learning. Figure 7B and C characterize ΔHbO₂ and ΔHb obtained after filtering out the heartbeat-related signals using a bandpass filter (0.01–0.1 Hz) (Materials and methods). Moreover, vector analyses reveal that the phase differences between ΔHbO₂ and ΔHb over time under the resting state are 180.9° and 180.0° for statistical learning and deep learning, respectively (Fig. 7B and C). When the concurrent resting-state fNIRS signals from the cortical areas of the brain and another peripheral site (fingertip) from the same healthy subject are analyzed, the time-averaged phase differences between ΔHbO₂ and ΔHb also return 184.3° and 184.8°, respectively (Fig. 7E and G). Particularly, these consistent phase differences corresponding to the brain and peripheral sites (i.e. inner eyelid and fingertip) indicate that low-frequency hemodynamic oscillations are nonneuronal and circulate to the different parts of the body. More importantly, these results validate the accurate and reliable extraction of hemodynamic parameters from smartphone video recordings.

Discussion

The two machine learning frameworks of statistical learning and informed deep learning are complementary. In statistical learning, l₂ norm minimization (i.e. least squares regression) of hyperspectral learning enables stable and robust hypercube recovery with high spectral and temporal resolutions from an RGB image. This method is analogous to compressed sensing (i.e. l₁ norm minimization) such that incomplete measurements can be used to recover an entire image. The recovered hypercube is an intermediate product to be processed to extract key features or classifications of interest, requiring an additional step. On the other hand, in deep learning, the sampling of hyperspectral data in a subset area facilitates the direct incorporation of tissue optics modeling into the algorithm, resulting in direct parameter extractions. In addition, this spectrally informed deep learning approach is advantageous for featuring a transparent network architecture, realizing explainable and interpretable learning. Moreover, this reported snapshot ability will offer ample opportunities for applying hyperspectral imaging to emerging areas where instantaneous data acquisitions have been the bottleneck.

Hyperspectral learning from an RGB image can potentially be enhanced by using the state-of-the-art three-color image sensors and high dynamic range images. First, conventional three-color sensors interpolate missing color information in selected pixel positions using a demosaicing algorithm, because each pixel records only one of the RGB colors in a format of a grid or checkerboard (also known as Bayer filter). Foveon X3 sensors capture all incident light (three colors) within the same pixel without having three separate color channels (101). An image sensor with a color router can directly detect the entire color content without using a filter to select the intended color channel (102, 103). Second, a higher color depth (also known as bit depth) in each pixel can further improve hyperspectral learning to avoid metamerism such that different spectral profiles result in the same RGB values. Because the color depth in each R, G, and B channel is 10 bit in our case, the number of different colors (combinations of RGB values) is 2¹⁰ × 2¹⁰ × 2¹⁰ (= 1.07 billion) colors. In other words, it is extremely unlikely that metamerism occurs. The higher the color depth of an RGB image, the more colors the image can store, further reducing the possibility of metamerism in hyperspectral learning.

In conclusion, the main advantages of this snapshot imaging method are the independence between the spectral and spatial resolutions, the hardware simplicity that could enable the use of commercially available smartphones, and the versatility offered by different machine learning techniques. We further envision that spectrally informed hyperspectral imaging will lay the foundation for mHealth technologies to noninvasively access hemodynamic information from an easily accessible peripheral tissue site using off-the-shelf smartphones in resource-limited settings or at-home settings.

Materials and methods

Dual-channel imaging setup mounted with a smartphone camera

First, we constructed a dual-channel imaging spectrograph setup to acquire an RGB image in the entire field of view and hyperspectral data in a subarea; a line-scan hyperspectral system was mounted with a trichromatic camera (i.e. RGB camera and smartphone camera). This configuration was inspired by an image-guided hyperspectral line-scan system mounted on a telescope, which is a common configuration used in amateur astronomy (80). In this dual-channel setup, a photometric slit, which is a mirror with a slit opening (slit width = 23 μm), enabled the simultaneous acquisition of an RGB image and hyperspectral line-scan data. An image reflected by the photometric slit (mirror portion) was captured using a trichromatic (RGB) camera (GS3-U3-120S6C-C, FLIR). The light passing through the slit was dispersed by a diffraction grating (groove density = 150 mm⁻¹) and captured using a monochrome camera (GS3-U3-120S6M-C, FLIR). A tungsten halogen light source (HL-2000-HP Light Source, Ocean Insight) was used as a broadband illumination source. Spectral calibration of the spectrograph was performed using a xenon calibration light source emitting multiple narrow peaks at specific wavelengths. An objective lens (4× and NA = 0.10) was mainly used to image the microvessels, and the field of view was as small as 2.5 mm × 2 mm with a spatial resolution of 55 µm (Fig. S1). The spectral resolution was determined via a helium–neon laser that emits monochromatic radiation with a narrow bandwidth at 543.5 nm. The spectral range of hyperspectral line-scan data was 380–720 nm, and the spectral resolution Δλ was 0.5 nm (Fig. S2). The typical sizes of the RGB image and hyperspectral data were 750 × 300 pixels and 750 × 1 pixels in the spatial domain, respectively. Second, we incorporated a smartphone (Samsung Galaxy S21) into the dual-channel system using a cube beamsplitter (Supplementary Methods and Fig. S8). The color depths (bit depth) were 10 bit (file format: DNG) for photos and 8 bit (MP4) for videos, respectively. The smartphone's video recording function (i.e. ultraslow-motion recording) was used to acquire videos at frame rates of 60 or 960 fps (Videos S1–S3, Supplementary Methods, and Fig. S9). Third, we employed a motorized linear stage (T-LSR150A, Zaber) to move the entire system to directly acquire the ground-truth hyperspectral image cube (hypercube) in the entire field of view in the same manner as the pushbroom-type hyperspectral imaging system. A mechanical linear scan step was performed at 10 μm. Mechanical scanning was performed only for the validation studies. The data acquisition was synchronized using a custom-built MATLAB interface.

Frontend data processing

We implemented frontend data processing for both hyperspectral and RGB imaging to factor out the spectral contributions from the illumination, system responses, and background ambient light. The measured spectral intensity $I_{\mathrm{M}}(\lambda )$ reflected from the sample is expressed as a function of wavelength λ:

$$I_{\mathrm{M}}(\lambda )=L(\lambda )C(\lambda )S(\lambda )I_{\mathrm{R}}(\lambda ),$$ (3)

where $L(\lambda )$ denotes the spectral shape of the illumination light source, $C(\lambda )$ represents the spectral response of all the optical components in the system, $S(\lambda )$ depicts the spectral response function (also known as the sensitivity function) of the image sensors, and $I_{\mathrm{R}}(\lambda )$ symbolizes the intensity reflected in the sample (26, 30, 53–65). First, a white reflectance standard (SRS-99-010, Labsphere) was used to compensate for the spectral response of the system, including the illumination. The white reflectance standard possesses a reflectivity of 99% in the visible range, and its measured intensity can be expressed as:

$$I_{\mathrm{White}}(\lambda )=L(\lambda )C(\lambda )S(\lambda ).$$ (4)

Second, a black reflectance standard (SRS-02-010, Labsphere) with a reflectivity of 2% in the visible range was acquired to eliminate the ambient stray and background room light. After subtracting the black standard intensity $I_{\mathrm{Black}}$ from each measurement, the normalized intensity reflected from the sample can be obtained:

$$I_{\mathrm{R}}(\lambda )={\displaystyle \frac{I_{\mathrm{M}}(\lambda )-I_{\mathrm{Black}}(\lambda )}{I_{\mathrm{White}}(\lambda )-I_{\mathrm{Black}}(\lambda )}.}$$ (5)

Hyperspectral statistical learning

We established a statistical learning framework to reliably reconstruct a hyperspectrum from RGB values. Conceptually, hyperspectral learning solves a pseudoinverse of ${\mathit{S}}_{3\times k}$ in Eq. (1) where ${\mathit{S}}_{3\times k}$ is the spectral response function in the R, G, and B channels of the camera. As the method of least squares (l₂ norm minimization), we took advantage of fixed-design linear regression with polynomial expansions to solve $[{\mathit{S}}_{3\times k}{]}^{+}$ (55, 62, 81, 83). First, the sampled hyperspectral and RGB data (m = training data size) were used to change this underdetermined problem to an overdetermined problem such that ${\mathit{X}}_{3\times m}$ and ${\mathit{Y}}_{k\times m}$ were formed by adding ${\mathit{x}}_{3\times 1}$ and ${\mathit{y}}_{k\times 1}$ of the training data. The relationship in Eq. (1) is rewritten:

$${\mathit{X}}_{3\times m}={\mathit{S}}_{3\times k}{\mathit{Y}}_{k\times m}.$$ (6)

The relationship in Eq. (6) can be expressed in the following inverse format:

$${\mathit{Y}}_{k\times m}={\mathit{T}}_{k\times 3}{\mathit{X}}_{3\times m},$$ (7)

where ${\mathit{T}}_{k\times 3}=[{\mathit{S}}_{3\times k}{]}^{+}$. Second, to incorporate the nonlinearity between RGB and hyperspectral data and improve the hyperspectral learning, we added fourth degree polynomial expansions by including a bias term such that ${\mathit{X}}_{3\times m}$ was expanded to ${\hat{\mathit{X}}}_{p\times m}$ with p = 34 (55, 62, 83):

$$\begin{array}{c}{\hat{\mathit{X}}}_{p\times m}=[1,1,1,R,G,B,R^2,G^2,B^2,RG,GB,RB,\dots \\ R^3,G^3,B^3,R{G}^2,R{B}^2,G{R}^2,G{B}^2,B{R}^2,B{G}^2,RGB,\dots \\ {R^4,G^4,B^4,R^{3}G,R^{3}B,G^{3}R,G^{3}B,B^{3}R,B^{3}G,R^2{G}^2,R^2{B}^2,G^2{B}^2,R^{2}GB,G^{2}RB,B^{2}RG]}^{\mathrm{T}}.\end{array}$$ (8)

Subsequently, Eq. (7) becomes:

$${\mathit{Y}}_{k\times m}={\hat{\mathit{T}}}_{k\times p}{\hat{\mathit{X}}}_{p\times m}.$$ (9)

Third, the inverse of the expanded matrix $\hat{\mathit{T}}$ can easily be solved via QR decomposition or the Moore–Penrose pseudoinverse (55, 62, 81–83). Finally, upon substituting the RGB values ($\mathit{x}=[\mathrm{R},\mathrm{G},\mathrm{B}{]}^{\mathrm{T}}$) into Eq. (7), the corresponding hyperspectral intensity values ($\mathit{y}=[I({\lambda }_1),I({\lambda }_2),\dots ,I({\lambda }_k){]}^T$) were obtained. Thus, $\hat{\mathit{T}}$ transforms RGB data to hyperspectral data.

Tissue reflectance spectral modeling

We make use of a tissue reflectance spectral model to extract key hemodynamic parameters from the ground-truth and recovered hyperspectral data. Light propagation in tissue can be modeled in accordance with the theory of radiative transport and robust approximations (e.g. diffusion, Born, and empirical modeling) (85–90). Specifically, we conducted parameter extractions using an extensively used empirical modeling method. The intensity reflected from a biological sample can be expressed as a function of λ in the visible range:

$$\begin{array}{rl}{I}_{\mathrm{R}}(\lambda )& =\left[b_1{\left({\displaystyle \frac{\lambda }{{\lambda }_0}}\right)}^{b_2}+b_3{\left({\displaystyle \frac{\lambda }{{\lambda }_0}}\right)}^{-4}\right]\\ & \times \exp [-b_4\times \{b_5\times {\epsilon }_{\mathrm{Hb}{\mathrm{O}}_2}(\lambda )+(1-b_5)\times {\epsilon }_{\mathrm{Hb}}(\lambda )\}],\end{array}$$ (10)

where $b_1$, $b_2$, and $b_3$ are associated with the scattering (Mie or Rayleigh) contributions at ${\lambda }_0$ = 800 nm, ${\epsilon }_{\mathrm{Hb}{\mathrm{O}}_2}(\lambda )$ denotes the absorption coefficient of HbO₂, ${\epsilon }_{\mathrm{Hb}}(\lambda )$ denotes the absorption coefficient of Hb, $b_4$ is the hemoglobin concentration multiplied by the optical path length, and $b_5$ is the blood sPO₂. In the chicken embryo study, the lipid content of the egg yolk is high, as avian embryos rely on over 90% of the caloric requirement of fatty acids (104). Thus, an additional absorption coefficient term for lipid was added to Eq. (10) to account for this high lipid content (105). All fitting parameters were computed using the simplex search (Nelder–Mead) algorithm (106). The differences (residuals) between the original and fitted hyperspectra were plotted with 95% CIs as a function of λ (Figs. 3, S5, and S6).

Deep neural network designing and training

We designed a deep neural network considering the polynomial expansions of hyperspectral learning (Eq. (8)) and tissue reflectance spectral modeling (Eq. (10)). First, as an alternative to the sequence of hyperspectral learning and hemodynamic parameter extraction in statistical learning, a deep neural network was established that utilized RGB values as the input and returned the hemodynamic parameters of HbO₂ and Hb (or total hemoglobin and sPO₂) as the output. Hemodynamic parameters were calculated using the tissue reflectance spectral model (Eq. (10)). The two hemodynamic parameters as outputs were fed to the network for supervised learning, resulting in rapid convergence and high prediction accuracy. The neural network was trained and validated using 80% (600 data points) and 20% (150 data points) of locally sampled data.

Second, in the fully connected network, 18 nodes (neurons) in the first hidden layer are intended to have optimal selections of RGB values with different weights on the R, G, and B values and their combinations. It should be noted that the first hidden layer is conceptually understandable in a manner similar to the polynomial expansions in statistical hyperspectral learning (Eq. (8)), thereby offering a transparent neural network model:

$$\begin{array}{c}{N}_{1,1}=w_{1,\mathrm{R}}\mathrm{R}+w_{1,\mathrm{G}}\mathrm{G}+w_{1,\mathrm{B}}\mathrm{B},\\ N_{1,2}=w_{2,\mathrm{R}}\mathrm{R}+w_{2,\mathrm{G}}\mathrm{G}+w_{2,\mathrm{B}}\mathrm{B},\\ ⋮\\ N_{1,i}=w_{i,\mathrm{R}}\mathrm{R}+w_{i,\mathrm{G}}\mathrm{G}+w_{i,\mathrm{B}}\mathrm{B},\end{array}$$ (11)

where R, G, and B are the input RGB values, $N_{1,i}$ indicates the ith node of the first hidden layer, and $w_{i,\mathrm{RGB}}$ is a weight connecting the R, G, or B input nodes and the ith node of the first hidden layer, respectively. Overall, the network included the input layer with three nodes, four hidden layers, and the output layer (Fig. S3). Batch normalization, the softplus activation function (i.e. a smooth approximation to ReLU), and the ADAM optimization were employed for the network. The weight decay (factor for l₂ regularization) was specified to be 10⁻⁵ to avoid overfitting. The initial learning rate was set to 0.01, but the learning rate was set to drop for a given number of epochs. The batch size was 20 and the value of the epoch was 15 (Fig. S3).

Spectral angle mapper

The spectral angle mapper directly compares two spectra (e.g. measured spectrum and reference spectrum) allowing for a geometric interpretation in a space with dimensionality equal to the number of discretized wavelengths (k). Specifically, the spectral angle mapper calculates an angle between the spectra in the k-dimensional space, quantifying spectral similarity in a pixel-by-pixel manner (107).

$$\alpha ={\cos }^{-1}\left({\displaystyle \frac{{\sum }_{i=1}^k{I}_{\mathrm{A}}({\lambda }_i)I_B({\lambda }_i)}{\sqrt{{\sum }_{i=1}^k{I}_{\mathrm{A}}{({\lambda }_i)}^2}\sqrt{{\sum }_{i=1}^k{I}_{\mathrm{B}}{({\lambda }_i)}^2}}}\right),$$ (12)

where I_A is the spectrum to be compared and I_B is the reference spectrum.

fNIRS measurements

We utilized an fNIRS neuroimaging system (NIRx Medical Technologies) to concurrently measure hemodynamic signals from the brain and fingertips during the resting state of healthy adult volunteers. In brief, the head cap and finger clip had two sets of laser sources and detectors at 785 and 830 nm, respectively, and the separation distance between the laser and the detector was 30 mm. Six detection channels were used to cover all brain areas and two detection channels were employed for a fingertip. The sampling rate of fNIRS was 10 Hz. HbO₂ and Hb signals were synchronized with the dual-channel imaging setup and smartphone camera. The fNIRS signals were processed for motion correction and spatial smoothing with a full width at half maximum of a 5-mm isotropic Gaussian kernel. A bandpass filter (0.01–0.1 Hz, third-order Butterworth) was used to extract the low-frequency signals of ΔHbO₂ and ΔHb. The phase differences between the HbO₂ and Hb signals (ΔHbO₂ relative to ΔHb) were calculated after application of the bandpass filter (100).

Ethics Approval Statement

This study was conducted with IRB approval (IRB-2022-1202) at Purdue University and informed consent was obtained from all participants.

Supplementary Material

Supplementary material is available at PNAS Nexus online.

Funding

This work was supported by the NIH Fogarty International Center (R21TW012486), the Technology Accelerator Challenge Prize from the NIH National Institute of Biomedical Imaging and Bioengineering, Veterans Affairs Merit Award (I01BX005816), the Ministry of Trade, Industry & Energy (MOTIE) in Korea (P0017306), and the Ralph W. and Grace M. Showalter Trust.

Author Contributions

Y.J. and Y.L.K. conceived the idea, developed the setup, and design the experimental design. Y.J., S.M.P., J.W.L., and V.V.N. worked on the experimental setup, imaging, and analyzing the data. Y.J., S.M.P., S.K., V.V.N., Y.T., and Y.L.K. participated in the analyses. Y.J. and Y.L.K. wrote the manuscript. Y.L.K. directed the overall research. All of the authors discussed the results and the manuscript.

Data Availability

Modified data and source codes are available at https://purr.purdue.edu/publications/4244/1 (Kim, Y. L. (2023). mHealth hyperspectral learning for hemodynamics. Purdue University Research Repository. doi:10.4231/VAFP-DW68). Data have been modified to protect potentially identifying or sensitive patient information.