Real-time, wide-field, and quantitative oxygenation imaging using spatiotemporal modulation of light

Abstract.

Quantitative diffuse optical imaging has the potential to provide valuable functional information about tissue status, such as oxygen saturation or blood content to healthcare practitioners in real time. However, significant technical challenges have so far prevented such tools from being deployed in the clinic. Toward achieving this goal, prior research introduced methods based on spatial frequency domain imaging (SFDI) that allow real-time (within milliseconds) wide-field imaging of optical properties but at a single wavelength. However, for this technology to be useful to clinicians, images must be displayed in terms of metrics related to the physiological state of the tissue, hence interpretable to guide decision-making. For this purpose, recent developments introduced multispectral SFDI methods for rapid imaging of oxygenation parameters up to 16 frames per seconds (fps). We introduce real-time, wide-field, and quantitative blood parameters imaging using spatiotemporal modulation of light. Using this method, we are able to quantitatively obtain optical properties at 100 fps at two wavelengths (665 and 860 nm), and therefore oxygenation, oxyhemoglobin, and deoxyhemoglobin, using a single camera with, at most, 4.2% error in comparison with standard SFDI acquisitions.

1. Introduction

Diffuse optical imaging using either temporal or spatial modulation of light has shown strong promise for quantitatively assessing functional parameters in living tissue, such as content in oxyhemoglobin, deoxyhemoglobin, water, and lipids. Such parameters can be used to determine the physiological condition of the tissue and provide critical information to healthcare professionals for decision making. For instance, temporal frequency domain monitoring of breast tissue during neoadjuvant chemotherapy treatment can help differentiating responding versus nonresponding patients within 10 days of treatment.^1,2 In other studies, spatial frequency domain imaging (SFDI) has demonstrated the potential to diagnose patients early with diabetic foot ulcers, providing means to adapt treatment toward limiting morbidity and decreasing healthcare burden.^3,4

While these methods have demonstrated a strong potential to aid in decision making and solve concrete clinical problems, their adaptation to challenging clinical environments has been very limited. Surgery is one of these environments where information must be provided quantitatively (i.e., repeatable and interpretable), in real time (i.e., $>25$ images per seconds), and over a large field-of-view (i.e., $>10\times 10\mathrm{c}{\mathrm{m}}^2$).⁵ To attain this challenging objective, our group has been developing solutions based on SFDI. Because SFDI works in the frequency domain, it allows the capture of information for all the pixels of an image at once, providing a significant advantage over raster-scanning methods. The original developments of SFDI relied on the acquisition of several images at single wavelengths, providing quantitative multipixel optical properties measurements.^6–8 More recent developments have introduced near-real-time [e.g., 10 frames per second (fps)] and real-time (i.e., $>25\mathrm{fps}$) imaging methods such as single snapshot of optical properties (SSOP).^9–11 These methods process the acquired information directly in the frequency domain offering the opportunity to drastically reduce the acquisition time down to a single acquired image for obtaining optical properties maps of living tissues. However, these methods still acquire only one wavelength at a time, making them suboptimal for imaging living tissue parameters (e.g., chromophores concentration) in real time.

In order to deploy such technology in the operating room, methods capable of performing multi- or hyperspectral imaging quantitatively, over a large field-of-view and in real time, must be developed. Recent work has proposed dual-wavelengths imaging for extracting blood parameters such as the concentration of oxyhemoglobin, deoxyhemoglobin, or oxygenation. For instance, SSOP was used in conjunction with speckle imaging using a scientific CMOS camera by sequentially acquiring wavelengths, leading to 16 fps imaging.¹² However, this work uses a mechanical optical chopper and low-power (150 mW) laser diodes, making it difficult to achieve higher frame rates and limiting its scalability to a higher number of wavelengths. Another, more recent work used square patterns to improve switching time using a standard digital micromirror device (DMD) to project three wavelengths sequentially at three phases (standard SFDI acquisition), and compressed single pixel imaging leading to 12.5 fps imaging.¹³ However, relying on sequential frames to acquire optical properties maps naturally restricts the possibility to achieve higher frame rates. As of today, because of these instrumental limitations, no method has been developed for real-time (i.e., $>25$ images per seconds), wide-field (i.e., $>10\times 10\mathrm{c}{\mathrm{m}}^2$), and quantitative imaging for clinical applications.

In this work, we present a method using both spatial and temporal modulation of light to provide images interpretable by clinicians in time-constrained environments such as surgery. We propose here a method where sources are modulated sinusoidally in time using signal generators (no mechanical chopper) and projected in the form of single-frame patterns for processing using the SSOP method (no need for sequential frames). Sources are modulated in time to encode wavelengths and combined in a multifiber bundle connected to a spatial-light modulator. Light is then projected with specific patterns onto the field-of-view and images captured using a fast camera. The obtained three-dimensional (3-D) cube of data (two-dimensions + time) is then processed in the temporal frequency domain to separate wavelengths, and the resulting images are processed in the spatial frequency domain using the SSOP methodology to extract optical properties for each wavelength and then calculate blood parameters (oxyhemoglobin, deoxyhemoglobin, and oxygenation). In the first part of this paper, we present the imaging method, processing, and hardware. Then, we validate the method in comparison with standard, non-real-time SFDI onto tissue-mimicking phantoms. Finally, we perform a validation on a living hand during an arm-cuff occlusion. Altogether, this study contributes to the translation of endogenous diffuse optical imaging for surgical guidance.

2. Material and Methods

2.1. Spatiotemporal Modulation of Light

The concept of multispectral imaging using temporal modulation of light sources for wavelength encoding is not new, nor are multispectral spatial frequency domain methods.^13–18 However, until now, imaging living tissue parameters over large fields-of-view at high resolution, quantitatively, and in real time remained a significant challenge. The principle underlying the spatiotemporal modulation of light method is shown in Fig. 1. Discrete sources are modulated in time, each wavelength being at a different specifically chosen temporal frequency. The light outputs from all sources are combined using a multifiber bundle connected to a spatial light modulator. Patterns of light combining all temporally modulated wavelengths are projected onto the field-of-view and a fast camera is used to acquire the scene. The acquired images are first processed using a temporal discrete Fourier transform (DFT) to accurately separate wavelengths contributions. Finally, the resulting images are processed in the spatial frequency domain using SSOP methodology to obtain the optical properties at the chosen wavelengths.

Fig. 1.

Spatiotemporal modulation of light: acquisition and processing. Sources at different wavelengths are modulated in time at specifically chosen frequency. Light from these sources is combined and modulated in space using a spatial light modulator. The scene is acquired using a fast camera. The acquired images are then processed in the temporal frequency domain to separate the contribution from the different light sources, and processed in the spatial frequency domain to extract the optical properties at all wavelengths.

One of the significant hurdles in implementing such method is in ensuring that wavelengths are precisely and correctly modulated, sampled, and processed to unambiguously extract the information content contained in each temporally modulated source. Not performing these operations correctly would result in inaccuracies in the extracted optical properties and therefore in the assessment of tissue parameters. Two strategies can be used to perform these measurements: either a fine temporal sampling of the scene allowing to use a generic fast Fourier transform (FFT) function to separate any temporal frequency contribution, or a custom DFT function allowing to sample only a chosen number of frequencies. While a generic FFT may be more suited to analyze a large amount of contributions, it can be more costly in terms of acquisition and processing time compared with a custom DFT. In this work, because we focus on acquiring real-time images, we chose to implement a custom DFT to allow ultrafast computation of the chosen sources’ contributions. The DFT of one pixel in time is given as

$$X(k)=\sum _{n=1}^{N}x(n).e^{-\frac{j2\pi k(n-1)}{N}},$$ (1)

and the proper frequencies that should be used to modulate are exact frequencies in the discrete Fourier domain, for instance, in the case of four sources:

$$f_1=\frac{k_1.F_e}{N};f_2=\frac{k_2.F_e}{N};f_3=\frac{k_3.F_e}{N};f_4=\frac{k_4.F_e}{N},$$ (2)

with $F_e$ being the sampling frequency, $N$ is the number of acquired images, and $k_i$ determined such that the captured signal spans an integer number of periods.^19,20

If these conditions are gathered, on top of obvious considerations such as avoiding harmonics or standard electrical current frequencies, and along with a precise synchronization of the camera and signal generators, the contribution of each source can be precisely and unambiguously determined.

2.2. Acquisition and Processing

In this work, we performed acquisitions at a sampling frequency $\mathrm{Fe}=100\mathrm{Hz}$, processing $N=10$ images, and modulated two wavelengths to determine oxygenation, ${\lambda }_1=665\mathrm{nm}$ with $k_1=3$ ($f_1=30\mathrm{Hz}$) and ${\lambda }_2=860\mathrm{nm}$ with $k_2=2$ ($f_2=20\mathrm{Hz}$).²¹ The number of images ($N=10$) was chosen such that all parameters have been adjusted to satisfy the conditions for proper DFT processing (Sec. 2.1) while maintaining a proper signal-to-noise ratio (SNR) in the acquired images. We implemented a custom rolling window algorithm of the DFT method onto Matlab with the parallel computing toolbox (using a graphics processing unit, GPU) by adding the $N+1$ term of the DFT sum and subtracting the first term of the DFT sum in real time. More details regarding the GPU implementation of the processing method are available elsewhere.²⁰

All acquisitions have been performed on the imaging system described in the next section. In order to handle the large flux of data, as well as control the hardware and perform GPU processing of the acquired images, a personal computer with the following characteristics was used: Intel i7-7800x 3.5 GHz central processing unit, 16 GB of RAM, four 1 TB solid-state drives for data acquisition and one 500 GB solid-state drive for system operation, and an NVIDIA GeForce GTX1080TI GPU.

2.3. Imaging System

The instrumental setup was custom built using a DMD (Vialux, Germany) for the projection of custom patterns, fiber-coupled to two laser diodes at 665 and 860 nm (LDX Optronics, Maryville, Tennessee). Diodes were mounted in a temperature-controlled mount (TCLDM9, Thorlabs, Newton, New Jersey). Intensity from the diodes was controlled using current controllers (TDC240C, Thorlabs, Newton, New Jersey) and temperature controlled using thermoelectric cooler controllers (TED200C, Thorlabs, Newton, New Jersey). Diodes were modulated using a dedicated 20-MHz dual channel function generator (4047B, BK Precision, Yorba Linda, California). The laser diodes were coupled into a 2-to-1 fiber bundle (BFY1000HS02, Thorlabs, Newton, New Jersey) connected to the DMD. The projection system projects a sine wave pattern over a $200\times 150{\mathrm{mm}}^2$ surface at 45-cm working distance. Images were acquired over a $130\times 130\mathrm{m}{\mathrm{m}}^2$ field-of-view and at a resolution of $1024\times 1024\text{\hspace{0.17em}\hspace{0.17em}pixels}$ using a scientific-grade monochrome sCMOS camera (PCO Edge 5.5, Kelheim, Germany). The camera and signal generators were synchronized together using a second 20-MHz dual channel function generator as a reference (4047B, BK Precision, Yorba Linda, California). Polarizers (PPL05C, Moxtek, Orem, Utah), arranged in a crossed configuration, were used to minimize the contribution from specular reflections at the surface of the sample. A silicone-based optical phantom was used for calibration and built using titanium dioxide (${\mathrm{TiO}}_2$) as a scattering agent and India ink as an absorbing agent.²² Its large size ($210\mathrm{mm}\times 210\mathrm{mm}\times 20\mathrm{mm}$) accommodates the system’s field-of-view with reduced scattering ${\mu }_s^{\prime }=1.08{\mathrm{mm}}^{-1}$ and absorption ${\mu }_a=0.012{\mathrm{mm}}^{-1}$ at 665 nm, and ${\mu }_s^{\prime }=0.78{\mathrm{mm}}^{-1}$ and ${\mu }_a=0.022{\mathrm{mm}}^{-1}$ at 860 nm.

2.4. Validation Experiments

2.4.1. Acquisition protocol

SFDI and spatiotemporal modulation of light experiments were performed sequentially. SFDI was used as a reference by acquiring a total of six images for each wavelength (with no temporal modulation) at spatial frequencies of 0 and $0.3{\mathrm{mm}}^{-1}$ with three evenly spaced phases each to extract the amplitude modulation and the calibrated diffuse reflectance. A lookup table (LUT) generated from Monte–Carlo was used to extract the optical properties for each wavelength following acquisition. Extensive details regarding the SFDI acquisition and LUT processing method can be found elsewhere.^7,23,24 Spatiotemporal acquisitions were performed at 30 Hz for 665 nm and 20 Hz for 860 nm. Acquisition rate of the camera was set at 100 fps. The SSOP projection frequency was $0.3{\mathrm{mm}}^{-1}$ and the acquired data postprocessed, first in time to extract contributions from the sources, and then in space to extract the optical properties for each wavelength. More details regarding the temporal processing can be found in Ref. 20, and details regarding the SSOP acquisition and processing method can be found in Refs. ⁹ and ¹¹.

2.4.2. Phantom experiments

A set of 10 tissue-mimicking phantoms were built using titanium dioxide (${\mathrm{TiO}}_2$) as a scattering agent and nigrosin as an absorbing agent.²⁵ Phantoms were all acquired using both SFDI and spatiotemporal modulation of light, postprocessed, and the mean error in percent calculated over $30\times 30\mathrm{m}{\mathrm{m}}^2$ region of interest taken at the center of each phantom.

2.4.3. In vivo experiments

A human hand was used to illustrate the capacity of the method to work properly during in vivo conditions and in real time. First, a still hand was imaged using SFDI and spatiotemporal modulation of light to compare in vivo results using both methods. Mean error in percent calculated over $50\times 50\mathrm{m}{\mathrm{m}}^2$ region of interest taken on the back of the hand. Then, a simple arm-cuff occlusion experiment was performed during 10 min, with 3 min of rest, 4 min of occlusion, and 3 min of recovery. Images were acquired and postprocessed using the spatiotemporal modulation of light method and the absorption coefficients used to determine oxygen saturation through the calculation of the concentrations of oxy- and deoxyhemoglobin.²¹

3. Results

3.1. Phantom Experiments

Results from the phantom experiments are summarized in Fig. 2. On average, optical properties obtained using spatiotemporal modulation of light show good agreement with the standard SFDI method. Mean errors were calculated for all the pixels in a $30\times 30\mathrm{m}{\mathrm{m}}^2$ region taken in the middle of each phantom. For 665 nm, the mean error is $1.8\%\pm 1.2\%$ in absorption and $3.3\%\pm 1\%$ in reduced scattering. For 860 nm, the mean error is $4.2\%\pm 2.1\%$ in absorption and $2.5\%\pm 1.1\%$ in reduced scattering.

Fig. 2.

Results from phantoms experiments. Absorption and reduced scattering are shown (mean and standard deviation) over a wide range of optical properties. Results are shown for both wavelengths using standard SFDI as a reference and spatiotemporal modulation of light.

Actual images of four phantoms are shown in Fig. 3 for both wavelengths, in absorption and reduced scattering. Results from standard SFDI are shown, as well as spatiotemporal modulation of light, a percentage error map comparing both methods, and a line profile. Overall good visual agreement is confirmed in these images.

Fig. 3.

Actual images of tissue mimicking phantoms. Absorption and reduced scattering images of four phantoms are shown over a range of optical properties. Results are shown for both wavelengths using standard SFDI as a reference and spatiotemporal modulation of light. In addition, relative error maps are shown and line profiles compared (dashed blue: standard SFDI, solid red: spatiotemporal modulation of light).

3.2. In Vivo Experiments

Images from the still hand are shown in Fig. 4 for both wavelengths, in absorption and reduced scattering. Results from standard SFDI are shown, as well as spatiotemporal modulation of light, a percentage error map comparing both methods, and a line profile. Overall good visual agreement is confirmed in these images. Mean errors were calculated for all the pixels in a $50\times 50\mathrm{m}{\mathrm{m}}^2$ region on the back of the hand. For 665 nm, the mean error is $1.9\%\pm 2.0\%$ in absorption and $2.0\%\pm 0.7\%$ in reduced scattering. For 860 nm, the mean error is $3.4\%\pm 1.9\%$ in absorption and $0.3\%\pm 1.7\%$ in reduced scattering. These results confirm the phantom experiments with similar error values on average and allow to appreciate the image quality of the extracted optical properties maps. In particular, the vein structures are clearly visible on the absorption images confirming the potential for the spatiotemporal modulation of light method to perform adequately for clinically relevant use.

Fig. 4.

Actual images of a still hand. Absorption and reduced scattering images of a still hand are shown over a range of optical properties. Results are shown for both wavelengths using standard SFDI as a reference and spatiotemporal modulation of light. In addition, relative error maps are shown and line profiles compared (dashed blue: standard SFDI, solid red: spatiotemporal modulation of light).

Finally, results from the arm-cuff occlusion experiment are shown in Fig 5. Significant decrease in oxygen saturation is visible during the occlusion, with values going from 78% down to 49%, following by a hyperemia phase after the release of the occlusion with a slow decrease back to the resting state.

Fig. 5.

Arm-cuff occlusion results. An arm-cuff occlusion was performed for 4 min, following 3 min of resting state, and followed by 3 min of recovery. Images were acquired using spatiotemporal modulation of light at 100 fps and processed for oxygenation. Top images show oxygen saturation at three time points during the baseline resting measurement, the occlusion, and after the release. The saturation of a point in the hand (blue square) is shown in the plot below. Time points of the above images are indicated by arrows.

A movie is shown in Fig. 6. Note that the movie frame rate has been decreased from 100 to 25 fps to reduce the size of the file.

Fig. 6.

Video of the arm-cuff occlusion (Video 1, MP4, 74.3 MB [URL: https://doi.org/10.1117/1.JBO.24.7.071610.1]).

4. Discussion

In this work, we present the application of a method, spatiotemporal modulation of light, to perform real-time (100 fps), wide-field ($13\times 13\mathrm{c}{\mathrm{m}}^2$), and quantitative imaging of blood parameters in vivo. This method relies on the modulation of light sources in time to encode wavelengths and on the modulation of the projection of these sources in space to extract the optical properties of living tissues. Combined together, these spatiotemporal modulations allow to gather multispectral quantitative optical information in real time (here at 100 fps and postprocessed). Altogether the method enables to provide healthcare professionals with interpretable information that can be used in real time to guide decision making in challenging clinical situations, such as surgery.

Compared to previous work, this study augments the capacity for performing multispectral diffuse optical imaging quantitatively at a high frame rate and over a large field-of-view. For instance, sequential acquisition of two wavelengths using both SSOP and laser speckle imaging acquisition methods led to 16 fps imaging of simultaneous blood parameters and perfusion imaging over a wide field-of-view ($15\times 15\mathrm{c}{\mathrm{m}}^2$).¹² In more recent work, compressed sensing was used along with square patterns projections and phase shifting leading to 12.5 fps imaging of blood parameters over a smaller field-of-view ($3.5\times 3.5\mathrm{c}{\mathrm{m}}^2$).¹³

However, this work comes with limitations. First, in this implementation, the SSOP method results in slight image degradation ^9–11 and the 3-D profile of the sample is not measured, which can cause errors in extracting optical properties onto complex, non-flat objects ($\sim 10\%$ error per cm for absorption).²⁶ Second, noise in oxygenation data, mainly caused by the poor quantum efficiency of the camera at 860 nm, as well as slight synchronization errors between the camera and the sources, resulted in the impossibility to observe fine variations such as the pulsatile nature of blood flow. Improvements in the imaging system will include stronger illumination at 860 nm and a custom light source for optimal camera-source synchronization to allow the observation of finer variations of oxygenation. Third, currently images are acquired in real time but postprocessed following acquisition. We are currently working on a hardware and software solutions enabling both real-time acquisition and real-time processing, making this technology deployable for real-time visualization of the surgical field.^24,27,28 Finally, while the concept can be expanded to virtually many more wavelengths, there is a significant instrumental challenges in acquiring properly all wavelengths with a good enough SNR, and within the limitations of the spectral quantum efficiency and the dynamic range of the camera. For instance, in this work, the 860-nm optical properties, necessary to optimally separate oxy- and deoxyhemoglobin, shows significantly more noise compared with 665 nm due to the decrease of the spectral quantum efficiency of silicon-based camera technology. While manageable for a few wavelengths, this certainly limits the acquisition of a large number of wavelengths concurrently.

One advantage of this method though is its capacity to work in real time. Current CMOS technology offers cameras at a very reasonable price permitting good resolution and dynamic range imaging up to 500 fps. Our GP-GPU implementation of the temporal DFT in CUDA permits processing down to $18\mu \mathrm{s}$ per wavelength at a resolution of $512\times 512\mathrm{pixels}$.²⁰ However, the rolling window processing approach comes at the cost of a short delay between the acquired scene and the displayed information (here $1/10\mathrm{th}$ of a second, i.e., 100 ms). One potential solution to decrease this delay is to increase the acquisition rate (500 fps for instance) but it requires to increase the light fluence from the sources to maintain proper SNR and prevent noise in the images.

Finally, current state-of-the-art processing methods in the spatial frequency domain include optimized LUTs (down to 18 ms for $572\times 672\text{\hspace{0.17em}\hspace{0.17em}pixels}$),²⁴ as well as deep learning and machine learning methods (down to 200 ms for $696\times 520\text{\hspace{0.17em}\hspace{0.17em}pixels}$),^28,29 and soon, faster methods that will enable real-time processing and visualization of data will certainly be developed. Efforts are ongoing to combine rapid processing of SSOP data, with spatiotemporal modulation of light, onto a clinically compatible device to test and validate the capacity for diffuse optical imaging to provide relevant data in real time during concrete clinical applications.

5. Conclusion

In this work, we presented an approach for real-time, wide-field, and quantitative imaging of physiologically and clinically relevant data onto living tissues. Spatiotemporal modulation of light enables true real-time quantitative multispectral imaging opening unprecedented possibilities for translating such technology to clinically relevant applications where time is constrained, such as surgery. This study contributes to the translation of diffuse optical imaging methods to relevant clinical applications.

Biography

Biographies of the authors are not available.

Disclosures

B.J.T. discloses patents in SFDI technology that are assigned to the Regents of the University of California. These patents have been licensed to Modulated Imaging (MI), Inc.; B.J.T. is a cofounder of MI. All other authors declare that there are no conflicts of interest related to this article.