Bayer Filter Snapshot Hyperspectral Fundus Camera for Human Retinal Imaging

Abstract

Purpose

To demonstrate the versatility and performance of a compact Bayer filter snapshot hyperspectral fundus camera for in-vivo clinical applications including retinal oximetry and macular pigment optical density measurements.

Methods

12 healthy volunteers were recruited under an Institutional Review Board (IRB) approved protocol. Fundus images were taken with a custom hyperspectral camera with a spectral range of 460–630 nm. We determined retinal vascular oxygen saturation (sO₂) for the healthy population using the captured spectra by least squares curve fitting. Additionally, macular pigment optical density was localized and visualized using multispectral reflectometry from selected wavelengths.

Results

We successfully determined the mean sO₂ of arteries and veins of each subject (ages 21–80) with excellent intrasubject repeatability (1.4% standard deviation). The mean arterial sO₂ for all subjects was 90.9% ± 2.5%, whereas the mean venous sO₂ for all subjects was 64.5% ± 3.5%. The mean artery–vein (A–V) difference in sO₂ varied between 20.5% and 31.9%. In addition, we were able to reveal and quantify macular pigment optical density.

Conclusions

We demonstrated a single imaging tool capable of oxygen saturation and macular pigment density measurements in vivo. The unique combination of broad spectral range, high spectral–spatial resolution, rapid and robust imaging capability, and compact design make this system a valuable tool for multifunction spectral imaging that can be easily performed in a clinic setting.

Introduction

Hyperspectral retinal imaging is a novel technique that can capture spatially resolved spectral information from the retina noninvasively. Conventional fundus imaging detects either monochromatic light or red, green, and blue (RGB) color channels that are reflected by retinal structures, providing great spatial details, but limited spectral information, of the retina.^1,2 Hyperspectral imaging adds the ability to quantify the spectral composition of the light reflected by the retina. As different structures and chemicals have unique spectral reflectance properties, hyperspectral imaging provides another dimension of information for researchers studying metabolism and chemical structure in human tissue.³

Spectral measurements of retinal vessel oxygenation allow for the study of tissue perfusion and metabolic demands of healthy retina, as well as changes that occur in diabetes and diabetic retinopathy.^4–8 Several popular commercial oximetry devices such as Oxymap and Imedos are based on a two-wavelength method for computing retinal vessel oxygenation.^9,10 Newer hyperspectral systems have the ability to use the entire absorbance spectra of hemoglobin in determining the oxygen saturation,^11,12 a method which reduces the influence of vessel thickness and backscatter from whole blood cells.^13–15

Similarly, retinal reflectance spectra can be used to study molecules (hemoglobin, macular pigment, melanin) and cell types within the retina.^16–18 These analyses have clinical utility in studying diseases of the retina such as age-related macular degeneration (AMD),¹⁹ where loss of macular pigment and retinal pigment epithelium (RPE) cells are characteristic features. For example, the study of macular pigment optical density (MPOD) has helped researchers understand the protective role of macular pigment in AMD.^20,21 Hyperspectral retinal reflectometry has greatly simplified the study of MPOD compared to prior methods.²²

Hyperspectral imaging in ophthalmology has evolved from coarsely adapted industrial prototypes with lengthy imaging time to specially designed snapshot systems that have improved potential for clinical applications. Early hyperspectral systems collected spectral information in an image by scanning either the wavelength (λ) or spatial dimension (x,y) in the image plane.^11,23 These scanning-type systems required seconds to minutes to acquire a complete dataset and relied on complicated image rectification when imaging a rapidly moving object, like the eye.¹² More modern hyperspectral systems have combined high-resolution detectors with sophisticated optics to allow for simultaneous imaging at multiple wavelengths. Without lag or scanning time, snapshot systems collect an information-dense hyperspectral data cube in a single image.²⁴ Unfortunately, these existing systems are often bulky, require precision alignment, and/or specialty optics, such as prisms,²⁴ multiple aperatures,²⁵ or gratings,¹⁷ which make them difficult to install, align, and operate in a clinic setting.

Herein we describe a snapshot hyperspectral imaging system that is compact and versatile and allows for quick, noninvasive spectral study of the retina without moving parts for spectral or spatial scanning, imaging lag, or complicated optical components. We realized 16 wavelength channels from 460 to 630 nm and 256 × 512 pixels of resolution using a mosaic layout of mono-lithographically fabricated Fabry–Perot filters atop a complementary metal-oxide-semiconductor (CMOS) detector array (CMV2000, CMOSIS, Imec, Belgium). This novel design does not require any additional optical image splitting or complicated filters.^17,24,26 It can capture hyperspectral images with the ease and flexibility of a normal CMOS camera, allowing for simple integration to standard fundus cameras such as those commonly found in ophthalmology clinics.²⁷

In this work, we demonstrate the versatile and robust data collection capabilities of this compact snapshot hyperspectral camera in clinical ophthalmology applications including spectra-based retinal oximetry measurements and MPOD measurements. These analyses provide the proof-of-principle of the performance of the system and allow us to study the strengths and limitations of this tool.

Methods

System

The hyperspectral fundus camera system is shown in Figure 1. We installed the hyperspectral detector onto the camera port of a commercial fundus camera (Topcon TRC-50EX). The hyperspectral imaging detector (inset in Figure 1a) is a prototype. It consists of a Bayer filter mosaic pattern of 16 different Fabry–Perot narrow-band spectral filters, with transmission bandwidths of 10–15 nm, ranging from 460 to 630 nm, manufactured mono-lithographically onto a CMOS imaging sensor (1024 × 2048 CMOS pixels, CMOSIS CMV2000, Imec, Belgium). The filters are arranged in a 4 × 4 mosaic pattern, with each filter covering a single CMOS pixel. The mosaic pattern of filters is repeated across the CMOS sensor, resulting in an effective spatial resolution of 256 × 512 hyperspectral pixels consisting of 16 wavelength bands as shown in Figure 1c. The detector chip is housed in an Adimec quartz series camera body (Q-2A340, Adimec, The Netherlands) measuring 80 mm² and weighing 400 g. The optical schematic is shown in Figure 1b. For clarity, we only show the path of light reflected from retina. The detector was installed on a commercial fundus camera via an off-the-shelf relay lens system (C-mount, TL-209 relay lens, Topcon, Japan). To prevent second-order transmission of Fabry–Perot filters outside the spectral range of the detector, a band-pass filter set (long-pass filter: OD4-450 nm, EdmundOptics, band-pass filter BG-38 VIS, EdmundOptics) was installed in the lens mount of the camera to pass light between 460 and 630 nm. The spectral response of each channel can be found in our previous report.²⁷ More detailed information regarding this hyperspectral imaging detector can be found at Imec’s website.²⁸

Figure 1.

Clinical imaging system. (a) The compact Imec snapshot hyperspectral detector (inset) is integrated with a commercial fundus camera. (b) Optical schematic of detector and relay lens system. (c) Diagram of the detector with the filter mosaics. Each filter mosaic is a 4 × 4 array of filters printed on a 4 × 4 area of pixels.

The camera was connected to a computer via the Camera Link data port. The electronic shutter was controlled by software provided by Imec, and the raw data was processed off-line with Matlab software (The Mathworks Inc., Natick, MA, USA).

Subjects

This study was approved by the Institutional Review Board of Northwestern University and adhered to the tenets of the Declaration of Helsinki for research involving human subjects. Each participant signed an informed consent form prior to inclusion.

We recruited 12 volunteers without known ocular disease for fundus imaging with the snapshot hypersectional system. Volunteers were included regardless of refractive errors. The mean age of the group was 44.9 ± 20.7, range 21–80 with 7 males and 5 females. Volunteers underwent pupil dilation with a combination of tropicamide 1% and phenylephrine hydrochloride 2.5% eye drops. For each subject, images were taken of the optic disc at 20° field of view (FOV) for oxygen saturation measurements and of the macula at 35° FOV for MPOD mapping.

Retinal oximetry

From the collected hyperspectral fundus images, the vessels were manually segmented, and the vessel oxygen saturation (sO₂) was measured using a multi-wavelength curve fitting analysis. The method was described in our previous reports.²⁷ We first calculated the optical density (OD) spectra of blood vessels by

$$\text{OD}=-{\text{log}}_{10}\left(\frac{I_{\text{in}}}{I_{\text{out}}}\right)$$ (1)

where I_in is the spectrum of a vessel extracted from the hyperspectral cube and I_out is the spectrum from the area adjacent to the vessel. We then used least squares method to fit the OD to the equation,

$$\begin{array}{cc}\hfill \text{OD}(\lambda )=& B-N\text{ln}(\lambda )\hfill \\ \hfill & +A\left[{\mu }_{\text{HbR}}(\lambda )+(1-{\text{sO}}_2){\mu }_{{\text{HbO}}_2}(\lambda )\right]\hfill \end{array}$$ (2)

where B and N represent wavelength-independent and wavelength-dependent optical scattering, respectively; λ is the optical wavelength; A is the coefficient related to the experimental geometry and vessel diameter; μ_HbR(λ) and μ_HbO2(λ) are the effective attenuation coefficients of the fully deoxygenated and oxygenated blood, respectively. To increase the accuracy of the calculation, we convolved the oxygenated/deoxygenated whole blood spectra with the spectral response of the camera in the fitting. This method is visually represented in Figure 2.

Figure 2.

Example vessel selection and analysis. Vessels near the optic disc are selected, and their spectra are analyzed to determine the content of oxygenation and deoxygenated blood, after convolving the measured spectra with the spectral response of the camera.

The retinal oximetry measurements were validated experimentally with bovine blood specimens. The specimens were exposed to either room air, or 100% nitrogen to simulate oxygenated and deoxygenated blood. The specimens were then placed in 150 μm deep chambers made of glass slides and coverslips and imaged with the hyperspectral camera. For validation, the same slides were imaged with a well-calibrated spectrometer (Shamrock 303i, Andor. 150 line grating, 0.88 nm spectral resolution). Using Equation 2 we determined the sO₂ of the blood specimens. The sO₂ of oxygenated and deoxygenated specimens were 0.97 and 0.68, respectively, as measured with the spectrometer. The measurements made with the hyperspectral camera varied by less than 3% (0.99 and 0.68 for oxygenated and deoxygenated blood, respectively).

Macular pigment optic density

We calculated MPOD as

$$\begin{array}{cc}\hfill D_{\text{MP}}\left(465\right)& =\frac{0.5}{K_{\text{MP}}\left(465\right)-K_{\text{MP}}\left(547\right)}\hfill \\ \hfill & \times \left(\text{log}\frac{R_P\left(465\right)}{R_F\left(465\right)}-\text{log}\frac{R_P\left(547\right)}{R_F\left(547\right)}\right)\hfill \end{array}$$ (3)

where D_MP(465) is the macular pigment OD with 465 nm wavelength; K_MP(465) and K_MP(547) are optical extinction coefficients with 465 and 547 nm, respectively; R_F and R_P are light reflectance at fovea and perifovea, respectively.¹⁶

Retinal reflectance maps were extracted from the corresponding channels in the hyperspectral image datacube. We chose the 95 percentile of MPOD as the maximum value, and normalized the MPOD to that maximum value. The purpose of this normalization is to provide a fair comparison between each subject and minimize the estimation bias from the calculation method.

In order to better visualize the location of macular pigment, we overlaid MPOD maps with vessel maps extracted from hyperspectral image datacube. The vessel maps were generated by comparing the optical densities between 562 and 509 nm:

$$D_V=\text{log}\frac{R\left(509\right)}{R\left(562\right)}$$ (4)

where D_V is the relative OD difference between 562 and 509 nm; R(509) and R(562) are the reflectance extracted from the corresponding channels in the hyperspectral image datacube. These channels were specifically chosen because they highlight blood, which shows a large difference in OD at these wavelengths, while the retinal tissue does not. The vessel contrast was further enhanced using Hessian-based Frangi filter for better visualization.²⁹

Results

An example of visualization of the hyperspectral datacube output is shown in Figure 3. Spectral analysis of the fundus images enhances visualization of distinct features in the retina. For example, vessel detail is best seen in wavelengths greater than 550 nm where hemoglobin strongly absorbs light causing the vessels to appear dark.

Figure 3.

Visualization of hyperspectral datacube with false coloring applied to emphasize the different channels (50° field of view).

An example of the generated sO₂ map is shown in Figure 4. The colored maps help highlight variation between arteries and veins, as well as sO₂ variation within the same vessel. The mean sO₂ variation within arteries ranged from 3.2% to 14.3% across subjects. In veins, the mean sO₂ variation was between 3.2% and 14.9%. In both cases, variability was greatest inside the optic disc.

Figure 4.

Example vessel oxygen saturation (sO₂) map. (a) Vessels near the optic disc are manually segmented and then the OD is computed per Equation 2. (b) sO₂ for each vessel is shown, and then (c) averages are computed for the image. Arteries are labeled in red and veins are labeled in blue.

The results of the vessel oxygenation analysis are shown in Table 1. The arterial sO₂ value reported for each subject represents the mean sO₂ across all arteries, where the sO₂ of each artery is computed by averaging the sO₂ of all pixels in that vessel. Likewise, the reported standard deviation (std) represents the mean, across all arteries, of the standard deviation of sO₂ within each artery. The venous measurements were computed in the same fashion. The mean arterial sO₂ for all subjects was 90.9% ± 2.5%, whereas the mean venous sO₂ for all subjects was 64.5% ± 3.5%. The mean A–V difference varied between 20.5% and 31.9% among the group.

Table 1.

Oximetry results for healthy controls.

			Arteries sO2 (%)		Veins sO2 (%)		A–V difference sO2 (%)
Subject ID	Sex	Age	Mean	std	Mean	std	Mean	std
p019	M	28	92.1	5.0	62.0	3.2	30.1	6.0
p020	M	29	93.0	3.2	61.1	2.2	31.9	3.9
p021	M	24	89.3	8.0	64.4	7.5	25.0	10.9
p022	F	31	89.9	13.2	58.8	14.3	31.1	19.5
p069	F	21	92.0	12.9	64.9	14.9	27.1	19.7
p070	M	59	85.2	6.6	61.2	5.0	24.0	8.3
p076	F	55	92.6	14.3	62.9	11.0	29.8	18.0
p077	F	80	89.1	7.1	66.7	5.3	22.4	8.8
p080	M	27	91.3	6.0	70.8	5.9	20.5	8.4
p081	F	74	95.4	9.7	70.0	7.7	25.4	12.4
p083	M	61	91.4	8.0	64.7	8.2	26.7	11.4
p084	M	50	91.2	8.2	67.5	6.0	23.6	10.2

Note: sO₂: oxygen saturation; std: standard deviation.

To assess the repeatability of the system, we compared the vessel sO₂ from six consecutive images of the same eye within 5 minutes. We first spatially registered the six image datasets to ensure that each pixel in different datasets represents the same location in the retina. We then calculated sO₂ from the six datasets separately. Finally, we calculated the sO₂ standard deviation of each pixel from the six sO₂ maps. The standard deviations of sO₂ of single pixels within the vessel area are all less than 5%, with an average value of 1.4%, indicating excellent sO₂ measurement repeatability.

MPOD was calculated (per Equation 3) and mapped in combined with vessel identification (per Equation 4). In the image overlay (Figure 5), we pseudocolored the vessel maps in red and MPODs in yellow. Here we see a uniform distribution of macular pigment in the macula of the 24-year-old subject. In comparison, the macular pigment distribution of the 80-year-old subject is less focal and more variable in density.

Figure 5.

Normalized macular pigment optical densities (yellow) overlaid with contrast-enhanced retinal vessel maps (red) from (a) a 24-year-old volunteer and (b) an 80-year-old volunteer. Field of view, 35°.

Discussion

In this report we demonstrate a snapshot hyperspectral system capable of acquiring high-quality spectral images in vivo. To our knowledge, this is the first Bayer filter mosaic type snapshot hyperspectral detector demonstrated in ophthalmology. This system is compact, easily integrated, and straightforward to operate. The mosaic filter design provides good spectral and spatial resolution without the need for complicated post-processing or image reconstruction.

We have also demonstrated the clinical utility of the system by performing retinal oximetry measurements on a series of healthy subjects. The data from our small cohort is in accordance with previously published retinal oximetry data showing arterial sO₂ of healthy subjects between 93–98% ± 7–10% and venous sO₂ of 65% ± 5–12%.^5,7 The variation in the measurement between subjects could be due to physiologic difference in retinal O₂ consumption, subclinical pathology (small vessel disease) or differences in media opacity or fundus pigmentation.^30,31 In addition, the data from short-period consecutive images on one subject reveals excellent repeatability (average standard deviation 1.4%) in vessel sO₂ measurements compared with a previous study (2.5–3.3%).⁵ These results provide evidence that the system is capable of reliable performance suitable for in-vivo studies.

A limitation of this oximetry method is that it is not valid for vessels overlying the optic nerve head (ONH). As seen in Figure 4, the oximetry maps overlying the ONH were highly variable. From Equation 1, I_out and I_in represent the light reflected in a single pass from the area in and around the vessel. For normal retina, this is a reasonable assumption. However, the turbidity of light reflected from the ONH makes the single pass assumption invalid for vessels overlaying that area.³² For this reason, these vessel segments were exclude from the analysis.

As demonstrated in Figure 5, spectral decomposition of retinal reflectance helps to highlight specific structures in the retina. Using multiple channels of the hyperspectral datacube we have demonstrated the ability to identify, localize, and quantify vessels and macular pigment. Our analysis of MPOD reveals a distribution of macular pigment that is in agreement with previous publications.^17,33,34 In the example shown in Figure 5, we see evidence of macular pigment disturbances in the macula of an 80-year-old subject. This likely represents normal physiologic changes in the setting of aging,^35–37 although decreased MPOD has also been implicated in pathologic conditions.²⁰ The inherent variation in MPOD measurements, combined with our small sample size would make quantitative comparisons difficult.^{16,33,35,38,39} Instead, we chose the MPOD maps as a preliminary demonstration, while ultimately a larger cohort is needed to assess the performance of this analysis. We feel that the high resolution spectral–spatial data provided by this system can be harnessed with improved software tools to improve the detection, localization, and even identification of retina components and pathological elements.

One limitation of the system is the presence of significant second-order effects in several channels. In many cases, the second-order contributions of the filters were up to 50% of the intensity of the primary response, making spectral decomposition a challenge. In the current analysis, we corrected for these effects by convoluting the measured spectra with the response of the filters. We are currently exploring options to improve the performance of the system by using different optical filters.

An additional downside of this system is the limited spectral range of the detector. While the 460 nm to 630 nm spectral range is well suited for oximetry and MPOD measurements, it is limited near the UV and IR boundaries of the visible spectra. The mosaic design of the detector creates a tradeoff between spatial and spectral resolution as shown in Figure 1. The mosaic uses 4 × 4 pixels (each pixel is 5.5 μm on a side), making the spatial resolution 22 μm at the detector and roughly 15 μm at the retina. A larger mosaic with more spectral channels will result in a decrease in spatial resolution. Fortunately, a reduction in pixel size can improve the resolution. Future detectors with an expanded spectral range will allow for better color approximation of the data and may help to visualize additional spectral components of the retina that lie outside the current spectral range of this detector.

Conclusion

In summary, we have demonstrated a Bayer filter snapshot hyperspectral camera capable of instantaneous spatial–spectral datacube acquisition from the fundus of healthy subjects. The size and optical design of the camera make it simple to integrate and operate. We demonstrated robust vessel oximetry and measurement of the MPOD with good repeatability. The versatility and simplicity of this tool make it well suited for further research and clinical applications.