A Dual-Camera High-Resolution Hyperspectral Imaging System for the Retina

Abstract

Hyperspectral imaging (HSI) emerged as a powerful tool for biomedical applications, particularly in the analysis and discrimination of biological tissues. In this study, we developed a dual-camera system that integrates hyperspectral imaging with high-resolution RGB imaging to capture detailed retinal images. We tested different pan-sharpening algorithms to enhance the spatial resolution of the hyperspectral images, combining the spectral details of HSI with the spatial details of RGB imaging. We developed algorithms to estimate the diameter of retinal vessels and the oxygenation rate. We validated our algorithms by imaging a mouse retinal phantom and then by imaging mice under anesthesia. Our systems showed the ability to resolve fine structural details, including small blood vessels. We found that out of the tested methods, PSGAN offered the best pansharpened image both quantitatively and qualitatively, with a root-mean-squared error (RMSE) score of 2.15 × 10⁻². Using the pansharpened hyperspectral image, we measured vessels diameter and vessel oxygenation rate. We found the average diameter for arterioles and venules to be 45.7 μm and 31.5 μm, respectively. The average oxygenation rate for arterioles and venules were 96.2% and 76.3% respectively. This study represents a significant step towards the development of a versatile retinal imaging tool, with potential applications in both research and clinical diagnostics. Future work will focus on in vivo testing, algorithm refinement, and the exploration of specific retinal disease markers using this imaging system.

1. INTRODUCTION

Biological tissues hold essential spectral information that reflects their physiological and pathological states [1]. Traditionally, spectrometers were used to analyze these spectral signatures, but their limited spatial coverage restricted their utility. Hyperspectral imaging (HSI) has emerged as a transformative technology to overcome this limitation, offering a three-dimensional dataset, or hypercube, that integrates both spatial and spectral data. By capturing an extensive range of spectral signatures across large spatial areas, HSI has proven to be superior to traditional RGB imaging in various biomedical applications, such as quantifying blood oxygenation [2–4]. Over the last two decades, advancements in manufacturing and optical technologies have significantly improved the accessibility of HSI systems, making cameras more compact, portable, and affordable, further fueling interest in biomedical diagnostics [5].

The retina, an extension of the central nervous system (CNS) [6], is particularly suited for non-invasive in vivo imaging, providing a unique window into CNS-related conditions such as multiple sclerosis, Alzheimer’s disease (AD), and stroke. Conventional retinal imaging techniques, including fundus imaging, retinal angiography, and optical coherence tomography (OCT), are widely employed to investigate retinal biomarkers such as oxygen saturation and retinal layer thickness. However, these methods have inherent limitations. Angiography is invasive, while fundus imaging and OCT, although non-invasive, are costly and incapable of measuring blood vessel oxygenation. In contrast, HSI offers a compelling alternative by detecting specific spectral signatures associated with retinal biomarkers. Specifically, several retinal biomarkers can be detected using HSI. Oxygenation rate, or sO₂ for short, measured the percentage of oxygen bound to hemoglobin as a percentage of maximum possible oxygen bound. Because oxygenated and deoxygenated hemoglobin has different spectra [1], HSI can estimate the ratio of bound hemoglobin to unbound ones. The diameter of the arterioles and venules can serve as predictors of retinal metabolism [7, 8]. They can be measured if the magnification factor of the HSI system is known. Using either of these biomarkers in the retina, HSI has demonstrated its effectiveness in monitoring diabetic retinopathy, radiation retinopathy, age-related macular degeneration (AMD), and AD [7–12].

Despite these advances, retinal imaging using HSI presents unique technical challenges. Protecting the retina from excessive exposure requires minimizing imaging duration and ensuring that illumination levels remain within safe limits [5]. These constraints influence the choice of spectral acquisition hardware. Push-broom and spectral scanning cameras, while capable of high spectral resolution, often take several minutes to complete a single scan, making them impractical for retinal imaging. Snapshot imaging systems, particularly those leveraging spectrally resolved detector arrays (SRDAs), offer a viable alternative by capturing hyperspectral data in real-time. SRDAs utilize CMOS sensors with mosaic-patterned filters to create hyperspectral cubes through processing. Recent engineering innovations have made SRDA systems increasingly attractive for retinal imaging due to their compact form factor—often weighing under 100 grams—which facilitates the development of portable imaging devices [5]. However, SRDA-based snapshot systems face trade-offs, most notably in spatial resolution. The lower raster size of their images can obscure critical microvascular details necessary for accurate diagnosis.

To address these limitations, pan-sharpening techniques, which were used in remote sensing, have been employed to combine low-resolution hyperspectral images with high-resolution monochromatic images to create high-resolution hyperspectral data. Although pan-sharpening has been extensively applied in remote sensing [13–16], its use in biological imaging remains limited, with only a few notable examples, such as the work by Ma et al. [15] Building on this idea, we propose a novel dual-camera system that integrates snapshot hyperspectral imaging with high-resolution RGB imaging. Our approach is an extension of the topical endoscopy fundus imaging (TEFI) system, which is a cost-effective method to image retina of small mammals [17]. To enhance this system, we developed a software workflow capable of aligning and merging hyperspectral and RGB images, resulting in high-resolution hyperspectral images. Unlike previous TEFI-based method that utilized a beamsplitter for simultaneous fluorescence and RGB imaging [18], our system is the first to incorporate hyperspectral pan-sharpening in a beamsplitter configuration within TEFI.

Our contributions are: (1) The development of a high-speed, high-resolution hyperspectral imaging system capable of robust imaging across various animal models. (2) An in-depth investigation of pan-sharpening methods applied to hyperspectral images of biological tissues. (3) A validation study using retinal phantoms and in vivo mouse imaging, wherein we extracted retinal vessel diameters and measured blood vessel oxygenation rates. This dual-camera system, combined with advanced pan-sharpening techniques, offers a promising solution for high-resolution hyperspectral retinal imaging, opening new avenues for both diagnostic and research applications.

2. METHODS

Acquisition System

Figure 1 shows the configuration of our hyperspectral retinal imaging system. The system consisted of a snapshot HSI camera (Imec, Belgium), a high-resolution RGB imaging camera (ToupTek Photonics, China), a 50:50 optical beam splitter (ThorLabs, USA), imaging endoscopes, and fixed focal length imaging lenses. The HSI snapshot camera captured 16 bands of images at image resolution of 270 × 512 pixels from 460 – 600 nm [19]. The RGB camera is a standard scientific CMOS camera (ToupTek C3CMOS series) and can acquire RGB images at pixel resolution of up to 2048 × 3072 pixels. We designed our system for mouse retinal imaging. However, the system can be adapted for human retinal imaging by the use of a larger endoscope. For the imaging of mouse retina, the diameter of the scope should be similar to that of mouse pupil (around 3 mm) [20]. We chose an optical otoscope (Karl Storz 1218 AA) with a diameter of 3 mm for this purpose. Illumination was provided from a Xenon illumination source through a fiber optic wire that travels in the same direction as the scope. Finally, a series of fixed focal length lenses were needed to provide magnification of the scopes’ images. Table 1 shows the configurations of our systems. The systems were placed on a translation platform (ThorLabs, NJ, USA), which in turn was placed on top of a rotational platform (Thorlabs, USA) This gives our system four degrees of freedom (three rotations plus forward/backward translation).

Figure 1.

Diagram of the system used to produce both high-resolution RGB and hyperspectral images at the same time.

Table 1.

Configurations of our systems.

RGB camera	2048 × 3072-pixel imaging camera
HSI camera	270 × 512 pixel × 16 channels HSI camera
Beamsplitter	50 – 50 optical beamsplitter
Imaging scope	3 mm diameter borescope
RGB imaging lens	35 mm FFL lens with 2× magnifier
HSI imaging lens	35 mm FFL lens

Optical Properties

We identified the closest focusing distance using a United States Air Force (USAF) 1951 resolution chart (ThorLabs, NJ, USA), with 54 markings divided into 8 groups of different resolutions. The resolution is determined by the smallest pairs of markings that can be resolved. A line is drawn through each pair of markings, and then a sine curve with values ranging from ${\mathrm{S}}_{\text{min}}$ to ${\mathrm{S}}_{\text{max}}$ is fitted to the intensity value. Resolved markings have the Michelson contrast above 0.2, calculated using the equation: $\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{s}\mathrm{t}=\left({\mathrm{S}}_{\mathrm{m}\mathrm{a}\mathrm{x}}-{\mathrm{S}}_{\mathrm{m}\mathrm{i}\mathrm{n}}\right)/\left({\mathrm{S}}_{\mathrm{m}\mathrm{a}\mathrm{x}}+{\mathrm{S}}_{\mathrm{m}\mathrm{i}\mathrm{n}}\right)$ [21]. With the FFL lens at the nearest possible focus, we placed the target touching the tip of the scope. Then, we slowly moved the target away until it was in focus. The corresponding distance was the nearest focus distance of the system. The system is viable for mouse retinal imaging if the image distance is farther than the nearest focus distance. The farthest distance was found to be 10 meters.

Spectral performance of the snapshot camera was measured using small tiles made out of Spectralon (LabSphere, NH, USA). There are eight tiles total (blue, cyan, green, orange, purple, red, violet, yellow). The tiles were placed 5 mm from the tip of the scope. For comparison, we independently measured the reflectance using a spectrometer (Ocean Insight, FL, USA). The values measured using snapshot camera and spectrometer were compared against the calibrated value, which was measured by the manufacturer using a sphere spectrophotometer. The method used to calculate errors was root mean squared error (RMSE), which was defined as

$$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\sqrt{\frac{\sum _{i=1}^N{\left(x_i-{\hat{x}}_i\right)}^2}{N}}$$

Where $x_i$ and ${\hat{x}}_i$ are the factory calibrated wavelength and the measured wavelengths, respectively, and $N$ is the number of wavelengths sampled by the spectrometer or the HSI camera.

We wanted to convert measurements made in retinal images to physical size i.e., converting pixel values to micron. We did not attempt to perform any theoretical evaluations to estimate the retinal magnification factor of the mouse eye. Instead, we proposed an engineering-based approach to calculate the magnification factor. First, we imaged the mouse retina under anesthesia. With the tip of the endoscope touching the cornea, we adjusted the FFL lens so that the retina was in focus. The distance between the tip of the endoscope and the back of the retina is the object distance (${\mathrm{d}}_{\mathrm{o}}$ for short). This distance is not exactly known; we approximated it using the diameter of the eyeball. Keeping the same focus on the FFL lens, we placed the USAF target at the tip of the endoscope. Then, we slowly moved the USAF target away from the lens until the markings were in focus. The distance between the tip of the endoscope and the USAF target is the image distance (${\mathrm{d}}_{\mathrm{i}}$ for short). With the assumption that the eye contains a single crystalline lens, we found the magnification factor of the mouse eye:

$${\mathrm{M}}_{\mathrm{e}\mathrm{y}\mathrm{e}}=\frac{{\mathrm{d}}_{\mathrm{i}}}{{\mathrm{d}}_{\mathrm{o}}}$$

To find the pixel-to-micron ratio of the optical system, we placed a microscope graticule (a glass piece with measurement scale on its surface) at the same distance away from the endoscope tip as the image distance. The micron-to-pixel conversion factor of the endoscope is:

$${\mathrm{m}}_{\text{system}}=\frac{1000}{\text{How}\text{many}\text{pixels}\text{to}\text{cover}1\mathrm{m}\mathrm{m}}$$

The total magnification of the system when imaging mouse eyes is the product of the lens magnification factor and the endoscope system magnification factor. Its unit is micron/pixel:

$${\mathrm{m}}_{\text{conversion}}={\mathrm{m}}_{\text{system}}\times {\mathrm{M}}_{\text{eye}}$$

In the next section, we detail the procedures to validate our calculation of retinal magnification factors.

Phantom Acquisition Process

Prior to imaging animal retina, we validated our system on a series of mouse retinal phantoms. Mouse retina phantom was 3-D printed by modifying a design by Hosseinaee et al.[22]. Figure 2 depicts the 3D representation of the mouse eye phantom. The lens holder mimicked the natural curvature and dimensions of a mouse lens. For the mouse lens, we used a semi-spherical lens with a diameter of 3.10 mm and a focal length of 3 mm. The design was developed using Creo Parametric CAD software. Printing was completed on a Bambu Labs X1 Carbon 3D printer with white polylactic acid (PLA) materials. The interior of the retina phantom was filled with carbon fiber strands. The retina phantom was used to validate our magnification factor formula. Using the same procedures described in the previous section, we identified the focus of the FFL lens such that carbon fiber strands in the optical model were clearly seen. Then, we removed the lens from the optical model and measured the size of the fiber strands using an upright microscope. The diameter of the carbon fiber strands calculated using the magnification formula was compared with the diameter of the strands measured using the microscope.

Figure 2.

Design of the mice retinal phantom. (a) The printed system with the lens in place. (b) The interior chamber is filled with carbon fiber strands.

In Vivo Image Acquisition in Mice

We acquired high-resolution HSI of the retina in C57BL/6 black mice (Jackson National Labs, USA). Our procedure modified the procedure described by More et al. [23]. Fifteen minutes prior to the anesthesia, we applied 1% tropicamide (Mydriacyl, Fisher Scientific) onto each eye for pupil dilation. Mice were anesthetized by inhalation of 5% isoflurane until immobile, then remained anesthetized by inhalation of 1.5% isoflurane. Mice were then placed on top of a heating pad and stayed warmed during the entirety of imaging. We applied local anesthetic (Proparacaine, Fiser Scientific), followed by a glycol-based lubricant eye drop (Systane, Alcon) to the cornea. We placed mice on top of the vertical translation platform, then rotated the imaging platform toward the surface of the cornea. Then, we translated the entire imaging system forward until the scope contacted the cornea. We then changed the focus on the FFL lens until the retina was in focus. Mouse retina was in direct view when the head of the mouse is slightly tilted toward the scope, and the scope made a 45-degree angle with the snout. The exposure time for the RGB camera was 70 ms, whereas the exposure time for the HSI camera was 700 ms. The entirety of imaging lasted less than 5 minutes per mouse, which was enough for the anesthetized mouse to return to their cage and recover. Figure 3 shows the imaging platform with a mouse.

Figure 3.

Hyperspectral imaging setup and image data acquisition of the mouse retina.

Generating High-Resolution Images

To produce high-resolution images, we created an automatic imaging workflow. The workflow has three stages: preprocessing, registration, and pan-sharpening. For preprocessing, separate pipelines were used for RGB images and hyperspectral images. The RGB image was converted to grayscale. We registered the grayscale, high resolution images under the conditions that only translation, scaling, and rotation are involved (similarity transformations). Using manual examination, we found the rotation and translation necessary to produce proper registration.

For clarity, we used the term “hyperspectral images”/HSI to refer to the original image of 512 × 270 × 16 pixels, and use the term “pansharpened images” to refer to the 3072 × 2048 × 16 images that is the product of combining the RGB image and the HSI. We tested several different methods for image pan-sharpening. Because pan-sharpening is a well-studied problem in remote sensing, we applied multiple published methods to our application. A list of the tested methods is shown in Table 2. The choices of methods were taken from a review paper by Javan et al. [24]. Bicubic interpolation (BCB) resamples the image using cubic convolution. The Brovey algorithm (BV) combines spectral and panchromatic images by normalizing and scaling each band [25]. Gram-Schmidt (GS) uses orthogonalization to merge the images [24]. Adaptive Gram-Schmidt (GSA) uses least-square minimization to generate images prior to merging using GS [24]. Generalized intensity-hue-saturation (GIHS) method transforms the spectral image into intensity, hue, and saturation components, replaces intensity with the panchromatic image, and converts it back [25]. High-pass filtering (HPF) adds spatial details from the panchromatic image [24]. Undecimated discrete wavelet transform (UDWT) injects details using wavelet decomposition [26]. In addition, we tested several deep learning-based architectures that were published in recent years. Masi et al. [14] proposed a deep learning network with convolutional neural network (CNN) architecture. Yang et al. [27] and Ma et al. [16] proposed methods based on generative adversarial networks (GAN) to improve the quality of the images. Ma et al. [15] proposed an unsupervised method with U-Net backbone. Our dataset to train and evaluate pan-sharpening models came from a set of 1,203 HSI captured using a customized snapscan camera. The low-resolution, mosaic images and RGB images were generated using a customized algorithm. Training was done on a high-performance GPU cluster consisting of 8 NVIDIA 64Gb GPUs. Hyper-parameter tuning was done using five-fold cross validation of the data. Low-resolution HSI were divided into non-overlapping patches of size 64 × 64 × 16 pixels, with the goal of producing high-resolution HSI patches of size 256 × 256 × 16 pixels.

Table 2.

Overview of the methods that were evaluated to produce pan-sharpened images.

Method	Abbreviation
Classical Methods
Bicubic interpolation	BCB*
Brovey algorithm [25]	BV*
Generalized intensity–hue–saturation [25]	GIHS*
Adaptive Gram-Schmidt	GSA*
Gram-Schmidt	GS*
High-pass filtering using 13×13 box filter	HPF*
Undecimated/“ à trous“ wavelet substitution [26]	UDWT*
Deep Learning Methods
Convolutional neural network [14]	CNN
Generative adversarial network [27]	PSGAN
Unsupervised generative adversarial network [16]	PAN-GAN*
U-Net [15]	UN*

A star (*) marks that the method is unsupervised.

We employed several metrics to evaluate pan-sharpening techniques. The goal is to ensure that the pan-sharpened, high-resolution image preserved both the spectral fidelity (compared to the hyperspectral image captured by the snapshot camera) and the spatial fidelity (compared to the RGB image captured by the CMOS camera). The final pansharpened images were compared to the ground truth captured by our camera. We used spectral angle mapper (SAM), root mean squared error (RMSE), error relative global dimensionless synthesis index (ERGAS), and two-dimensional correlation coefficient score (CC) to compare the spatial and spectral distortion [13]. Let H be the ground truth hyperspectral image, and P be the result of the pan-sharpening algorithm. Let $\mathrm{x},\mathrm{y}$, and $\mathrm{z}$ be the coordinates of the pixels in the images, with $\mathrm{z}$ being the spectral dimension. SAM compares the angle between the corresponding pixels $\mathrm{i}$ in the fused image and the original image using the following equation:

$$\mathrm{S}\mathrm{A}\mathrm{M}=\frac{1}{\mathrm{N}}\sum _{\mathrm{x},\mathrm{y}=1}^{\mathrm{N}}\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{c}\mathrm{o}\mathrm{s}\left(\frac{⟨\mathrm{H}(\mathrm{x},\mathrm{y}),\mathrm{P}(\mathrm{x},\mathrm{y})⟩}{\Vert \mathrm{H}(\mathrm{x},\mathrm{y})\Vert \Vert \mathrm{P}(\mathrm{x},\mathrm{y})\Vert }\right)$$

The SAM metric is obtained by averaging the angles measurements across all pixels. Smaller SAM is better, with a minimum value of 0.

ERGAS is a refinement of RMSE by accounting for errors across different bands. ERGAS is defined as follows:

$$\mathrm{E}\mathrm{R}\mathrm{G}\mathrm{A}\mathrm{S}=\frac{100}{4}\sqrt{\frac{1}{\mathrm{c}}\sum _{\mathrm{z}=1}^{\mathrm{c}}\left(\frac{\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}\left(\mathrm{H}\right[\mathrm{z}],\mathrm{P}[\mathrm{z}\left]\right)}{\mathrm{\mu }\left(\mathrm{H}\right[\mathrm{z}\left]\right)}\right)}$$

The ratio 100/4 came from the ratio between the size of the small hyperspectral image and the large RGB image. C is the number of channels, $\mathrm{H}\left[\mathrm{j}\right]$ indicates the j^th spectral channel, and $\mathrm{\mu }\left(\mathrm{H}\right[\mathrm{z}\left]\right)$ indicates the mean value of $\mathrm{H}\left[\mathrm{z}\right]$. Smaller ERGAS is better, with a minimum value of 0.

The RMSE is defined as follows:

$$\text{RMSE}=\sqrt{\frac{\sum _{\mathrm{x},\mathrm{y},\mathrm{z}}^{\mathrm{N}}(\mathrm{H}(\mathrm{x},\mathrm{y},\mathrm{z})-\mathrm{P}(\mathrm{x},\mathrm{y},\mathrm{z}){)}^2}{\mathrm{N}}}$$

Here, $\mathrm{N}$ is the number of total pixels that we want to take the RMSE over. Similar to ERGAS, RMSE has a minimum value of 0, and smaller RMSE is better.

2-D correlation coefficient ($\mathrm{C}\mathrm{C}$) score measures how well two images agree with each other. It is defined as:

$$\mathrm{C}\mathrm{C}=\frac{\sum _{\mathrm{x},\mathrm{y}}\left(\mathrm{H}-{\mathrm{\mu }}_{\mathrm{H}}\right)\odot \left(\mathrm{P}-{\mathrm{\mu }}_{\mathrm{p}}\right)}{\sqrt{\sum _{\mathrm{x}}\sum _{\mathrm{y}}{\left(\mathrm{H}-{\mathrm{\mu }}_{\mathrm{H}}\right)}^2\odot {\left(\mathrm{P}-{\mathrm{\mu }}_{\mathrm{p}}\right)}^2}}$$

Here, ${\mathrm{\mu }}_{\mathrm{H}}$ and ${\mathrm{\mu }}_{\mathrm{P}}$ indicates the mean of $\mathrm{H}$ and $\mathrm{P}$, respectively. The symbol $\odot$ indicates an element-wise product, also known as Hadamard product. CC score ranges from 0 to 1, with higher CC score being better.

Analysis of Biomarkers

The final image was color-corrected with white and dark references using the following formula:

$$\mathrm{I}=\frac{{\mathrm{I}}_{\text{raw}}-{\mathrm{I}}_{\text{dark}}}{{\mathrm{I}}_{\text{white}}-{\mathrm{I}}_{\text{dark}}}$$

where ${\mathrm{I}}_{\text{raw}}$ is the raw acquired image, ${\mathrm{I}}_{\text{dark}}$ is the dark reference, ${\mathrm{I}}_{\text{white}}$ is the white reference image, and $\mathrm{I}$ is the final corrected image. The image raster size of an RGB image was 3072 × 2048 × 3 pixels, and the image resolution of a hyperspectral image was 512 × 270 × 16 pixels. Using the best performing pan-sharpening algorithm (both qualitatively and quantitatively), we extracted two biomarkers simultaneously: the retinal vessels diameter and the retinal vessels oxygenation rates. We identified the branches of the optic vessels manually, then we used ImageJ software to measure the diameter of the vessels in pixels. The diameters of the vessels were converted to physical values:

$$\text{Size}\text{in}\mathrm{\mu }\mathrm{m}=\text{Size}\text{in}\text{pixel}\times {\mathrm{m}}_{\text{conversion}}$$

We manually masked the retinal vessels using the pansharpened image as a guide. Then, we used multi-wavelength curve fitting to estimate the oxygen saturation values [28]. The white-reference corrected image is converted to optical density using the formula: $OD=-{\mathrm{l}\mathrm{o}\mathrm{g}}_{10}I$. Here, OD is the optical density, and I is the white-reference corrected image. Then, we found the solution of the following equation with a non-negative constraint:

$$\mathrm{O}\mathrm{D}(\mathrm{\lambda })=\mathrm{B}-\mathrm{N}\mathrm{l}\mathrm{n}(\mathrm{\lambda })+\mathrm{A}\left[{\mathrm{\mu }}_{{\mathrm{H}\mathrm{b}\mathrm{O}}_2}(\mathrm{\lambda })+\left(1-{\mathrm{S}\mathrm{O}}_2\right){\mathrm{\mu }}_{\mathrm{H}\mathrm{b}}\right]$$

Here, B and N are the constant term that model the linear and non-linear scattering contributions, respectively. A is the thickness of the blood vessel, which scales the absorbance according to Beer-Lambert law. SO₂ is the oxygen saturation which we wanted to find. The value of ${\mu }_{Hb}(\lambda )$ and ${\mu }_{{HbO}_2}(\lambda )$ are the wavelength-dependent absorption coefficients of deoxygenated and oxygenated hemoglobin, respectively. We sourced these values from experimental coefficients by Prahl et al.[29]. To validate the algorithm, we mixed sodium dithionite into fully oxygenated whole sheep blood. Sodium dithionite concentration in blood correlates with the oxygen saturation values [30]. As such, we captured HSI of blood with different concentrations of sodium dithionite and estimated the oxygen saturation. A linear correlation was found, validating the algorithm. Vessels with oxygen saturation higher than 90% were considered arterioles, whereas vessels with oxygen saturation less than 90% were considered venules [31].

3. RESULTS

System Optical Performances

The nearest imaging distance of the system was 1.9 mm. Experiments showed that the imaging distance of the mouse retina ranges from 3.5 – 4 mm, which means that the system is fully capable of imaging mouse retina in focus. Figure 4 shows the respective curves from the HSI camera, spectrometer, and factory calibrated values for each of the eight calibration tiles. Table 3 shows the RMSE score of the HSI camera intensity and the spectrometer measured value. The camera has an RMSE of 3.87 ± 1.89 ×10⁻², and the spectrometer has an RMSE of 2.39 ± 1.85 ×10⁻². The results showed that the HSI camera can achieve spectral accuracy comparable to that of a spectrometer.

Figure 4.

Comparison between HSI camera intensity, spectrometer recorded values, and factory calibrated values.

Table 3.

Comparison between reflectance values measured by HSI camera and spectrometer.

Tiles color	Camera RMSE (×10−2)	Spectrometer RMSE (×10−2)
Blue	6.15	1.13
Cyan	4.52	3.54
Green	2.39	1.15
Orange	3.01	5.36
Purple	0.95	0.89
Red	3.58	0.44
Violet	3.70	4.55
Yellow	6.65	2.09
Average ± St. Dev.	3.87 ± 1.89	2.39 ± 1.85

We measured the magnification factor of the mouse eye to be 1.2×, the optical magnification of the endoscope system at imaging distance to be 2.15 μm/pixel, and the conversion factor for retinal image to be 1.80 μm/pixel. Comparing the lengths of strands that we calculated against the actual length gave an error rate of 7.6 ± 5.6%. This shows that our conversion algorithm produces reasonable approximation of small details within the phantom retina.

Performance of Pan-Sharpening Algorithms

Table 4 shows the quantitative result of the different pan-sharpening algorithms. Figure 5 shows the results of applying pan-sharpening onto the retinal images. The results show that Brovey method (BV), high-pass filtering (HPF), and methods based on neural networks best preserved the high-resolution structural details that we wanted. PAN-GAN and U-Net models, owing to the fact that they were trained on unsupervised data, produced metrics that were slightly worse compared to other deep learning-based methods. We found that PSGAN produced the right balance between quantitative metrics and qualitative output. As such, for the analysis performed in the next sections, we use the super-resolution hyperspectral image produced by PSGAN. Figure 6 shows a selection of single-band images from the super-resolution hyperspectral image produced by PSGAN. Figure 7 compares the spectrum between the original HSI and the super-resolution image. To generate this image, we took samples from two different blood vessels marked A and B and calculated their mean value. Figure 7 shows that PSGAN properly preserved the spectra of the image.

Table 4.

Quantitative result of pan-sharpening applied onto the test dataset (n = 100). Method with the best performance (average ± standard deviation) is bolded for each column.

Method	CC	RMSE (×10−2)	ERGAS	SAM
BCB	0.94 ± 0.08	4.64 ± 1.87	3.87 ± 2.38	0.04 ± 0.02
BV	0.94 ± 0.03	6.31 ± 2.76	6.15 ± 5.97	0.04 ± 0.02
GIHS	0.94 ± 0.03	6.11 ± 2.53	6.18 ± 6.27	0.11 ± 0.09
GS	0.94 ± 0.03	6.10 ± 2.53	6.09 ± 6.17	0.11 ± 0.09
GSA	0.94 ± 0.05	6.27 ± 2.74	6.65 ± 8.85	0.11 ± 0.09
HPF	0.96 ± 0.03	5.65 ± 1.66	4.80 ± 2.29	0.08 ± 0.04
UDWT	0.98 ± 0.01	2.74 ± 0.86	2.57 ± 1.93	0.05 ± 0.02
CNN	0.95 ± 0.05	2.33 ± 0.70	2.73 ± 2.45	0.07 ± 0.03
PSGAN	0.96 ± 0.05	2.15 ± 0.64	2.37 ± 1.71	0.06 ± 0.03
PAN-GAN	0.95 ± 0.05	4.01 ± 0.99	4.22 ± 2.96	0.10 ± 0.08
UN	0.88 ± 0.08	5.29 ± 1.34	6.78 ± 5.21	0.15 ± 0.07

Figure 5.

Comparison of different pan-sharpening methods used to produce super-resolution image of the mouse retina. For purposes of visualization, the hyperspectral images were converted into pseudo-RGB using a method by Ma et al.[15]. Because the conventional RGB image is captured by an RGB camera with unknown response curve, we did not try to accurately recreate its color. The methods were described in Section 2. (a) Conventional RGB image. (b) Bicubic interpolation. (c) Brovey method, (d) IHS, (e) GS, (f) GSA, (g) HPF (h) UDWT (i) CNN (j) U-Net (k) PSGAN (l) PAN-GAN.

Figure 6.

Super-resolution hyperspectral image of the retina produced using the PSGAN super-resolution method.

Figure 7.

Spectra sampled at two different vessels—marked A and B in the retinal image. Dashed line indicates original spectra from the hyperspectral image. Solid lines indicate spectra from the pansharpened image produced using PSGAN.

Retinal vessels diameter measurement

Within each mouse, 7–9 major vessels were found surrounding the optic nerve. The average lumen diameter of the venules was 45.7 ± 13.6 μm, whereas the average lumen diameter of the arterioles was 31.5 ± 8.7 μm. The diameter of retinal vessels in mice can range from 3.2 to 45.8 μm [32], which agrees with our measurements. Figure 8(b) shows the box plot of measured vessels diameter from three mice. There is a difference in the diameter of venules versus arteries. Venules in general have larger lumen diameter, due to the fact that venules have more flexible lumen.

Figure 8.

(a) Oxygen saturation mapping of the mouse retina, showing 8 different vessels (5 arterioles and 3 venules). (b) Average diameter of the venules and arterioles measured in three mice. (c) Average oxygen saturation level measured from the eight selected vessels in Figure 8(a).

Retinal Vessels Oxygenation

Figure 8(a) shows a generated oxygen saturation map. Overall, we found consistent oxygen saturation within vessels, with the exception being along the edges of the vessels and close to the optic nerve. This is likely due to the fact that there is a significant presence of macular pigment near the optic nerve, making correct observation difficult. This problem is also observed previously in human retina [28]. For the subject in Figure 8(a), we identified 5 arterioles and 3 venules. Within the measured subjects, the average arteriole-venules oxygenation difference was 20%. Many of the arterioles have close to 100% oxygenation rate, likely because mouse breathe pure oxygen during anesthesia. The average arterial sO₂ was 96.2 ± 5.8%, whereas the average venous sO₂ was 76.3 ± 9.4%. This is comparable to values reported in the literature [33], 96.4% and 74.1% in the arterioles and venules, respectively.

4. CONCLUSIONS AND DISCUSSIONS

In this study, we developed a dual-camera system combining hyperspectral imaging (HSI) and high-resolution RGB imaging to capture detailed retinal images. The system produced high-resolution images of animal retinal phantoms, resolving fine structural details crucial for analysis. By applying pan-sharpening algorithms, we enhanced the spatial resolution of hyperspectral images, merging the spectral detail of HSI with the spatial clarity of RGB imaging. Imaging results confirmed the system’s ability to detect small blood vessels, essential for identifying subtle retinal changes and abnormalities. Our evaluation of pan-sharpening methods showed that deep learning-based approaches, such as convolutional neural networks (CNNs) and generative adversarial networks (GANs), delivered higher fidelity in both spatial and spectral dimensions. Originally developed for satellite imagery, these methods proved effective for biological imaging. To our knowledge, this is one of the first studies to apply pan-sharpening techniques to hyperspectral images of biological tissues.

We developed tools to extract and validate biomarkers from the pansharpened images, focusing on vessel diameter and oxygenation rate. These biomarkers reveal important details about retinal metabolic activity and demonstrate the strengths of our system. High-resolution RGB images captured small structures with minimal motion blur, while hyperspectral images provided the necessary spectral data. Notably, RGB images mitigated the motion blur introduced by the hyperspectral camera’s longer exposure time, improving overall image quality. This suggests that future systems could use RGB imaging to address the trade-off between speed and resolution in hyperspectral imaging. Moving forward, we aim to validate the system further through in vivo studies in animal models. Early results suggest that the 600–800 nm spectral range offers better differentiation between oxygenated and deoxygenated blood, which we plan to explore for in vivo imaging. We also intend to improve our phantom design by incorporating whole blood for greater realism. Additionally, we will integrate advanced machine learning algorithms to enhance image registration, pan-sharpening, and analysis workflows. Our ultimate goal is to create a system capable of high-resolution, hyperspectral video imaging of the retina in vivo, allowing us to observe real-time changes in retinal structure and function. This could significantly advance both research and clinical applications in retinal imaging.