Unsupervised Super Resolution Network for Hyperspectral Histologic Imaging

Abstract

Hyperspectral imaging (HSI) has many advantages in microscopic applications, including high sensitivity and specificity for cancer detection on histological slides. However, acquiring hyperspectral images of a whole slide with a high image resolution and a high image quality can take a long scanning time and require a very large data storage. One potential solution is to acquire and save low-resolution hyperspectral images and reconstruct the high-resolution ones only when needed. The purpose of this study is to develop a simple yet effective unsupervised super resolution network for hyperspectral histologic imaging with the guidance of RGB digital histology images. High-resolution hyperspectral images of hemoxylin & eosin (H&E) stained slides were obtained at 10× magnification and down-sampled 2×, 4×, and 5× to generate low-resolution hyperspectral data. High-resolution digital histologic RGB images of the same field of view (FOV) were cropped and registered to the corresponding high-resolution hyperspectral images. A neural network based on a modified U-Net architecture, which takes the low-resolution hyperspectral images and high-resolution RGB images as inputs, was trained with unsupervised methods to output high-resolution hyperspectral data. The generated high-resolution hyperspectral images have similar spectral signatures and improved image contrast than the original high-resolution hyperspectral images, which indicates that the super resolution network with RGB guidance can improve the image quality. The proposed method can reduce the acquisition time and save storage space taken up by hyperspectral images without compromising image quality, which will potentially promote the use of hyperspectral imaging technology in digital pathology and many other clinical applications.

1. PURPOSE

Hyperspectral imaging (HSI) is a non-contact and label-free imaging modality that has been emerging in recent years. The three-dimensional (3D) data cube acquired with HSI contains both the spatial and spectral information of the object, which offers more fine features that are potentially useful for image segmentation and classification. Many studies have proven the usefulness of hyperspectral imaging in microscopy applications, such as pathology diagnosis [1–8], hematology [9–11], cell autofluorescence imaging [12], nanomaterial imaging [13], et cetera. It could be extremely advantageous to employ HSI for whole-slide imaging (WSI) and automatic cancer detection in histological slides, because HSI not only provides reproducible and quantitative diagnosis of the slides, but also improves the classification results compared to RGB [1, 3].

There are, however, a few difficulties when applying hyperspectral microscopy in real clinical situations. Firstly, the size of a hyperspectral image file is large. For example, one hyperspectral image with a dimension of 3600 pixels × 2048 pixels × 150 bands saved in single precision can take about 5GB of storage space. To image a whole slide under 10× magnification, it can take tens of images, thus resulting in one whole slide image of more than 100 GB. In some applications such as hematology, magnification as high as 60× is necessary, which may put an extreme burden on data storage devices. This limitation of data storage makes it quite difficult to establish a comprehensive hyperspectral microscopic dataset, which is essential for a thorough study of deep learning methods. Secondly, the acquisition of hyperspectral images can be time-consuming. Most hyperspectral microscopy studies use push-broom or spectral-scanning hyperspectral imaging systems to obtain a sufficient spectral and spatial resolution, and the scanning process of each image can take several seconds, which greatly increases the acquisition time of a whole-slide image. Even though a snapshot hyperspectral camera can speed up image acquisition, its intrinsic low spatial resolution would not meet the requirement of many applications where fine structures in the slides need to be seen. Thirdly, it is more difficult to implement autofocusing in a hyperspectral camera than a color cameras. During the scanning of the histological slides, the subtle changes of the slide thickness can cause the acquired images to be out of focus. State-of-the-art microscopes and whole-slide scanners are integrated with autofocusing algorithms, which is based on the acquisition of a stack of many RGB images along the Z axis. However, applying the same method for hyperspectral cameras would prolong the scanning time tremendously. In addition, due to the chromatic distortion of optics and the high spectral resolution of a HSI camera, very sophisticated algorithms may be needed to ensure good focus for all wavelength bands. Finally, some bands in hyperspectral images, especially the first and last several bands, can have large amounts of noise due to the low sensitivity. Both the unsatisfying focus and noise can result in low image quality and missing spatial details, i.e., high frequency information, as shown in Figure 1.

Figure 1.

Incidences of low image quality of hyperspectral images. (a) High resolution digital histology image of a thyroid slide with high image quality. (b) HSI-synthesized RGB image of the same region as (a) but has lower contrast because of the unsatisfying focusing during the slide scanning without autofocusing. (c) The first band from the hyperspectral image of the same region as (a) with a lot of noise due to low sensitivity.

On the other hand, whole-slide scanning with a color camera has become a routine in the field of pathology. Digital histology images that are acquired with high-quality color cameras usually have very high image resolution and contrast. Since many state-of-the-art histology microscopes and whole-slide scanners are integrated with advanced algorithms such as auto-focusing and extended focal imaging (EFI), significant improvements can be made to the quality of the digital histology images.

HSI super resolution, also known as HSI pansharpening, is a technique that uses the spatial information from a high-resolution panchromatic image (PAN) and the color information from a low-resolution multispectral image to generate high-resolution multispectral image. It has been investigated mainly for remote sensing applications. Previously proposed methods can be roughly divided into four types, namely component substitution (CS), multiresolution analysis (MRA), hybrid, and model-based methods[14, 15]. CS methods can result in a certain degree of spectral distortion, while hybrid methods may generate blurry results. The other two methods, despite good performance, add complexity to calculation and implementation. In addition, some methods were developed for multispectral images and may not be able to handle the wide spectral range in hyperspectral images [15]. In recently years, deep learning algorithms have been explored for the super resolution reconstruction of hyperspectral images. Masi et al [16] adopted a three-layer convolutional neural network, named PNN, to reconstruct high-resolution multispectral images (MSI) from the stack of interpolated low-resolution MSI and high-resolution PAN. However, the network had a limited learning ability due to the shallow architecture. Yang et al [17] proposed a ResNet-based architecture named PanNet, which did not fully exploit the spatial information due to the use of high-pass filter. Yao et al [18] implemented pixel-wise regression for hyperspectral pansharpening using a U-Net. The regression model was trained to estimate high-resolution MSI with the low-resolution MSI and low-resolution PAN. Zheng et al [15] developed a two-part framework, which firstly enhanced the spatial resolution in hyperspectral images through contrast limited adaptive histogram equalization and then use a deep residual neural network to further boost the fusion accuracy.

Since many microscopes and whole-slide scanners offer options for more than one camera to be mounted on the microscope and share the same field of view (FOV), it is feasible to apply the HSI super resolution technique for hyperspectral microscopy. Considering the easy acquisition and wide utilization of high-quality RGB histology images in microscopy, we propose to use high-resolution RGB images to guide the super-resolution reconstruction of low-resolution hyperspectral images. The acquisition of low-resolution hyperspectral images can be achieved either using a video adapter with reduced magnification, or simply using a snapshot hyperspectral, as long as sufficient spectral bands are covered.

Despite a wide variety of methods, most previously-proposed methods are based on a supervised learning method [15–20]. Using this technique, the reconstructed high-resolution hyperspectral image might inherit the spectral noise from the reference hyperspectral image, especially in the first several bands. Ideally, the super-resolution reconstruction of hyperspectral microscopic images should take full advantage of the superb image quality of digital histology images while maintaining the spectral features that are critical for image classification. Therefore, we developed a simple yet effective unsupervised super-resolution network, which fuses the spatial information from the high-resolution RGB images and the spectral information from the low-resolution hyperspectral images. With the proposed method, it is possible to save the acquisition time and the storage space for hyperspectral images, as well as compensate for the low quality of some bands in the hyperspectral images, thus promoting the application of HSI for whole-slide imaging and automatic cancer detection.

2. METHODS

2.1. Histologic Slides and Hyperspectral Dataset

In this work, we utilized hematoxylin and eosin (H&E) stained histological slides of head and neck cancer. The tissue of each slide was resected during a routine surgery, after which the specimen was inked, formalin fixed, and paraffin embedded. The top section of each specimen was obtained using a microtome and stained with hematoxylin and eosin. High-resolution digital histology images of the slides were obtained using whole-slide scanning at 40× magnification right after the preparation of the slides [21].

For the acquisition of hyperspectral images, we used the hyperspectral microscopic imaging system that has been reported in our previous works [2, 3] to scan the slides. The magnification of the objective lens was 10×. The original size of the hyperspectral images was 2000 pixels × 2000 pixels × 87 bands, covering a wavelength range of 480~720 nm. During the image acquisition, we tried our best to focus the hyperspectral camera by looking at the preview image in the image acquisition software and manually adjusting the focusing knob.

To make sure that our super-resolution network is able to deal with the staining and thickness variation of different slides, which could cause the change of spectral signatures, we acquired hyperspectral images of 26 different slides from 20 patients. In total, 58 hyperspectral images were obtained, where 42 images from 14 patients were used for training, 7 images from 3 patients were used for validation, and 9 images from 3 patients were used for testing. Data from the same patient was never used in training, validation, and testing simultaneously.

2.2. Data preprocessing

All the raw hyperspectral images were calibrated with white reference and dark reference images to obtain the transmittance data. For each calibrated hyperspectral image, we synthesized an RGB image using a customized HSI-to-RGB transformation[3], and generated a grayscale image by calculating the average of all 87 bands. The high-resolution RGB images that are used to guide the super resolution were cropped from digital histology images, which were originally imaged at 40× magnification. We looked at each HSI-synthesized RGB image, found the corresponding region in the whole-slide digital histology image, and cropped the high-resolution RGB image with a slightly larger FOV than the HSI-synthesized RGB image. All high-resolution RGB images were registered to their matching hyperspectral images using the affine registration in the AnalyzePro software (AnalyzeDirect Inc., Kansas) and resized to a spatial size of 2000 pixels × 2000 pixels. Then, both the hyperspectral images and RGB images were divided into 100 patches of 200 pixels × 200 pixels, namely the high-resolution hyperspectral patches (HR-HSI) and high-resolution RGB patches (HR-RGB). All HR-HSI were downsampled using a “box” interpolation kernel by 2, 4, and 5 times, respectively, to generate low-resolution hyperspectral patches (LR-HSI) for three different super-resolution networks. Image patches with too little tissue (<50% area of the whole patch) were removed from the dataset.

2.3. Unsupervised Super-resolution Network

We developed an unsupervised super-resolution network, which took the LR-HSI and HR-RGB as inputs and generated HR-HSI as well as LR-HSI for outputs. To consider different acquisition methods of low-resolution hyperspectral images, we trained three different super-resolution networks. For the scenario where a video adapter with a reduced magnification is used (e.g. 1× reduced to 0.5×), we trained a 2× super-resolution network. For the cases where snapshot cameras are used, we trained a 4× and a 5× super-resolution network, because 4×4 and 5×5 are the two most common mosaic patterns of snapshot hyperspectral cameras. Figure 2 shows the structure of a 4× super-resolution network as example. The three channels (R, G, and B) of the HR-RGB were duplicated by 35, 35, and 17 times, respectively, and stacked to form an 87-band “stacked HR-RGB”. The LR-HSI input (50×50×87) was firstly upsampled to 200×200×87 by a deconvolution layer, and then concatenated with the stacked HR-RGB along the third dimension, i.e. the spectral dimension, to form a 200×200×174 patch. What followed is a modified U-Net architecture, which took the concatenated 174-band patch as input and output a HR-HSI patch. In addition, an average pooling layer was applied to the generated HR-HSI, in order to generate an LR-HSI patch with the same size as the input LR-HSI. Note that the dimensions of both the input and generated low-resolution HSI patches in Figure 2 are for a 4× super-resolution network. For the 2× and 5× network, the dimensions would be 100×100×87 and 40×40×87, respectively. The details of the network are shown in Table 1.

Figure 2.

Illustration of the unsupervised 4× super-resolution network architecture. For the 2× and 5× network, the dimensions of both the input and generated low-resolution HSI patch would be 100×100×87 and 40×40×87, respectively.

Table 1.

Super-resolution network architecture

Layer	Kernel/Strides/Padding	Output shape
Input_1: High-reso. RGB		200×200×87
Input_2: Low-reso. HSI		D × D × 87
Deconv2D	(L+1) ×(L+1), L, ‘same’	200×200×87
Concatenate	Concatenated feature maps	200×200×174
Conv2D	3×3, 1, ‘same’	200×200×196
Conv2D	3×3, 1, ‘same’	200×200×256
Conv2D	3×3, 2, ‘same’	100×100×384
Conv2D	3×3, 1, ‘same’	100×100×512
Conv2D	3×3, 2, ‘same’	50×50×640
Conv2D	3×3, 1, ‘same’	50×50×768
Conv2D_Transpose	3×3, 2, ‘same’	100×100×768
Concatenate	Skip connection	100×100×1280
Conv2D	3×3, 1, ‘same’	100×100×640
Conv2D	3×3, 1, ‘same’	100×100×512
Conv2D_Transpose	3×3, 2, ‘same’	200×200×256
Concatenate	Skip connection	200×200×512
Conv2D	3×3, 1, ‘same’	200×200×256
Conv2D	3×3, 1, ‘same’	200×200×194
Conv2D	3×3, 1, ‘same’	200×200×160
Conv2D	3×3, 1, ‘same’	200×200×128
Conv2D	3×3, 1, ‘same’	200×200×96
Conv2D	1×1, 1, ‘same’	200×200×87
AveragePool	L×L, L, ‘valid’	D × D ×87

^*For 2× network, L=2, D=100; for 4× network, L=4, D=50; for 5× network, L=5, D=40.

Our purpose in this work is to generate high-resolution hyperspectral images that have spatial contrast and spatial details as clear as the high-resolution RGB images, as well as spectral signatures as real as the original HSI. Thus, the network should be able to extract the spatial information and spectral information from the RGB images and hyperspectral images, respectively, and fuse them in the generated hyperspectral image. This was fulfilled by simultaneously minimizing the spatial loss and spectral loss of the network. For the spatial loss, both the input stacked high-resolution RGB and the generated high-resolution HSI were averaged along the spectral dimension, resulting in two high-resolution grayscale images; then the minimum squared error (MSE) of the two grayscale images was calculated and used as the spatial loss, as Equation (1) shows. The spectral loss calculates the minimum squared error between the generated and input low-resolution HSIs, as shown in Equation (2). The loss weights of the two loss functions were 0.5 and 0.5.

$$L_{\mathit{\text{spatial}}}=\frac{1}{M\times N}{\sum }_{i=1}^M{\sum }_{j=1}^N{\left(G_{ij}-{\widehat{G}}_{ij}\right)}^2$$ (1)

$$L_{\mathit{\text{spectral}}}=\frac{1}{m\times n\times b}{\sum }_{i=1}^m{\sum }_{j=1}^n{\sum }_{k=1}^b{\left(H_{ijk}-{\widehat{H}}_{ijk}\right)}^2$$ (2)

where M and N are the spatial dimensions of the high-resolution grayscale images; m and n are the spatial dimensions of the low-resolution HSI; b =87 is the number of bands in the low-resolution HSI; G and $\widehat{G}$ are the average grayscale images of the stacked high-resolution RGB and the generated high-resolution HSI, respectively; H and $\widehat{H}$ are the input and generated low-resolution HSI, respectively.

The unsupervised super-resolution network was implemented using Keras on a Titan XP NVIDIA GPU. We used the Adam optimizer [22] with a learning rate of 10⁻⁴. the network was trained with a batch size of 2. The three networks (2×, 4×, and 5×) were trained for 6~8 epochs, depending on how fast the validation loss stopped decreasing.

2.4. Evaluation

We firstly use the MATLAB implementations of peak signal-to-noise-ratio (PSNR) and mean absolute error (MAE) to quantify the super-resolution reconstruction quality. PSNR measures the global intensity difference between the generated HR-HSI and the real HR-HSI and is calculated by averaging the PSNR value in a band-by-band basis across all 87 bands, with one band (B) from the real HR-HSI and the corresponding band $(\widehat{B})$ from the generated HR-HSI:

$$\mathit{\text{PSN}}{R}_{\lambda }(B,\widehat{B})=10{\text{log}}_{10}\left(\mathit{\text{peak\hspace{0.17em}valu}}{e}^2/\mathit{\text{MSE}}\right)$$ (3)

$$\mathit{\text{PSNR}}\left(\mathit{\text{Real\hspace{0.17em}HSI}},\mathit{\text{\hspace{0.17em}Generated\hspace{0.17em}HSI}}\right)=\frac{1}{87}{\sum }_{\lambda =1}^{87}PSN{R}_{\lambda }\left(B,\widehat{B}\right)$$ (4)

where MSE is the mean squared error.

MAE was also calculated in a band-by-band basis across all 87 bands:

$$MA{E}_{\lambda }(B,\widehat{B})=\frac{1}{M\times N}{\sum }_{i=1}^M{\sum }_{j=1}^N\left|B_{ij}-{\widehat{B}}_{ij}\right|$$ (5)

$$\mathit{\text{MAE}}\left(\mathit{\text{Real\hspace{0.17em}HSI}},\mathit{\text{\hspace{0.17em}Generated\hspace{0.17em}HSI}}\right)=\frac{1}{87}{\sum }_{\lambda =1}^{87}MA{E}_{\lambda }\left(B,\widehat{B}\right)$$ (6)

Then, we used structural similarity index measure (SSIM) [23] to evaluate the spatial character of the generated high-resolution hyperspectral images. Usually for super resolution tasks, SSIM is calculated between the entire data cube of the original and generated hyperspectral images to evaluate how “real” the generated image can be. However, due to the obvious improvement of spatial contrast in the generated HR-HSI, which resulted in disparity between the generated and original hyperspectral images, we only obtained an SSIM in the range of 0.3 to 0.7. Considering that most spatial information came from the high-resolution digital histology images, we generated grayscale images for the generated HR-HSI $(\widehat{G})$ and the stacked HR-RGB (G) by calculating their average across all bands, and calculated SSIM between two grayscale images:

$$\mathit{\text{SSIM}}(G,\widehat{G})=\frac{\left(2{\mu }_G{\mu }_{\widehat{G}}+c_1\right)\left(2{\sigma }_{G\widehat{G}}+c_2\right)}{\left({\mu }_G^2+{\mu }_{\widehat{G}}^2+c_1\right)\left({\sigma }_G^2+{\sigma }_{\widehat{G}}^2+c_2\right)}$$ (7)

where μ and σ are the mean and standard deviation of the grayscale images; c₁ and c₂ are two constants with small values chosen by default in MATLAB.

For the evaluation of spectral signatures in the generated HR-HSI, we used spectral angle mapper (SAM) [24, 25] as the evaluation metric. Still, due to the improved image contrast of the generated HR-HSI, the spectral signature of each pixel in the generated HR-HSI is not always the same with that of the corresponding pixel in the original HR-HSI. Therefore, we manually extracted several regions of the nucleus, cytoplasm, thyroid colloid, as well as the blank area, and calculated their spectral angles, just to illustrate how well the spectra were reconstructed.

3. RESULTS

Our proposed unsupervised super-resolution network could fuse the spatial information from the high-resolution digital histology images and the spectral information from the low-resolution hyperspectral images and generate high-quality high-resolution hyperspectral images. The quantitative evaluation results of three super-resolution networks (2×, 4×, and 5×) are shown in Table 2. The PSNR and SSIM measures are more related to the spatial component of the hyperspectral images, while the MAE is more sensitive to the spectral reconstruction errors [26]. The PSNR and MAE of three networks indicate a satisfying reconstruction performance both spatially and spectrally. Due to the significant improvement in image quality of the generated HR-HSI, especially the recovered texture details that were missing in the original HR-HSI, we did not get high SSIM values on the entire hyperspectral data cube. However, we calculated SSIM between the grayscale images of generated HR-HSI and stacked HR-RGB and got an average SSIM of 89.4% for the 2× network, 92.1% for the 4× network, as well as 93.0% for the 5× network. Considering the inherent difference between the two grayscale images, which would have lowered the SSIM value to some extent, the obtained SSIM indicate very good maintenance of the spatial information.

Table 2.

Quantitative testing results of the unsupervised super-resolution network

Network	PSNR (dB)	MAE (%)	SSIM (%)
2×	22.7±1.3	6.0±0.9	89.4±4.1
4×	21.8±1.7	6.4±1.2	92.1±3.0
5×	19.96±1.3	8.2±1.2	93.0±2.4

Figure 3. shows the comparison between the original HR-HSI and the generated HR-HSIs from three super-resolution networks. Five wavelength bands as well as the HSI-synthesized RGB images from each HR-HSI are shown. The intensity of each wavelength band in the generated HR-HSI is close to that in the original HR-HSI. The spatial features were very well maintained in all wavelength bands, even if some texture details were missing in the original HR-HSI (e.g. 701 nm). It can be seen that the RGB-guided super resolution network can significantly reduce the noise in some wavelength bands (e.g. 470 nm) that was caused by the low sensitivity of the image sensor. Moreover, our network could also compensate the unsatisfying focusing issue on some hyperspectral images. The high-frequency spatial information that was fused from the RGB images also increased the contrast in the generated hyperspectral images, which was especially obvious for the nuclei and muscle fibers, as shown in Figure 4.

Figure 3.

Single-band images and HSI-synthesized RGB images of the original HR-HSI and generated HR-HSI from three different super-resolution networks showing the satisfying spatial reconstruction in various wavelength bands. Difference images between the original HSI bands and the generated HSI bands have very low values. The major difference is in the first (470 nm) band, where lots of noise exist in the original HSI but gets removed from the generated HSIs.

Figure 4.

Improvement of image quality in generated high-resolution hyperspectral images. (a) Real high-resolution RGB histology image with fine structures of the tissue. (b) Synthesized RGB of the original high-resolution higherpsectral image patch, where the fiber structures are missing due to unsatisfying focus. (c) Synthesized RGB of the generated high-resolution hyperspectral image, where the fiber structures are recovered. (d) Real high-resolution RGB histology image with clear margins of nuclei. (e) Synthesized RGB of the original HR-HSI, where the margins of nuclei become fuzzy. (f) Synthesized RGB of the generated HR-HSI, where the shape of nuclei is very well recovered.

Figure 5. shows the spectral signatures of extracted nucleus, cytoplasm, lymphocyte, and the blank area in the slide from the original HR-HSI and the generated HR-HSIs, respectively. The shape of spectra from different tissue types was very well maintained, which is critical for an effective image classification. We calculated the spectral angles between the spectra from the generated HSI and original HSI for the extracted nucleus, cytoplasm, lymphocyte, and blank area, as shown in the figure, which indicate a high similarity between the synthesized and original spectra. However, the spectral signatures in the generated HSI had slightly lower intensities than their original values. It was likely due to the difference between the two grayscale images when minimizing the spatial loss for the network. In the future, we will modify the grayscale images to reduce the spectral intensity variation. Nevertheless, this variation can be compensated by increasing the intensity of the whole synthesized HSI.

Figure 5.

Comparison of spectral signatures of various cellular components from the original HR-HSI and generated HR-HSI. The low spectral angle values and the well-maintained spectral shapes indicate a good spectral reconstruction. (a) A high-resolution digital histologic image patch from a head and neck cancer slide, including cancer cells and lymphocytes. (b) Spectral signatures of cytoplasm extracted from the same region in the original HR-HSI and the generated HR-HSIs. (c) Spectral signatures of the same lymphocyte extracted from the original HR-HSI and the generated HR-HSIs. (d) Spectral signatures of the same blank area extracted from the original HR-HSI and the generated HR-HSIs. (d) Spectral signatures of the same cancer nucleus extracted from the original HR-HSI and the generated HR-HSIs. All spectral angles are given in radians.

4. DISCUSSION & CONCLUSION

In this work, we developed an unsupervised convolutional neural network based on a modified U-Net architecture for hyperspectral super resolution reconstruction with the guidance of high-quality RGB digital histology images. The network can fuse the spatial information from the high-resolution RGB images and the spectral information from the low-resolution hyperspectral images to generate high-resolution hyperspectral images. The generated hyperspectral images from all three super-resolution networks (2×, 4×, and 5×) maintained the shape of the original spectral signatures while being enriched with high-frequency spatial information. In addition, the proposed method could improve the image quality for hyperspectral images, such as reducing image noise, increasing image contrast, and deblurring. Furthermore, the unsupervised method does not require any high-resolution hyperspectral image as ground truth for network training, which minimizes the complexity of the workflow.

One existing shortcoming of our network is that some spectra in the generated hyperspectral images have slightly lower values than those in the original hyperspectral images. It is possibly due to the intensity difference between the two grayscale images that were used to calculate the spatial loss. In addition, the color variation and different thickness of slides might have caused some minor noise in the generated HR-HIS, due to which the spectra were not as smooth as the original ones. Regarding these issues, we will modify the stack ratio of the RGB image patch and increase the training dataset in the future, in order to further improve the performance of the network.

With hyperspectral imaging getting more attention in the medical imaging field, the confliction between the acquisition speed, data storage, and resolution of hyperspectral images must be solved, in order to apply this technology in real clinical settings. Our proposed method makes it possible to obtain high-quality hyperspectral images with low-resolution camera and low-magnification objective lens, hence greatly reduce the acquisition time and file size of hyperspectral histology images. For instance, with a 4× network, the acquisition time of a line-scanning HSI system can be cut to 1/4, and the file size to 1/16. It also allows the use of snapshot cameras, which can easily reach video-rate imaging, without the concern of low spatial image resolution. Moreover, our method doesn’t change the routine workflow in pathology. Instead of development of a new complex system, it can be achieved simply by mounting a low-resolution hyperspectral camera onto a commercial whole-slide scanner. The acquired low-resolution hyperspectral images can be stored and used for automatic digital pathological analysis after super-resolution reconstruction, while the RGB digital histology images are still available for pathologist to peruse to confirm the results.

In the future, we plan to develop an automatic whole-slide scanning hyperspectral imaging microscope, with a low-resolution hyperspectral camera (e.g. a snapshot camera) and the proposed super-resolution network built into the system. By synchronizing the color camera and the hyperspectral camera, the system will be able to acquire both data simultaneously. With the mature autofocusing and image enhancement techniques of color cameras as well as our proposed method, the image quality of both the RGB and hyperspectral images are secured. We may be able to establish a comprehensive database of whole-slide hyperspectral histology images, which is extremely beneficial for a thorough investigation of pathological features in hyperspectral images as well as various deep learning algorithms. In conclusion, with the satisfying results that we got in this work, we believe that our proposed method is promising to prompt the application of HSI technology in pathology.