Hyperspectral and multispectral imaging in digital and computational pathology: a systematic review [Invited]

Abstract

Hyperspectral imaging (HSI) and multispectral imaging (MSI) technologies have the potential to transform the fields of digital and computational pathology. Traditional digitized histopathological slides are imaged with RGB imaging. Utilizing HSI/MSI, spectral information across wavelengths within and beyond the visual range can complement spatial information for the creation of computer-aided diagnostic tools for both stained and unstained histological specimens. In this systematic review, we summarize the methods and uses of HSI/MSI for staining and color correction, immunohistochemistry, autofluorescence, and histopathological diagnostic research. Studies include hematology, breast cancer, head and neck cancer, skin cancer, and diseases of central nervous, gastrointestinal, and genitourinary systems. The use of HSI/MSI suggest an improvement in the detection of diseases and clinical practice compared with traditional RGB analysis, and brings new opportunities in histological analysis of samples, such as digital staining or alleviating the inter-laboratory variability of digitized samples. Nevertheless, the number of studies in this field is currently limited, and more research is needed to confirm the advantages of this technology compared to conventional imagery.

1. Introduction

Traditional computational pathology, also known as digital pathology, is an emerging technology that promises quantitative diagnosis of pathological samples, reduction of inter-observer variability among pathologists, and saving time in the manual examination of histological samples [1,2]. Traditional computational pathology relies on RGB digitized histology images. Within computational pathology, several research groups have begun to explore if hyperspectral/multispectral (HS/MS) imaging (HSI/MSI) are technologies able to provide further advantages to this end.

The study of light propagation through biological tissues is useful to identify several diseases. Light propagation in biological tissues involves three different photophysical processes: refraction, scattering and absorption [3]. Refraction and reflection of light within biological tissues, which are usually non-homogeneous media, is related to the changes in speed and direction of light. The absorption of light involves the extraction of energy from light by molecules. Thus, absorption peaks are related to transitions between two energy levels in a molecule at a specific wavelength. These absorption peaks are used as a fingerprint of the molecules’ response to light, providing information that can be used for diagnostic purposes. Scattering of light occurs when there is a spatial variation of the reflective index within tissues. The scattering of some biological components shows variations under certain disease conditions, becoming useful for diagnosis purposes [4]. Finally, some tissues show fluorescence when are excited by an external light source. For example, the emission of proteins and nucleic acids can be observed after exciting tissue with ultraviolet light. Traditionally, these properties of tissue are measured in the spectral range known as therapeutic window, from 600 to 1300 nm [5], where tissues present weak absorption, and light is more likely to penetrate tissues.

These properties of the interaction between light and biological tissue motivate the use of technologies that exploit the information of light propagation through tissues to develop tools for diagnosis support. Raman Spectroscopy (RS) and Fourier Transform Infrared (FTIR) Spectroscopy make use of the information of the vibrating molecules produced by photons for diagnostics [6–8]. RS and FTIR spectroscopy are useful for identifying types of molecules, leading to their usage in biomedical applications. Both techniques are based on the vibrational state of the molecules, but while FTIR spectroscopy is more appropriate for absorption measurements, RS is more sensitive to scattering changes. This makes both technologies complementary. Additionally, Spatial Frequency Domain Imaging (SFDI) make use of modulated light sources and light transport models to extract information about absorption and scattering about different tissues, which can be subsequently used for diagnosis [9]. Finally, after the application of specific fluorescent agents to the sample, fluorescence spectroscopy techniques are able to measure the fluorescence spectra of the specimen after light excitation. Such fluorescence spectra can be associated with different disease states, leading in applications for biomedical diagnosis [10].

In opposition to these spectral technologies, HSI/MSI are optical spectroscopy imaging modalities, which directly measure the incoming radiance spectra of light. There are two major detection modes, depending on the incidence of light within the tissue: light reflection or light transmission. The spectral information measured by these technologies is usually related to the information about both scattering and absorption of light within the sample but can also be used for fluorescence measurements. In this sense, HSI/MSI are imaging techniques (also called imaging spectroscopy) able to obtain both spatial and spectral information within and beyond the human visual sensitivity, which is restricted to the spectral range from 380 to 740 nm [3]. HSI/MSI can obtain additional information within the electromagnetic (EM) spectrum by capturing the information regarding different wavelengths (also called spectral bands or spectral channels) up to 2500 nm. Similarly to the human eye capabilities, RGB imaging can be conceived as a multispectral image with only three spectral bands, related with the opsins of the retina (blue light – 430 nm – cianopsin; green light – 530 nm – cloropsin; red light – 650 nm – eritropsin)) [11]. Each pixel of an HS/MS image represent the light measured by the camera at each specific wavelength, creating a set of light measurements which comprise the spectral signature. This spectral signature can be understood as a fingerprint of each material that can allow differentiation of elements in a captured scene by using HS/MS processing algorithms [12].

In the medical field, several studies implement HSI/MSI for automated disease diagnosis and image-guided surgery [13], for example, both in-vivo and ex-vivo cancer detection [14], or gastroenterology applications [15], and many others. There are existing reviews related to the use of HSI/MSI for microscopy and digital histology available in the literature. Some of these reviews aim to present the basics on technology and the common processing approaches [16–18], but those manuscripts are limited in the number of applications covered and are not up to date. Other reviews are focused on technology aspects. Gao et al. performed a review describing the instrumentation used in HSI/MSI for microscopy applications [19], while Hermes et al. performed an overview of the different infrared sensing techniques [20]. Finally, Mansfield et al. presented a review about the application of only MSI for histopathological analysis more focused in the technical aspects [21].

In this systematic review, we analyze the use of both HSI and MSI for pathological diagnosis, digital staining and other applications. This review adheres to the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), and the objectives of this review can be summarized by using the PICOS (Participants, Interventions, Comparisons, Outcomes and Study design) criteria [22]. The subjects of studies found in this review are limited to specimens from healthy and diseased humans and mammals. Prior to conducting the search and systematic review, it was unknown if other studies would compare outcomes of HSI to other image analysis techniques using conventional RGB images. Therefore, the nature of this review is non-interventional and studies the applications of HSI/MSI only for microscopic examination. Papers that provide comparisons of imaging modalities are reported, but this systematic review is not restricted to them. The outcome is to summarize the current status of HSI/MSI in histological analysis and methodology, including details about the population of each study, sample preparation, instrumentation and data analysis methods. Lastly, regarding study design, all primary research publications that make use of HSI/MSI for analysis of all types of histological specimens (including both journal papers and conference proceedings) are considered for inclusion in this review. A detailed technological analysis and summary is provided for clinical applications, methodology, instrumentation, data analysis techniques, limitations, and quality of outcomes compared to conventional imagery.

2. HSI/MSI information processing overview

This manuscript is focused on the current advances in HSI/MSI for histological applications. As stated previously, HS/MS data contain information about both the spectral and the morphological features of the samples. However, in order to extract useful information for different applications, such data should be properly analyzed using image processing techniques. The available options for HS/MS image processing are currently wide, and there are multiple choices about how to analyze such data. One important stage in HS/MS image processing is data calibration methods. Such methods are used to alleviate the heterogeneities between HS/MS data, which are mainly caused by the instrumentation itself [23]. Nevertheless, the main trend in HSI/MSI is the use of machine learning algorithms to extract information about the different materials or substances which are present in a HS/MS image in order to make accurate predictions. The first attempt to exploit information from HS/MS images was carried out in a pixel-wise manner, where only the spectral information of the different pixels in the image was used to identify the different materials within the image. In a recent survey manuscript, Ghamisi et al. performed a comparison between the most common supervised classifiers used for information retrieval in HSI/MSI [24]. The authors discussed the most popular supervised classifiers for HS/MS classification, namely Support Vector Machines (SVM), Random Forests (RF), Neural Networks (NN) and Multinomial Logistic Regression (MLR). Critically, the aforementioned classifiers do not necessarily take into account the morphological properties of the samples. For this reason, other approaches are based in the joint exploitation of both the spectral and the spatial properties of data [25].

The aforementioned machine learning techniques can be applied either directly to the spectral data or after applying some feature extraction methods. On one hand, dimensionality reduction methods have the goal of performing a mathematical transformation in the data, where the most important information is kept, achieving a reduced dimensionality. On the other hand, band selection methods are devoted to identifying the most important spectral bands for a given application. More recently, deep learning techniques have shown their potential for automatic HSI/MSI feature extraction and classification [26,27]. Such approaches are mainly based on different Convolutional Neural Networks (CNN) architectures, which extract spatial and spectral features simultaneously. The main advantage of deep learning techniques for supervised classification is their capability to find which features from the original dataset are more relevant for the identification of different materials within a HS/MS image through stacked layers of abstraction. Beyond the usage of machine learning approaches, other processing techniques for HSI/MSI image processing are spectral unmixing [28], the use of Normalized Difference Index (NDI) estimation [29], or simply the statistical analysis of HSI/MSI data.

In this section, we provided a brief context about the processing techniques that can be used for HSI/MSI information retrieval. In conclusion, we find that the current information extraction methods are not standardized, and there is a wide range of options to reach the same goal. In our systematic review, researchers employ a variety of processing techniques to evaluate the disease detection ability of HSI/MSI. A more concise analysis about the processing methods for HSI/MSI image processing is outside the scope of this manuscript. However, we provide the basic context about HSI/MSI image processing and some state-of-art references for readers interested in deeper details about this topic.

3. Systematic review methodology

The methodology carried out for this systematic review adheres to the previously established PRISMA guidelines. The PRISMA guidelines consist of a four-phase flow chart and item checklist. The PRISMA statement [22] aims to improve the reporting of systematic reviews and meta-analyses. While the original focus of PRISMA is on randomized trials and interventions, it can also be used as the foundation for reporting systematic reviews of any type of research. Additionally, we used the PRISMA explanation and elaboration document [30] as a guideline, which is intended to enhance the use, understanding and dissemination of the PRISMA statement. The PRISMA statement establishes a methodology to perform a systematic review that includes a description of information sources to be used, the search strategy, the eligibility criteria of manuscripts, a report on how the study selection was carried out, and the protocol that researchers should follow when reviewing the manuscripts selected for the systematic review.

3.1. Eligibility criteria

The eligibility criteria of the manuscripts are highly related with the aforementioned objectives, which follow the PICOS guideline. To this end, the manuscripts eligible to be included in this systematic review consist of applications of MSI, HSI, or near-infrared (NIR) imaging. The former is included just in case that some studies exploit the use of only a few wavelengths, but not the standard RGB for histopathological analysis. Moreover, this review will only cover optical microscopy, which excludes point-wise and non-imaging spectroscopy, RS, SFDI, or FTIR from the scope of this review. Furthermore, the specimens used by the studies covered in this manuscript are restricted to be acquired from human or mammals. Regarding the data analysis methods, all data analysis techniques will be included in this manuscript. Articles considered must be published between 2004 and 2019 and written in English.

3.2. Information sources

The search for manuscripts to be included in this systematic review was performed in the Scopus and PubMed databases. The Scopus database [31] is the largest abstract and citation database of peer-reviewed literature. Scopus comprises a comprehensive collection of research outputs in the fields of science, technology, medicine, social sciences, and arts and humanities. On the other hand, the PubMed database [32] is a free resource developed by the National Center for Biotechnology Information (NCBI) and the National Institutes of Health (NIH) of United States of America. PubMed comprises over 30 million citations and abstracts from the fields of biomedicine and health, including life sciences, behavioral sciences, chemical sciences, and bioengineering. As an additional source of information, after performing the study selection, we considered all references cited by the selected manuscripts for the clinical diagnostic research category for potential inclusion if they adhere to the previous eligibility criteria.

3.3. Search

The database search for manuscripts was performed on August 13^th 2019. The search was limited to manuscripts published between 2004 and 2019 and written in English. Two different researchers performed the search (S.O. and M.H.), one in PubMed and the other one in Scopus. The keywords used in both searches were:

• (Hyperspectral OR Multispectral) AND (histology OR pathology OR histopathology)

The primary searches produced 2,213 citations and abstracts in total: 1,115 from PubMed and 1,098 from Scopus.

3.4. Study selection

The screening was performed by two researchers by reading the titles and abstracts of the manuscript citations found in the search. If the suitability of a manuscript was not clear from the abstract, the full manuscript was obtained and scrutinized to decide if the document should be included in the systematic review. The studies that adhere to the above eligibility criteria were selected.

3.5. Protocol and registration

As stated in the PRISMA guidelines, prior to the search and analysis of the manuscripts for this systematic review, we developed a protocol to be followed for reviewing manuscripts. This protocol is mainly focused in covering the objectives, i.e. questions to be addressed, of this systematic review, and it defines the data items to be sought during the evaluation of each manuscript. Our protocol is based on four major questions to be answered: the analysis of clinical data, the methodology used to carry out the experiments, the description of instrumentation employed, and the analysis of the data processing methods. For the analysis of clinical data, the authors should identify the types of tissues to be imaged, the number of patients, the types of patients, the histological preparation, e.g. staining, embedding, sectioning, etc., and the outcomes from a medical perspective. In the instrumentation, we searched for the type of HS/MS system employed and its characteristics, such as the spectral resolution, the spatial resolution, the number of bands, and the magnification. Lastly, regarding the data analysis methods, we sought for information regarding the calibration of data and the goal of the data analysis method, e.g. classification or segmentation. The information extracted from each manuscript is summarized in a table that contains all the data items previously mentioned.

3.6. Risk of bias

It is worth noting that the studies herein presented may suffer from publication bias or selection bias because these studies demonstrate positive results on the utility of HS/MS histological analysis. It is possible that only promising results are published and that comparison to other technologies, such as conventional imaging, could be intentionally omitted from experimental design or the final manuscript preparation.

3.7. Search results

In this section, we describe the study selection performed after the search. We make use of the PRISMA four-phase flow diagram (Fig. 1). We also used Mendeley reference manager (Elsevier B.V., Amsterdam, Netherlands) as a computer-aid tool for the study selection. The primary search provided 1,115 manuscripts from PubMed and 1,098 manuscripts from Scopus, having a total of 2,213 manuscripts. After removing duplicates, we analyzed abstracts from 1,648 manuscripts. From these records, we selected 311 for full-text review, and we rejected 1,337 records. We excluded 497 records, which were not related to HSI/MSI, and 840 records, which were not associated to histology. After reviewing the full-text articles, 118 were excluded because they were not in the scope of this systematic review. Finally, additional 20 manuscripts were included from the references of the selected articles.

Fig. 1.

PRISMA flowchart for the search of literatures and studies.

In total, we have included 193 research articles for this systematic review. The papers were categorized into five different sections: system development, color and staining, inmunohistochemistry (IHC), autoflorescence (AF) and diagnostic research for clinical routine practice. In addition to the systematic review of the selected papers, we have included some critical remarks on the current limitations and challenges of HSI/MSI at the end of each section.

4. Hyperspectral/multispectral system development

In this section, we provide readers with a basic background about the instrumentation used for HSI/MSI in histology applications, and we briefly summarize the main research carried out in the context of HS/MS instrumentation development and optimization.

There are three main characteristics of any HS/MS acquisition system: spectral range, spectral resolution, and spatial resolution. The spectral range is related to the range of EM wavelengths covered by the spectral camera. For biomedical applications, several light-tissue interactions can be observed in the diagnostic window, i.e. from 600 to 1300 nm. There are several types of commercial HS/MS acquisition systems which cover different spectral ranges: visible and near-infrared (VNIR) spectrum (400–1,000 nm), NIR spectrum (900–1,700 nm) or short-wavelength infrared (SWIR) spectrum (1,000–2,500 nm). The spectral resolution is defined as the resolution the EM is sampled, i.e. the difference between two consecutive spectral channels. A narrow spectral resolution may enable the identification of subtle absorption peaks, which is not possible to differentiate at lower spectral resolutions. Finally, the spatial resolution is related to the actual pixel size. Depending on the application, higher spatial resolutions are required to image small objects. In spectral imaging, the use of low spatial resolution can ultimately lead to spectrally-mixed measurements, i.e. finding the combined spectral signatures of several materials which are present in the current pixel. Low spatial resolution, and therefore high spectral-mixing, is admissible only when the application deals with homogenous materials, or at least materials that are homogenous enough for such spatial granularity.

In addition to the basic features of HS/MS instrumentation, there are several types of acquisition systems that can be found in the literature depending on the way of capturing the spectral and spatial information. Spatial-scanning techniques collect all the spectral information from a single point (point-scanning or whisk-broom sensors) or from a single spatial line (push-broom or line-scanning sensors) [33]. In order to collect the spatial information, spatial-scanning techniques require relative movement between the camera and the sample. In opposition, spectral-scanning methods aim to collect progressively all the spatial information for different wavelengths. In spectral-scanning methods, a single wavelength is captured each time, and the scan is performed by changing the central wavelength of the spectral channel to be imaged. Examples of spatial-scanning systems are filter wheels, liquid crystal tunable filters (LCTFs), and acousto-optic tunable filters (AOTFs) [34]. Finally, snapshot sensors are designed to simultaneously collect both the spatial and the spectral information of the sample in a single shot [34]. Further comparisons between HS/MS acquisition technologies in terms of spatial resolution, spectral resolution, spectral range and acquisition time requirements can be found in [35].

Most of the researchers who use HS/MS technologies for histopathological analysis use equipment based on a conventional microscope attached to a commercial HS/MS camera. Nevertheless, several research groups have focused their efforts in instrumentation development of HS/MS image acquisition systems for histological analysis of samples. These approaches include systems based on the following technology and sensors: AOTFs [36,37], tunable light sources [38], imaging scanning spectrometers [39,40], thin-film tunable filters [41], MS filter arrays [42], and push-broom scanning [43,44]. Most of the acquisition systems in HSI/MSI are in the proof-of-concept stage and they are not ready to effectively perform whole-slide spectral imaging over the samples. However, Jiang et al. proposed a whole-slide imaging (WSI) system for HSI based on slit-array projections [45]. The aforementioned studies contain valuable information about development of HS/MS microscopic systems, but the contents are too technical for the previously established scope of this review. For completeness, the references are provided for interested readers to explore in more detail.

In this section, we defined the most important parameters of HS/MS acquisition systems. HS/MS technology is shown as a promising technology for biomedical applications, since the spectral interaction between light and tissue has been proven to provide information about diagnosis. However, the instrumentation is still quite expensive, large amounts of data storage are needed, and computational requirements are extremely high. These storage requirements are more evident for histopathological applications, where histopathological laboratories are able to digitize hundreds of slides per day.

However, it is possible to reduce the cost of the HS/MS instrumentation. If a reduced subset of spectral bands, which are useful for a certain application (e.g. certain tissue diagnosis) are identified, it would be possible to develop a low-cost MS system with similar storage requirements compared to conventional RGB imagery. In this situation, different low-cost MS systems can be used for different applications, where different spectral bands are required. Nevertheless, to reach this situation, more research should be performed with high resolution and wide spectral range HS instrumentation to determine which bands are relevant for each application. Furthermore, the storage capacity requirements can be also alleviated by making use of HS/MS data compression algorithms. The impact of using lossless or lossy compression in the data analysis should be determined in the close future. More investigation is required for histological applications where HS/MS analysis may increase the diagnosis ability of disease compared to conventional RGB imagery.

5. Color enhancement and digital staining

The examination of pathological slides provides the final diagnosis for most diseases, and color plays an important role. In this context, Cukierski et al. made use of the concept of metamers for analyzing the advantages of MSI for histopathological imaging. If a color has a higher probability of being metameric, then MSI will likely improve the task of distinguishing two structures, which have similar color but different absorption spectra. Using the concept of metamers and linear algebra transformations, the authors concluded that the maximum spectral difference between two metamers is beyond the capabilities of the human eye, where MSI could boost the diagnostic capabilities of conventional RGB imagery [46,47]. Uneven color conditions among different samples can complicate the examination of samples and may be caused by different staining conditions (staining time, temperature or pH of the solution) and instrumentation (camera and microscope characteristics). To handle this problem, Abe et al. proposed a method where the amount of dye is estimated by using MS images in combination with the Beer Lambert law, and then the original image is weighted to obtain a color-corrected image. The method was proven to perform the color correction under different conditions of H&E (hematoxylin and eosin) stained slides of human liver, namely over-staining, under-staining, and excess of either hematoxylin or eosin [48,49]. Yagi et al. proposed a method for color standardization based on MSI and the use of a standardized color chart [50,51]. This procedure was proven to deal with the variations in appearance of H&E stained slide from different laboratories, which is one of the biggest challenges in whole slide imaging of histological slides. Additionally, the capabilities of HSI/MSI to measure colors accurately make it attractive to be employed as benchmark for measuring the color performance. Motivated by the lack of conventional color performance techniques, such as colorimetry or spectroradiometry, to measure microscopic biological tissues, Salehen et al. evaluated the color performance of two different whole-slide imaging systems using HSI and three different H&E stained histological samples as color targets [52,53].

Another interesting application of HSI/MSI in histopathology is digital staining. The main goal of this approach is to highlight molecular components in digitized slides without performing a physical stain of the samples. For example, Masson’s trichrome (MT) stain is used to emphasize fibrosis structures, which can aid in the diagnosis of chronic liver diseases. In this sense, Bautista et al. proposed several techniques to digitally stain H&E MS images and virtually transform into MT stained samples [54–60]. Apart from the digital stain to simulate MT, the same group also applied digital staining to visualize the color differences between tissue structures that displayed similar H&E staining patterns. This approach promises to be useful to visualize tissue structures that are not emphasized by the original stain, without the requirement of additional physical staining [61,62]. An example of this application can be observed in Fig. 2. Beyond digital staining of previously H&E stained samples, some researchers have focused their work in performing digital staining of unstained samples. In this field, Bautista et al. also performed digital generation of an H&E image from an unstained kidney slide using supervised classification of nucleus, cytoplasm and RBCs (Red Blood Cells), and then applied a linear transformation to stain those components into a H&E like image [63]. Additionally, Bayramoglu et al. digitally stained an unstained lung specimen, producing the appearance of H&E staining, using a conditional generative adversarial network (cGAN) [64].

Fig. 2.

Example of digital staining of H&E images to provide them with the appearance of MT stain. (a) Original H&E image (b) Digitally stained image (c) MT stained image. Reproduced from [62]; Creative Commons BY 4.0; published by SPIE (2012).

In summary, the current state-of-art uses of HSI/MSI in the field of digital staining and color optimization are mostly focused on color optimization, standardization of digitized slides, and digital staining of samples.

In the context of color enhancement and standardization, the research carried out by the HSI/MSI community has been limited to proof-of-concept research, where the color reproducibility and the inter-laboratory variations of digitized slides are improved by means of HS/MS image processing. Nevertheless, such studies have not demonstrated a significant difference between conventional WSI digitized slides and HSI/MSI. For this reason, HSI/MSI solutions have been proven as a realistic alternative to conventional RGB imagery for clinical environments despite increased instrumentation costs, data storage, and computational requirements for color enhancement and standardization. Furthermore, recently, Campanella et al. found a performance drop in a supervised classification for histological diagnosis when data used for training belong to different institutions, even when the number of training samples was high [65]. For this reason, and regardless of the cost associated with image acquisition in the preliminary stages, HSI/MSI technology should be further analyzed as an alternative to compensate the differences in color between data from different institutions or acquisition systems. Once such research is performed, there could be a well-founded argument discussion about the enhancements of using expensive HSI/MSI to alleviate inter-laboratory differences, and hence, compensating the problems in automatic machine learning approaches, which could benefit histopathological laboratories in the long-term.

In the context of digital staining, some of these works have the goal of generating trichromatic images from H&E stained images by processing MS/HS images. Some researchers have recently proposed new approaches to reach the same goal using conventional imaging techniques. Fereidouni et al. demonstrated the feasibility of HSI/MSI for the generation of trichrome images exploiting the combination of a fluorescence image and a brightfield image [66]; while Rivenson et al. suggested the use of deep learning to generate trichrome images from standard RGB data from H&E slides [67]. In fact, both approaches were presented as cost-effective alternatives to the use of HSI/MSI. However, these are only examples of the generation of one specific type of digital stain. Further research should be performed in order to investigate if HSI/MSI is able to outperform RGB or fluorescence technologies for this and other digital stain applications. For this reason, the research in the field of digital staining by using HSI/MSI should be focused in the exploration of unstained samples. The dyes used to stain tissues for subsequent examination of histological samples modify the spectral signature of the sample itself, restricting the spectral information only to the visible spectral range of light. Therefore, it should be investigated if the spectral information within unstained histological samples could provide advantages compared to conventional stained samples. The main opportunity of HSI/MSI technologies is to use unstained samples for digitally synthesized multiple-dye digital staining. According to the literature, this information can be found in the diagnostic window of the EM spectrum from 600 to 1300 nm [5]. In addition, Sordillo et al. have recently proposed the spectral window from 1600 to 1800nm to be also useful for diagnosis [68]. The investigation of histological samples at these wavelengths should be performed with unstained samples and with HS/MS cameras in the NIR spectral range. In summary, to explore the possibilities of this promising technology, more research should be performed in order to determine the potential clinical usage of HSI/MSI for such applications.

6. Immunohistochemistry and Immunofluorescence

Our extensive literature review located 59 research articles that present two commercially available MS microscope systems that are predominantly applied in the literature for immunohistochemistry and immunofluorescence (IF). These MS microscopes are clinically useful because they allow quantitative analysis of multiple molecular biomarkers in multi-label tissue specimens, both in bright-field and fluorescence modes. The Nuance CRi Multispectral Imaging System (Cambridge Research and Instrumentation, Woburn, MA; PerkinElmer, Inc., Hopkinton, MA) is available in three versions (VX, FX, and EX), which capture MS images in the wavelength ranges of 420 to 720 nm (VX and FX) and 450 to 950 nm (EX). A LCTF acquisition system is used for spectral imaging with spectral bandwidths of 20 or 40 nm for the EX/FX versions and 7, 10, or 20 nm for the VX version. The more recent Vectra Quantitative Pathology Imaging System (PerkinElmer, Inc., Hopkinton, MA) performs MS imaging and automated whole-slide scanning, available in 6 slides and 200 slides versions, using both the same camera. The Vectra captures MS images from 440 to 720 nm and uses a LCTF to capture MS images with a 10 or 20 nm spectral bandwidth. As summarized in Table 1, the MS microscopy has been used to study both IHC and IF in a variety of diseases, but it has been predominantly used to identify and quantify cancer biomarkers in human patients using the Nuance MS system, [69–94] and the Vectra MS system [95–123]. The IHC and IF studies reported below cover both human and animal subjects across all organ systems [69–128]. The detection of IHC biomarkers using MS microscopy is usually carried out by performing spectral unmixing of the samples, identifying the spectra of biomarkers that are similar to the spectra previously recorded in spectral libraries. An example of MSI for spectral unmixing of IHC stains for detecting biomarkers of follicular lymphoma is shown in Fig. 3.

Table 1. HS histopathological dataset summary.

IHC/IF Study Feature	No. (%)	Relevant Research Article References
MS Microscope
Nuance MS System	27 (46%)	[69–94]
Vectra MS System	28 (47%)	[95–123]
Other/ Unspecified	2 (5%)	[124–128]
Molecular Markers
IHC		[69–84,86–115,117–125,127]
IF		[85,116,126]
IHC and IF		[69–72,76,78,86,94,97–100,102,103,109,112–115,117,118,121–125,128]

Fig. 3.

Multiple IHC markers in one tissue specimen of follicular lymphoma. MS images acquired using the Vectra system and software for spectrally unmix each IHC stain component. (a) Triplex IHC composite image. (b-d) Spectral-unmixed grayscale images of IHC for CD3, FOXP3, and CD69, respectively. Reproduced from [107]; Creative Commons BY 4.0; published by Nature (2015).

The advantages of MSI are arguably most prominent for IHC, specifically regarding clinical translation. Firstly, with respect to the equipment needed, relatively small bands with ability for high-throughput slide scanning has been somewhat standardized and is commercially available in Vectra and Nuance. This is important because in much of HSI/MSI research, the technology is still research-grade, meaning that it is customized, developmental, and potentially cumbersome to deploy. Secondly, there is a clearly defined clinical need: the interpretation of IHC staining can be variable, user dependent, and often with an ill-defined threshold of immunopositivity [129]. Next, the data analysis techniques needed to solve the problem are quite rudimentary, with spectral-unmixing as straightforward and effective algorithms for identifying immunopositivity in the literature. Finally, there is the added ability of detection and quantification of multiple IHC stains and antigens in single specimens with some preliminary success.

On the contrary, however, pathology and laboratory medicine departments have not yet widely adopted standard RGB digital pathology. Therefore, it is unlikely that even more advanced digitization equipment, whose prohibitive cost has been already described above, would be widely adopted without convincing merit. The approach would have to be tested in multiple institutions with all types of stains, imaging parameters, and acquisition systems before it would ever be able to be clinically utilized and relied upon for routine diagnosis of immunopositivity in specimens. Therefore, though it is promising, there are numerous and substantial hurdles to the clinical translation of even this most plausible application of MSI technology.

7. Fluorescence and autofluorescence spectral imaging

Histological specimens can be excited by a specific, narrow band of light, and longer wavelengths can be imaged with MS or HS microscopy to observe the characteristic emission from the tissue. AF uses label-free specimens, allowing only endogenous fluorophores to create the measured signal, but both AF and dye-based fluorescence, such as green fluorescent protein (GFP), can be used for detecting pathologies. In the work performed by Pantalone et al., lymph node sections (5 microns thick) from unstained, frozen-section specimens were imaged with 365 nm excitation light and 375 to 750 nm emission. Lymph nodes with metastatic gastric or colon cancer showed differences from normal lymph nodes, which was reproducibly validated with multiple observers [130]. On the other hand, Constantinou et al. implemented a fluorescent slide scanner to image sections of a human colonic adenocarcinoma xenograft with excitation light of 488 nm that had detectable AF from elastin and flavin adenine dinucleotide (FAD) despite staining with a relevant antibody [131–133]. Similarly, Duong et al. developed a MS light-emitting diode (LED) array used to photo-irradiate tissue specimens, reducing artifactual AF. The appearance of artificial AF from formalin fixation degrades the ability to detect fluorescent signatures of interest. Therefore, fixed slices of mammalian brains were treated with the LED array with emissions from UV to IR, and successful and reproducible removal of artifact AF was observed. Treated IHC-stained slides were imaged using excitation/emission of 495 nm and 520 to 700 nm, and visible reduction of artificial AF were observed [134]. Ellingsen et al. demonstrated that using laser excitation of 800 nm, which produces 2-photon absorption at 400 nm, and capturing AF HS images with confocal microscopy allow the detection of amyloid plaque in cryo-sections of mouse brain [135,136].

Another study performed by Leavesley et al. employed HS imaging to capture data from cryo-sections of lung tissue for resolving the fluorescence of GFP [137,138]. Additionally, Dolloff et al. explored the application of spectral signatures of GFP in tissues for measuring autophagy [139]. The Leavesley et al. group also developed an excitation-scanning AF HS microscope imaging system for acquiring microscopic images of thin ex-vivo tissues (<1 mm), applied to rat organs [140] and human colon cancer [141,142]. Moreover, as shown in Fig. 4, this group applied the same technique to calculate the properties of FAD, nicotinamide adenine dinucleotide (NADH), elastin, and collagen from human colonic cancer tissues [143,144]. In ocular diseases, Dey et al. implemented a tensor decomposition approach to detect age-related macular generation using AF with a range of excitation and emission wavelengths [145,146]. Additionally, Habibalahi et al. used MS AF imaging to detect ocular squamous neoplasia in unstained, cover-slipped slides with multiple excitation and emission wavelengths [147].

Fig. 4.

Multiple IHC markers in one tissue specimen of follicular lymphoma. MS images acquired using the Vectra system and software for spectrally unmix each IHC stain component. (a) Triplex IHC composite image. (b-d) Spectral-unmixed grayscale images of IHC for CD3, FOXP3, and CD69, respectively. Reproduced from [87]; Creative Commons BY 4.0; published by Nature (2015).

In summary, there are various advantages and disadvantages of autofluorescence and fluorescence spectral imaging methods. The main advantage is that spectral autofluorescence can sense the concentrations of endogenous molecules in specimens using precise excitation and emission [143,144]. For example, it can be used to assess the concentration of NAD, FAD, collagen, and keratin, which have well-known excitation and emission signatures. One major attraction of this technique is that it works with unstained tissue specimens, so no time is lost by applying different histochemical stains, which supposes a loss of information. However, there are certain drawbacks of fluorescent spectral imaging for histopathology. Mainly, the equipment is not standardized, and most experiments are conducted with research-grade, custom-fitted microscopes that can be complex and expensive. As most tissues are comprised of numerous endogenous fluorophores, it is still unknown what wavelengths are best for excitation/emission for certain applications. Additionally, the fluorescence signals can be difficult to discern, especially if a stain or label is also used [148]. Therefore, while spectral autofluorescence imaging may seem promising, there are many challenges which currently hinder its widespread adoption.

8. Trends in diagnostic research for clinical routine practice

In this section, we summarize the research performed for clinical diagnosis of histological samples using HSI/MSI. A total of 84 research articles have been analyzed, and the systematic review of these manuscripts has been sorted according to the different fields within medical diagnosis, namely hematology, breast, central nervous system, gastrointestinal, genitourinary, head and neck, and skin. To conclude, we provide a summary table where the main characteristics of these researches are synthesized.

8.1. Central nervous system

HSI and MSI have been applied for aiding central nervous system (CNS) diagnosis and characterization, where these technologies have been investigated for the study of brain, nerves, and some indicators of different diseases. To investigate the normal histology of brain specimens, Bouzid et al. developed a customized MS microscopic system based on tunable light sources [149]. Cell nuclei segmentation was performed on a stained rat brain sample, and the results obtained were compared across 3 to 10 band MS images within the visual range. Ortega et al. investigated the use of HSI for detection of high-grade brain tumors in H&E stained slides using a customized push-broom VNIR microscopic system. Their results suggest that the differences in the spectral signatures of normal and tumor areas within the slides are sufficient to automatically provide a prediction of the diagnosis using supervised classifiers, such as SVM or NN [150,151].

Nerves are also a significant constituent of the CNS. The automatic identification and quantitative morphometry of nerve fibers can complement the clinical and histopathological evaluations of injured nerves in humans. Li et al. proposed an automatic segmentation of unstained nerve fiber specimens using HSI, and relying on RGB images of stained samples as ground truth. They found HSI successfully provided morphological parameters in unstained nerves, such as myelin thickness and area [152]. Additionally, the fiber diameter, perimeter, area, and myelin thickness and area were extracted in a later study, showing agreement with the manually labeled ground truth [153]. The conclusions of these studies suggest that HSI may be a promising analysis tool for unstained nerve sections, making possible a rapid characterization of nerve fibers compared to traditional techniques. On the other hand, in order to investigate pathological nerve samples, spinal cord tissue samples from mice were microscopically analyzed by Vazgiouraki et al. using MSI to develop a tool for diagnosis of spinal cord myelin loss associated with multiple sclerosis. The MS microscopic system used a custom-made rotating filter wheel, and the authors concluded that the maximum difference observable between normal and demyelinated lesion areas was at 500 nm [154]. Additionally, Kopriva et al. demonstrated the contrast enhancement within an unstained sample of sciatic nerve fibers from a mouse using nonlinear unsupervised segmentation [155].

The study of samples from CNS constituents can reveal the presence or progression of different diseases. In such context, More et al. studied the use of HSI as a potential tool for early detection of amyloidopathy in Alzheimer’s disease through statistical analysis of the spectral signatures from histological mouse retina and brain samples obtained with a push-broom VNIR microscopic system [156]. Diabetes can cause degeneration of the blood vessels in the eye, specifically in the retina, which can lead to blindness. To investigate this, Li et al. performed studies with control group, untreated, and treated diabetic rats to study HS analysis techniques of retina sections. The spectral signatures showed relevant differences in the range of 636 to 722 nm between the three groups, which motivates the quantitative study of the spectral differences for the evaluation of the therapeutic efficacy of drugs [157–160]. Finally, to investigate the detection of the Newcastle disease virus infection, which produces a neurological condition known as spongiosis, Abeysekera et al. obtained infected specimens from poultry [161]. In this work, a processing framework combined a customized feature extraction technique based on statistical indicators obtained from the MS data and supervised classifiers, SVM and LDA (Linear Discriminant Analysis), and the results demonstrated that MS data outperformed RGB data.

8.2. Head and neck

Cancer of the head and neck (H&N) can be diverse due to the anatomical diversity of this region. In the oral cavity and upperaerodigestive tract, approximately 90% of cancers are squamous cell carcinoma (SCC) [162]. Ou-Yang et al. combined transmission and fluorescence based HSI using a push-broom system with hundreds of VNIR spectral bands for detection of oral SCC in 34 patient samples. The cell nuclei were identified in the basal-cell layer manually and a five-fold method combined spectral and morphological features to yield good performance on the testing patients [163]. In order to investigate the detection of SCC after metastasizes, Akbari et al. used SVM supervised classification for identifying human oropharyngeal SCC cancer cell line xenograft that metastasized into lung and lymph node tissues in mice [164]. The SCC detection was performed macroscopically within H&E slides without a microscope.

The thyroid gland rests in front of the trachea in the neck, and masses can often be visually observed by the patient. To assess if a thyroid mass is benign or malignant, a fine needle aspiration (FNA) biopsy is performed and microscopically investigated, and HSI/MSI may be able to increase diagnostic ability. Mansoor et al. used MS histology for detecting non-cancerous thyroid and parathyroid adenomas in FNA, using 8 cases of follicular adenoma and 7 cases of parathyroid adenoma. Papanicolaou-stained cells were manually annotated, and cell-based classification was performed with a basic neural network [165]. Thyroid cancer biopsies were also studied by Hahn et al. using 100 cases of papillary thyroid carcinoma (PTC) and benign goiter (BG), developing a classifier to segment MS images and classify each region as background, PTC, or BG with successful results [166], as shown in Fig. 5. Additionally, the experiments were performed with different number of training cases, from 10 to 40 cases, without observing significant improvements in the classification results. However, the authors concluded that nuclear features offer improvement over nuclear and cytoplasmic regions when developing a classifier [166]. Finally, He et al. incorporated Muller matrix polarization HSI to distinguish papillary thyroid carcinoma tissues in unstained histological sections [167].

Fig. 5.

Thyroid FNA biopsies with Papanicolaou stain in RGB (left) and classification results with MSI spectral unmixing (right). (a,b) case of PTC. (c,d) case of benign goiter. Background (yellow), PTC (red), benign goiter (green), out-of-focus or crowded areas (blue). Reproduced from [166]; Creative Commons BY 4.0; published by ACS (2012).

Regarding the detection of thyroid nodules in FNA smears, several works performed by Shah et al. investigated MSI analysis techniques. After manually labeling the target cells to be detected, the authors performed a watershed image segmentation of the cells [168], finding that MSI yielded significantly fewer false positives compared to conventional image analysis. Furthermore, Wu et al. performed a minimally supervised band selection and reduction method of thyroid FNA to increase contrast of RBCs using histogram-based local descriptors evaluated by three distinct metrics [169]. On the other hand, Gabriel et al. used unsupervised k-means clustering for classification of cells in FNA thyroid lesion smears suspicious for cancer [170]. Lastly, Wu et al. also presented a conditional random field model segmentation scheme for classification of different thyroid nodules. Hyperplastic nodules, PTC, and follicular neoplasm were segmented successfully, and it was concluded that MS features offered increased accuracy compared to conventional image analysis [171–174].

8.3. Breast

Several approaches have been proposed in the literature for breast cancer identification in histological slides using HSI/MSI. Boucheron et al. performed a benign versus malignant nuclei classification using MSI, and compared the results with the ones obtained using 3 different synthetic RGB images generated from the MS images [175]. The authors did not find a significant boost on the classification using MSI compared to conventional RGB image processing. Nevertheless, Qi et al. also explored the utility of MSI for histological breast cancer diagnosis, finding an improvement in MSI data exploitation compared to RGB [176,177]. Although the instrumentation of both approaches was similar, there were some differences in the experimental procedures, especially in the type of staining, the magnification and the image analysis approaches. While Boucheron et al. used H&E stained samples acquired with 40× magnifications, Qi et al. employed hematoxylin-only specimens acquired with 10× magnification. Even if the contradictory conclusions can be caused by the differences in magnification or stain, the image analysis frameworks present several differences. Both studies used supervised classifier approaches, but while Boucheron et al. used the raw spectral bands of the MS image as features for classification, Qi et al. used the features, extracted from the MS image, that were supposed to maximize the underlying differences between normal and tumor tissue samples. More experimentation should be performed in this research line to provide a more relevant discussion about the importance of the staining, magnification, and the processing framework for these applications.

Beyond the discrimination between normal and tumor tissue, more specific approaches have been recently proposed to identify ductal carcinoma in situ (DCIS) or mitotic cells within breast histological slides. In one approach, Khouj et al. proposed the detection of DCIS with HSI using two types of specimen preparations of breast biopsies: unstained and H&E-stained [178]. Using as inputs some pixels manually annotated by pathologists, a semi-supervised k-means approach was applied to both types of images, suggesting a good discrimination between normal and tumor regions even in unstained samples. In Fig. 6, the spectral signatures of DCIS and normal tissue samples are depicted. Additionally, mitotic cells counting is an important indicator in breast cancer grading. Roux et al. proposed a contest for detecting mitotic cells within H&E stained breast cancer specimens. In this contest, three types of datasets were released: two of them composed of conventional RGB images, and a single dataset composed of MS images. In this challenge, the detection of mitotic cells was suggested to be superior to conventional imaging approaches [179]. Nevertheless, some authors have kept working in this dataset. Malon et al. proposed a CNN framework for mitotic cell detection, but the authors claimed that more images were necessary to successfully train deep learning models [180]. Furthermore, by using different band selection methods and different supervised classifiers, such as SVM, MLP (Multilayer Perceptron), LDA, Irshad et al. [181,182] and Lu et al. [183] were able to outperform the classification performance on this dataset stated by previous groups. Irshad et al. also found improvements in performance when exploiting the MS information compared to only the RGB information.

Fig. 6.

Spectral signatures of normal breast tissue (blue) and DCIS (red). Reproduced from [178]; Creative Commons BY 4.0; published by Frontiers (2018).

8.4. Gastrointestinal

The use of HSI/MSI has been also employed for gastrointestinal disease detection, where colorectal cancer is the most prominent application. In colorectal cancer diagnosis, an appropriate identification of different tissue constituents within pathological slides can help in tracking disease progression and improve the selection of an optimal treatment.

The preliminary studies for colorectal cancer detection using HSI/MSI were limited to discrimination between benign adenoma and malignant carcinoma tissues within H&E stained colon histological slides. To this end, Masood et al. employed spatial features extracted with local binary patterns, and then colonic specimens were classified using a SVM classifier [184]. Another approach, proposed by Maggioni et al., consisted of an initial segmentation of the MS images into nuclei, cytoplasm and background, and subsequently, nuclei were later classified into benign or malignant using a partial least squares (PLS) classifier based on morphological features [185]. A similar processing framework was followed by Rajpoot et al. and Masood et al., consisting of an initial segmentation of colon cell images into nuclei, cytoplasm, lumina propria and lumen, followed by the extraction of multiscale morphological features and supervised classification. Using this common framework, Rajpoot et al. employed a SVM classifier [186], while Masood et al. employed PCA (Principal Component Analysis) and LDA algorithms [187].

Beyond binary classification between benign and malignant colon tissues, a more detailed tissue identification scheme was followed by several authors, aiming to detect different types of cancer cells within colon samples: carcinoma, intraepithelial neoplasia, and benign hyperplasia. The results from Chaddad et al. suggested the utility of texture features of MS images for classification [188–190]. An example of this technique is demonstrated in Fig. 7. Using the same dataset, Peyret et al. demonstrated that exploiting morphological features in MSI improves the performance of panchromatic images in colon tissue classification [191,192]. Using a hybrid method which combines unsupervised clustering and supervised method using PCA and logistic regression, Nakaya et al. detected four stages of colon cancer progression from patients with ulcerative colitis, namely cancer, non-cancer, low grade dysplasia and high grade dysplasia [193]. In addition, Lao et al. used MS histology to directly quantify the optical signal obtained from in-situ hybridization (ISH) of colorectal adenocarcinoma tissues and counter-stained nuclei with methyl green. For the ISH, a small segment of microRNA was used to target cancer differentially, compared to low-grade and high-grade neoplasia and normal tissues [194]. In a later work, Chaddad et al. also revealed that the quantification of the spatial heterogeneity of the pathological tissues can help to detect the progression from benign cell proliferation to malignant lesions [195]. Also, Haj-Hassan et al. performed comparative research that demonstrated the boost in classification performance when using CNNs instead of feature-based approaches [196].

Fig. 7.

Segmentation of four types of tissue within colon pathological slides. (a) Stroma (b) Benign hyperplasia, (c) Intraepithelial neoplasia, (d) carcinoma. Reproduced from [190]; Creative Commons BY 4.0; published by Frontiers (2018).

Recently, Awan et al. improved the research in colorectal cancer detection within H&E pathological slides with two relevant innovations: a large patient dataset (n = 151) and the exploitation of information within near-infrared spectral bands (beyond 1,000 nm). The authors performed the classification in two different schemes: a two-class classification (normal versus tumor) and a four-class classification (normal, tumor, hyperplastic polyp and tubular adenoma with low-grade dysplasia). Using different types of feature extraction, band selection methods and SVM classification, the authors found that the use of a greater number of spectral bands significantly improves the discrimination of the different classes. Furthermore, they found that the use of near-infrared spectral bands improved the classification [197].

Extending beyond detection of primary colon cancers, Kopriva et al. applied MS histology for the detection of colorectal adenocarcinoma metastasis in the liver. Although the application is quite novel and the classification result was assessed with IHC staining as ground truth, the study was limited to a single patient, thus the results cannot be considered conclusive [198].

In addition to colorectal cancer detection, only pancreas, liver, spleen, and esophagus have been explored by means of HS/MS histological analysis. The first computer-aided tool for pancreas diagnosis was motivated by the drawbacks of conventional techniques for the identification of elastic and collagen fibers within pathological slides, which usually requires the use of Verhoef’s Van Gieson (EVG) staining, which is a more complex and expensive procedure compared to standard H&E staining. To deal with such limitations, Septiana et al. successfully exploited the spectral information within pancreatic ductal carcinoma H&E stained slides, showing a good capability of this technology to identify collagen and elastic fibers samples [199]. Similarly, motivated by the difficulty of distinguishing fibers and cytoplasm in H&E slides, Hashimoto et al. used a pixel-wise bag of features classification method over H&E stained liver samples by employing simultaneously spatial and spectral features for detecting five liver tissue components: nucleus, sinusoid, lymphocytes, fibers and cytoplasm. In this context, HSI was shown to outperform conventional RGB imagery [200]. Beyond the segmentation of tissue constituents, Wang et al. evaluated the ability of microscopic HSI to early detect bile duct carcinoma within H&E stained rat liver samples [201]. In this research, the authors were able to quantitatively measure the tumor areas in the biopsies at different time points through the analysis of the HS data with a feature extraction method based on the morphological watershed algorithm followed by SVM classification. On the other hand, Kopriva et al. applied a contrast enhancement technique, previously mentioned with application on sciatic nerves, for the evaluation of unstained spleen specimens [155]. Finally, Bautista et al. implemented MSI of an H&E stained slide from esophagus tissue to enhance the visualization of eosinophilic esophagitis [202]. The nuclei of eosinophils were automatically detected using spectral transmission PCA and a thresholding method. The authors concluded that this approach allowed tissue classification despite the nearby staining attributes, which facilitates a better specimen analysis compared to conventional RGB imagery.

8.5. Genitourinary

The genitourinary section comprises the organs of the urinary system, such as kidney and bladder, and both male and female reproductive organ systems. Prostate cancer is the second leading cause of cancer deaths for men [203]. It has been proposed in recent literature that histology using MS spectral analysis may improve the diagnosis of genitourinary cancers, such as cervical, prostate, ovarian, and bladder cancers.

Distinguishing between normal and cancerous cervical cells under Papanicolau stain requires the examination of texture, size, shape and contextual information of cells. For this reason, Zhang et al. proposed a method for the automatic segmentation of cervical cell nuclei that can automate the identification of the relevant cells to be examined carefully among other types of cells that are present in the specimens [204]. Another approach for identifying abnormal cervical cells that may be malignant in Papanicolau smears was implemented using cosine correlation analysis to exploit the differences of spectral signatures [205]. For the detection of cervical cancer cells in Pap smears, a segmentation method was applied to MSI based on a Gaussian mixture model (GMM) for unsupervised nuclear segmentation, and a similarity distance measurement was developed to quantity the similarity between the segmentation results and the original data, which were able to reveal intra-spectrum information achieving high nuclear segmentation accuracy despite wavelength reduction [206]. Another cervical cancer study that employed HSI for the discrimination of normal, precancerous, and cancerous cells demonstrated that it is possible to correctly classify high-grade precancerous cells, as shown in Fig. 8, but low-grade precancerous cells are more difficult to automatically distinguish from normal cervical cells [207]. The Mueller matrix provides a comprehensive characterization of the polarization properties of specimens, and contains information regarding optical properties of biological tissues that can be used for diagnosis. Using this principle, He et al. successfully analyzed the Muller matrix to distinguish cervical carcinoma tissues within unstained histological sections [167].

Fig. 8.

(a) Spectral plots of Papanicolaou-stained cells from cervical Pap smear: normal, low-grade (LG), high-grade (HG) and squamous cell carcinoma (SCC). (b-d) Classified HS images of normal (green), LG (yellow), and HG/SCC (red) cells. (e-f) RGB image with annotation of cervical SCC (e) and HS results of the automatically extracted and classified HG/SCC nuclei (f). Reproduced from [207]; Creative Commons BY 4.0; published by ACS (2008).

There is only a single research study in the literature that deals with HSI applied to ovarian cancer. After the extraction of a dataset carefully annotated by pathologist attending to cell morphology, Nakaya et al. suggested HSI as a suitable technology to differentiate between normal and cancer cells by using both supervised and unsupervised techniques [193].

For prostate cancer detection, several groups have implemented MSI for H&E stained prostate cancer pathological samples. Tahir et al. successfully performed a round-robin Tabu search algorithm along with a nearest neighbor classification method for classifying prostate cancer, benign prostatic hyperplasia, prostatic intraepithelial neoplasia, and normal stroma in MS histological images of H&E stained prostate specimens from nearly 600 cases [192,208–210]. Additionally, Khelifi et al. worked in the same task for performing a spatial-spectral feature extraction approach and SVM classification for the same four types of prostate H&E MS images [211]. Akbari et al. used spectral feature based SVM for detection of human prostate cancer in H&E slides imaged macroscopically [212]. On the other hand, for automated detection of glandular structures and nuclei in prostate cancer H&E slides using MSI, Zarei et al. proposed a method combining the PCA algorithm, to generate an artificial RGB image, and the k-means unsupervised segmentation [213].

With the goal of early detection of bladder cancers, MS examination of urine samples could lead to improved diagnosis and follow-up. Angeletti et al. proposed a genetic algorithm combining spatial features and MS spectral analysis of Papanicolau-stained cells in urine specimens with promising sensitivity and specificity values for detecting bladder cancer cells in urine specimens [214].

8.6. Hematology

For the diagnosis of a wide variety of diseases, blood sample examinations are usually performed by skilled hematologists through microscopic analysis and evaluation of blood smears. Although automatic hematological analyzers are available to perform this task, pure optical technologies and imaging processing tools are shown as cost-effective alternatives to this end. Furthermore, hematological analyzers are not able to consider information about cell morphology. To aid researchers and physicians in the analysis of blood samples, HS/MS technologies are proposed as an alternative to conventional RGB imagery, whose main limitations are the low capabilities to handle both the uneven staining of samples and the differences in the instrumentation used to digitize the samples [51].

In the field of RBC analysis, Li et al. evaluated the feasibility of exploiting HSI for RBC counting. After conducting the RBC counting using uniquely spatial or spectral features of blood cells, the authors found an improvement in the under-counting and over-counting rates when they performed the image analysis using both types of features together [215,216]. Some authors have also proposed HSI as a promising technology for white blood cells (WBCs) segmentation. After the careful annotation of nucleus, cytoplasm, erythrocytes and background within blood smears specimens by pathologists, different authors have demonstrated a successful differentiation between the aforementioned cell parts based only in their spectral profiles. While Guo et al. employed the SVM classifier [217], Guan et al. performed the same task with SID (Spectral Information Divergence), k-means and SAM (Spectral Angle Mapper) algorithms [218]. Besides the identification of blood cells, morphological characterization is further important for diagnosis, especially in WBCs, which present a complex taxonomy with more than 20 subtypes. For this reason, Li et al. proposed a method to automatically extract morphological features of leukocytes from blood smear samples by using HSI. After leukocyte segmentation, morphological parameters such as cytoplasm area, nuclear area, perimeter, nuclear ratio, form factor, and solidity were extracted from the segmented images [219]. Additionally, due to the complexity of contents of bone marrow smears, the identification of WBCs is even more challenging in such type of samples. Wu et al. presented an approach to identify WBCs within bone marrow samples, showing the capabilities of this technology to identify WBCs in both high and low magnifications (100× and 10×) [220]. Finally, another interesting approach in the context of blood cell detection was proposed by Verebes et al., who proposed an approach to analyze blood samples without prior sample preparation, i.e. with no requirement of stains. The authors suggest the capability of darkfield HS microscopy to identify different types of red blood cells (ordinary RBCs, stacked erythrocytes), WBCs, and neutrophils within unstained samples [221], which can lead in a reduction in the cost and time required for blood sample preparations.

In addition to identification and examination of RBCs and WBCs, HS/MS analysis of blood samples has been also used for the identification of diseases. In leukemia analysis, Li et al. were able to identify leukemic cells in blood smears based only on their spectral differences from RBCs [222]. Additionally, Wang et al. proposed a method to differentiate between lymphoblasts and lymphocytes, which is an important task in diagnosis of acute lymphoblastic leukemia (ALL) [223]. Due to the high similarity between lymphoblasts and lymphocytes, the examination of these samples is challenging by both visual examination and RGB analysis. The authors employed a neural network for classification, using three types of inputs: spatial features, spectral features and spatial-spectral features. The results of this study suggest that the exploitation of both the spatial and the spectral features significantly improves the quality of the classification. Fig. 9 shows some RGB images of lymphoblasts and lymphocytes, and the classification maps extracted from this study.

Fig. 9.

Differentiation between lymphoblast and lymphocytes. Top: conventional microscope images. Bottom: Identification of lymphoblast (red) and lymphocytes (green). Reproduced from [223]; Creative Commons BY 4.0; published by OSA (2017).

In another application, some authors proposed the use of MSI to identify malaria in blood smear samples by using a multi-mode LED illuminated microscope. These authors performed a PCA transformation of normal blood smears and their counterpart infected by malaria. Their findings indicate that the parasite presents differences in spectra compared to normal blood samples in the spectral range from 590 to 700 nm, making the identification of malaria possible using such information. The use of MSI can save time in malaria detection compared to traditional methods [224–227].

Finally, the use of HS/MS analysis of blood samples is not limited to cell examination. Qian et al. presented a proof-of-concept system for the visualization enhancement of vessels within histological slides, which consisted in the application of a spectral correction technique to the MS image [228]. This pre-processing algorithm showed a reduction in the variance of the spectral signatures of the specimen, and it was shown as a promising method to be applied prior to subsequent segmentation and classification of the cells within histological samples.

8.7. Skin

Mammalian skin is comprised of epithelial cells forming a stratified squamous epithelium above an inner layer that contains connective tissue, glands, and vessels. Normal histological skin samples from rats were studied by Li et al. to enhance the visualization of different microanatomical skin structures. Different combinations of wavelengths were used to generate false color images for visual inspection, and 3D surface views of skin sections were generated by combining spectral and textural information from HS data [229]. Normal skin also contains melanin to protect from UV exposure, and its quantification can be of clinical utility. Melanin identification was performed in H&E sections directly using MSI by Kalleberg et al., comparing the results obtained with the traditional gold standard Fontana Masson (FM) silver-staining, which directly targets melanin [230]. It was discovered that spectral unmixing of MSI can identify melanin in H&E sections directly and more accurately, without the need of an additional FM staining. Additionally, spectrally separated MSI of FM stains allowed even more sensitive identification of melanin, which led the authors to conclude that FM staining was not required and that MSI would yield faster and more accurate results [230]. The ability of HSI for melanin detection was further illustrated by Wilson et al. in unstained specimens of melanocytic lesions in the skin (cutaneous) and the eye (conjunctival) using PCA and false-color representations [231].

Melanoma is a malignant form of skin cancer arising in melanin-producing melanocytes. In this field, Gaudi et al. used a HS push-broom microscope to evaluate over 100 H&E sections from different patients with melanocytic lesions. A clustering method using spectral Euclidean distance classified the data into 12 clinically-relevant spectral classes that were correlated with benign and malignant melanocytic lesions [232]. Melanocytic skin cancer has been studied using HSI microscopic systems to capture H&E pathological samples for the detection of normal skin, benign nevus, and malignant melanoma samples, taking into account the minimum correlation coefficient between their spectra [233]. This study also found that the use of higher magnifications can reveal more spectral differences between the diverse samples than lower magnifications, due to the ability to observe intracellular and extracellular components. On a cellular level, Wang et al. proposed a custom spatial-spectral SVM classification method for the analysis of H&E skin samples to detect malignant melanoma cells from normal melanocytes with high specificity and sensitivity [234].

8.8. Summary table

In Table 2, we show a summary of the main applications of HSI/MSI for diagnostic research targeting clinical histological practice, which is intended to provide readers with a quick overview of the current usage of HSI/MSI in histopathological diagnosis assessment. The table is organized by different fields, following the same structure previously presented for this section. The information of the table is related to the number of patients involved in the study, the magnification used for specimen’s acquisition, the type of staining and the details on the instrumentation, and the image processing techniques used for the analysis of data. Specifically, the instrumentation details consist of the type of technology of the HS/MS camera, the spectral range, the spatial resolution, and the number of bands. As can be observed, not every field in the table was available for every article. For this reason, and following the PRISMA recommendations, we emailed every author whose information was missing. In the cases that we got a reply with the information, the table was updated.

Table 2. Summary table of applications of HSI/MSI for diagnostic research targeting clinical histological practice ^a .

^aA list of acronyms is provided at the end of the manuscriptin the Appendix (Table 3).

8.9. Concluding remarks

In this section, we summarize the current status of the usage of HSI/MSI technologies for histopathological analysis and diagnosis. According to the results depicted in the analyzed research articles, it can be concluded that HSI/MSI technologies are able to succeed in histological disease detection. The range of applications which make use of HSI/MSI diagnosis research within pathological slides is wide, and mostly focused in the detection of cancer. There are some positive outcomes on the use of HSI/MSI technologies for histological analysis. First, researchers have found differences in spectral information between diseased and normal tissue to be discriminant enough to detect some illnesses (e.g. [150,151,156] or [217], among others). These results suggest that the spectral signatures of tissues can be a complementary source of information for disease detection in histopathology, where the disease identification is commonly based on the morphological analysis of tissue components. Second, the advantages of HSI/MSI compared to conventional RGB digitized slides have been reported in various clinical research applications [161,176,177,183,200,202]. Nevertheless, in the computational pathology community, there is a need to further quantify such improvements in disease detection compared to conventional RGB digitized slides. Additionally, most investigations on the use of HSI/MSI technologies were performed in applications where conventional RGB digital pathology has been proven to be successful. In order to really prove the utility of HSI/MSI, future investigations should be highly focused in diseases in which current diagnostic procedures are not effective, thus allowing the benefit of HSI/MSI to be demonstrated. Third, some researchers suggest the utility of the analysis of unstained samples [155,167,178,224–227,231]. In most current state-of-art research using HSI/MSI, histological dyes are used for the observation of samples. Such dyes modify the spectral response of tissue to light, and thus transform the spectral information of the tissue. For this reason, the future HSI/MSI analysis should also be focused in the exploitation of the spectral information of tissues with no external dyes. Finally, the real capabilities of HSI/MSI in diagnostic detection are likely hidden by the broad options for image processing information retrieval techniques. In most of the research carried out at this moment, different processing techniques are used with successful results.

There may be some challenges to the adoption of MSI/HSI for digital pathology in a clinical setting. Firstly, the spectral acquisition equipment is unstandardized, complex, research-grade, and expensive. The imaging technology is inconsistent between research groups. It is impossible at the moment to know which imaging parameters, wavelengths, or spectral resolution are best for each application. The experimental results reported in the literature are difficult to compare since each of them are usually performed using a single HS camera with data from a single institution. Additionally, the CAD and analysis algorithms are often different. Most CAD methods also use custom software, which is not suitable for easy clinical deployment. It is difficult to determine if experiments are performed objectively or contain bias. Additionally, publication bias may be present in the literature, which yields only positive results, and the negative results remain unpublished. Moreover, it is extremely difficult to extrapolate the results obtained in one particular diagnostic application in one specific organ system to another application/organ, mainly due to the heterogeneity of the spectral properties of the different tissues. In this sense, to generalize and standardize a methodology capable of achieving accurate results for several different frameworks, more research is required, exploring several organ systems and applications, performed in large experiments, employing a wide spectral range HS system. Finally, one major challenge that cannot be ignored is that storage size of data is greater than RGB digital pathology, which is already a challenge with no straightforward solution. The use of HSI/MSI for digital and computational pathology is promising and is still in its infancy, requiring more investigation and creative solutions to the problems listed above for clinical translation.

9. Discussion

In this manuscript, the most relevant aspects of HSI/MSI for the analysis of histopathological specimens have been systematically reviewed. To this end, we employed the PRISMA guidelines for systematic reviews to provide a rationale about the motivations of performing this review and details about how the review process was performed. We have analyzed the manuscripts selected for this review in four major categories: digital staining and color correction, autofluorescence, IHC, and histology clinical diagnostic research. Additionally, papers related to the development of HSI/MSI instrumentation targeting the acquisition of histological samples were also selected. We considered such research out of the main scope of this manuscript, and we only provided a brief overview of this topic. The main goal of this manuscript is to provide readers with the current context of the applications of HSI/MSI in this field and to illustrate the main limitations and challenges for future research.

We have shown the current status of color correction and digital staining of histological samples using HSI/MSI. First, together with image analysis techniques, HSI and MSI have demonstrated to be able to correct the variations on the digitized slides among different laboratories. Such differences are mainly caused by the differences in the instrumentation and in the histopathological process, e.g., differences in the staining conditions. For these reasons, MSI and HSI are presented as a suitable technology to deal with inter-laboratories variability of digitized slides, which is one of the main challenges in computational pathology. Nevertheless, as stated before, HSI/MSI instrumentation is expensive and not standardized. Therefore, more research should be done to determine the cost-benefit trade-off on the use of HSI/MSI for improving intra-laboratory variability or for digital staining. Second, HSI/MSI has been also applied to perform digital staining of samples. In this field, one trend consists of applying image processing techniques to standard H&E samples to highlight tissue structures that are barely visualized using such staining. The other trend is to directly use image processing techniques for synthetic staining of unstained samples, avoiding the physical staining. Although the research regarding digital staining is promising, the number of works in this field is still limited and more research should be conducted.

Autofluorescence using HSI techniques was demonstrated for measuring emission signals from intrinsically fluorescent tissue components. This technique was utilized for applications such as measuring concentrations of molecules within tissue and differentiating strong emission signals of exogenous fluorophores, such as GFP, from more subtle, intrinsic autofluorophores.

In the field of IHC, MSI is a well-established research technology, where both the instrumentation and the imaging processing techniques have been commercially standardized. The main goal of MSI in this field is the identification of different biomarkers within a specimen, which has been achieved, even in cases of multiple staining in a single specimen, by applying spectral unmixing techniques. Commercial instrumentation and analysis software are available for IHC applications. However, multi-institutional, large patient studies should be performed.

Finally, the main focus of this manuscript details clinical diagnosis research applications of HSI/MSI in histopathology. Given the research works summarized, we conclude that HSI/MSI is useful for the identification of diverse diseases and tissues, where cancer detection is the most common application. The instrumentation strongly varies among different studies. It is not clear which instrumentation parameters are more appropriate for HSI/MSI histological analysis. The most important challenge is to determine which spectral range is more informative. Regarding the spectral range, most of the studies covered in this systematic review are restricted to wavelengths below 1,000 nm. The exception is the research performed by Awan et al., whose results suggest an improvement in performance of the classification of colon cancer tissues when information from NIR bands was also incorporated [197]. Thus, the exploration of the performance of the spectral range beyond 1,000 nm is a challenge in HSI/MSI. Additionally, for the full exploitation of the spectral range, it should be taken into account how different stains limit the spectral range of the sample [235]. Furthermore, there are substantial differences among data analysis methods across the different studies. Most approaches target automated classification of different types of tissues or diseases using machine learning techniques, and others deal with image visualization enhancement of different tissue constituents. In order to reach an agreement about an adequate common framework for HSI/MSI data processing for histopathological applications, there is a need for publically available datasets, where a fair comparison across different methods could be performed. In this context, to allow a fair comparison between different studies, we strongly recommend future authors in this field to report in their manuscripts the details regarding the number of patients involved in the study, the type of staining, the magnification, and details of the instrumentation, namely spectral range, spectral resolution, and number of bands. Most importantly, the experimental design of the papers should minimize bias, especially in machine learning approaches, i.e. performing data partitions with independent patient data for testing.

Although promising results have been obtained using HSI/MSI technology for histopathological diagnosis, there are still challenges to be investigated. There are several studies that point out that HSI/MSI are able to outperform standard RGB for disease detection [161,176,177,183,200,202]. However, HSI/MSI technologies are associated with unique challenges, such as large data storage, expensive processing requirements, and whole-slide-imaging acquisition. These problems are already solved in conventional RGB digitize slides, where obstacles related to WSI and image processing are well-established. For this reason, to state if HSI/MSI has a future in computational pathology, more performance comparisons should be carried out to definitively demonstrate the suggested superiority of HSI/MSI for disease detection.

Finally, one of the most promising fields for future research in HSI/MSI for histopathological applications is the exploitation of unstained samples. The use of unstained samples is mainly motivated by the fact that the stains usually employed for the manual sample examination reduce the spectral range of the samples [235]. In the case of H&E, this staining limits the spectral response of the specimens to the visible spectral range, i.e. to wavelengths between 400 and 750 nm. With unstained specimens, there is the possibility to exploit information in a wider spectral range, which may provide more information about some tissue constituents. For example, spectral information about collagen or lipids are more evident beyond 800 nm, and cell fuel sources that are upregulated in certain diseases states, such as FAD and NAD, have spectral responses from 300 to 500 nm [4,144]. Theoretically, these signals would still be present in stained specimens, but they may have been washed out by the strong visible signals from stains. In this systematic review, we found some successful diagnosis applications using unstained samples [152,153,155,167,178,221,224–227]; however, the influence of staining on the spectral response of tissue should be further quantified in the future.

In this manuscript, we have focused on the current status of HSI/MSI for histopathological applications. However, as stated in the introduction, there are other spectral imaging techniques, such as RS, FTIR spectroscopy or fluorescence spectroscopy techniques, which can be used for similar clinical purposes. The main differences between those technologies are in the instrumentation and sample preparation. On the one hand, the instrumentation used in RS, FTIR spectroscopy, and HSI/MSI is not standardized and is usually expensive, which makes their integration difficult in clinical environments. HSI/MSI requires less time for data collection, and the spatial resolution is higher. On the contrary, the molecular detection in RS and FTIR seems to be more accurate. Also, the processing framework on RS and FTIR spectroscopy seems to be more straightforward, and focused in the biomarker identification, while the processing frameworks in HSI/MSI are usually based on empirical machine learning approaches. On the other hand, fluorescence spectroscopy techniques require both specialized instrumentation and additional sample preparation. There is a tradeoff between advantages and disadvantages of different spectral imaging modalities for histological applications, and also for generic biomedical applications. However, there is no current research about comparisons between different spectral imaging technologies, and which ones are more suitable for clinical environments. This problem is still valid for applications beyond the clinical context, where only a few researchers have focused their work in comparing the performance of different spectral imaging technologies [236,237]. There is a current need for the spectral imaging community in the biomedical field to perform a methodological comparative study of different imaging modalities for diagnostic applications. After such investigations, the community will be able to make a more informed decision about which spectral technology is more appropriate for the clinical environment, taking into account both cost-efficiency and the benefits for clinical diagnosis.

In conclusion, this paper analyzed the current status of HSI/MSI for histopathological analysis of biomedical samples. On the one hand, in the field of IHC research, the use of MSI is currently established as a technology demonstrated to be useful for evaluation of biomarkers. On the other hand, the results pointed out by researchers for autoflorescense and for histopathology diagnostic research were demonstrated to be promising, but these technologies still present several challenges. In the field of color correction, HSI/MSI is presented as a suitable technology for dealing with the inter-laboratory variability of digitized slides. Finally, digital staining of samples is presented as one of the most promising future trends for HSI/MSI in histological analysis, either for the analysis samples with no requirement of physical stain, or to digitally improve the visualization of tissue structures within stained slides that are difficult to identify with conventional stains. HSI/MSI should be further investigated to be proven as an effective and accurate alternative to conventional technologies for digital and computational pathology.

Appendix 1

Table 3. Acronyms.

AF	Autofluorescence	LED	Light-emitting diode
ALL	Acute lymphoblastic leukemia	LR	Logistic regression
AOTF	Acousto-optic tunable filters	MED	Minimum euclidean distance
BG	Benign goiter	ML	Maximum likelihood
cGAN	Conditional generative adversarial network	MLP	Multilayer perceptron
CNN	Convolutional neural network	mRMR	Minimum redundancy maximum relevance
CNS	Central nervous system	MSI	Multispectral imaging
DCIS	Ductal carcinoma in situ	MT	Masson’s trichrome stain
DT	Decision trees	NADH	Nicotinamide aenine dinucleotide
EM	Electromagnetic	NCBI	National Center for Biotechnology Information
EVG	Verhoef’s Van Gieson staining	NIH	National Institutes of Health
FAD	Flavin adenine dinucleotide	NIR	Near-infrared
FCM	Fuzzy c-means	NN	Neural networks
FE	Feature extraction	PCA	Principal component analysis
FLDA	Fisher linear discriminant analysis	PICOS	Participants, interventions, comparisons, outcomes and study design
FM	Fontana Masson silver-staining	PLS	Partial least squares
FNA	Fine needle aspiration	PRISMA	Preferred reporting items for systematic reviews and meta-analyses
FTIR	Fourier-transform infrared spectroscopy	PTC	Papillary thyroid carcinoma
GA	Genetic algorithm	RBC	Red blood cell
GFP	Green fluorescent protein	RF	Random forests
GMM	Gaussian mixture model	RGB	Red, Green, and Blue
H&E	Hematoxylin and eosin	RS	Raman spectroscopy
H&N	Head and neck	SAM	Spectral angle mapper
HSI	Hyperspectral imaging	SCC	Squamous cell carcinoma
ICA	Independent component analysis	SFDI	Spatial frequency domain imaging
IF	Immunofluorescence	SID	Spectral information divergence
IHC	Inmunohistochemistry	SVM	Support vector machine
ISH	In-situ hybridization	SWIR	Short-wavelength infrared
kNN	K-nearest neighbors	VNIR	Visible and near-infrared
LBP	Local binary pattern	WBC	White blood cells
LCTF	Liquid crystal tunable filters	WSI	Whole-slide imaging
LDA	Linear discriminant analysis

Funding

Cancer Prevention and Research Institute of Texas10.13039/100004917 (RP190588); National Institutes of Health10.13039/100000002 (R01CA156775, R01CA204254, R01HL140325, R21CA231911); Agencia Canaria de Investigación, Innovación y Sociedad de la Información10.13039/501100007757 (ProID2017010164, TESIS2018010140); Ministerio de Economía y Competitividad10.13039/501100003329 (TEC2017-86722-C4-4-R).

Disclosures

The authors declare that there are no conflicts of interest related to this article.