Spectral-Spatial Classification Using Tensor Modeling for Cancer Detection with Hyperspectral Imaging

Abstract

As an emerging technology, hyperspectral imaging (HSI) combines both the chemical specificity of spectroscopy and the spatial resolution of imaging, which may provide a non-invasive tool for cancer detection and diagnosis. Early detection of malignant lesions could improve both survival and quality of life of cancer patients. In this paper, we introduce a tensor-based computation and modeling framework for the analysis of hyperspectral images to detect head and neck cancer. The proposed classification method can distinguish between malignant tissue and healthy tissue with an average sensitivity of 96.97% and an average specificity of 91.42% in tumor-bearing mice. The hyperspectral imaging and classification technology has been demonstrated in animal models and can have many potential applications in cancer research and management.

1. INTRODUCTION

Head and neck cancer refers to malignancy starts in the nasal cavity, oral cavity, pharynx, larynx, paranasal sinuses, nasal cavity, and salivary glands. Over 90% of head and neck cancers are squamous cell carcinoma of the mucosal surfaces at the head and neck region [1]. More than a million patients are diagnosed each year with squamous cell carcinoma of the head and neck (HNSCC) worldwide [2]. Survival and life quality of the patients correlate directly to the size of the primary tumor at first diagnosis, therefore, early detection of malignant lesions could improve both the incidence and the survival [3].

Traditionally, a conventional oral examination using incandescent light is the standard method for oral abnormality [4] [5] and cancer screening, but it does not identify all potentially premalignant lesions, nor accurately detect the small proportion of biologically relevant lesions that are likely to progress to cancer [6]. Visual examination followed by biopsy of suspicious tissue sites is usually performed to make definitive diagnosis [7]. However, this process is subjective and limited by the experiences and skills of the clinicians. Furthermore, it could be challenging to delineate the margin of the lesion, which makes the choice of biopsy sites difficult [8]. Optical Imaging has the potential to solve these clinical challenges.

It has been reported that optical spectroscopy could detect malignant lesions before they become macroscopically visible, by correlating early biochemical and histological changes in oral tissue with spectral features in fluorescence, reflectance, and light scattering spectra [7]. However, spectroscopy is only a point measurement method, which is generally used to obtain an entire spectrum of a single tissue site within a wavelength of interest. It needs to scan the area over time in order to acquire a spatial distribution about the chemical composition of tissues.

Recently, hyperspectral imaging has exhibited great potential in the diagnosis of cancer such as cancers in the cervix, breast, colon, gastrointestine, skin, ovarian, urothelial carcinoma, prostate, esophageal, trachea, oral tissue, tongue, and brain [9]. HSI acquires images over continuous spectral bands across a wide range of electromagnetic spectra, which combines both the chemical specificity of spectroscopy and the spatial resolution of imaging [10]. Compared to optical spectroscopy, HSI is able to capture images of a large area of tissue noninvasively, without the administration of contrast agents.

HSI has been reported to detect head and neck cancer by several researchers. In our group, we imaged pathological slides using a hyperspectral camera and reported the detection of HNC metastasis ex vivo with promising sensitivity and specificity [11]. Liu et al. [12] measured and analyzed the reflectance spectra of human tongue based on the sparse representation of spectral information of each pixels. However, these classification methods only utilized the spectral information of individual pixels without considering the spatial relationship of neighboring pixels. Roblyer et al. [13] [14] reported the use of a multispectral digital microscope (DMD) for the detection of oral neoplasia in a pilot clinical trial. The proposed DMD was a multimodal imaging method which combined the fluorescence, narrow-band (NB) reflectance, and orthogonal polarized reflectance (OPR) modes. They observed decreased blue/green autofluorescence and increased red autofluorescence in the lesions and increased visibility of vasculature with NB and OPR imaging. However, their method was limited to a few spectral bands within the ultraviolet (UV) and visible (VIS) light regions. In our research, we explored the spectral difference of normal and cancerous tissue spectra ranging from the visible to near infrared region, which has higher penetration depth and could potentially provide more valuable information for cancer diagnosis.

HSI generates large amount of high dimensional data, therefore automatic processing methods are indispensable for the interpretation of hyperspectral data. However, classification of hyperspectral images is challenging due to the high spatial resolution, high dimensionality of spectral bands, and high redundancy caused by high correlation in adjacent bands [9]. Traditional classification methods suffer from two aspects: (1) Pixel-based methods treat each pixel independently without considering the spatial relationship of neighboring pixels, and the classification methods based on high dimension of the spectral features are time-consuming. (2) Dimension reduction methods, such as principal component analysis (PCA) [15], independent component analysis (ICA), maximum noise fraction (MNF), sparse matrix transform [16] [17], etc. require spatial rearrangement by vectorizing all images into two-way data. In order to utilize both spectral and spatial information, and to perform dimension reduction without changing the three dimensional structure of hyperspectral data, we introduce a new spectral-spatial classification scheme, which involves spectral-spatial representation and dimension reduction based on tensor modeling for head and neck cancer detection. To our best knowledge, this study represents the first application of tensor-based methods in cancer detection with hyperspectral imaging.

2. METHODS

2.1 Hyperspectral Imaging System

Reflectance images were acquired by a CRI Maestro in-vivo imaging system, which mainly consists of an internal optics, a flexible fiber-optic lighting system, a 16-bit high-resolution charge-coupled device (CCD), a solid-state liquid crystal filter (LCTF), and a spectrally optimized lens. The system is a light-tight apparatus that uses a Cermax-type, 300-Watt, Xenon light source, which spans the electromagnetic spectrum from 500 nm to 950 nm. The active light sensitive area of the CCD is 1,392 pixels in the horizontal direction and 1,040 pixels in the vertical direction. For image acquisition, the wavelength setting can be defined within the range of 450-950 nm with 2 nm increments; therefore the data cube collected was a three-dimensional array of the size 1,040 × 1,392 × n. n is determined by the wavelength range and increments as chosen by the user. The field of view (FOV) is from 3.4 × 2.5 cm to 10.2 ×7.6 cm with variable zoom.

2.2 Hyperspectral Imaging Experiment in Animals

We used tumor-bearing mice for the HSI experiments. A head and neck tumor xenograft model using HNSCC cell line M4E was adopted in the experiment. The HNSCC cells (M4E) were maintained as a monolayer culture in Dulbecco’s modified Eagle’s medium (DMEM)/F12 medium (1:1) supplemented with 10% fetal bovine serum (FBS) [18]. M4E-GFP cells which are generated by transfection of pLVTHM vector into M4E cells were maintained in the same condition as M4E cells. Animal experiments were approved by the Animal Care and Use Committee of Emory University. Four female mice aged 4-6 weeks were injected with 2 × 10⁶ M4E cells with green fluorescence protein (GFP) on the lower back.

HSI scans were performed approximately three weeks post tumor cell injection. For each mouse, we acquired two HSI cubes at two different time points. During the image acquisition, each mouse was anesthetized with a continuous supply of 2% isoflurane in oxygen. First, both the interior infrared and the white excitation were opened for reflectance image acquisition with 50 ms exposure time. Reflectance images contain 251 spectral bands from 450 to 950 nm with 2 nm increments. Then, blue excitation and 50 ms exposure time were selected for the fluorescence image acquisition. Tumors show green signals in fluorescence images due to tumor cells with GFP. Since tumors grow mostly on one side of the body, a mirror was used to aid in capturing the entire tumor while imaging. In this study, we used GFP fluorescence images as the gold standard for the evaluation of cancer detection by HSI.

2.3 Data Preprocessing

Data preprocessing consists of three steps. First, in order to remove the influence of the dark current, raw image cubes and white reference image cubes were corrected by subtracting a dark image cube from each acquired image cubes. To get the white and dark reference image cubes, a standard white reference board was placed in the field of view, and the dark currents were measured by keeping the camera shutter closed. Second, in order to produce the spectra such that a higher absorbance of light gives a positive peak, the data cubes were converted to optical density by taking the negative decimal logarithm of the ratio between the images of the tissue and the images of the white reference at each pixel in the cube [19]. Moreover, after the conversion to optical density, data can be analyzed using standard spectroscopic algorithms. Finally, the GFP spectral bands, i.e. 508 nm and 510 nm in our case, were removed from the image cubes to avoid the effect of GFP signals on the classification process. Filtering methods may be used as a pre-processing step to reduce noise in images [20] [21].

2.4 Tensor Decomposition for Feature Extraction and Classification

Hyperspectral data is composed of hundreds of gray images over a wide range of electromagnetic spectra. For each pixel in the image, the intensity over each spectral band forms a spectral curve, which is the spectral feature used in pixel-based classification method as shown in Figure 1.

Figure 1.

Hyperspectral imaging and illustration of pixel-based Classification

Tensor provides a natural representation for hyperspectral data. In the remote sensing area, tensor modeling has been increasingly utilized for target detection [22], denoising [23], dimensionality reduction [24] [25], and classification [26] [27-29]. An N-way or N^th-order tensor X ∈ R^{I₁×I₂×…×I_n} is a multidimensional array represented using N indices. A first-order tensor is a vector, a second-order tensor is a matrix, and tensors of order three or higher are called higher-order tensors [30]. The order of a tensor is the number of dimensions, also known as modes. In this study, we used the Tucker tensor model [31]. Tucker decomposition is a form of high-order principle component analysis (PCA). An N-way Tucker tensor X can be decomposed into a core tensor G ∈ℝ^{R₁×R₂×…×R_N} multiplied or transformed by a set of component matrices ${\mathrm{A}}^{\left(\mathrm{n}\right)}=[{\mathrm{a}}_1^{\left(\mathrm{n}\right)},{\mathrm{a}}_2^{\left(\mathrm{n}\right)},\cdots ,{\mathrm{a}}_{{\mathrm{R}}_{\mathrm{n}}}^{\left(\mathrm{n}\right)}]\in {\mathrm{R}}^{{\mathrm{I}}_{\mathrm{n}}\times {\mathrm{R}}_{\mathrm{n}}}(\mathrm{n}=1,2,\dots ,\mathrm{N})$ [32]:

$$\begin{array}{cc}\hfill \mathrm{X}=& \sum _{{\mathrm{j}}_1=1}^{{\mathrm{R}}_1}\sum _{{\mathrm{j}}_2=1}^{{\mathrm{R}}_2}\cdots \sum _{{\mathrm{j}}_{\mathrm{N}}=1}^{{\mathrm{R}}_{\mathrm{N}}}{\mathrm{g}}_{{\mathrm{j}}_1{\mathrm{j}}_2\cdots {\mathrm{j}}_{\mathrm{n}}}\left({\mathrm{a}}_{{\mathrm{j}}_1}^{\left(1\right)}\circ {\mathrm{a}}_{{\mathrm{j}}_2}^{\left(2\right)}\circ \cdots {\mathrm{a}}_{{\mathrm{j}}_{\mathrm{N}}}^{\left(\mathrm{N}\right)}\right)+\epsilon \hfill \\ \hfill & =\mathrm{G}\times {}_1\mathrm{A}^{\left(2\right)}\times {}_2\mathrm{A}^{\left(2\right)}\cdots \times {}_{\mathrm{N}}\mathrm{A}^{\left(\mathrm{N}\right)}+\epsilon \hfill \\ \hfill & =\widehat{\mathrm{X}}+\epsilon \hfill \end{array}$$

where symbol “°” represents outer product, G×₁A⁽¹⁾×₂A⁽²⁾…×_NA^(N) denotes the multiplication in all possible modes of a tensor G and a set of matrices A⁽ⁿ⁾. $\widehat{\mathrm{X}}$ is an approximation of X and ε presents the approximation error.

To fully exploit the natural multi-way structure of hyperspectral data, we construct a spectral-spatial representation by dividing each image in the hypercube with a dimension of 1,024×1,392×249, where 1,024 and 1,392 denotes the row and column number of the hypercube, respectively and 249 denotes the number of spectral bands, into small local patches, each of which consists of I₁×I₂ pixels. We assume that within each neighborhood, each pixel has the same label (tumor or normal) since the spectral property of each pixel is similar. Therefore, we can form a 3 way tensor X∈ ℝ^{I₁×I₂×I₃}(I₃ = λ represents the number of spectral bands), which incorporates both the spatial and spectral information. Figure 2 illustrates the spectral-spatial tensor representation of the hypercube.

Figure 2.

Spectral-spatial representation of an HSI hypercube.

In this study, eight hypercubes from four mice with head and neck xenograft tumors were used for the hyperspectral image analysis. The leave-one-out cross validation method was used for the evaluation by dividing the eight image cubes with one of these cubes as testing data and the rest as training data. For each hypercube, we used the GFP composite image as the gold standard to delineate the tumor margin. The training data was constructed by concatenating K sample patches as a 4-D tensor of size I₁×I₂×I₃×K, and the testing data was formed in the same manner. We first performed the third-order orthogonal tucker tensor decomposition along the mode-4 on the training data, using higher order discriminant analysis (HODA) [32], which is a generalization of linear discriminant analysis (LDA) for multi-way data. After the Tucker decomposition, the core tensor G∈ℝ^P×Q×R, which expressed the interaction among basis components, was vectorized into a feature vector with a length of P×Q×R as the training feature. The dimension of the tensor feature can be much less than that of the original pixel-based feature. Therefore, dimension reduction can be achieved by projecting the original tensors X to the core tensors G with proper dimensions for P, Q, R. To extract features from testing data, the basis matrices A⁽ⁿ⁾ found from training data is used to calculate the core tensor, and the corresponding core tensor was then converted into a testing feature vector. If the feature dimension is still high after the feature extraction step, feature ranking or feature selection method can be applied to further reduce the feature dimension. Finally, a classifier is used to classify the tissue to be tumor or normal tissue. The flowchart for the classification framework is illustrated in Figure 3.

Figure. 3.

Flowchart of the Classification Algorithm

2.5 Evaluation Methods

Accuracy, sensitivity, specificity are commonly used performance metrics in medical image processing literatures [33] [14] [34] [35]. To evaluate the performance of classifiers, accuracy, sensitivity, specificity and F-score were investigated in the experiment. Table 1 shows the confusion matrix, which contains information about actual and predicted classification results performed by a classifier.

Table 1.

Confusion Matrix

		Predicted Results
		Negative	Positive
Gold Standard	Negative	True Negative (TN)	False Positive (FP)
	Positive	False Negative (FN)	True Positive (TP)

The definitions of accuracy, precision, sensitivity, specificity are as follows:

$$\text{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}+\mathrm{TN}};\text{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}};\text{Sensitivity}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}};\text{Specificity}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$$

F-score is the harmonic mean of precision and sensitivity:

$$\mathrm{F}=2\times \frac{\text{precision}\times \text{sensitivity}}{\text{precision}+\text{sensitivity}}$$

3. EXPERIMENTAL RESULTS AND DISCUSSIONS

In this study, we chose the patch size of 3 × 3 to retain the local spatial and spectral information in a tensor structure. Each HSI cube was divided into 346 × 464 = 160,544 patches, including both tumor and normal patches. Since a tumor region was much smaller than the normal region, we randomly chose the same number of patches from the normal tissue region as that of the tumor patches, and then concatenated all the tumor and normal patches from seven of the HSI cubes as the training tensor with a dimension of 3 × 3 × 249 × K (K is the number of all the training patches). Similarly, testing tensor was constructed by the patches from the rest mouse.

After feature extraction by Tucker tensor decomposition, the dimension of the features was still high for classification, so we ranked the extracted features using fisher scores [36]. A K-nearest neighbor (KNN) classifier was used to classify the data with different feature dimensions. As can be seen from Figure 4 (a), although the sensitivity went up slightly and the specificity went down as feature dimension went higher, the classification accuracies of the eight mouse data were not sensitive to feature dimensions, which indicate that the feature dimension can be significantly reduced without sacrificing the accuracy. Therefore, we selected the top-one dominant feature for classification.

Figure 4.

Comparison of two classification methods with different feature dimensions

Furthermore, we also implemented a pixel-based classification method using the averaged absorbance spectra of each pixel over each local patch. First, a spectral feature vector consisting of optical density values from all the image bands for each pixel of a mouse was formed, and then centered by subtracting the mean values. Afterwards, PCA was performed to reduce the feature dimension. Finally, the original hypercube was approximated by the inverse principle component transformation, with the first few bands containing the majority of information residing in the original hypercube [37]. KNN classifier was employed to classify the data with different feature dimensions. Figure 4 (b) shows that PCA feature dimension has significant effect on the classification accuracy. As the feature dimension grew higher, the classification accuracy, sensitivity, and specificity became much higher. Although the first PCA image band has the highest contrast or variance, the classification accuracy only achieved about 50%. While the top tensor feature alone achieved an accuracy of around 90%. Therefore, higher feature dimension and longer classification time is needed in order to achieve better accuracy for the pixel-based PCA method. As illustrated in Figure 4 (b), more than 100 features were required in order to obtain an accuracy of 80% with the PCA dimension reduction method. The tensor-based method kept the nature structure of hyperspectral data, and the dimension reduction based on tensor features significantly reduced the classification time.

After feature dimension reduction, the top tensor feature was selected for classification. We used Support Vector Machine (SVM) as our classifier, and chose the Gaussian radial basis function (RBF) as the kernel function for SVM. We performed three-fold cross validation (CV) and a grid search on the training data to select the optimal value for parameter C and g over the range of log₂ C ={−5, −4, …, 0, …, 4, 5} and log₂ g ={−5, −4, …, 0, …, 4, 5}. Then a new SVM model was trained with the optimal parameters on seven mice data, and the performance of that model was tested on the rest mouse. For each mouse data, tumor dataset consisted of all the patches within the tumor region, and normal dataset was randomly selected from the normal tissue region with the same size as the tumor. With only one feature dimension, the classification time was significantly reduced.

To compare the pixel-wise method with the tensor-modeling method, we ranked the significance of pixel absorption value at each wavelength from 450 nm to 950 nm (excluding the two GFP bands) according to Fisher’s score, and picked the pixel with the top absorption value for classification. Similarly, we used the SVM classifier and compared it with the proposed method. Three-fold cross validation (CV) and a grid search were performed to find the optimal parameter C and g over the range of log₂ C ={−5, −4, …, 0, …, 4, 5} and log₂ g ={−5, −4, …, 0, …, 4, 5}.

The classification accuracy, sensitivity, specificity, and F-score of the above two methods are listed in Table 1. With only one feature dimension, the proposed method achieved promising results. Our proposed classification method outperformed the pixel-based method. For the pixel-wise classification method, the result was very poor with only one feature dimension. This indicated that even the most sensitive wavelength band didn’t perform as well as the top tensor feature, and that feature extraction based on tensor modeling was effective in extracting the most important information for distinguishing tumor from normal tissue.

Table 1.

Comparison of Classification Performance

Mouse ID	Days after tumor cell injection	Methods	Accuracy	Sensitivity	Specificity	F-score
1	21	Tensor Modeling	91.52%	99.82%	83.22%	0.92
		Pixel-Wise	50.01%	100.00%	0.03%	0.67
	26	Tensor Modeling	94.82%	96.93%	92.72%	0.95
		Pixel-Wise	68.34%	63.36%	73.33%	0.67
2	21	Tensor Modeling	91.96%	90.03%	93.90%	0.92
		Pixel-Wise	50.00%	100.00%	0.00%	0.67
	26	Tensor Modeling	91.74%	94.77%	88.72%	0.92
		Pixel-Wise	60.38%	72.04%	48.72%	0.65
3	21	Tensor Modeling	82.84%	99.37%	66.30%	0.85
		Pixel-Wise	49.98%	99.96%	0.00%	0.67
	35	Tensor Modeling	92.96%	99.97%	85.95%	0.93
		Pixel-Wise	65.52%	69.87%	61.17%	0.67
4	21	Tensor Modeling	67.71%	35.58%	99.84%	0.52
		Pixel-Wise	50.00%	100.00%	0.00%	0.67
	36	Tensor Modeling	97.26%	96.22%	98.30%	0.97
		Pixel-Wise	65.79%	60.15%	71.43%	0.64

For each mouse, HSI dataset at two time points after tumor injection were used for classification. In general, the classification performance was better at the second time point than that at the first time point, probably because the spectral difference between the cancerous and healthy tissue became more distinct as tumor grew larger. The tumor sizes at the first time point were about half of the sizes at the second time point. For all four mice, the average accuracy by tensor modeling achieved 83.51% at the first time point and 94.20% at the second time point.

We also noticed that the 4^th mouse gave the worst classification accuracy compared to other mice with the tensor-modeling method. The classification performance of the 4^th mouse at the first time point was relatively poor compared to all other mice as shown in Table 1. Figure 5 shows the tumor detection result of the 4^th mouse using the tensor modeling method.

Figure 5.

Tumor detection results with the tensor modeling method. (a) HSI composite images of a tumor-bearing mouse. The tumor was mirrored in order to capture the whole tumor. (b) GFP composite images that serve as the gold standard for evaluation. (c) Detection results where green color represents the cancer pixels detected by the tensor modeling method.

To further compare the mice at two different time point, quantitative analysis was performed to compare the absorption spectra i.e. optical density of the cancerous and normal tissue. The average absorption spectra of all the pixels within the cancerous region and the normal region of the eight mouse data were computed and compared for two different time points. The absorption spectra of cancer tissue differed significantly from those of normal tissue for all the mouse data. Figure 6 shows the absorption spectra of the 4^th mouse at two different time points. The left one is at 21-day after tumor cell injection, and the right one is at the 35-day after injection. It was found that the absorption spectra of the 4^th mouse from two different time points were very different after normalization. In particular, the spectra at the range of 550 nm to 900 nm has larger difference between the tumor and normal tissue at the second time points, which facilitated the classification task and gave better performance as shown in Table 1.

Figure 6.

Absorption spectra of cancerous and normal tissue. The left figure represents the spectra of a mouse 21 days after tumor cell injection. The right figure represents the spectra of the same mouse 35 days after injection.

4. CONCLUSIONS

Hyperspectral imaging combines both the chemical specificity of spectroscopy and the spatial resolution of imaging. Compared to optical spectroscopy, hyperspectral imaging is able to capture images of a large area of tissue noninvasively, without the administration of contrast agents.

In this study, we introduced a new spatial-spectral classification framework based on tensor modeling for hyperspectral imaging in the application of the head and neck cancer detection. This method incorporates both spatial and spectral information of the hypercube and performs dimensionality reduction. With the proposed classification framework, we are able to distinguish between tumor and normal tissue in animal experiments with different tumor sizes. Preliminary results demonstrated that the hyperspectral imaging and classification methods are promising for head and neck cancer detection. With the proposed classification method, we are able to distinguish between malignant tissue and healthy tissue with an average sensitivity of 96.97% and an average specificity of 91.42% for tumor-bearing mice one month after tumor injection. To move forward to achieve the goal of early cancer detection, future study for differentiating between normal, pre-malignant, and malignant cancerous tissues is planned.