Hyperspectral Imaging for Cancer Surgical Margin Delineation: Registration of Hyperspectral and Histological Images

Abstract

The determination of tumor margins during surgical resection remains a challenging task. A complete removal of malignant tissue and conservation of healthy tissue is important for the preservation of organ function, patient satisfaction, and quality of life. Visual inspection and palpation is not sufficient for discriminating between malignant and normal tissue types. Hyperspectral imaging (HSI) technology has the potential to noninvasively delineate surgical tumor margin and can be used as an intra-operative visual aid tool. Since histological images provide the ground truth of cancer margins, it is necessary to warp the cancer regions in ex vivo histological images back to in vivo hyperspectral images in order to validate the tumor margins detected by HSI and to optimize the imaging parameters. In this paper, principal component analysis (PCA) is utilized to extract the principle component bands of the HSI images, which is then used to register HSI images with the corresponding histological image. Affine registration is chosen to model the global transformation. A B-spline free form deformation (FFD) method is used to model the local non-rigid deformation. Registration experiment was performed on animal hyperspectral and histological images. Experimental results from animals demonstrated the feasibility of the hyperspectral imaging method for cancer margin detection.

1. INTRODUCTION

Cancer is the second most common cause of death in the US, accounting for nearly 1 of every 4 deaths. In 2013, about 1,66,290 new cancer cases and 580,350 cancer deaths are expected to occur in the United States in 2013 [1]. The central objective of surgical procedures for treatment of cancer is to maximize removal of all local malignant tissue, without leaving residual malignant cells. Disease free local recurrence has been shown to increase with positive resection margins [2]. However, standard guidelines on evaluating tumor margins intraoperatively have not been established yet. If a margin was initially positive on frozen sections of resection specimen, but further resections resulted in a negative margin, that tumor would then be considered by most surgeons to have negative margins [3]. However, intraoperative frozen sections diagnosis may suffer from errors that occur during sampling and histological interpretation; and the histological processing can take time, which is not ideal during surgery [4].

Optical diagnostic technology may provide a means to improve surgical margin delineation in real time. Optical imaging techniques, such as fluorescence spectroscopy, Raman spectroscopy, elastic scattering spectroscopy, microendoscopy, and optical coherence tomography, has been used to detect head and neck malignancy [5]. It is desirable to detect tumor margins in real time without the use of additional molecular stain [6]. Hyperspectral imaging (HSI) combines both the chemical specificity of spectroscopy and the spatial resolution of imaging [7], with the capability of capturing images of a large area of tissue noninvasively and without the administration of contrast agents. With potential of extending surgeon’s vision beyond the visible light spectrum region, the ability of medical hyperspectral imaging (MHSI) as an intraoperative visual aid tool has been explored in a variety of surgeries [8], such as gall bladder surgery, cholecystectomy, nephrectomy, and intestinal surgery, etc. Panasyuk et al. [9] reported the intraoperative assessment of residual tumor using HSI in a rat breast tumor model. They successfully identified and differentiated tumors, blood vessels, muscle, and connective tissue by HSI. A sensitivity of 89% and a specificity of 94% for the detection of residual tumors, comparable to that of histopathological examination of the tumor bed, were reported. With the aid of HSI, more extensive resection and more effective biopsy locations may be identified.

Many quantitative method has been applied to medical image analysis [10] [11] [12] [13] [14] [15]. Registration of medical images from the same imaging modalities, such as pre-operative magnetic resonance imaging (MRI) and intro-operative MRI, has been investigated intensively for cancer detection and treatment [16] [17] [18] [19] [20] [21]. Leverage of complementary information from different imaging modality requires multimodal registration [22] [23] [24] [25]. The ground truth of tumor margins is only available in histological images, therefore registering histological images with HSI images is necessary to validate the tumor margin and to incorporate complementary information [26] [27].

However, registration between HSI and histological images is not a trivial task. First, tumor margins are defined differently on HSI and histological images. In HSI, it is assumed that the biochemical and morphological changes associated with lesions alter the absorption, scattering, and fluorescence properties, therefore the tissue optical characteristics can in turn provide valuable information for cancer screening and detection. So the differentiation between cancerous and normal tissue is usually achieved by investigating the difference of the reflectance or absorbance spectra between the two types of tissue [28]. While in H&E stained tumor images, tissue components are identified according to different staining colors, shapes and morphological information. Second, during the histological preparation phase, tissue undergoes deformations, including shrinkage, tearing, and distortion, which make the registration between in vivo and ex vivo images a challenge task.

The objective of the registration is to relate each pixel on the HSI macro images to the histological micro images, allowing a delineation of the tumor in the macro images. To overcome above-mentioned difficulties, we first projected the high dimensional HSI raw data into a subspace by PCA, and utilized the PCA component image which reflected the most variations as the macro image. Then, we performed a control point based affine registration to get an initial correlation between the two images, and further used a B-spline free-form deformation to refine the deformation at the local regions. The proposed registration method was tested on tumor-bearing mice, and the feasibility of the hyperspectral imaging technology for cancer margin detection were demonstrated in animal experiments.

2. METHODS

2.1 Hyperspectral Imaging System

Reflectance images were acquired by a CRI Maestro in-vivo imaging system, which mainly consists of an internal optics, a flexible fiber-optic lighting system, a 16-bit high-resolution charge-coupled device (CCD), a solid-state liquid crystal filter (LCTF), and a spectrally optimized lens. The system is a light-tight apparatus that uses a Cermax-type, 300-Watt, Xenon light source, which spans the electromagnetic spectrum from 500 nm to 950 nm. The active light sensitive area of the CCD is 1,392 pixels in the horizontal direction and 1,040 pixels in the vertical direction. For image acquisition, the wavelength setting can be defined within the range of 450–950 nm with 2 nm increments; therefore the data cube collected was a three-dimensional array of the size 1,040 × 1,392 × n. n is determined by the wavelength range and increments as chosen by the user. The field of view (FOV) is from 3.4 × 2.5 cm to 10.2 ×7.6 cm with variable zoom.

2.2 Hyperspectral Imaging Experiment in Animals

We used GFP cells as well as non-GFP cells for the hyperspectral imaging experiments. A head and neck tumor xenograft model using head and neck cancer cell line M4E was adopted in the experiment. The cells were maintained as a monolayer culture in Dulbecco’s modified Eagle’s medium (DMEM)/F12 medium (1:1) supplemented with 10% fetal bovine serum (FBS) [29]. M4E-GFP cells which are generated by transfection of pLVTHM vector into M4E cells were maintained in the same condition as M4E cells. Animal experiment was approved by the Animal Care and Use Committee of Emory University. Five female mice aged 4–6 weeks were injected with 2 × 10⁶ M4E cells with GFP on the lower back.

Surgery was performed approximately three weeks after tumor cell injection. Before surgery, mice were anesthetized with a continuous supply of 2% isoflurane in oxygen. After the anesthesia administration, the skin covering the tumor was moved to expose the tumor. HSI images with the interior infrared and the white excitation and using an auto setting for exposure time, were captured over the exposed tumor. Acquired reflectance images contain 251 spectral bands from 450 to 950 nm with 2 nm increments. Subsequently, a blue excitation and auto-exposure time were selected for the fluorescence image acquisition. Tumors show green light in the fluorescence images due to GFP. In this study, we used GFP fluorescence images as the gold standard for tumor margin demarcation. After imaging, tumors were removed horizontally from the bottom using a blade, and were inked with four different colors which represent the head, tail, left and right orientations of the tumors in the mice. Histological samples were kept in formalin and sent for histological evaluation after 24 hours. Histological slides with H&E staining were digitized for further analysis.

2.3 Data Preprocessing

HSI image preprocessing consists of four steps. First, in order to remove the influence of the dark current, raw image cubes and white reference image cubes were corrected by subtracting a dark image cube from each acquired image cubes. To get the white and dark reference image cubes, a standard white reference board was placed in the field of view, and the dark currents were measured by keeping the camera shutter closed. Second, each HSI image was divided by its autoexposure time to get the pixel intensity per unit time. Third, in order to produce spectra such that high absorbance of light gives a positive peak (high absorbance correspond to low reflectance, so in reflectance curve, high absorption gives a negative peak), the data cubes were converted to optical density by taking the negative decimal logarithm of the ratio between the images of the tissue and the images of the white reference at each pixel in the cube [30]. Moreover, after the conversion to optical density, data can be analyzed using standard spectroscopic algorithms. Fourth, the GFP spectral bands, i.e. 508 nm and 510 nm in our case, were removed from the image cubes to avoid the effect of GFP signals on the HSI image processing. Finally, tumors in GFP images were manually segmented as the gold standard; therefore, an accurate in vivo tumor margin was delineated by the GFP tumor mask. Tumors in histological images were also manually segmented, and the orientations of the tumor were recognized by different color inks.

2.4 Principle Component Images for Registration

Principal component analysis (PCA) is a multivariate technique for spectral data analysis, which is based on the fact that neighboring bands of hyperspectral images are highly correlated and often convey almost the same information about the object. The analysis is used to transform the original data in order to remove the correlation among the bands. In the process, the optimal linear combination of the original bands accounting for the variation of pixel values in the image is identified. After the pre-processing procedures, PCA is utilized to extract the principle component images with the highest contrast or variance for registration with histological images.

The registration method is shown in Figure 1. First, the grayscale histological image is registered to the first principle component image based on an affine transformation; and the transformation matrix is saved. Second, the affine transformation matrix is applied to the binary mask of the above two images; and a registered histological tumor mask is obtained. Third, B-spline free-form deformation is utilized to register the two binary masks and a deformed transformation grid is generated. Finally, the deformed grid is applied to transform the grayscale histological image from step one in order to model the local deformation.

Figure 1.

The registration method for histological and hyperspectral images

For the global registration, control-point based affine registration allows manually picking points in a pair of images that identify the same feature or landmark in both images. A spatial transformation is then inferred from the positions of these control points. Affine transformation is parameterized by 12 degrees of freedom, describing the translation, rotation, scaling, and shearing. Therefore, affine registration is used to model the global rigid deformations as an initial point for the local non-rigid registration.

The process of freezing, fixation, cryotoming, and staining of the tissue to create histological images introduces non-rigid deformations and significant contrast changes, which makes it difficult to describe the local deformation via parameterized transform. Since affine registration cannot address the local deformations caused by the tissue shrinkage after the fixation of the tumor specimen, a non-rigid transformation model is required to registering HSI with histological images. We chose B-spline free-form deformation (FFD) to model the local deformation of tissue. The basic idea of FFD is to deform an image by manipulating a regular grid of control points that are distributed across the image. The resulting deformation produces a smooth and C2 continuous transformation. The detailed description of B-spline FFD transformation can be found in [31, 32]. B-spline deformation can provide a local control. The resolution of the mesh of control points defines the degree of nonrigid deformation and computational complexity. Small spacing of control points allows modeling of local nonrigid deformation, while large spacing models global deformations. To relate a deformed histological image to the HSI image, we used the sum of squared differences between two images.

2.5 Evaluation

Target registration errors (TRE) [33] and Dice similarity coefficient (DSC) [34] [35] [36]are calculated to evaluate the image registration accuracy. The target points are selected from both the reference and transformed images. These points are deliberately excluded from the calculation of the mapping parameters. The mean and standard deviation of the errors are computed.

3. EXPERIMENTAL RESULTS

Principle component images were extracted from hyperspectral images and serve as the fixed image, while histological images were converted to grayscale images and are the moving images. Since the histological images were inked with different colors, the orientation could be determined using these colors. Meanwhile, the texture and shape of necrotic tissue and tumors from both hyperspectral and histological images could be clearly differentiated. The ink, texture, and shape information of tumors provided important guidance for the selection of the control points during the affine registration process.

Figure 2 shows an example of the registration result of a tumor-bearing mouse. Tears and deformations are very common in the H&E stained micro images as shown in Figure 2(a). The in vivo HSI composite tumor images are shown in Figure 2(b). HSI images appeared very different with glares and blood, which made it difficult to correlate them with histological images based on intensities. After the global and B-spline FFD registration, the HSI and histological images were matched well, indicating the ability of the registration approach for this hyperspectral imaging application.

Figure 2.

Registration of hyperspectral and histological images

Table 1 shows the target registration errors and Dice similarity coefficients of five mice. For each registered histological image and PCA image, five pairs of data points were selected and the mean distance and standard deviation were calculated. The spatial resolution of the HSI images measured in the experiment was 26.5 μm/pixel. As shown in the table, the proposed method produced low target registration errors and achieved over an overlap of more than 98%.

Table 1.

Target Registration Errors and DICE

Mouse Label	TRE (mean ± STD, mm)	DSC
1	0.13± 0.19	98.73%
2	0.21 ± 0.14	98.32%
3	0.11± 0.12	98.64%
4	0.17 ± 0.16	98.76%
5	0.21 ± 0.09	98.80%

4. DISCUSSIONS

We used PCA to extract the image band with the highest variance, and registered the resulting PCA image with the corresponding histological image. The feasibility of HSI technology for cancer detection and the registration approach was tested in tumor-bearing mice. To the best of our knowledge, this study represents the first report on the registration of in vivo hyperspectral images and ex vivo histologic images.

The proposed method has two advantages. First, by extracting principal component images from hyperspectral images, we are able to match hyperspectral images with pathological images based on the shape information of head and neck tumors. Our goal is to first identify the image band which provides the best contrast between the cancerous and normal tissue, and better delineate the tumor margin in HSI images. Texture information is not utilized, as they are very different on pathology images and HSI PCA images. HSI acquires three-dimensional dataset with two spatial dimensions and one spectral dimension across a wide range of electromagnetic spectra. The spatially resolved spectra obtained by HSI reflects diagnostic information about the tissue physiology, morphology, and composition. However, it is not clear how to match such high-dimensional information with the morphological and composition information provided by pathological slides. Further investigation between their relationships are needed. Second, the proposed registration strategy captured non-rigid deformation between digital images of histological slides and HSI PCA image, and allowed us to align histological images with HSI images of head and neck cancer. Histological slides give a well-defined tumor margins, however, these images suffer from tumor deformations during the histological preparation. FFD based on B-spline was used to model the local tissue deformation, and produced a smooth transformation map that facilitate the registration.

The limitation of our method lies on the fact that HSI provides a two-dimensional (2D) mapping of a three-dimensional (3D) tumor, while histological slides are only one 2D slice from a 3D tumor. Their corresponding relationship is difficult to determine. In our experiment, we cut the tumor slices horizontally from the bottom of the 3D tumors in vivo, and the 2D mapping of the tumors in HSI images were expected to have an appropriate match with the tumor histological slides.

5. CONCLUSIONS

In this study, hyperspectral imaging experiments were performed for tumor margin detection in animals. Image registration was applied to validate in vivo hyperspectral images using corresponding histological images. The results demonstrated the feasibility of using HSI for intraoperative tumor margin delineation in animals.