A color-based tumor segmentation method for clinical ex vivo breast tissue assessment utilizing a multi-contrast brightfield imaging strategy

Abstract

We demonstrate an automated two-step tumor segmentation method leveraging color information from brightfield images of fresh core needle biopsies of breast tissue. Three different color spaces (HSV, CIELAB, YCbCr) were explored for the segmentation task. By leveraging white-light and green-light images, we identified two different types of color transformations that could separate adipose from benign and tumor or cancerous tissue. We leveraged these two distinct color transformation methods in a two-step process where adipose tissue segmentation was followed by benign tissue segmentation thereby isolating the malignant region of the biopsy. Our tumor segmentation algorithm and imaging probe could highlight suspicious regions on unprocessed biopsy tissue to guide selection of areas most similar to malignant tissues for tissue pathology whether it be formalin fixed or frozen sections, expedite tissue selection for molecular testing, detect positive tumor margins, or serve an alternative to tissue pathology, in countries where these services are lacking.

Graphical Abstract

We demonstrate an automated two-step tumor segmentation method using color space features from brightfield images of breast tissue biopsies, highlighting suspicious regions for pathology and molecular testing. This technique aids in detecting tumor margins and serves as an alternative to traditional pathology in settings with limited services.

1. INTRODUCTION

Cancer diagnosis relies on access to centralized laboratories and board-certified surgical pathologists who analyze histopathology slides created from patients’ tissue specimens^{1, 2}. Generating histology slides of breast specimens requires a complex, multi-step tissue processing procedure, including tissue fixation, paraffin embedding and sectioning, which needs, on average, 24 to 48 hours of processing time³. In addition to tissue processing, pathologists often examine multiple sections from the same tissue block to reduce sampling errors and increase diagnostic accuracy. For biopsies, the total turn-around time varies from 1 to 2 days, depending on the workload of the pathologist and the need for additional work-up, and for large cases, 1–2 weeks. While pathology is essential to confirmatory tissue diagnosis, there are several challenges which prevent pathology from being widely accessed.

First and foremost, there is a shortage of the pathologist workforce. During the period of 2007 to 2017, the U.S. physician workforce grew from 765,688 to 892,856 (+16.61%); however, the pathologist workforce decreased from 15,568 to 12,839 (−17.53%)⁴. Though the U.S. has the highest density of pathologists compared to other high-income countries, the shortage of this essential workforce can cause increased workloads, significant delays, and limited time for resident training. This problem is exacerbated in low- and middle-income countries. Two-thirds of the global pathology workforce is in just 10 countries. Other countries have only a few laboratory medicine professionals to serve millions of patients’ healthcare needs and the least resourced countries have none.

A second challenge is gross examination of the tumor specimen to assess whether there is tumor in the tissue. Tissue preparation is the first step in the pathology workflow which involves dissection of tissue, identification of gross abnormalities and transportation of the specimen to the histology lab in a labeled cassette. If the tumor is not sampled, then that would lead to a false negative diagnosis. Partial mastectomy (lumpectomy) is a common procedure for the removal of breast tumors, as it is a less surgically involved procedure than mastectomy and can be performed in an outpatient setting. However, the status of tumor margins is critical to the management of the cancer from lumpectomies since tissue is left behind in the patient. Histologic evaluation of the margins after the tissue has been removed means that patients often undergo a subsequent surgery to re-excise tissue where tumor is close to or involves the margin. While it has been reported that intraoperative frozen sections reduce positive margins for breast cancer^{5, 6}, very few health facilities have the personnel and resources to perform them. Finding a simple way to evaluate margins at the time of lumpectomy is therefore of high importance.

There are additional opportunities to streamline breast cancer evaluation. Molecular tests, such as Oncotype Dx, Mammaprint, and Breast Cancer Index, are ancillary tests that provide predictions on disease prognosis and help identify appropriate treatment options for different breast cancer molecular subtypes [29]. Currently, all of these tests are developed for formalin-fixed paraffin-embedded (FFPE) tissue because all specimens must first undergo histology evaluation to confirm that adequate amount of tumor is sampled before being used for downstream molecular testing. However, a large number of molecular assays used in breast cancer clinical trials are developed using fresh frozen tissue for either mRNA analysis, or because DNA quality may not always be sufficient from FFPE specimens [31]. For example, in the case of the Mammaprint, it was actually originally designed in 2007 on fresh tissues but later needed to be modified for FFPE in 2015 to promote broader uses [30]. Guiding selection of tumor specific regions in fresh tissue samples could both increase the specificity of the sample being evaluated as well as potentially obviate the need for FFPEs which involves the time intensive process as described earlier.

Towards the goal of improving efficiency of breast tissue evaluation, we have developed an optical imaging device which identifies tumors in fresh tissue specimens at the time of excision and color segmentation algorithm to where the tumor is in the excised tissue. Unprocessed human breast tissue, containing fat, fibroglandular tissue, and tumor cells, has unique color properties that can be leveraged for brightfield imaging. High levels of Vitamin A in human breast adipocytes lead to the yellow-orange color of human adipose tissue⁷. Fibroglandular and cancerous tissue contain more collagen and epithelial cells than adipose tissues and possess a white-gray appearance. Pathologists leverage these properties to perform gross examination of tissue at the macroscopic level as part of post-operative assessment before sending specimens for downstream histology analysis^{8, 9}, creating the rationale to perform color-based segmentation on brightfield optical images from fresh breast tissue specimens.

For this study, we collected optical images of breast core needle biopsies using a portable imaging device previously developed by our group^{10, 11}, and color transformation (HSV, YCbCr, and CIELAB) to segment cancer, from adipose and benign tissue prior to pathologic evaluation. HSV, YCbCr, and CIELAB color transformations are advantageous over RGB for segmentation tasks because they can separate luminance from chrominance information (chrominance refers to color and luminance refers to lightness of image). By leveraging the color information from both white-light and green-light images, we were able to identify two different types of color transformations that could distinguish between benign and cancerous tissues. We leveraged these two distinct color transformation methods in a two-step process in which adipose tissue segmentation was followed by benign tissue segmentation thereby isolating the malignant region of the biopsy. While this study is specific to breast tissue, it should be noted that our segmentation algorithm could be generalized to process RGB images for other clinical applications. For example, in the area of gastrointestinal (GI) disease detection, white-light endoscopy is one of the gold standard procedures to detect abnormalities. Chronic skin wounds also present distinct color features^{12, 13}. Since skin wounds often present with a mixture of different tissue types¹², our method could be incorporated into the wound monitoring system by identifying different skin types.

2. MATERIALS AND METHODS

2.1. Clinical ex vivo biopsy multi-contrast imaging

All image collections were compliant with Duke IRB protocol Pro00008003. At the Duke Cancer Center Mammography Clinic, 32 patients undergoing ultrasound-guided core needle biopsies (USGCNB) consented to enroll in this study from March 2019 to May 2022. After excision, each biopsy was rinsed with phosphate-buffered saline (PBS) to remove excess blood, and the specimen was placed on a black cloth to provide dark background. For all biopsy cases, the first biopsy from each patient was imaged with our brightfield portable microscope using white- and green-light illuminations^{11, 14}. The biopsy was positioned 20mm from the detector to facilitate wide-field imaging, achieving a resolution of 20 μm and securing a diagonal field of view of 20 mm. This arrangement guarantees comprehensive capture of each biopsy within a single snapshot. The biopsy was illuminated with white- and green light from LEDs, and images were collected using auto-exposure of our system. This process was conducted without any supplementary lighting in the room to ensure optimal imaging conditions. After imaging, biopsies were colored with three different inks to indicate the biopsy orientation for future co-registration and sent for standard histology processing. The pathologist performed a diagnosis on the specimen and analyzed every millimeter along the biopsy for co-registration with our images, including the percentage of tissue, adipose, and fibroglandular areas.

2.2. Data Preparation

Of all 32 biopsy cases, 6 malignant and 2 benign cases were excluded from analysis due to fragmented histology (no ground truth), low image quality (blurry images with no identifiable morphology features), or lack of green light images, resulting in 15 malignant cases and 9 benign cases for segmentation analysis. For the colormap clustering analysis, we sectioned white- and green-light biopsy images into 1-mm biopsy image sections based on site-level pathology. Non-bloody biopsy sections with more than 80% of one type of tissue were selected, resulting in 86 biopsy sections from 14 biopsies (10 malignant and 4 benign). These 14 biopsies were used as the training set to establish the segmentation workflow. The remaining 10 biopsies (5 malignant and 5 benign) were used as the testing set to evaluate the segmentation workflow.

2.3. Specular Reflection Removal

All images were processed using MATLAB. The background of each biopsy image was removed, and all images were manually cropped based on the size of the biopsy specimen. A well-established specular removal method was applied for all cropped images in Python¹⁵. The area of specular reflection was defined as pixels above a certain percentage (0.8–0.95 for white-light and 0.6–0.8 for green-light) of the maximum grayscale and below a certain percentage (0.4–0.6 for white-light and 0.6–0.7 for green-light) of the maximum saturation intensity of images. The cutoff thresholds for each image were determined case by case to visually cover the most specular reflection regions. After specular removal, filtered pixels were filled with the Telea inpainting method using surrounding pixels^{15, 16}.

2.4. Color Space Transformation

All images were processed using MATLAB. After specular removal, all biopsy images were transformed from RGB color space to HSV, CIELAB, and YCbCr color spaces using MATLAB built-in functions (rgb2hsv, rgb2lab, rgb2ycbcr). Individual color channels, Hue (H), Saturation (S), Value (V) from HSV, Lightness (L), red/green(a), and yellow/blue(b) from CIELAB, and Luminance (Y), blue chroma (Cb) and Red chroma (Cr) from YCbCr, were extracted¹⁷.

However, it is unsuitable for image processing due to high correlation between the three-color channels and the mixing of chrominance and luminance information^{18, 19}. HSV performs a non-linear transformation of RGB color space into hue (shades of color), saturation (amount of color) and values, with the value channel representing the lightness of the images²⁰. YCbCr is primarily used in image compression and television transmission, and its luminance component (Y) can also be separated from two chrominance channels (Cb and Cr)²¹. Cb (blue chroma) and Cr (red chroma) each presents chrominance different between blue/red component and their reference values. Lastly, CIELAB is most analogous to human vision, where L describes the luminance component, and a (red/green) and b (yellow/blue) channels describe chrominance. Among these channels, V, Y, and Y were not chosen for the colormap generation because they were luminance-related channels. S, V, a, b, Cb, and Cr were explored for colormap generation and segmentation.

2.5. Colormap Clustering Analysis

For adipose segmentation, colormaps of biopsy images were generated first using H, S, a, and b color channels through multiplication and addition. All ten combinations of H, S, a, and b were explored: a*H, a*S, b*H, b*S, a*H+b*H, a*H+b*S, a*S+b*H, a*S+b*S, a*H+a*S, and b*H+b*S using non-bloody biopsies image sections with larger than 80% one type of tissue. Scatterplots were generated using white-light colormap values as the x-axis and green-light colormap values as the y-axis to compare the adipose and non-adipose clusters (fibroglandular and tumor). The Silhouette score of each cluster was calculated using MATLAB’s built-in function to analyze the compactness of individual clusters. The average Silhouette scores were reported for all ten combinations. Silhouette scores use intra-cluster and inter-cluster distances to measure cluster separation²².

The exact process was repeated for tumor segmentation with H, S, Cb, and Cr color channels. All ten combinations of H, S, a, and b were explored: Cb*H, Cb*S, Cr*S, Cr*H, Cb*H+Cr*H, Cb*H+Cr*S, Cb*S+Cr*H, Cb*S+Cr*S, Cb*H+Cb*S, and Cr*H+Cr*S using the same set of non-bloody biopsies image sections as tumor clustering analysis. Scatterplots were generated using white-light colormap values as the x-axis and green-light colormap values as the y-axis to compare the tumor and fibroglandular clusters. The average Silhouette scores were reported for all ten combinations.

2.6. K-means Clustering

For adipose segmentation, a*H+b*S and a*S+b*S colormaps were generated using all white- and green-light images of entire biopsies from the training set. The colormap ratios were calculated by dividing green-light colormaps from white-light colormaps. Resulting colormap ratios were inputted into MATLAB’s built-in K-means-clustering-based image segmentation function, imsegkmeans, with three clusters (k=3) and ten iterations. The segments that had the highest overlapping region with the ground truth were selected as the adipose segment. We computed and compared eight types of statistics of the a*H+b*S and a*S+b*S colormap values from adipose and other tissue segments: minimum, maximum, mean, median, and means of the bottom 1%, 5%, 10%, and 25% colormap values. For the statistical analysis, Student T-tests were used to calculate the statistical significances of all four comparisons. The cutoff threshold was determined by the midpoint between the adipose segment and the other segment using the mean value of the bottom 1%. Using cutoff thresholds, we performed an automated selection of adipose segments for all training biopsies using both colormap transformations. The percentage of intersection over union (% IoU) was calculated by dividing the union of ground truth areas and segmenting from the overlapped areas. The % IoU of a*H+b*S and a*S+b*S colormaps were compared to the overall percentage of adipose (% adipose) for each biopsy. For the statistical analysis, the Pearson coefficient and p-values were calculated for the correlation analysis.

After adipose segmentation, the same K-means clustering segmentation was repeated for tumor segmentation with Cr*S and Cb*S+Cr*S colormaps with the same hyperparameters. To determine the cutoff threshold, eight types of statistics were computed: minimum, maximum, mean, median, and means of bottom 1%, 5%, 10%, and 25% colormap values. For the statistical analysis, Student T-tests was used for all comparisons. The cutoff threshold was determined by the midpoint between the tumor segment and the fibroglandular segment. The % IoU of automated segmentation was compared to the percentage of tumors (% tumor). For the statistical analysis, the Pearson coefficient and p-values were also calculated for the tumor correlation analysis. All Box-whisker plots were generated R-studio, and all statistical analyses were performed with JMP Pro 16.

3. RESULTS AND DISCUSSION

3.1. Patient Enrollment Distribution in Clinical Biopsy Study

A pilot clinical study was performed at the Duke Cancer Center Mammography Clinic under IRB protocol Pro00008003 to collect anatomical images of breast ultrasound-guided core needle biopsies. A handheld probe developed by our group (and now commercialized) was used to collect images using white and green light illumination from the entire biopsy in a single capture¹⁰. The images were collected as 8-bit RGB color images with a resolution of 20 μm and a diagonal field of view of 20 mm. The biopsy was color-inked to record the orientation allowing downstream comparison with the histology slides. All imaged biopsies were processed at the Duke Pathology histology lab into standard hematoxylin and eosin (H&E) slides for routine clinical pathology analysis. Site-level pathology analysis was also performed by board certified pathologists (AH and RF), providing the percentage of adipose, tumor, and fibroglandular tissue for every 1mm segment of each biopsy using the H&E slides acquired from the top surface. The tumor region was outlined on the H&E histology slides, which served as the control for image segmentation. Table 1 shows the demographic and pathology information of the patients enrolled in the study. Of the 32 enrolled patients, there were 22 malignant and 10 benign cases. The receptor subtypes of the 22 malignant cases were 16 ER/PR+, 2 HER2+ cases, and 3 Triple Negative Breast Cancer (TNBC) cases. After excluding biopsy cases with incomplete/out-of-focus images, 15 malignant and 9 benign biopsies were processed using our two-segmentation method.

Table 1:

Demographic and Pathology Breakdown of Clinical Biopsy Study

Characteristic	Biopsies
Number of Patients
Ultrasound (US) biopsy	32
Patient Demographics
Average Age (range)	62.3 (37–89)
Average BMI (range)	27.3 (18–38.5)
Pathology Breakdown
Malignant	22 (69%)
Benign	10 (31%)
Receptor Status (malignant only)
ER+, −	18, 3 (82%, 14%),
PR+, −	16, 5 (73%, 23%)
Her2+, −	2, 19 (9%, 86%)
TNBC	3 (14%)
Menopausal Status
Pre-menopause	5 (16%)
Post-menopause	24 (75%)
Peri-menopause	2 (6%)
Breast Density
Fatty	2 (6%)
Scattered Fibroglandular	13 (41%)
Heterogeneously Dense	14 (44%)
Extremely Dense	3 (9%)

3.2. Colormaps Generated from CIELAB and HSV Color Spaces Enable Adipose Segmentation

In this study, we developed a two-step segmentation method where we first segmented adipose tissue and then benign tissue to isolate the malignant regions. Previous research has shown that color space transformation is very important for color-image segmentation^{18, 19}. Therefore, we exploited the color differences between the three tissue types transforming the anatomical images from RGB to other color spaces. Prior to image segmentation, the biopsy images were preprocessed. The background of each biopsy image was removed, and all images were cropped based on the size of the biopsy specimen.

After specular removal and pixel inpainting, all preprocessed images were converted from RGB into CIELAB (L*a*b) and HSV (hue, saturation, and value) color spaces. Hue, saturation, a, and b channels were extracted from white and green light images of core needle biopsies. Color channel values from HSV and luminance from CIELAB was excluded because they are used to describe the lightness or the brightness of the images, which are not relevant to color information. Colormaps were generated by combining individual color channels (a, b, hue, saturation) using linear mathematical operations, such as multiplication and addition. All possible combinations of colormaps were compared to histology ground truth and site-level pathology.

Figure 1A shows white light and green light images and the corresponding histopathology of a representative cancer biopsy. The white/green group is defined as the colormap from white light images divided by the colormap of green light images of the same biopsy. Figure 1B shows six representative colormap combinations of hue, saturation, a, and b color channels, including a*H (a*Hue), b*H (b*Hue), a*S (a*Saturation), b*S (b*Saturation), a*H+b*H (a*Hue+b*Hue), and a*S+b*S (a*Saturation+b*Saturation). These combinations are shown for white light images, green light images, and the ratio of white and green light images. We further analyzed the colormaps through clustering analysis of all biopsy images, excluding samples with blood. Based on site-level pathology, each biopsy image was sliced into sections representing 1mm segments each of the biopsy. We used site-level histopathologic assessment to divided biopsy sections (across multiple samples) to correspond to predominantly one tissue type (>80% of one type of tissue) from 10 malignant and 4 benign biopsies. Next, a scatter plot was created where the x-axis is the colormap value from white light images, and the y-axis is the colormap value of green light images of all pixels from adipose and non-adipose tissue segments from across 108 image sections of 14 biopsy samples. A standard metric called the Silhouette score was used to analyze the compactness of the individual clusters. Average Silhouette scores were obtained by calculating the mean of both clusters and are reported for all possible combinations of a, b, hue, and saturation channels in the table in Figure 1C. a*S+b*S and a*H+b*S have the top 2 average Silhouette scores of 0.37 and 0.2631, respectively. Figure 1D shows the scatterplots of a*H+b*S (a*Hue+b*Saturation) and a*S+b*S (a*Saturation+b*Saturation), which correspond to the best separation between the adipose cluster (blue), and the other cluster corresponding to all tissues other than adipose tissue (orange).

Figure 1: Colormap generation using CIELAB and HSV color spaces for adipose segmentation.

A) Representative white-light, green-light, H&E images, and site-level pathology of one core needle biopsy sample. The adipose tissue appears orange, and the tumor is circled in the H&E histology image and appears whitish grey. For the site-level pathology, of every 1mm segment, the % Tumor, % Fat, and % Fibroglandular tissue were annotated by a pathologist. B) Different combinations of color channels are shown. From top to bottom are a*H, b*H, a*S, b*S, a*H+b*H, and a*S+b*S, where H stands for hue, and S stands for saturation. The left panel shows the color combination results of white light images, the middle shows the color combination results of green light images, and the right panel shows the ratio of white light over green light color combination results. C) Table of average silhouette scores obtained from adipose and other clusters for all combinations of colormaps in CIELAB and HSV color spaces. The top two combinations are a*H+b*S and a*S+b*S, with an average silhouette score of 0.2631 and 0.37. D) Scatterplots of pixels from different tissue types obtained from the site level bins based on detailed histopathology for color maps with the highest two Silhouette scores. For all panels, the x-axis represents the color transformation of white-light images, and the y-axis represents the color transformation of green-light images. Only representative biopsy cases with more than 80% of either adipose, tumor, or fibroglandular tissue (n = 14) are selected in this analysis. Regions with blood are excluded. The top panel shows the a*H+b*S, and the bottom panel shows a*S+b*S. For all images, a and b are color channels of CIELAB color space. S refers to saturation, and H refers to hue from HSV color space

Representative colormap images for both sets of variables, a*H+b*S and a*S+b*S generated from white/green light images are shown in Figure 2A. For adipose segmentation, K-means clustering (k=3) was applied to each of the two colormaps to generate three different segments: background, adipose, and cancer/fibroglandular and this was performed for all tissue segments within each biopsy for all 14 biopsies. In other words, clustering was not limited to just the tissue segments with > 80% of a given tissue type. The clustering results are shown in the middle panel in Figure 2A. The representative segmentation result is shown in the bottom panel of Figure 2A and the threshold for segmentation is described in the next paragraph. The magenta, green, and black regions correspond to the algorithm, ground truth, and the overlapping region. The colormap ratios were calculated by dividing green-light colormaps from white-light colormaps. The colormap corresponding to the variable a*S+b*S had the higher Silhouette score and was selected as the variable to use.

Figure 2: Combining color features of white and green light images enable adipose segmentation.

A) Comparison of K-Means (k=3) clustering result of one representative biopsy case using colormaps generated from white/green light images. The top 2 colormap combinations, a*H+b*S, and a*S+b*S, based on clustering analysis, are shown here. The top row is the colormap images of two different colormaps generated using white light images divided by green light images. The middle row contains the K-Means clustering results. The bottom row shows the overlap between the clustering result (magenta) and ground truth (green). The black color indicates the overlap region. The a*S+b*S colormap achieves the best segmentation performance, shown by the largest black area. B) Box-whisker plots to illustrate the range of adipose a*H+b*S colormap values compared to the other tissue a*H+b*S colormap values (n=14). The mean values of the lowest 1%, 5%, 10%, and 25% colormap values of both the adipose and other segments are compared. For all comparisons, there is a statistical significance of p<0.0001. C) Box-whisker plots to illustrate the range of adipose a*S+b*S colormap values compared to the other tissue a*S+b*S colormap values (n=14). The mean values of the lowest 1%, 5%, 10%, and 25% colormap values of both the adipose and other segments are compared. For all comparisons, there is a statistical significance of p<0.0001. D) Scatterplots of intersection over union (IoU) values of adipose segmentation over the percentage of adipose for all biopsy cases in the training set (n=14). Colormaps a*H+b*S and a*S+b*S results are shown in each scatterplot, and the correlation coefficient and p-value are calculated.

To implement an automated segmentation method, we need to establish a systematic method to select adipose segments among three different segments generated by K-means clustering. Therefore, we calculated the mean, median, minimum, and maximum of the a*H+b*S colormaps. Mean values of the lowest 1%, 5%, 10%, and 25% intervals of the colormaps were compared. Figure 2B shows the box-whisker plots of the mean values of the lowest 1%, 5%, 10%, and 25% of the a*H+b*S colormap values from adipose and other tissue segments. Supplement Figure 1A shows minimum, maximum, and mean and median comparisons. For all comparisons, statistical analysis using the Student T-test shows that the adipose segment is significantly higher than other tissue types (p<0.0001). Figure 2C shows the box-whisker plots of the mean values of the lowest 1%, 5%, 10%, and 25% of the a*S+b*S colormap values from adipose and other tissue segments. Supplement Figure 1B shows minimum, maximum, and mean and median comparisons. Similar to the a*H+b*S result, the adipose segment is statistically higher than other tissue types (p<0.0001). No overlap is observed between two tissue segments for all four statistical comparisons indicating that the adipose segment can be separated by a clear cutoff threshold using minimum, maximum, mean, median, or mean values of the bottom 1%, 5%, 10%, and 25% of colormaps. Given that the minimum and maximum or more subject to noise we chose our cutoff threshold based on the mean values of the bottom 1% for best separation between the adipose and the other group. The 1% threshold was selected so as to maximize sensitivity for tumor detection though this can be varied based on the false-negative criteria. The midpoint between the adipose and other segments was used as the cutoff threshold, for use on an independent test set which is 0.371 for a*H+b*S and 0.370 for a*S+b*S. The colormaps for the color transformation corresponding to a*S+b*S were selected for automated segmentation. Of the 24, 14 biopsies were used in the training set, and the rest were used as the test set the results of which will be shown in Figure 5.

Figure 5: Evaluation of segmentation performance on the testing set.

A) Schematic of two-step tumor segmentation workflow. B) Representative tumor prediction based on the segmentation of a*S+b*S and Cb*S+Cr*S colormaps (top), overlayed images with tumor prediction and ground truth (middle), and corresponding histology (bottom). The magenta region is the algorithm-predicted tumor region, and the green region is the tumor ground truth based on histology. The black region indicates an overlap between ground truth and prediction. The performance of adipose segmentation at first step and tumor segmentation at final step is summarized in the table below. The percentage of intersection over union (IoU) between algorithm prediction and ground truth is used as evaluation metric. C) Representative tumor predictions based on the segmentation of a*S+b*S and Cb*S+Cr*S on two benign biopsies (top) and corresponding histology (bottom). The magenta region is the algorithm-predicted tumor region. In this case, a magenta region on the benign biopsy indicates an algorithm misclassification. A blank prediction suggests that the algorithm predicts no tumor on biopsy. The performance of adipose segmentation at first step and tumor segmentation at final step is summarized in the table below. The percentage of intersection over union (IoU) between algorithm prediction and ground truth is used as evaluation metric and the presence of misclassified tumor in each benign case is also included.

We performed automated selection of adipose segments for all 14 biopsies using both colormap transformations using our previously selected cutoff thresholds. The percentage of intersection over union (% IoU) was calculated between the control and segmented areas for a*H+b*S and a*S+b*S colormaps and compared to the overall % adipose for each biopsy. Figure 2D shows scatterplots of intersection over union (% IoU) values of adipose segmentation vs. the percentage of adipose tissue in the sample using a*H+b*S colormap (left) and a*S+b*S colormap (right). The a*S+b*S colormap has a higher % IoU mean of 37% compared to the a*H+b*S mean % IoU of 26%. There is a higher correlation between segmentation performance (% IoU) and tissue purity (% Adipose) in the a*H+b*S colormap values as compared to that of the a*S + b*S values, indicating that segmentation based on a*H+b*S could potentially be more sensitive to tissue purity. The correlation coefficients and p-values for a*H+b*S and a*S+b*S are 0.53 (p=0.053) and 0.68 (p<0.01), respectively.

3.3. Colormaps Generated from YCbCr and HSV Color Spaces Enables Tumor Segmentation

After the 1st level of segmentation, the adipose segment was removed from the original RGB images. Figure 3A shows the representative white-light and green-light segmented images in which the adipose has been removed from the 1^st step and the corresponding histology ground truth. From the previous analysis, colormaps generated from CIELAB color space was not sufficient for tumor segmentation, as shown in Supplement Figure 2. Therefore, the YCbCr color space was further explored for this task. The Y channel is defined as the luma component representing the brightness of the image. Cb and Cr are the blue-difference and red-difference channels describing the color information of the image. Similar to the previous adipose segmentation, Cb and Cr channels were combined with the hue and saturation channels through multiplication and addition. Figure 3B shows four colormaps, Cb*H+Cr*H (Cb*Hue+Cr*Hue), Cb*S+Cr*S (Cb*Saturation+Cr*Saturation), Cb*H+Cb*S (Cb*Hue+Cb*Saturation), and Cr*H+Cr*S (Cr*Hue+Cr*Saturation), generated using white light only, green light only, and white/green light images. A set of high-purity site-level data (>80%) was used to reduce heterogeneity. Again, the same 14 non-bloody samples and fibro glandular biopsies were used for the tumor segmentation clustering analysis. The average Silhouette scores were reported for all possible combinations of Cb, Cr, hue, and saturation channels, as shown in Figure 3C. Cr*S (Cr*Saturation) gave the highest average Silhouette score (0.77), and Cb*S+Cr*S was the second highest (0.71). The tumor colormap values were compared to the fibro glandular tissue colormap values assigned based on site-level pathology. Figure 3D shows the scatterplot generated by plotting values from each colormap of white-light images vs. colormap values of green-light images. Both Cr*S and Cb*S+Cr*S with the top 2 average Silhouette scores showed a clear separation between the tumor cluster (blue) and the fibroglandular cluster (orange).

Figure 3: Colormap generation using YCbCr and HSV color spaces for tumor segmentation.

A) Representative white-light, green-light after adipose segmentation, and the H&E image of one core needle biopsy sample (same as that in Figure 1). The tumor is circled in the H&E histology image. B) Different combinations of color channels. From top to bottom are Cb*H+Cr*H, Cb*S+Cr*S, Cb*H+Cb*S, and Cr*H+Cr*S, where H stands for hue, and S stands for saturation. The left panels show the color combination results of white light images, the middle shows the color combination results of green light images, and the right panels show the ratio of white light over green light color combination results. The last two panels from bottom Cb*H+Cb*S and Cr*H+Cr*S of the white light image only and white/green group yield the best representation of the feature of tumor tissue of this core needle biopsy. C) Table of average silhouette scores obtained from tumor and fibroglandular clusters for all combinations of colormaps in YCbCr and HSV color spaces. The top two combinations are Cr*S, and Cb*S+Cr*S, with an average silhouette score of 0.77 and 0.71, accordingly, where H stands for hue and S stands for saturation. D) Scatterplots of pixels from different tissue types and their distribution in different colormaps generated from white-light and green-light images. For all panels, the x-axis represents the color transformation of white-light images, and the y-axis represents the color transformation of green-light images. Only representative biopsy cases with more than 80% of either tumor or fibroglandular tissue (n = 14) were selected in this analysis. The top panel is Cr*Saturation, and the bottom panel is Cb*Saturation+Cr*Saturation.

For tumor segmentation, K-means clustering (k=3) was applied to the colormaps of the two variables with the highest Silhouette scores, and the segmentation results are shown in the middle panel of Figure 4A. We chose k = 3 to generate three segments: background, cancer, and fibroglandular. Representative segmentation results are shown in the bottom panel of Figure 4A. The magenta, green, and black regions correspond to the algorithm, ground truth, and overlapped region, respectively.

Figure 4: Combining color features of white and green light images enables tumor segmentation.

A) Comparison of K-Means (k=3) clustering result of one representative biopsy case using white/green image. The top 2 colormap combinations, Cr*S and Cr*H+Cr*S-based clustering analysis, are shown here. The top row is the colormap images of two different colormaps generated using white light images divided by green light images. The middle row contains the K-Means clustering results. The bottom row shows the overlap between the clustering result (magenta) and ground truth (green). The black color indicates the overlapping region. The Cr*H+Cr*S colormap achieves the best segmentation performance, shown by the largest black area. B) Box-whisker plots to illustrate the range of tumor Cr*S colormap values compared to the fibroglandular Cr*S colormap values. The mean of the bottom 1%, 5%, 10%, and 25% colormap values of both the tumor and fibroglandular segments are compared. For each comparison, statistical significances are labeled in each graph. (**** =p<0.0001, *** = p<0.001, ** = p<0.01, * = p<0.05, ns = not significant) C) Box-whisker plots to illustrate the range of tumor Cb*S+Cr*S colormap values compared to the fibroglandular Cb*S+Cr*S colormap values. The mean of the bottom 1%, 5%, 10%, and 25% colormap values of both the tumor and fibroglandular segments are compared. For each comparison, statistical significances are labeled in each graph. (**** =p<0.0001, *** = p<0.001, ** = p<0.01, * = p<0.05, ns = not significant) D) Scatterplots of intersection over union (IoU) values of tumor segmentation over the percentage of tumor for all malignant biopsy cases in the training set. Colormaps Cr*S and Cb*S+Cr*S results are shown in each scatterplot, and the correlation coefficient and p-value are calculated.

To determine the cutoff threshold for tumor segmentation, we compared the Cr*S and Cb*S+Cr*S colormap values of tumor segments to fibro glandular segments, as was shown for adipose segmentation. We calculated the mean, median, minimum, and maximum of the Cb*S+Cr*S colormaps. Mean values of the lowest 1%, 5%, 10%, and 25% intervals of both colormaps were compared. Supplement Figure 3A shows minimum, maximum, and mean and median comparisons of all pixels. Figure 4B shows the box-whisker plots of the mean values of the lowest 1%, 5%, 10%, and 25%comparisons of the Cr*S colormap values. The tumor segment is statistically higher than the fibroglandular segment (p<0.05) using the Student T-test for all comparisons except the median (Supplement 3A). Figure 4C shows the box-whisker plots of the mean values of the lowest 1%, 5%, 10%, and 25%comparisons of the Cb*S+Cr*S colormap value. Supplement Figure 3B show box-whisker plots of minimum, maximum, and mean and median comparisons of all pixels from the Cb*S+Cr*S colormap values. There is a statistically significant difference between tumor and fibroglandular segments when comparing the mean of the lowest 1% (p<0.01) and 5% (p<0.05) of the minimum (p<0.001). Even though the minimum value of tumor and fibroglandular segments can be separated by a cutoff threshold for both color maps, we decided to calculate the cutoff threshold between fibroglandular clusters and tumor clusters using the mean of the bottom 1% colormap values. The cutoff threshold was determined as 0.094 for Cr*S and 0.185 for Cb*S+Cr*S, with the fewest misclassified clusters in the training set. As before 14 of the total 24 biopsies were used as the training set and the rest were used in the test set, the results of which will be shown in Figure 5.

Using cutoff thresholds, we performed automated selection of tumor segments for all training biopsies using both colormap transformations. The % IoU was calculated for both Cr*S and Cb*S+Cr*S colormaps and compared to the overall % tumor for each biopsy. Figure 4D shows scatterplots of % IoU values of tumor segmentation for the Cr*S colormap (left) and Cb*S+Cr*S colormap (right). The Cb*S+Cr*S colormap shows consistently better performance than the Cr*S colormap in the training set with a higher mean of % IoU of 45% compared to the Cb*S+Cr*S mean % IoU of 41%. Neither Cr*S nor Cb*S+Cr*S shows a strong correlation between segmentation performance (% IoU) and tissue purity (% tumor), indicating that the segmentation performance of neither transformation is sensitive to tissue purity. The correlation coefficients and p-values for Cr*S and Cb*S+Cr*S are 0.07 (p=0.85) and −0.20 (p=0.61).

Lastly, we evaluated the two-step segmentation workflow with the test set of 10 samples as shown in Figure 5A. The test set contains 5 malignant biopsies and 5 benign biopsies. a*H+b*S and a*S+b*S colormaps were evaluated for adipose segmentation, and Cr*S and Cb*S+Cr*S were assessed for tumor segmentation, resulting in four combinations in total. The representative a*S+b*S and Cb*S+Cr*S biopsy segmentation predictions on malignant and benign cases and their corresponding histology are shown in Figures 5B and 5C and Supplement 4. The adipose and tumor segmentation in terms of percentage intersection over union (IoU%) are summarized in the tables of Figure 5B and 5C. The algorithm identified 5 out of 5 malignant biopsies as positive (Supplement 4B) and 3 out of 5 benign biopsies (Supplement 4C). Two benign cases were misclassified as tumor by the algorithm. The a*S+b*S and Cb*S+Cr*S combinations demonstrated the best overall segmentation performance and achieved a sensitivity of 100% and a specificity of 60% in predicting the testing biopsy, the best among all four combinations. The prediction results of a*S+b*S and Cr*S combination, a*H+b*S and Cb*S+Cr*S combination, and a*H+b*S and Cr*S combination are shown in Supplement Figures 5, 6, and 7, respectively. a*S+b*S shows improved adipose segmentation performance than a*H+b*S, indicated by higher adipose % IoU in the test set. Even though the a*S+b*S and Cr*S combination achieved a higher tumor % IoU and classified all tumor cases correctly, this combination misclassified all benign cases as tumor cases in the test set. A similar trend can be observed in the a*H+b*S and Cr*S combination, where all benign cases were diagnosed as tumor, indicating that Cb*S+Cr*S is more appropriate for the tumor segmentation task.

3.4. Discussion

Detection of breast tumors on fresh tissue has intriguing implications for expediting diagnosis, reducing cost, and collecting tissue for molecular studies. Here, we demonstrate an automated two-step tumor segmentation method leveraging color information from brightfield images of fresh breast core needle biopsies. In most instances, we were able to achieve at least partial overlap between the predicted and ground truth tumor areas. Assuming that any level of overlap reflects the presence of a tumor, we were able to identify all malignant samples in the test set. Color channels from three different color spaces (HSV, CIELAB, YCbCr) were explored in the segmentation task. We generated colormaps representing the different tissue types by performing simple mathematical transformations of the extracted color channels. Two different colormaps were chosen for the adipose and tumor segmentation task, and a two-step segmentation was performed. Adipose tissue was delineated and removed in the first level of segmentation, and benign tissue was identified and eliminated in the second segmentation step. This simple approach of identifying at least some of the tumor allows a provider to determine if tumor is present in the sample and where it is. Though 1% was chosen as the threshold to minimize false negatives, the thresholds can be varied depending on the tradeoff between sensitivity and specificity. Another important point to note that while we chose the color feature variable with the highest silhouette scores for tumor segmentation, other color space features might segment complementary areas of the tumor thereby increasing the overlap. The use of colormaps were effective in identifying tumor tissues. In fact, the RGB color space is commonly used for semantic segmentation tasks of medical images and in the realm of breast oncology, it has been applied for delineation of tumors in histology²³, thermograms^{24, 25}, brightfield microscopic⁷ images and spectral and polarization images of tumor margin⁷.

Compared to traditional color-based segmentation, our method introduces the use of green light illumination on ex vivo tissue specimens, which has been applied in GI endoscopy imaging²⁶ and vascular imaging²⁷ for enhancing imaging contrast. Green light illumination lightens the regions with abundant green color and darken the regions with abundant red color, which provides supplemental imaging data for color-based segmentations. In line with prevalent color-based segmentation methods, our approach employs YCbCr and CIELAB color spaces over the conventional RGB, facilitating the separation of luminance from chrominance information. This ensures that image brightness does not influence subsequent segmentation. Distinct from other color-based cancer segmentation techniques necessitating additional modalities such as fluorescence or polarization imaging, our method leverages the information provided by green-light images to achieve segmentation, thereby eliminating the requirement for auxiliary imaging modalities. Through the use of mathematical color transformations that combine saturation/hue with Cb/Cr or a/b color spaces, our method facilitates precise tissue type differentiation. These transformations were identified by analyzing clusters from images of different tissue types, providing a strong rationale and high interpretability for the segmentation process. Consequently, simple thresholding based on transformed colormap values allows for the identification of adipose and tumor tissues from surrounding tissue.

A variety of other optical imaging technologies have been developed for assessment of breast tissues and tumor margins. These include OCT^{28, 29}, Raman³⁰, diffuse reflectance³¹, and fluorescence-based imaging technologies^32–34. Many of these technologies are based on endogenous contrast. The most commonly used method is fluorescence spectroscopy and/imaging. Many of these technologies image or measure from small fields of view relative to the area of the excised tissue specimen. Our handheld optical probe and algorithm can complement these approaches by identifying suspicious areas for placement of these probes, thereby increasing specificity as well as reducing time for sampling. In addition to intrinsic fluorescence contrast, several promising fluorescent molecular probes have been explored to improve tumor identification in tumor margins. For example, our lab has explored the imaging of fluorescence labeled heat shock protein 90 (Hsp90) on core needle biopsies for rapid breast cancer diagnosis^{35, 36}. By modifying our device to perform fluorescence imaging, our device can generate co-registered brightfield and fluorescence images³⁷. The fluorescence molecular probe can identify tumors and the bright field images can identify regions of nonspecific fluorescence from the fluorophore.

There are several limitations to the study. First, spatial distortion and fragmentation may occur due to the processes of fixation and paraffin during histology slide preparation. Because of this, co-registration between the histology slide (ground truth) and the input images from fresh tissue may not be concordant, resulting in a mismatch of ground truth and annotations. Second, the brightfield images reflect the surface tissue morphology of the core needle biopsy. However, the histology ground truth is determined by a thin tissue slice from the core needle biopsy. Given that tissue morphology changes at different tissue depths, the shape and size of the tumor region could be different on the biopsy surfaces than from the center. Variations in tissue composition also create a challenge in the segmentation task. Our results showed that adipose segmentation is correlated to the percentage of adipose in the biopsy sample. A pure tissue biopsy is more likely to be better segmented with our method than a biopsy with mixed tissue composition. Lastly, we observed that blood on the tissue could also affect the segmentation performance as the bloody regions in the testing set had similar colormaps values as the adipose tissue and thus created another source of false negative classification.

The results from this study indicate that the automated, two-step tumor segmentation approach utilizing color information from fresh core needle biopsies of breast tissue could present a practical alternative to conventional histology. Its ability to identify malignant regions underscores its potential utility promptly and efficiently in augmenting diagnostic accuracy and increasing accessibility to pathology services, particularly in resource-constrained settings. The promising potential of this approach in guiding accurate tissue selection for molecular testing and intraoperative tumor margin assessment is noteworthy, contributing to the field of personalized cancer management. Looking forward, as our dataset expands, we aim to integrate deep learning techniques, building upon the optimized colormap transformations established in this study. This advancement is expected to further enhance segmentation accuracy and efficiency, particularly in complex diagnostic scenarios, offering a substantial leap forward in digital pathology.

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available from the corresponding author upon reasonable request.