Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging

Abstract

The segment anything model (SAM) was released as a foundation model for image segmentation. The promptable segmentation model was trained by over 1 billion masks on 11M licensed and privacy-respecting images. The model supports zero-shot image segmentation with various segmentation prompts (e.g., points, boxes, masks). It makes the SAM attractive for medical image analysis, especially for digital pathology where the training data are rare. In this study, we evaluate the zero-shot segmentation performance of SAM model on representative segmentation tasks on whole slide imaging (WSI), including (1) tumor segmentation, (2) non-tumor tissue segmentation, (3) cell nuclei segmentation.

Core Results:

The results suggest that the zero-shot SAM model achieves remarkable segmentation performance for large connected objects. However, it does not consistently achieve satisfying performance for dense instance object segmentation, even with 20 prompts (clicks/boxes) on each image. We also summarized the identified limitations for digital pathology: (1) image resolution, (2) multiple scales, (3) prompt selection, and (4) model fine-tuning. In the future, the few-shot fine-tuning with images from downstream pathological segmentation tasks might help the model to achieve better performance in dense object segmentation.

Introduction

Large language models (e.g., ChatGPT [6] and GPT-4 [7]), are leading a paradigm shift in natural language processing with strong zero-shot and few-shot generalization capabilities. This development has encouraged researchers to develop large-scale vision foundation models. While the first successful “foundation models” [8] in computer vision have focused on pre-training approaches (e.g., CLIP [9] and ALIGN [10]) and generative AI applications (e.g., DALL·E [13]), they have not been specifically designed for image segmentation tasks [14]. Segmenting objects (e.g., tumor, tissue, cell nuclei) for whole slide imaging (WSI) data is an essential task for digital pathology, deep learning models typically necessitate well-delineated training data. Obtaining these gold-standard data from clinical experts can be challenging due to privacy regulations, intensive manual efforts, insufficient reproducibility, and complicated annotation processes [16]. Hence, zero-shot image segmentation [20] is desired, where the model can accurately segment pathological images without prior exposure to the domain data during training.

Recently, the “Segment Anything Model” (SAM) [14] was proposed as a foundation model for image segmentation. The model has been trained on over 1 billion masks on 11 million licensed and privacy-respecting images. Furthermore, the model supports zero-shot image segmentation with various segmentation prompts (e.g., points, boxes, and masks). This feature makes it particularly attractive for pathological image analysis where the labeled training data are rare and expensive.

In this study, we assess the zero-shot segmentation performance of the SAM model on representative segmentation tasks, including (1) tumor segmentation [18], (2) tissue segmentation [19], and (3) cell nuclei segmentation [21]. Our study reveals that the SAM model has some limitations and performance gaps compared to state-of-the-art (SOTA) domain-specific models.

Experiments and Performance

We obtained the source code and the trained model from https://segment-anything.com. To ensure scalable assessments, all experiments were performed directly using Python, rather than relying on the Demo website. The results are presented in Figure 1 and Table 1.

Figure 1. Qualitative segmentation results.

The SOTA methods are compared with SAM method with different prompt strategies.

Table 1.

Compare SAM with state-of-the-art (SOTA) methods.(Unit: Dice score)

Method	Prompts	Tumor	Tissue						Cell
		0.5×	5×		10×			40×	40×
		Tumor	CAP	TUFT	DT	PT	VES	PTC	Nuclei
SOTA	no prompt	71.98	96.50	96.59	81.01	89.80	85.05	77.23	81.77
SAM	1 point	58.71	78.08	80.11	58.93	49.72	65.26	67.03	1.95
SAM	20 points	74.98	80.12	79.92	60.35	66.57	68.51	64.63	41.65
SAM	total points	n/a	88.10	89.65	70.21	73.19	67.04	67.61	69.50
SAM	total boxes	n/a	95.23	96.49	89.97	86.77	87.44	87.18	88.30

total points/boxes: we place points/boxes on every single instance object (based on the known ground truth) as a theoretical upper bound of SAM. Note that it is impractical in real applications.

Tumor Segmentation.

The whole-slide images (WSIs) of skin cancer patients were obtained from the Cancer Genome Atlas (TCGA) datasets (TCGA Research Network: https://www.cancer.gov/tcga). We employed SimTriplet [18] approach as the SOTA method, with the same testing cohort to make a fair comparison. In order to be compatible with the SAM segmentation model, the WSI inputs were scaled down 80 times from a resolution of 40×, resulting in an average size of 860×1279 pixels. We evaluated two different scenarios: (1) SAM with a single positive point prompt, and (2) SAM with 20 point prompts (10 positive and 10 negative points). The prompts were randomly selected from manual annotations, with positive prompt points being selected from the tumor region and negative prompt points being selected from the non-tumor region.

Tissue Segmentation.

A total of 1,751 regions of interest (ROIs) images were obtained from 459 WSIs belonging to 125 patients diagnosed with Minimal Change Diseases. These images were manually segmented to identify six structurally normal pathological primitives [12], using digital renal biopsies from the NEPTUNE study [11]. To form a test cohort for multi-tissue segmentation, we captured 8,359 patches measuring 256×256 pixels. For comparison, We employed Omni-Seg [19] approach as the SOTA method. The tissue types consist of the glomerular unit (CAP), glomerular tuft (TUFT), distal tubular (DT), proximal tubular (PT), arteries (VES), and peritubular capillaries (PTC). For the SAM method, we evaluated four different scenarios: (1) SAM with a single positive point prompt, (2) SAM with 20 point prompts (10 positive and 10 negative points), and (3)/(4) SAM with all points/boxes on every single instance object, which served as a theoretical upper bound for SAM. We randomly selected point prompts from the manual annotations and eroded each connected component with a 10×10 filter to generate at most one random point. For the box prompts, we used the bounding box of each connected component.

Cell nuclei Segmentation.

The dataset for nuclei segmentation was obtained from the MoNuSeg challenge [17]. It contains H&E stained images at 40× magnification with 1000×1000 pixels from the TCGA dataset, along with corresponding annotations of nuclear boundaries. The MoNuSeg dataset includes 30 images for training and 14 for testing. We evaluated the performance of SAM models against the BEDs model [21], a competitive nuclei segmentation model trained on the MoNuSeg training data. The prompt method and evaluation are as described in §Tissue Segmentation.

Limitations on Digital Pathology

The SAM models achieve remarkable performance under zero-shot learning scenarios. However, we identified several limitations during our assessment.

Image resolution.

The average training image resolution of SAM is 3300×4950 pixels [14], which is significantly smaller than Giga-pixel WSI data (> 10⁹ pixels). Moreover, analyzing WSI data at the patch level may result in an impractical number of interactions, even if only a small number of points or bounding boxes are marked for each patch.

Multiple scales.

Multi-scale is a significant feature in digital pathology. Different tissue types have their optimal image resolution (as shown in Table 1). For instance, at the optimal resolution for CAP segmentation (5× scale), it is difficult to achieve good segmentation for PTC. However, zooming in (40× scale) would result in nearly 100 times more patches.

Prompt selection.

Firstly, to achieve decent segmentation performance in zero-shot learning scenarios, a considerable number of prompts are still necessary. Secondly, the segmentation performance heavily depends on the quality of prompt selection. Another concern related to segmentation performance is inter-rater and intra-rater reproducibility of prompt-based segmentation.

Model fune-tuning.

Currently, tedious manual prompt placements are still necessary for segmentation tasks with significant domain heterogeneities. A reasonable online/offline fine-tuning strategy is necessary to propagate the knowledge obtained from manual prompts to larger-scale automatic segmentation on Giga-pixel WSI data.

Conclusion

The zero-shot setting of SAM enables domain users to segment heterogeneous objects in digital pathology without undergoing a heavy training process. The results suggest that the zero-shot SAM model achieves remarkable segmentation performance for large connected objects. However, it does not consistently achieve satisfying performance for dense instance object segmentation, even with 20 prompts (clicks/boxes) on each image. Nonetheless, several limitations still exist and require further investigation for digital pathology.

Biography

Ruining Deng is a PhD candidate in Computer Science at Vanderbilt University, working with Dr. Yuankai Huo. He received his Bachelor’s degree from China University of Mining and Technology, Beijing. In addition to being a research assistant at Vanderbilt University, Mr. Deng was a visiting scholar at the University of Notre Dame and a member of the research staff at the Guangdong Provincial Cardiovascular Institute. Recently, he also served as an imaging scientist intern at Roche Diagnostics USA.