TriSAM: Tri-Plane SAM for zero-shot cortical blood vessel segmentation in VEM images

Abstract

While imaging techniques at macro and mesoscales have garnered substantial attention and resources, microscale Volume Electron Microscopy (vEM) imaging, capable of revealing intricate vascular details, has lacked the necessary benchmarking infrastructure. In this paper, we address a significant gap in this field of neuroimaging by introducing the first-in-class public benchmark, BvEM, designed specifically for cortical blood vessel segmentation in vEM images. Our BvEM benchmark is based on vEM image volumes from three mammals: adult mouse, macaque, and human. We standardized the resolution, addressed imaging variations, and meticulously annotated blood vessels through semi-automatic, manual, and quality control processes, ensuring high-quality 3D segmentation. Furthermore, we developed a zero-shot cortical blood vessel segmentation method named TriSAM, which leverages the powerful segmentation model SAM for 3D segmentation. To extend SAM from 2D to 3D volume segmentation, TriSAM employs a multi-seed tracking framework, leveraging the reliability of certain image planes for tracking while using others to identify potential turning points. This approach effectively achieves long-term 3D blood vessel segmentation without model training or fine-tuning. Experimental results show that TriSAM achieved superior performances on the BvEM benchmark across three species. Our dataset, code, and model are available online at https://jia-wan.github.io/bvem.

I. Introduction

With around 2% of body weight, our brain receives around 20% of blood supply. Most of the energy and nutrients are consumed by the neurons, and neuron function is sensitive to the blood supply [3], [4]. The blood supply can even be adjusted following the consumption of brain regions, called neurovascular coupling [5]. Alterations of blood vessel structures are observed in many brain diseases, e.g., Alzheimer’s and vascular dementia [6]. Thus, blood vessels in the brain have been extensively investigated with various imaging modalities at different resolutions (Figure 1a). Compared to the macro-level imaging (e.g., CT [1] , MRI [7]) and mesoscale-level imaging (e.g., light microscopy [2]), volume electron microscopy (VEM) [8] can further reveal the detailed ultrastructure including all vascular cells (Figure 1b) for in-depth analysis. However, no large-scale annotated VEM dataset exists to develop and evaluate automated 3D blood vessel segmentation methods.

Fig. 1.

Imaging modalities for blood vessel analysis. (a) Both microtomography (μCT) [1] and light microscopy (LM) [2] can only capture blood vessels in the cortex at the sub-micron resolution without ultrastructure details. (b) Volume electron microscopy (VEM) can show unbiased details of the vasculature including all types of cells at a higher resolution.

Traditionally, the imaging methods at the macro and mesoscale are widely used and have produced a large amount of data, and a variety of image segmentation algorithms, public datasets, and evaluation methods have been developed [9], [10]. At the microscale level, the sample size of VEM is normally limited and most image analyses focus on neuron reconstruction, and blood vessels are largely ignored. Recently, owing to the rapid improvement of imaging technology, the sample size of VEM is significantly increased covering all the layers of the cerebral cortex of mouse [11]and human brain [12], as well as the whole brain of fly [13]. Moreover, imaging the whole mouse brain using VEM technology is under planning [14].

Thus, we first curate the BvEM dataset, the largest-to-date public benchmark dataset for cortical blood vessel segmentation in VEM images to foster segmentation method development. The raw image volumes are from recent publications, which are the largest for each of the three mammals: mouse, macaque, and human acquired at different VEM facilities. We downsampled the volumes to a consistent resolution and performed extensive blood vessel annotation, including manual proofreading, semi-automatic segmentation error correction, and quality control, involving multiple rounds of scrutiny by neuroscience experts to ensure accuracy and completeness. However, on the BvEM dataset, existing 3D blood vessel segmentation methods suffer from two major challenges: the diversity of the image appearance due to variations in the imaging pipeline and the complexity of the 3D blood vessel morphology. Conventional blood vessel segmentation heavily depends on a substantial volume of manually annotated data, a resource that is notably scarce in the existing literature.

To address these challenges, we propose a zero-shot 3D segmentation method, TriSAM, leveraging the recent 2D segmentation foundation model, Segment Anything Model (SAM) [15], to handle the appearance diversity. To re-purpose SAM for 3D segmentation, we developed a multi-seed segment tracking framework based upon the video object segmentation method with SAM [16]. As it is easier to track 2D blood vessel segments along the blood flow direction, TriSAM selects the best 2D plane for SAM-based tracking and further introduces a recursive seed sampling strategy that performs tri-plane selection at the potential turning points for efficiency. The proposed method is similar to Flood-Filling Networks (FFN) [17] by extending the mask in 3D space. Instead of training a 3D neural network, we utilize SAM for prediction since it generalize better. Second, FFN densely extends to 3D while the proposed method extends along the blood vessels. Finally, FFN uses only one seed while we are using growing seeds for long-term tracking. Experimental results demonstrate that the proposed TriSAM method significantly outperforms the prior state-of-the-art methods on the proposed BvEM benchmark across all three species.

II. Related Work

Blood vessel segmentation methods.

Most existing VEM image segmentation algorithms were developed for neurons [18] and synapses [19]–[21]. The traditional VEM dataset size is too small to study blood vessel architecture. Recently, with rapid technology improvement, VEM sample size reached the cubic millimeter scale covering all the layers of cerebral cortex [11], [12], providing rich details of vasculature structure at scale. However, due to the lack of efficient and effective automatic methods, the blood vessels were segmented manually in the human cortex dataset [12]. On another front, automatic blood vessel segmentation methods have been developed for other image modalities, such as light microscopy and MRI, at a lower resolution. We refer to [9], [10] for detailed reviews on filter-based methods [22], [23] and deep learning methods [24], [25]. Due to the burden of annotation, efforts have been made to decrease the need for annotations [26]. In this paper, we propose the first VEM cortical blood vessel benchmark to foster novel methods.

Segment anything-based methods.

As a foundation model for image segmentation, the recently proposed Segment Anything Model [15] has garnered significant attention and has been extended to a variety of domains [27]–[30] including object tracking [16], [27], image inpainting [28], image mattting [31], super-resolution [32], 3D point cloud [33], and image editting [34]. Despite SAM’s remarkable generalization capabilities, it still encounters some challenges in practical applications, One of these challenges is the huge computation costs due to the heavyweight image encoder. FastSAM [35] adopted a conventional CNN detector with an instance segmentation branch for the segment anything task with real-time speed. MobileSAM [36] proposed decoupled distillation to obtain a small image encoder, which achieved approximately five times faster speed compared to FastSAM while also being seven times smaller in size. Therefore the MobileSAM is employed in our proposed method. Although SAM demonstrates impressive generalization capabilities, it faces challenges in specialized domains like medical [37], [38] or biological images [39], especially in handling 3D data. [30] assessed the SAM model’s zero-shot segmentation performance in the context of digital pathology and showed scenarios where SAM encounters difficulties. [40] extensively evaluates the SAM for medical image segmentation across 19 diverse datasets, highlighting SAM’s performance variability. To address the domain gap between natural and medical images, SAM-Adapter [37], MA-SAM [41], 3DSAM-adapter [42], SAM-Med2D [43], and Medical SAM Adapter [44] introduced Adapter modules and trained the Adapter with medical images. They attained good performance on various medical image segmentation tasks. MedSAM [29] adapted SAM with more than one million medical image-mask pairs and attained accurate segmentation results. MicroSAM [39] also presented a segment anything model for microscopy by fine-tuning SAM with microscopy data. Unlike these approaches that require model fine-tuning for adaptation, our method lifts the blood vessel segmentation capabilities of SAM from 2D images to 3D volumes without any model fine-tuning.

III. BvEM Dataset

A. Dataset Description

Tissue samples.

We built the BvEM dataset upon the largest publicly available VEM image volumes for three mammal species: visual cortex from an adult mouse [11], superior temporal gyrus from an adult macaque [45], and temporal lobe from an adult human [12]. Each volume was acquired with different protocols at different facilities and we refer to respective papers for more details. As shown in Fig. 2, the imaging quality and the appearance of blood vessels vary drastically across these three volumes, showcasing the cutting-edge large-scale VEM imaging pipelines.

Fig. 2.

The proposed BvEM dataset. We proofread the blood vessel instance segmentation (displayed in different colors) in the three largest publicly available VEM volumes: (a) mouse [11], (b) macaque [45], and (c) human [12] samples acquired at different VEM labs.

Image volumes.

We processed the original image volumes into a standardized form that is suitable for benchmarking. We first downsampled all three VEM image volumes to a near-isotropic resolution (~200–300 nm) along each dimension, which is a good balance between rich image details for biological analysis and the nature of capillary diameter. Then, we trimmed the image boundary of the BvEM-Mouse and BvEM-Human volumes, where the blood vessels are hard to annotate due to the missing image content.

B. Dataset Annotation

Initial annotation.

The paper on the BvEM-human volume provides a manual proofread blood vessel segmentation, where many segments are incomplete or disconnected due to missing annotations. The papers for BvEM-mouse and BvEM-macaque volumes only provide dense 3D instance segmentation and we manually selected the blood vessel segments. Due to the large scale of the BvEM-Mouse and BvEM-Human volumes, we only annotated every 4 slices, where the z-dimension resolution is around 1μm.

Automatic error detection.

For the false split errors, which are the majority source of error, we computed the skeleton of the segmentation and detected all the skeleton endpoints that do not touch the volume boundary as candidates. For the false merge errors, we computed the intersection-over-union (IoU) of 2D segments on neighboring slices and detected areas of small IoU value as candidates.

Manual proofreading.

We used the VAST lite software [46] to accelerate the manual proofreading process by using the provided dense segmentation results as templates. Instead of manually delineating segments manually, proofreaders can coarsely draw or fill the segment mask that is snapped to the detailed boundary in the template. Assisted with the 3D blood vessel visualizations, two neuroscience experts proofread each volume in multiple rounds until no disagreement.

C. Dataset Statistics

As shown in Tab. I, the BvEM-Macaque volume has around 0.5G voxels, and the BvEM-Mouse and BvEM-Human volumes are around 80 and 121 times bigger, respectively. From the blood vessel instance segmentation annotation, we automatically extracted skeleton centerlines [47] and computed the length for each blood vessel instance. Due to the hyper-connectivity nature of cortical blood vessels, the length of the largest instance is around 99%, 95%, and 85% for each volume. The histogram of the blood vessel radius in the proposed dataset is shown in Figure 3. The mouse and the human dataset have similar radius distributions peaked around capillaries, while the macaque volume has no capillaries.

TABLE I.

Dataset information. Despite the difference in the scale and the geometry of the image volume, the largest blood vessel instance (rendered in red) is significantly bigger than the rest combined. The last column shows the max/total length of the blood vessel instances.

Sample	Resolution (nm)	Size (voxel)	Length: (mm)
Mouse (TEM)	320× 256×256	2495×3571×2495	1.6/1.7
Macaque (SBEM)	240×176×176	450×1271×995	713.3/714.5
Human (MultiSEM)	264×256×256	661×7752×13500	107.2/126.7

Fig. 3.

Dataset statistics. Empirical distribution of blood vessel radius.

IV. Method

A. Overview

We aim to leverage the 2D generalist Segment Anything Model (SAM) to build a zero-shot 3D VEM blood vessel segmentation method, which can be widely applied to various image volumes. Given an input seed position, existing SAM-based tracking methods work well when the blood vessel travels only in one axis. However, in reality, blood vessels not only turn in alternating directions but also bifurcate, leading to drastic shape and scale changes.

Instead, we represent the whole connected blood vessel segment that contains the input seed as a graph, $G=(V,E)$, where each node $v\in V$ is a tubular branch segment that does not bifurcate. Thus, the segmentation problem can be solved with the graph traversal algorithm. After visiting one node (i.e., running SAM-based tracking methods given the seed), we sample new seed positions to find neighboring nodes to segment recursively.

Thus, the proposed TriSAM framework has the following three modules: (1) Tri-plane selection module to pick the best 2D plane to segment the object (Fig. 4a); (2) SAM-based tracking module to predict 2D masks along the selected axis (Fig. 4b); (3) Turning point sampling module to propose potential turning points as new seeds based on existing segmentation results (Fig. 4c) to recursively grow all branches of the blood vessel. Note that the second module can be any existing SAM-based tracking method.

Fig. 4.

TriSAM framework. (a) Tri-plane selection is first proposed to select the best plane for tracking. (b) SAM-based tracking leverages SAM to perform short-term tracking given a seed location and a tracking axis. (c) Recursive seed sampling exploits potential turning points for long-term tracking.

B. TriSAM Framework

We first explain the three modules in the TriSAM framework (Fig. 4) and then provide the algorithmic description (Alg. 1). In practice, we sample many initial seeds, run the TriSAM framework for each seed, and fuse the segmentation results. Tri-plane selection module. Unlike videos, 3D VEM image volumes $I$ can be segmented on 2D planes along different axes, $p\in \{x,y,z\}$, based on the object morphology. However, tracking along some axis $p$ can be harder than others due to the irregular shape of the 2D cross-section of the 3D blood vessels on the corresponding 2D plane $I_p$. We later verify such a phenomenon empirically as shown in Table III in which tracking along the $y$ axis is more effective than tracking along the $x$ axis for a given seed position.

TABLE III.

Ablation studies on different plane selection strategies. The proposed tri-plane approach achieves the best overall accuracy with comparable speed.

Method	Pre. (%)	Rec. (%)	Acc. (%)	Speed (sec)
Single-plane (z)	75.28	48.48	41.82	324
Single-plane (y)	79.37	58.47	50.75	307
Single-plane (x)	69.73	12.14	11.53	345
Single-plane (fusion)	71.78	74.15	57.41	976
Tri-plane	78.41	74.97	62.14	335

This module aims to select the best axis $p^{\ast }$ for tracking, given the SAM segmentation results, SAM $\left(I_p,s\right)$, on each tri-plane $I_p$ centered at the seed position s as the approximation of the blood vessel cross-sections. In theory, the ideal axis for tracking is along the blood flow direction, where the corresponding 2D image plane is tangent to the blood flow direction with the smallest cross-section segment area (Fig. 4a, xy-plane). In addition, SAM outputs the probability, $P_{\text{SAM}}$, for the segment result, reflecting the confidence in the naturalness of its shape. Thus, to combine the two cues above, we pick the axis with the smallest segment size on the 2D plane and a probability of at least the threshold $\tau$ (PLANE-SELECT in Alg. 1).

$$p^{\ast }=\arg \underset{p}{\min }\text{Area}\left(\text{SAM}\left(I_p,s\right)\right),\text{s}.\text{t}.P_{\text{SAM}}\left(I_p,s\right)>\tau .$$ (1)

To avoid an infinite recursion, traditional graph traversal algorithms check if a node is marked visited. In our case, a tubular branch segment may grow into different blood vessel segments along different tracking axes, e.g., bifurcation regions. Thus, we mark the combination of the seed and the tri-plane tracking axis $\left(s,p^{\ast }\right)$ visited if seed $s$ falls in the segment that is predicted through tracking along $p^{\ast }$ axis (VISITED in Alg. 1). Note that $p^{\ast }=\varphi$ if no plane is selected and VISITED($\left(s,p^{\ast }\right)$) is True. To implement the “Visited” function, we store both the segmentation result and the $p^{\ast }$, which is omitted in Alg. 1 for simplicity.

SAM-based tracking module.

Given the seed position s, the selected tracking axis $p^{\ast }$ and SAM’s initial 2D segment mask, we need to produce a 3D segment by tracking the 2D mask with SAM in both directions (e.g., $x+$ and $x-$). Note that naively propagating the mask center or bounding box as the SAM prompt for the next image slice leads to poor segmentation results, as SAM may output segments with inconsistent sizes or shrinking sizes along the axis respectively. Many existing sophisticated SAM-based tracking methods can be directly used.

This module aims to provide a simple yet effective approach by generating better SAM prompts for the next image slice. Empirically, we find prompting SAM with both the enlarged bounding box and the center of the segment generates better segment tracking results (SAM-TRACK in Alg. 1).

$$prompt=\{(x,y),(x,y,\gamma w,\gamma h)\},$$ (2)

where $(x,y,\gamma w,\gamma h)$ is the bounding box of the segmentation and $\gamma$ is the scaling factor. To ensure the quality of the predicted SAM mask, the proposed tracking module terminates when the SAM probability for the predicted mask is lower than the threshold $\tau$.

Turning point sampling module.

As discussed above, SAM-based tracking module can not handle blood vessel segments that change directions or bifurcate, where the 2D segment along the original direction changes drastically. Thus, when the SAM-based tracking module terminates in the middle of the volume, we need to find turning points as new seeds to track segments along other directions. Naively, we can densely sample points from the segmentation result and run the PLANE-SELECT module to find points that prefer other directions. However, such an approach can be inefficient as most points may prefer the original tracking direction, and ineffective due to false turning points caused by SAM errors.

Instead, we design this module to sample turning points around the last point prompt position from the SAM-based tracking module. We first predict 2D SAM segmentation $se{g}_p$ along the other two directions $\left(p\in \{x,y,z\}\setminus \left\{p^{\ast }\right\}\right)$ at the last point prompt position, which has a high probability of capturing the tangent cross-section of the swerving blood vessel. Then, we sample $K$ points from each of the 2D

Algorithm 1.

TriSAM Framework

Require: 3D image volume $I$ and initial seed $s_0$, threshold $\tau$  Initialize the prediction: $\mathbf{P}=\varphi$  Initialize the seed list: $\mathbf{S}=\left\{s_0\right\}$  while $\mathbf{S}\ne \varphi$ do   Take $s$ from $\mathbf{S}$   $p^{*}=\mathrm{PLANE}-\mathrm{SELECT}(I,s)$   if not $\mathrm{Visited}\left(s,p^{*}\right)$ then    $seg=\mathrm{SAM}-\mathrm{TRACK}\left(I,s,p^{*}\right)$    if $seg\ne \varphi$ then     $seeds=\text{TURNINGPOINT-SAMPLE}\left(seg,p^{*}\right)$     $\mathbf{P}=\mathbf{P}\cup seg$     $\mathbf{S}=\mathbf{S}\cup seeds$    end if   end if  end while  return P

segmentation $se{g}_p$ with the Farthest Point Sampling (FPS) method as turning points added to the seed list.

$$seeds={\displaystyle \underset{p\in \{x,y,z\}\backslash \left\{p^{*}\right\}}{\cup }\mathrm{FPS}\left(se{g}_p,K\right),}$$ (3)

where FPS selects well-spaced points from a dataset by starting with a random point and iteratively adding the farthest point from the current selection. This method ensures good spatial coverage and uniformity.

TriSAM Framework Algorithm.

Given the initial seed s₀, the proposed TriSAM framework applies the graph traversal algorithm to segment the whole connected blood vessel segment (Alg. 1). As either breadth-first-search or depth-first-search works, we use a generic “list” data structure $\text{(}\mathbf{S}\text{)}$ to store the seeds to visit. At each step, we take out a seed $s\in \mathbf{S}$, predict the tubular branch with the PLANE-SELECT and SAM-TRACK modules (i.e., visit a node), and then find its neighboring branches to segment with the TURNINGPOINT-SAMPLE module (i.e., add neighbors as new seeds). The TriSAM framework runs recursively until the seed list $\mathbf{S}=\varphi$.

C. Implementation Details

Image Pre-processing.

The large-scale datasets have many image artifacts and missing slices, which degrade the SAM-based tracking results. We utilize the temporal smoothing method along the z-axis to deflicker the images.

Initial seed generation.

Initial seeds can be effectively generated with global color thresholding since the pixels of blood vessels are brighter than the background. To improve efficiency, we only keep the center of each connected component as the final seeds.

SAM details.

Unless stated otherwise, we use MobileSAM [36] instead of the standard SAM [15] in all conducted experiments to improve the inference speed. To clean up SAM results, holes are filled and small connected components are removed through binary morphological operations.

Hyperparamter selection.

As a zero-shot approach, TriSAM sets the hyperparameters to reasonable values instead of using ground truth data. In later ablation studies, we verify the robustness of hyperparameters within the reasonable range. For the global color thresholding for seed generation, $\eta$ is set to 98 percentile. For SAM-based tracking, the probability threshold $\tau$ is set to 0.8.

Inference.

For BvEM-Macaque, we directly predict the whole volume. For others, $k\times 1024\times 1024$ subvolumes are used since 1024×1024 is the default resolution for SAM where $k=661$ for BvEM-Human and $k=818$ for BvEM-Mouse. The subvolume results are later fused to form the final prediction. All experiments are conducted on an NVIDIA-A100 GPU.

V. Experiments

A. Experimental Settings

Evaluation Metrics.

We use the Precision, Recall, and Accuracy metric defined in [48], where $Accuracy=\frac{TP}{TP+FP+FN}$.

$$Precision=\frac{TP}{TP+FP},$$ (4)

$$Recall=\frac{TP}{TP+FN},$$ (5)

$$Accuracy=\frac{TP}{TP+FP+FN},$$ (6)

where $TP$, $FP$, and $FN$ are instance-level true positive, false positive, and false negative respectively. We use instance-level metrics since it is more sensitive to split errors. In particular, the Hungarian algorithm is used to match ground-truth instances and predicted instances with negative Accuracy as the cost matrix. We used the whole dataset for evaluation and computed the score on the largest instance segment of each volume due to its dominating size.

B. Benchmark Results

Methods in Comparison.

We compare the proposed TriSAM with both zero-shot baselines and supervised methods. The compared zero-shot methods are global color thresholding and SAM+IoU tracking. For color thresholding, we first perform (3D) Gaussian blurring with $\sigma =1$ on 3D chunks (10×512×512). Then we label all voxels that are 3 standard derivations above mean as positive. Finally, connected components with less than 1000 voxels are filtered out. For SAM+IoU Tracking, we segment all objects in each z-slice of the dataset using automatic mask generation. Then we track each blood vessel using the first labeled slice as seeds. Our simple tracking algorithm finds the mask in the next slice with maximum IoU with the current slice. If the max IoU is above a threshold, we assign this mask to the current object and continue tracking. We also tried SAM+IoU tracking with microSAM [51] weights that have been finetuned on EM images. This model however does not work well on our dataset. We expect this is because microSAM has been finetuned on high-resolution EM images and does not generalize to our low-resolution dataset. We further compare TriSAM with the supervised method 3D U-Net [11]. We use the implementation from [52]. For supervised methods, we cropped subvolumes at the center of each volume composing approximately 10% in size. These annotated subvolumes were divided into a 1–1 train-val split.

Results Analysis.

The results are shown in Tab. II. First, both the Color Thresholding and SAM+IoU Tracking methods exhibit significant performance variability across three volumes, highlighting the diversity of our dataset and the sensitivity of these methods to different species. Furthermore, both of these unsupervised methods demonstrate relatively poor performance, underscoring the challenges of the zero-shot setting in the BvEM dataset. Additionally, the 3D UNet, as a supervised learning approach, also yields subpar results, indicating poor generalization of models trained with limited data. Finally, TriSAM significantly outperforms other methods as it not only accurately segments the boundary but also tracks the blood vessels in the long term.

TABLE II.

Benchmark results on the proposed BvEM dataset. We evaluate the initial blood vessel annotation to show the significant amount of our proofreading effort.

Method	Setting	BvEM-Mouse			BvEM-Macaque			BvEM-Human
		Pre	Rec	Acc	Pre	Rec	Acc	Pre	Rec	Acc
Initial annotation	N/A	93.74	36.62	35.74	1.65	23.17	1.57	100.00	25.68	25.68
Color thresholding	N/A	86.45	37.32	35.26	95.14	21.65	21.42	41.77	1.92	1.87
MAESTER [49]	unsupervised	2.16	18.95	0.94	22.08	40.30	16.64	0.29	5.03	0.27
3D UNet [11]	supervised	13.45	0.91	0.67	16.56	86.95	16.16	68.63	2.46	2.43
nnUNET [50]	supervised	3.54	49.55	6.59	24.34	29.57	23.74	3.44	23.20	5.86
SAM+IoU tracking [15]	zero-shot	63.59	0.27	0.27	74.39	1.89	1.88	18.19	23.58	12.92
TriSAM (ours)	zero-shot	84.12	66.75	59.28	78.41	74.97	62.14	31.35	25.57	16.39

Qualitative Results.

The final instance segmentation results on BvEM-Macaque are shown in Fig. 5. Color thresholding segments bright pixels, inadvertently capturing nuclei cells while overlooking darker pixels corresponding to blood vessels. Training the 3D UNet model with limited data results in confusion with background elements. IoU tracking fails to capture a significant portion of the blood vessel, revealing its ineffectiveness in tracking. TriSAM prediction emerges as the most accurate method, affirming its effectiveness. We further visualize the TriSAM’s performance on complex vessel structures in both the mouse and human datasets in Fig. 6. These additional examples demonstrate our method’s ability to handle challenging cases, including tortuous vessels and U-turns. While performance may vary depending on vessel complexity, our tri-plane selection strategy effectively adapts to changes in direction by dynamically choosing optimal viewing planes at potential turning points.

Fig. 5.

Qualitative results of instance segmentation on the BvEM-Macaque dataset. Different colors represent different instances of blood vessels. The methods of color thresholding and 3D UNet often produce false positives, meaning they mistakenly identify parts of the image as blood vessels when they are not. On the other hand, SAM+IoU tracking tends to miss many blood vessels altogether. Among all the methods tested, TriSAM segmentation performs the best and is the most accurate in identifying blood vessels.

Fig. 6.

3D visualizations on complex vessel structures from mouse and human datasets.

C. Ablation Studies

We conducted a comprehensive series of ablation studies exclusively using the BvEM-Macaque dataset due to the computation constraints.

Oracle analysis.

We conduct an oracle analysis to showcase the advantage of the proposed TriSAM framework for its capability to deal with complicated vascular geometry. For the SAM-based tracking module in TriSAM, we plug in an oracle 2D segmentation method which returns the connected component of the ground truth mask containing the prompt. As shown in Fig. 7b, the TriSAM framework can perfectly segment the whole vasculature with the oracle 2D segmentation. In comparison (Fig. 7a), existing SAM-based video object tracking methods can only track along one axis, missing bifurcated branches.

Fig. 7.

Results with oracle 2D segmentation.

Tri-plane selection module.

We first compare our method with the single-plane methods to evaluate the effectiveness of the tri-plane. For the single-plane method, we choose one plane as the main plane and only track along the chosen plane. The results on the BvEM-Macaque and BvEM-Mouse volumes are shown in Tab. III and IV. We have observed significant differences in performance among the three single-plane methods, with accuracy ranging from 11.53% to 50.75% on BvEM-Macaque, depending on the chosen tracking plane, which indicates the importance of the chosen tracking plane. This variability can be attributed to the tubular nature of blood vessel extensions within biological organisms, resulting in the generation of intricate mask shapes in certain planes, while simpler mask shapes are produced in others. Then we fuse the results of three Single-Plane methods and attain a higher accuracy, which demonstrates that the segmentation results from different planes exhibit a high degree of complementarity with each other. Instead, We see that tri-plane selection exploits the blood vessel 3D structures by tracking along a suitable plane and attains the highest accuracy of 62.14% and 59.28% on BvEM-Macaque and BvEM-Mouse, respectively.

TABLE IV.

Ablation studies on different plane selection strategies on BvEM-Mouse.

Method	Pre. (%)	Rec. (%)	Acc. (%)
Single-plane (z)	89.05	30.05	28.98
Single-plane (y)	60.44	35.41	28.75
Single-plane (x)	53.20	33.56	25.91
Single-plane (fusion)	49.46	51.93	33.92
Tri-plane	84.12	66.75	59.28

We visualize the segmentation results with one initial seed on BvEM-Macaque in Fig. 8. The origin point is the seed location. We see that the performance is sensitive to the selection of the tracking plane. If the plane is not well-selected, the segmentation result can be empty as the example of tracking along the x-axis shows. The best result is tracking along the y-axis which is still worse than the proposed method since it considers potential turning points to leverage the 3D structure. The best result is tracking along the y-axis. However, it still falls short of the proposed method’s performance, as the latter takes into account potential turning points to exploit the 3D structure and perform long-term tracking.

Fig. 8.

Results for different plane selection strategies without recursive seed sampling.

SAM-based tracking module.

We examine the effect of different SAM variants, i.e., MobileSAM, on the TriSAM performance. As shown in Tab. V, the inference time of MobileSAM is 22% of the original SAM which confirms that MobileSAM significantly improves the inference speed. Moreover, the performance of MobileSAM is even better than the original SAM, possibly because the distilled small model is less prone to overfitting to the original natural image domain.

TABLE V.

Ablation study results on the choice of SAM models. The MobileSAM model achieves better performance with faster inference speed and smaller model size.

Backbone	Pre (%)	Rec (%)	Acc (%)	Speed	Parameters
SAM [15]	77.65	66.03	55.48	1535s	615M
MobileSAM [36]	78.41	74.97	62.14	335s	9.66M

Turning point sampling module.

To perform long-term tracking and fully leverage the 3D blood vessel structure, we introduced recursive seed sampling by considering potential turning points. To validate its effectiveness, we report the results on the BvEM-Macaque volume in Tab. VI where the runtime comparison for segmentation prediction on the entire BvEM-Macaque data is also included. Compared to the baseline without recursive seed sampling “naive” and dense seed sampling “dense”, the proposed method achieves the best performance with less than 10% running time increase. One naive baseline is to remove the recursive seed sampling component and not consider any potential turning points. This strategy is simple and fast but it fails to exploit the 3D shape prior leading to poor performance. Another strategy is to select the best plane for every tracking step/slice, which densely performs SAM segmentation on each step across three planes. Therefore, the performance is limited. The proposed method achieves the best performance with less than 10% running time increase on the BvEM-Macaque dataset. Unfortunately, it significantly increases the computation cost. Compared to our method, DenseSAM’s running time is 5.97 times longer since it needs to segment 3 planes for every tracking step. However, we were surprised to observe that the performance of the Dense Redirection strategy was even worse. This could be attributed to the frequent axis changes potentially leading to the omission of certain parts of the blood vessel and causing splitting errors.

TABLE VI.

ABLATION STUDIES ON DIFFERENT SEED SAMPLING STRATEGIES. THE PROPOSED RECURSIVE SEED SAMPLING APPROACH ACHIEVES THE BEST OVERALL ACCURACY WITH COMPARABLE SPEED.

Strategy	Pre. (%)	Rec. (%)	Acc. (%)	Speed (sec)
Naive	79.37	58.47	50.75	307
Dense sampling	86.94	24.84	23.95	2001(↑552%)
Recursive sampling	78.41	84.97	62.14	335 (↑ 9%)

Tri-plane dynamics.

We delve deeper into the plane dynamics in Fig. 9 and 10. The majority of the selected plane predominantly tracks along the y-axis. This observation aligns with the experimental results presented in Tab.III, where it is evident that a single plane tracking with the y-axis outperforms the z-axis and x-axis. This is because the blood vessel flows mainly along the y-axis in the test volume. In Fig. 11, we explore the size dynamics using various methods, where the mean derivation is shown in parentheses. The size variation observed with our proposed method is relatively smaller compared to tracking along the y-axis and significantly less than when tracking along the z-axis and x-axis.

Fig.9.

Histogram of the tri-plane selection.

Fig.10.

Dynamics of the tri-plane selection.

Fig.11.

Dynamics of segmentsize.

Initial seed planes.

We then conducted experiments by selecting seeds from the X-, Y-, and Z-planes, respectively. As shown in Table VII, using the Z-plane for initial seed generation consistently achieved the best overall performance in terms of precision, recall, and accuracy. This demonstrates that the Z-plane provides a better starting point for vessel tracking, likely due to the natural imaging orientation and more consistent vascular cross-sections in the Z-direction.

TABLE VII.

COMPARISON OF DIFFERENT PLANES FOR GENERATING INITIAL SEEDS.

Method	Pre. (%)	Rec. (%)	Acc. (%)
initial seed (z)	78.41	74.97	62.14
initial seed (y)	86.73	28.16	27.00
initial seed (x)	81.32	13.19	12.80

Therefore, all initial seeds in TriSAM are generated from the Z-plane to ensure optimal segmentation performance.

Hyperparameters.

We compute the accuracy of TriSAM with varying hyperparameters for seed generation $(\eta )$ and SAMbased tracking module $(\tau )$. As shown in Fig. 12, TriSAM’s performance is robust within the reasonable range of hyperparameters.

Fig. 12.

Ablation study results on hyperparameters $\eta$ and $\tau$.

D. Failure Cases

The main source of error in the proposed TriSAM framework is the SAM-based tracking module, where SAM can not produce accurate 2D mask given the prompt. For the false negative case (Fig. 13a), SAM predicts an empty mask when the shape and appearance of the blood vessels are complex (e.g., conjunction point) for all three axes. For the false positive case (Fig. 13b), SAM falsely segment the neighboring region with the blood vessel due to the similar image appearance.

Fig. 13.

Failure cases of the SAM-based tracking module.

VI. Conclusion

In this paper, we contribute the largest-to-date public benchmark, the BvEM dataset, for cortical blood vessel segmentation in 3D VEM images. We also developed a zero-shot blood vessel segmentation method, TriSAM, based on the powerful SAM model, offering an efficient and accurate approach for segmenting blood vessels in VEM images. With tri-plane selection, SAM-based tracking, and recursive seed sampling, our TriSAM effectively exploits the 3D blood vessel structure and attains superior performance compared with existing zero-shot and supervised technologies on BvEM across three species, marking a critical step towards unlocking the mysteries of neurovascular coupling and its implications for brain health and pathology. With the availability of the BvEM dataset and the TriSAM method, researchers are now equipped with valuable tools to drive breakthroughs in VEM-based cortical blood vessel segmentation and further our understanding of the brain’s intricate vascular network. By addressing a significant gap in the field of neuroimaging, we have laid the foundation for advancing the understanding of cerebral vasculature at the microscale and its intricate relationship with neural function.