tUbeNet: a generalizable deep learning tool for 3D vessel segmentation

Abstract

Deep learning has become an invaluable tool for bioimage analysis but, while open-source cell annotation software such as Cellpose is widely used, an equivalent tool for three-dimensional (3D) vascular annotation does not exist. With the vascular system being directly impacted by a broad range of diseases, there is significant medical interest in quantitative analysis for vascular imaging. We present a new deep learning model, coupled with a human-in-the-loop training approach, for segmentation of vasculature that is generalizable across tissues, modalities, scales, and pathologies. To create a generalizable model, a 3D convolutional neural network was trained using curated data from modalities including optical imaging, computational tomography, and photoacoustic imaging. Through this varied training set, the model was forced to learn common features of vessels’ cross-modality and scale. Following this, the pre-trained ‘foundation’ model was fine-tuned to different applications with a minimal amount of manually labelled ground truth data. It was found that the foundation model could be specialized to a new datasets using as little as 0.3% of the volume of said dataset for fine-tuning. The fine-tuned model was able to segment 3D vasculature with a high level of accuracy (DICE coefficient between 0.81 and 0.98) across a range of applications. These results show a general model trained on a highly varied data catalogue can be specialized to new applications with minimal human input. This model and training approach enables users to produce accurate segmentations of 3D vascular networks without the need to label large amounts of training data.

Introduction

A broad range of diseases directly involve the vascular system, and as such, the organization and structure of blood vessels are of inherent medical interest. For example, blood vessel networks become highly disrupted following angiogenesis in cancer [1], inflammation in response to infection induces changes in vascular structure [2], and blockages causing loss of perfusion has serious consequences in cardiac infarction [3], stroke [4], and diabetes [5]. Many bioimaging techniques have been developed to characterize both normal and diseased blood vessel networks, across a range of length scales, organs, and pathologies. However, the ability to quantify and fully characterize the complex, hierarchical nature of blood vessels still presents a significant challenge, particularly when consideration of their three-dimensional structure is required.

Deep learning (DL) with convolutional neural networks has rapidly become the state of the art approach for semantic segmentation (labelling each pixel according to a predefined set of classes) in medical imaging due its accuracy and generalizability. However, it usually requires a large and varied array of training data, complete with paired ground truth labels. Whilst image data is generally widely available, acquiring manually defined labels is a substantial and often challenging undertaking and introduces a significant bottleneck to implementation. Accordingly, most DL models to date have been trained using a limited set of data from a single pathology or organ, imaged with a single modality, at a single length scale. In particular, much of the field is focused on segmenting retinal vasculature from two-dimensional colour fundus images [6–10]. Models have also been created for segmenting brain vasculature in three dimensions [11, 12]. Typically, each new application requires a new model, trained on new data.

In the field of cellular imaging, Cellpose [13] and Cellpose 2.0 [14] have become highly popular open-source tools that employ an alternative, a human-in-the-loop workflow: where a general model is trained most of the way and then fine-tuned on a small amount of the end-users’ data in order to specialize the model to their needs. This approach minimizes manual labelling and increases the accessibility of deep learning tools for users without the computational resources or sufficient labelled data to train a model from scratch. Similar approaches are also available for vessel segmentation in the form of proprietary software (for example, the Imaris 10.0 Filament Tracer and the Amira-Avizo XFibre extension), but no open-source tools exist for this purpose.

In this study, we aimed to leverage the ubiquitous tube-like structuring of blood vessels to create a model that can be used to segment three-dimensional image data containing blood vessels, acquired with multiple modalities and from multiple organs. This is achieved by implementing a training strategy that prioritizes a highly varied multi-modal training set over volume of data. Employing this strategy, we develop a generalizable vessel-finding model with far less pre-training data than typically reported and demonstrate that it can be fine-tuned to new tasks with a minimal amount of user-labelled data.

Semantic segmentation with U-Nets

Convolutional neural networks (CNN) consist of a sequence of convolutional layers (a deep network) with weights that can be trained to enable it to detect image features at multiple length scales. U-Nets are a class of CNN architecture consisting of encoding layers that feed into a symmetric set of decoding layers [15]. In the encoding arm, alternating convolutional and pooling layers downsample the data and convert the input data into a latent representation. Once encoded, the decoding arm converts the latent representation into a new image space, by sequential upsampling using deconvolutional (or transpose convolutional) layers. The downsampling and upsampling branches are symmetric, giving a ‘U’-shaped architecture. Skip connections enable direct communication between corresponding encoding and decoding layers, primarily to help overcome the vanishing gradient problem that affects very deep CNNs and has been found to improve the accuracy of segmentation of biological images [16].

Bottlenecks in ML-based segmentation

Collating sufficient data to accurately train a deep neural network is one of the major challenges in deep learning. In the field of medical imaging, there is generally no shortage of raw data, but the corresponding ground truth labels are more challenging to obtain. Images must be manually segmented to create paired datasets, which is highly labour-intensive and potentially error-prone [10]. The training data must also comprise a representative sample of the range of the data the model will be expected to classify upon application, to enable sufficient generalizability [17].

Multiple approaches have been developed to overcome these challenges. In addition to creating architectures that require less training data, such as the U-Net, researchers have attempted to make smaller datasets go further using data augmentation [18, 19] and have employed pre-training (or transfer learning) strategies [20–22]. Pre-training leverages the ability of CNNs to detect features that are characteristic of a broad range of image domains; as such, a CNN trained on one domain can be transferred to another domain using less data than would be required to train it from scratch. Similarly, synthetic data has been used to address training bottlenecks: models have been trained using either purely synthetic data [23, 24] or a combination of real and synthetic data [25, 26], for which the ‘ground truth’ labels are inherently known. However, the extent to which models trained on synthetic data may be meaningfully applied to physiological data remains an open question [27].

Semantic segmentation of blood vessels

CNNs have been successfully used to label blood vessels in two-dimensional (2D) images and have been widely applied in the segmentation of retinal vasculature from image data [6–10]. This is aided by the fact that several hand-labelled data sets are available to allow developers to benchmark new models and approaches (e.g. DRIVE [28], Structured Analysis of the Retina (STARE) [29], and FIVES [30]).

Segmentation of three-dimensional (3D) vasculature, on the other hand, presents a greater challenge due in part to the much higher demands of manual segmentation, higher computational expense, and the lack of ready-labelled libraries against which to benchmark. Furthermore, much of the research in this area has been limited to brain vasculature; for example, previous study [11] used a 3D CNN to segment the vascular network from dual-stained mouse brains, imaged using light-sheet microscopy, in their VesSAP pipeline. Likewise, DeepVesselNet [12] was trained with two sets of brain data: clinical MRA and rat brain micro-CT. There are few examples of applications outside of the brain [31], which demonstrated the effectiveness of an integrated U-net and graphical neural network on two clinical contract-enhanced liver CT datasets. To our knowledge, no work has been published in which DL is used to segment 3D microvasculature outside of the brain.

While these examples point to many potential applications for 3D CNN image analysis, both in research and in the clinic, there is clearly a need for models that can be applied to a wider range of tissues and data sources. In particular, there is a need for models that can segment vasculature at the micron scale in organs other than the brain and retina. Here, we explore the use of transfer learning to address bottlenecks in blood vessel segmentation and aim to create a ‘generalized’ 3D vessel segmentation model that can be fine-tuned to multiple applications.

Methods

Model architecture

A convolutional neural network was produced in tensorflow version 2.20.0 [32] by adapting the U-Net architecture for 3D data. The model consists of six convolution block followed by five deconvolution blocks. The convolutional blocks (purple, Fig. 1) consist of two 3D convolution layers (kernel size 3 $\times$ 3 $\times$ 3) followed by a Leaky ReLU activation function (alpha = 0.2) and group normalization. A pooling function follows each block, downsampling the data by a factor of two in each dimension—effectively halving the resolution. Deconvolution blocks (orange) increase the resolution of the data using a deconvolving function (‘conv3DTranspose’), followed by two standard convolution layers, again with a Leaky ReLU activation function. The output layer is a convolution with a single filter and a softmax activation function to give a pseudo-probabilistic output for each pixel belonging to each class (i.e. inside or outside a vessel). Concatenations are used to ‘skip’ layers by transferring feature maps directly from the encoding branch to the symmetric layer of the decoding branch, which has been shown to improve convergence speed in deep networks [16]. A 30% dropout is included after each layer to reduce the risk of overfitting to the training data.

Figure 1.

(a) This schematic shows the architecture of the tUbeNet model: the purple and blue boxes represent multi-channel feature maps, with the number of channels denoted by the number above the box. The dimensions to the side of each layer give the x-y-z dimensions of the feature maps in that layer. The network takes an image (grey) as an input and gives a map of probabilities as an output (black). The arrows represent operations, as summarized in the image key. (b) An overview of the data pre-processing pipeline, and method of loading data subvolumes from the disk storage during training. Utilising the Zarr file format and dask lazy loading, the model can be trained on larger datasets that cannot fit within the computer’s RAM. The custom data header is automatically generated during preprocessing provides the necessary metadata for the training script. (c) Example images taken from the three-dimensional image data used in training. 3D vessel images from four different modalities, and three different tissue types, were manually segmented to create a varied training set. Here, a 2D slice is shown from each dataset (top row), along with the corresponding manual labels (bottom row).

Attention blocks

A second version of the model was created in which skip connections were replaced with attention layers. Attention mechanisms are used to capture correlations in the data across scales or channels. Here, an attention map was computed between feature maps of the same resolution in the encoding and decoding branches, similar to the method proposed in a previous study [33], but using the Luong model of attention [34] rather than additive attention. The attention map for a given layer of the U-net is produced by first transforming both the encoding branch feature map (q) and decoding branch feature map (k) via a convolution with a trainable 3 × 3 × 3 kernel. The resulting matrices are combined and then passed through another convolution layer with a 1 × 1 × 1 kernel and sigmoid activation to produce the final attention map. The attention map was then multiplied element-wise with the decoding branch feature map.

Model hyperparameters

Model hyperparameters were determined using a random search function (15 iterations) to compare a range of initial learning rates (0.00001, 0.0001, 0.001), activation functions (ReLU, Leaky ReLU alpha = 0.1, Leaky ReLU alpha = 0.2, Leaky ReLU alpha = 0.3), and dropout rates (10%, 20%, 30%). Evaluating the model performance on hold-out data (see section ‘Training, testing and validation data’), the optimal combination of hyperparameters was found to be an initial learning rate of 0.001, leaky ReLU activation function with alpha = 0.2, and a dropout rate of 30%.

Correcting for class imbalance

In classification problems, where one class is more common than the other, ML algorithms can become biased towards the better represented class. This is common in image data containing blood vessels, as blood volume in biological tissue typically 3%–5%. To counter this, the training data generator was weighted towards batches of image subvolumes containing at least 0.1% vessels. This method meant that the model would still see some batches containing no vessels, but far less often than would otherwise be the case.

Three loss functions designed to handle imbalanced classes were compared: weighted categorical cross entropy (WCCE), DICE binary cross entropy (DICE BCE), and focal loss. WCCE used pixel-wise cross entropy, a measure of similarity between distributions, commonly used for segmentation and classification tasks, and scales the contribution of each pixel based on the ground truth class. Class weights were assigned in proportion to the inverse of the class size. DICE-BCE is a sum of the DICE coefficient and Binary Cross Entropy (BCE). The DICE coefficient is robust to imbalanced classes as it takes into account recall and precision, while BCE provides a smooth gradient for backpropagation [35]. Focal loss is designed to more highly weight pixel classifications that the model is ‘uncertain’ about, allowing the training to focus on sparse examples over easily classified background pixels. This is achieved by scaling standard cross entropy by a factor of ${(1-p)}^y$, where p is the model-estimated probability of the pixel belonging to the ground truth class, and y is a tuneable hyperparameter [36].

Training, testing, and validation data

A training library of paired images and manual labels was compiled from three-dimensional image data acquired both in-house and externally. Images from four modalities were chosen to represent a range of sources of contrast, imaging resolutions, and tissues types (Training 1–4). As such, the training set encompasses a huge amount of variability with relatively little data. Volumes representing 25% of both Training 1 and Training 2 were held out for model evaluation during hyperparameter tuning and training. An additional 3D MRI dataset (Test 1) provided unseen test data for model evaluation during training. Finally, additional data from modalities not included in the training/testing libraries were used to validate the cross-modality generalizability of the model (Validation 1–3). A summary of all datasets, with image sizes in pixels, is given in Table 1. Details of how the images were generated are given below:

Table 1.

Summary of data used in training and validation.

Dataset	Modality	Tissue	Volume (pixels)	Volume (mm)	Voxel size ($\mu$ m)
Training 1	Eosin HREM	Mouse liver	4080$\times$3072$\times$416	9.2$\times$6.9$\times$0.72	2.25 $\times$ 2.25 $\times$1.72
Training 2	CT	Mouse	1000$\times$1000$\times$682	20$\times$20$\times$13	20$\times$20$\times$20
		Subcutaneous
		Tumour
Training 3	RSOM	Mouse	191$\times$221$\times$400	3.8$\times$4.4$\times$1.6	20$\times$20$\times$4
		Subcutaneous
		Tumour
Training 4	OCT-A	Human retina	500$\times$500$\times$64	12$\times$12$\times$1.5	24$\times$24$\times$24
Test 1	MRI	Porcine liver	400$\times$400$\times$15	360$\times$360$\times$75	900$\times$900$\times$5000
Validation 1	MF-HREM	Mouse	2659$\times$2034$\times$1561,	9.2$\times$6.9$\times$2.7	3.46$\times$3.4$\times$1.72
		Subcutaneous	From which a subvolume of
		Tumour	480$\times$480$\times$480 was
			used in fine-tuning
Validation 2	MF-HREM	Mouse cortex	4080$\times$3072$\times$99	2.3$\times$1.75$\times$0.17	0.56$\times$0.56$\times$1.72
Validation 3	Two photon	Mouse	1536$\times$1536$\times$79	0.3$\times$0.7$\times$0.4	0.2$\times$0.5$\times$5
	Microscopy	Olfactory bulb	From which a subvolume of
			500$\times$250$\times$79 was
			used in fine-tuning

1. Training 1: A high-resolution episcopic microscopy (HREM) dataset, acquired in house by staining a healthy mouse liver with Eosin B and imaged using a standard HREM protocol [37].

2. Training 2: X-ray microCT images of a microvascular cast, taken from a subcutaneous mouse model of colorectal cancer (acquired in house).

3. Training 3: Raster-Scanning Optoacoustic Mesoscopy (RSOM) data from a subcutaneous tumour model (provided by Emma Brown, Bohndiek Group, University of Cambridge) [38].

4. Training 4: retinal angiography data obtained using Optical Coherence Tomography Angiography (OCT-A) (provided by Dr Ranjan Rajendram, Moorfields Eye Hospital) [24].

5. Test 1: T1-weighted Balanced Turbo Field Echo Magnetic Resonance Imaging (MRI) data from a machine-perfused porcine liver (provided by Dr Zainab Rai of the Royal Free Hospital London NHS Foundation Trust and UCL Hawkes Institute) [39].

6. Validation 1: a subcutaneous colorectal tumour mouse model was imaged in house using Multi-fluorescence HREM in house, with Dylight 647 conjugated lectin staining the vasculature [37].

7. Validation 2: an MF-HREM image of the cortex of a mouse brain, stained with Dylight-647 conjugated lectin, was acquired in house [23].

8. Validation 3: two-photon data of mouse olfactory bulb blood vessels, labelled with sulforhodamine 101, was kindly provided by Yuxin Zhang at the Sensory Circuits and Neurotechnology Lab, the Francis Crick Institute [40].

Animal imaging data acquired in house (Training 1-2, Validation 1-2) was collected according to a UK Home Office project license (PPL: PP9134635) in compliance with the UK Animals (Scientific Procedures) Act 1986 and the UK National Cancer Research Institute (NCRI) guidelines. For data provided by collaborators, details of image acquisition protocols and ethics approval can be found in the associated references.

Manual labelling of blood vessels was carried out using Amira (2020.2, Thermo-Fisher, UK). Figure 1 shows representative 2D sections from the training and test images alongside the manual labels.

Training

Training data were prepared by normalizing pixel intensities between zero and one. Optionally, the data can be cropped at this point; for example, in datasets where only a small region was of interest. Pixel data and true labels were converted into dask arrays and saved in Zarr file format [41]. A custom header class was created to store all relevant metadata to pass to the machine learning model during training and inference (Fig. 1).

A major obstacle when working with large 3D images is management of computer memory. The Dask python library [42] was used to facilitate lazy loading of image subvolumes: instead of loading entire data sets into memory, subvolumes were read in from the disk at the start of each training step. The Zarr file format is optimized for efficiently handling of large datasets through chunked data storage. In this case, a chunk size of 64 × 64 × 64 pixels was chosen.

During training, batches of image subvolumes (64 × 64 × 64 pixels) were read from randomly selected datasets at random coordinates within the training data (note: random datasets selection was weighted in a ratio of 5:4:1:1 for Training sets 1–4, respectively, to avoid over-fitting to smaller datasets). This method of reading image subvolumes from random coordinates doubled as a translation augmentation. Additional augmentations were then performed on the generated subvolumes, randomly selecting from the following: rotations in the xy plane between –30° and 30°; rescaling by a factor between 1.0 and 1.25; and reflection in one or both of the in-plane axes.

A range of optimisers and learning rate schemes were compared to improve speed of convergence and final accuracy of the model, with the Adam optimiser [43] and an exponential decay learning rate schedule ultimately being chosen for training. The model was trained with a batch size of six, which was the maximum that could be stored in the VRAM of the GPUs used for training, and stopped when no further improvement was seen in the accuracy of the validation set. Training took approximately 24 h on two 8 GB Nvidia GeForce GTX 1080 GPUs. Peak memory usage was 13.38 GB when trained on two GPUs and 5.49 GB when training on a single GPU. Inference time was 222 ms per 64 × 64 × 64 volume when run on a single GPU.

Evaluation

Model performance was measured by calculating the area under the Receiver Operating Characteristic (AUROC) curve, Average Precision (AP), and DICE score. Precision and recall were calculated voxel-wise. The ability of the model to generalize to new data was assessed by calculating AUROC, AP, and DICE scores for manually labelled subvolumes of Validation images 1, 2, and 3 (in the case of Validation 1 and 3, where fine tuning was performed, a manually labelled subvolume was withheld for evaluation).

Comparison to state of the art

VesSAP, a machine learning model for 3D vascular segmentation of light-sheet microscopy data [11], was applied according to the instructions available alongside the open-source code repository. Preprocessing and use of the crosshair filter were both enabled, and the batch size was set to 6 in accordance with the batch size used to train tUbeNet. Otherwise, all training options were left as their default values. The fully trained model weights were used as the ‘base model’, which was finetuned to create two updated model—one finetuned with MF-HREM data and one with two-photon data. As VesSAP was originally trained on two-channel image data (both channels representing vessel markers, Wheat Germ Agglutinin and Evans Blue, respectively), input data was duplicated in the channel axis to matched the expected input shape.

Additionally, tUbeNet was compared to Vista3D, a biomedical image segmentation foundation model [44]. Applying their zero-shot pipeline, the GUI provided in the code repository was used to select at least 20 points within the positive class and 5 points within negative class per validation image. As blood vessels are not a supported class, the class label was set to > 133 (as recommended in the zero-shot instructions).

3D rendering

3D rendering of image data and segmentations was performed in Amira (2020.2, Thermo-Fisher, UK). Amira’s auto-skeletonize tool (smoothing coefficient = 0.5, attach to data coefficient = 0.25) was used to generate skeletonized networks from vessel segmentations.

Results

Accounting for class imbalance

Three loss functions designed to address imbalanced classes were compared in order to identify the best loss function for the tUbeNet model. Training with the weighted categorical crossentropy (WCCE) loss function produced the most accurate segmentation, achieving an AUROC of 0.9 on the validation data (Fig. 2), but required iterative hyperparameter optimization to identify the best combination of weights (in this case, 1:7). DICE-Binary crossentropy (DICE-BCE) outperformed focal loss (AUROC of 0.86 versus 0.80) and had the advantage of not requiring hyperparameter tuning. As such, DICE-BCE is used throughout this work unless otherwise stated.

Figure 2.

Comparison of ROC curves for three different loss functions (WCCE, DICE-BCE and focal loss), for two validation datasets. The left-hand panel shows the central slice of example 64 × 64 × 64 volumes, along with the ground truth and predicted labels for each model. DICE BCE outperformed both the WCCE and focal loss functions, with focal loss producing the lowest AUROC score and visibly worse segmentations.

Cross-branch attention

The inclusion of a cross-branch attention mechanism sped up the initial rate at which the model learned the segmentation, with a AUROC score of 0.9 being achieved within 60 epochs, versus 200 epochs for the model without attention (Fig. 3). However, the attention model did not improve beyond 60 epochs, and both model versions converged at approximately the same level of accuracy. This result differs from the marked improvement in segmentation accuracy seen in previous studies. Despite converging in fewer epochs, the attention model had a much higher computational cost and a slower inference step. For these reasons, the standard model structure without attention was preferred and will be used throughout the rest of this work.

Figure 3.

(a) A schematic illustration of the attention method used in this model. Concatenation operations, represented by the grey arrows in Fig. 1, were replaced with an attention layer as shown here. The feature maps from corresponding encoding and decoding layers (q and k, respectively) are convolved with trainable kernels to produce Wq and Wk. These outputs are combined to find the dot product. A final convolution and sigmoid activation function produces the attention map for this layer, which is multiplied by the decoding feature map. (b) Receiver operating characteristic curves for the model with (top) and without (bottom) the inclusion of attention layers. The inclusion of attention did not improve segmentation quality when tested on the two validation datasets.

Cross-modality generalization

Through training on a library of data from different imaging modalities, the aim was to create a vessel-labelling model that would generalize to any volumetric dataset, even those acquired by previously unseen modalities, with minimal fine tuning. To test whether this aim had been achieved, tUbeNet was evaluated on positive-contract MF-HREM images of a subcutaneous hypopharyngeal cancer model (Validation 1 in Table 1) [37]. This modality was chosen due to the relative newness of MF-HREM: there is very little labelled data available for training, meaning existing deep learning approaches would not be suitable for analysing this data.

Applying the trained model without fine-tuning to this new data type, we produced a segmentation of the vasculature with high recall of the vessel class (96%) but low precision (23%) (2 s.f.). Erroneous labelling of out-of-plane fluorescence as vasculature was noted, leading to the low precision score (Fig. 4). However, after 2 h of fine-tuning the model using a labelled subvolume of the data (120 $\times$ 120 $\times$ 120 pixels) representing just 0.3% of the total image volume, the recall and precision were improved to 0.98 and 0.97, respectively (DICE score = 0.97, AP = 0.87) (2 s.f.) (Table 2). Increasing the train set size to 480 $\times$ 480 $\times$ 480 pixels (1.2% of the total image volume) further improved the segmentation accuracy [precision 0.99, recall 0.97, DICE 0.98, AP 0.97 (2 s.f.)]. This subvolume was manually annotated in a matter of hours, compared to the days/weeks that would be required to label the entire dataset. This result demonstrates that our pre-trained model can be fine-tuned to new data with a minimal amount of manual annotation, making it a practical option for applications where large libraries of labelled data do not exist.

Figure 4.

(a) 2D sections from an MF-HREM dataset of a fluorescence-stained subcutaneous tumour (top row) are shown alongside the corresponding manual labels (second row) and the predicted labels generated by the pre-trained model prior to fine-tuning (third row) and following fine-tuning (bottom row). (b) A skeletonized subvolume of the tumour and corresponding vessel radii distribution (c). (d) 3D views of the entire dataset segmented using the fine-tuned model.

Table 2.

DICE and Average Precision (AP) scores achieved on test data following fine-tuning of the tUbeNet model on different-sized subvolumes of labelled data.

Training volume	DICE score	AP
120$\times$120$\times$120	0.97	0.87
240$\times$240$\times$240	0.98	0.94
480$\times$480$\times$480	0.98	0.97

Moreover, the fine-tuned model was then applied to another MF-HREM dataset from a different tissue: the cortex of a mouse brain stained with DyLight-647 conjugated lectin (Validation 2 in Table 1) (Fig. 5). Despite differences in cortical and tumour vasculature, the model was able to generalize to the cortex dataset with a high level of accuracy and no additional fine-tuning [recall = 0.97, precision = 0.82, DICE score = 0.89 (2 s.f.)]. This suggests that fine-tuning may only need to be done once per image type, further saving on manual labelling.

Figure 5.

(a) 2D section of an MF-HREM image of the cortical vasculature of a mouse. (b) 3D vessel segmentation produced by the tUbeNet model fine-tuned using the tumour MF-HREM dataset seen in Fig. 4. The red square highlights a region of the data that was manually labelled to provide a ground truth for validation. (c) A Precision–Recall curve was calculated using the manually labelled ground truth (d) and model predictions for this subvolume (e) show that the prediction is highly accurate [AP = 0.97 (2 s.f.)].

Finally, the model was applied to images of the blood vessels of the mouse olfactory bulb, acquired by two photon microscopy (Validation 3 in Table 1) (Fig. 5) [40]. This dataset provided an additional challenge as, not only it an imaging modality that the model had not previously seen, but it exhibits spatially varying imaging artefacts due to resolution drop off with depth. In order to specialize the model to this data, a subvolume that spanned the whole z-axis (79 virtual sections) was manually labelled for fine-tuning. Training on a volume of 79 $\times$ 500 $\times$ 250 pixels (5% of the whole image volume) resulted in a model that labelled vasculature with a precision of 0.80 and recall of 0.83 (DICE = 0.81) (2 s.f.) (Fig. 6).

Figure 6.

(a) Two photon fluorescence microscopy data showing labelled vasculature in the mouse olfactory bulb. (b) A Precision–Recall curve plotted following fine-tuning on a manually labelled subvolume of the data shows that the model is accurately labelling vasculature [AP = 0.90 (2 s.f.)]. (c) A 3D visualization of the predicted vessel labels. (d) Skeletonized vasculature, colour coded by vessel radius.

Comparison to state of the art

To evaluate the effectiveness of tUbeNet as a generalizable model, the cross-modality validation results presented above must be compared to the performance of existing 3D segmentation models. To this end, our three validation datasets (Table 1) were segmented using the pre-trained VesSAP model [11], originally trained on a combination of synthetically generated vessels and light-sheet microscopy data. The VesSAP model was then fine-tuned in the same manner as above, using a small manually labelled image subvolume for training and another for evaluation. It was found that VesSAP did not readily generalize to the validation data sets; segmentation accuracy improved after fine-tuning but remained low compared to tUbeNet (Table 3).

Table 3.

DICE, Average Precision (AP), and area under ROC curve (AUROC) achieved on validation datasets using tUbeNet, VesSAP, and Vista3D models. The best performing model in each category is highlighted in bold.

Modela	MF-HREM tumour			MF-HREM brainb			Two-photon
	DICE	AP	AUROC	DICE	AP	AUROC	DICE	AP	AUROC
tUbeNet base model	0.37	0.72	0.99	0.73	0.90	0.99	0.70	0.82	0.98
tUbeNet fine-tuned	0.98	0.94	0.99	0.89	0.97	0.99	0.81	0.90	0.98
VesSAP base model	0.26	0.14	0.80	0.00	0.08	0.50	0.15	0.08	0.65
VesSAP fine-tuned	0.33	0.28	0.68	0.31	0.30	0.66	0.37	0.27	0.81
Vista3D zeroshot model	0.04	0.06	0.51	0.15	0.08	0.54	0.21	0.10	0.57

^aFor tUbeNet and VesSAP, both the pre-trained (base) model and fine-tuned model are included.

^bMF-HREM Brain data was segmented using models fine-tuned with MF-HREM tumour data.

Additionally, tUbeNet was compared to Vista3D, a biomedical image segmentation foundation model that incorporates a human-in-the-loop training approach [44]. Using their zero-shot pipeline, the segmentation accuracy was consistently outperformed by tUbeNet (Table 3). However, this may be expected as the model was designed for segmenting larger anatomical features from CT data. Currently, there is no equivalent open-source model for vascular segmentation.

Discussion

In this work, a 3D U-net for vessel segmentation was trained on a highly varied dataset, comprised of four imaging modalities and three tissue types. The general model was then specialized for new applications—segmenting data from previously unseen modalities—with minimal fine-tuning. Our results show that this two-step training approach to deep learning for image analysis is a viable strategy. Moreover, we demonstrate the value of training a model on highly varied data to encourage generalizability. This is seen in the favourable performance of tUbeNet compared to models that were only trained on one imaging modality, even after fine-tuning. Fine-tuning the general tUbeNet model with a small subvolume of the new data of interest (as little as $<1\%$ of the data volume) was sufficient to generate highly accurate label predictions for the entire dataset. This approach to deep learning is especially advantageous when analysing data from new or lesser-used imaging modalities, where a back catalogue of data is not available to train a model from scratch.

In the future, this model could be made more accessible to non-computational researchers creating a complementary graphical user interface for manual labelling and fine-tuning, or by integrating it into existing open-source data visualization programs (for example, as a Napari plug-in). Overall, the move towards an open-source, generalizable DL tool for vessel segmentation expands the options available to researchers looking to quantify vasculature in 3D and overcomes many of the bottlenecks preventing wider use of DL in the 3D imaging community.

Author contributions

Natalie Aroha Holroyd(Conceptualization [Lead], Data curation [Lead], Formal analysis [Lead], Methodology [Lead], Writing—original draft [Lead], Writing—review & editing [Equal]), Zhongwang Li(Methodology [Supporting]), Claire Walsh(Data curation [Equal], Supervision [Equal], Writing—review & editing [Supporting]), Emmeline E Brown(Methodology [Supporting], Writing—review & editing [Equal]), and Rebecca Shipley(Funding acquisition [Equal], Resources [Equal], Writing—review & editing [Equal]), Simon Walker-Samuel(Funding acquisition [Equal], Methodology [Equal], Resources [Equal], Supervision [Equal], Writing—review & editing [Equal]).

Conflict of interest statement: None declared.

Funding

This research was funded by Cancer Research UK (C44767/A29458 and C23017/A27935) and EPSRC (EP/W007096/1). Dr Walsh was supported by an MRC Skills Development fellowship (MR/S007687/1).

Data availability

The code used in this study is available on Zenodo (https://doi.org/10.5281/zenodo.15683547), and all data used for training, validation, and testing are accessible via the UCL Research Data Repository (https://doi.org/10.5522/04/25715604.v1). The pre-trained model weights are also available through the same repository (https://doi.org/10.5522/04/25498603.v2).