WSA-MP-Net: Weak-signal-attention and multi-scale perception network for microvascular extraction in optical-resolution photoacoustic microcopy

Abstract

The unique advantage of optical-resolution photoacoustic microscopy (OR-PAM) is its ability to achieve high-resolution microvascular imaging without exogenous agents. This ability has excellent potential in the study of tissue microcirculation. However, tracing and monitoring microvascular morphology and hemodynamics in tissues is challenging because the segmentation of microvascular in OR-PAM images is complex due to the high density, structure complexity, and low contrast of vascular structures. Various microvasculature extraction techniques have been developed over the years but have many limitations: they cannot consider both thick and thin blood vessel segmentation simultaneously, they cannot address incompleteness and discontinuity in microvasculature, there is a lack of open-access datasets for DL-based algorithms. We have developed a novel segmentation approach to extract vascularity in OR-PAM images using a deep learning network incorporating a weak signal attention mechanism and multi-scale perception (WSA-MP-Net) model. The proposed WSA network focuses on weak and tiny vessels, while the MP module extracts features from different vessel sizes. In addition, Hessian-matrix enhancement is incorporated into the pre-and post-processing of the input and output data of the network to enhance vessel continuity. We constructed normal vessel (NV-ORPAM, 660 data pairs) and tumor vessel (TV-ORPAM, 1168 data pairs) datasets to verify the performance of the proposed method. We developed a semi-automatic annotation algorithm to obtain the ground truth for our network optimization. We applied our optimized model successfully to monitor glioma angiogenesis in mouse brains, thus demonstrating the feasibility and excellent generalization ability of our model. Compared to previous works, our proposed WSA-MP-Net extracts a significant number of microvascular while maintaining vessel continuity and signal fidelity. In quantitative analysis, the indicator values of our method improved by about 1.3% to 25.9%. We believe our proposed approach provides a promising way to extract a complete and continuous microvascular network of OR-PAM and enables its use in many microvascular-related biological studies and medical diagnoses.

Graphical abstract

1. Introduction

Photoacoustic imaging (PAI) can non-invasively detect the internal information of the biological tissues from cells to organs based on the photoacoustic effect, which is of great significance for non-destructive imaging research in life science [1], [2], [3]. Due to the high blood absorption in the near-infrared region, PAI exhibits unique superiority in imaging vascular structures without exogenous contrast agents. In addition, the PAI also obtains the functional information of tissues in vivo, such as blood oxygen saturation and metabolic rate of oxygen [4]. Optical resolution photoacoustic microscopy (OR-PAM), a major form of PAI, images vascular networks at very high resolution, especially the capillaries. Segmenting microvasculature structures in OR-PAM imageis essential for preclinical and clinical applications. For example, the segmented network provides a clear and complete vessel visualization for tracing and monitoring hemodynamics in brain function study; the quantitative analysis, such as vessel density and tortuosity, of the segmented vessel network could provide clinically relevant morphology indicators to evaluate the angiogenesis in tumor development and blood vessel normalization in cancer treatment [5], [6], [7], [8]. For quantitative analysis, the accurate segmentation of the microvascular network is the most critical and primary step in image analysis. The lack of precise segmentation would lead to suboptimal applications.

Many approaches have been reported to implement the vessel segmentation of photoacoustic images from OR-PAM. Initially, the vessel segmentation for OR-PAM was implemented based on the traditional thresholding method [9]. This method is simple and easy to use and is based only on the intensity information of signals. This technique failed to detect many capillaries and could not extract continuous vessels. In recent years, vessel segmentation based on Hessian matrix enhancement has become popular in photoacoustic imaging. This technique enhances low-contrast microvessels with tubular structures [10], [11], [12], [13], [14]. In 2014, Yang et al. were the first to develop a method of integrating multi-scale Hessian matrix enhancement and region-growing algorithm to segment the vascular structures in OR-PAM images [10]. Based on Yang et al.’s method, H. Zhao et al. proposed a 3D Hessian matrix-based method to segment rat iris vessels from images acquired by OR-PAM [11]. Recently, Sun et al. presented a 3D two-branch vessel segmentation framework based on intensity information (i.e., thresholding) and structure information (i.e., Hessian matrix). Compared to the single-branch method, this model could extract more complete tumor vessels [12]. Recently, several studies for segmenting the OR-PAM images have been reported. Q. Zhao et al. developed an advanced version of Ref. [10] to extract the binary vascular network in the tumor OR-PAM images [13] and applied it to monitor tumor growth. Mai et al. applied multi-scale Hessian enhancement and adaptive threshold segmentation to mouse ear tumor images and quantitatively studied the changes in blood vessels before and after photodynamic therapy [14].

In all the approaches mentioned above, even though the multi-scale Hessian matrix can enhance blood vessels of different sizes, it is very sensitive to the scale of the Gaussian kernel. For example, when the maximum scale is too large, it results in thin vessels appearing blurred and deformed; conversely, when it is too small, it makes thick vessels appear thinner [15]. To overcome this limitation, Zhang et al. proposed a hybrid segmentation method where segmentation results from the Hessian matrix, adaptive threshold, and OSTU were fused and applied to analyze mouse vessel images of OR-PAM. The loss of connectivity and distortion of vessels resulting from the Hessian filter was repaired by upward region growth and full diameter at half maximum correction on each pixel along the vessel skeletons [16].

The tracking method is another typical approach for vessel segmentation and has been used in photoacoustic image analysis [17], [18], [19], [20]. This technique traces the vessels iteratively by starting from a pre-selected vessel point and locating the next vessel until the entire network is extracted [17]. Based on the concept of vessel tracking for OR-PAM images, Li et al. developed an automatic vascular tree extraction algorithm from the background and other crossed vascular trees [20]. Although this method exhibited superiorities in segmenting large vessels, it failed to extract the low-contrast capillaries.

Deep learning (DL) has become popular in medical imaging in recent years due to its ability to automatically extract essential features from sample data. Several researchers have applied DL for microvascular segmentation in photoacoustic images. Boink et al. proposed a DL algorithm that simultaneously realized the reconstruction and segmentation of PACT images and validated them using phantom data [21]. Yuan et al. proposed a hybrid network composed of U-Net and FCN for vascular segmentation of OR-PAM images using ground truth images containing large vessels only [22]. The above two demonstrate the performance of DL in thick-vessel extraction but have limitations on thin-vessel segmentation. Recently, a DL model integrated with Hessian matrix computation was developed by H. Zhao and applied to the vessel segmentation of OR-PAM images [23]. This method improves the extraction of continuous small vessels.

In summary, the existing work is limited in extracting the vessels from OR-PAM images in many ways. The thresholding and tracing methods cannot provide sufficient segmentation accuracy for microvessels. Even though the Hessian matrix methods can improve the continuity in microvascular extraction, it is difficult to select appropriate scale parameters of the Gaussian kernel that can extract thick and thin vessels, often leading to vascular deformation. The DL techniques have the potential to overcome the limitations of existing techniques. Still, they cannot achieve high-quality microvascular segmentation, especially for the thin vessels with low contrast, vital for biological studies such as tumor angiogenesis. In this paper, to overcome the above limitations, we propose a novel WSA-MP-Net model, where the weak signal attention (WSA) module is integrated with a multi-scale perception (MP) module to implement the microvascular segmentation for OR-PAM images. In this integrated model, the WSA enhances the extraction of thin vessels, while MP captures the multi-scale features from both thick and thin vessels. In addition, to address the difficulty in generating ground truth for the imaged method, a semi-automatic method was developed to label microvessels in OR-PAM images. We also incorporated pre- and post-processing operations to improve microvessel continuity by incorporating Hessian-matrix enhancement. We conducted experiments on in vivo OR-PAM images to verify the effectiveness and superiority of our proposed method over other existing methods.

2. Method

The flowchart of the proposed microvascular segmentation model is shown in Fig. 1. It consists of three main parts: pre-processing, WSA-MP-Net, and post-processing.

Fig. 1.

The flowchart of the proposed WSA-MP-Net for extraction of microvasculature from OR-PAM images.

The pre-processing has a two-branch framework. One branch is used to improve the continuity of small vessels. A high-frequency emphasis filtering is applied to the input images to enhance the high-frequency components while preserving low-frequency components. It is defined as $H_{hfe}=a+b{H}_{hp}(u,v)$ where $H_{hp}$ is the transfer function of the high-pass filter. $a$ is the offset from the origin and $b$ is a weighting factor controlling the contribution from high-frequency. This operator improves the contrast between blood vessels and the background. However, some capillaries remain discontinuous with unclear boundaries. The multi-scale Hessian matrix enhancement, as proposed by Ref. [10], is then used to further improve the capillaries’ segmentation. In the second branch of the pre-processing, gamma transformation is used to enhance the contrast between weak and strong signals in the input OR-PAM images. The results from the two branches are fed into the WSA-MP-Net as two-channel data. The WSA-MP-Net provides richer information on vasculature through network optimization.

The post-processing further enhances the continuity of segmented results of the DL network. Due to the low intensity of small signals, many capillaries in the predicted map may remain discontinuous. The post-processing module performs Hessian matrix enhancement on the predicted grayscale image to improve the continuity of small blood vessels. Since it is difficult for the Hessian matrix to select a scale parameter adaptable to both thick and thin vessels simultaneously, we set the maximum scale in the vascular response function to be smaller to focus on enhancing small vessels.

2.1. WSA-MP-Net

The WSA-MP-Net is a four-layer encoder–decoder network with U-Net as the backbone, as shown in Fig. 1 [24]. Thus, our network has the same U-shape structure as the U-Net. We have integrated novel WSA and MP modules on this structure. Two 3 × 3 convolutions are used at the beginning to extract low-level features in the input two-channel images for the encoder. The image size is then reduced to half the original using 2 × 2 max-pooling operations. WSA modules are employed in the subsequent levels to enhance weak signals. The MP module, placed at the lowest level, extracts multi-scale signal features. MP is located at the lowest level because the image size is smallest at the lowest layer, making it easy to capture features in a broad view. Also, the number of parameters used in the MP is minimal due to the small size of the image. For the decoder, deconvolutions are used to up-sample the semantic feature maps level by level and concatenate the feature maps from the encoder.

2.2. WSA module

The structure of the WSA module is shown in Fig. 2. $X^i$ and $Y^i$ $(i=1,2)$ are the input and output of this module, where $i$ represents the $(i+1)$ layer of the network. $H^i$, $W^i$, $C^i$ define the feature maps’ height, width, and channels. In WSA, a 3 × 3 convolution with batch normalization (BN) is applied to extract the features of the input feature maps. Then, data normalization ($N_a$) and gamma transformation are used to enhance the contrast of the weak signals. Traditional spatial attention module (SAM) is then used, with input and output represented by $G^i\in R^{H^i\times W^i\times C^i}$ and $M^i\in R^{H^i\times W^i\times C^i}$, respectively, and are computed as follows:

$$G^i=Gamma\left(N_a\left(BN\left(Con{v}_{3\times 3}\left(X^i\right)\right)\right)\right),i=(1,2)$$ (1)

$$M^i=\sigma \left(Con{v}_{7\times 7}\left(Concat(C_{Avg}\left(G^i\right),C_{Max}\left(G^i\right))\right)\right)\otimes G^i,i=(1,2)$$ (2)

Fig. 2.

The structure of WSA module. (a) The flowchart of WSA; (b) Illustration of SAM; (c) Illustration of WSE block.

In Eqs. (1) and (2), $\delta$ represents the ReLU activation, $\sigma$ is the sigmoid function, $C_{Avg}$ and $C_{Max}$ are the average pooling and max pooling, respectively, and $\otimes$ represents element-wise multiplication.

Although SAM can leverage the network to focus on the desired features, it still cannot enhance weak vessels due to the low contrast between the weak signals and the background. To address this, we generate the location information maps for both weak and strong signals through threshold segmentation on $M^i$, denoted as $A^i$ and $B^i$. The weak signal enhancement (WSE) block is then designed to raise weak vessels’ brightness to match the strength of strong signals. The structure of the WSE block and visualization of feature maps corresponding to each operation are shown in Fig. 2(c). In WSE, $G^i$ is multiplied by $A^i$ and $M^i$ is multiplied by $B^i$ to obtain the weak signal feature map of $G^i$ and strong signal feature map of $M^i$, respectively. To improve weak vessels’ signal intensity while keeping strong signals’ brightness, we adjust the data range of the two feature maps to be consistent with the range of $G^i$ by normalization ($N_b$). Finally, we add the results of the two branches to obtain the enhanced maps $W^i$, and it is computed using Eq. (3):

$$W^i=N_b(G^i\otimes A^i)+N_b(M^i\otimes B^i),i=(1,2)$$ (3)

The residual connection is performed between inputs and $W^i$ to reduce structural information loss and to accelerate the network convergence. The formula for calculating $Y^i$ is given in Eq. (4).

$$Y^i=\delta (W^i+BN\left(Con{v}_{1\times 1}\left(X^i\right)\right)),i=(1,2)$$ (4)

The key operations in WSA are SAM, thresholding, and data normalization. SAM enhances high-contrast signals present in the feature map. The thresholding is then applied to the SAM results to separate strong and low-contrast signals. The data normalization then stretches the amplitude of signals in low-contrast areas to the same data range as the gamma-transformed features map. As a result, the intensity of the weak signals in the combined feature map increases to the same level as that of strong signals, facilitating the subsequent identification of weak signals. In the above process, the key parameters are exponent in gamma transformation and threshold, which are set to 0.7 and 0.3, respectively, in our work.

2.3. MP module

We designed an MP module and added it to the bottom layer of the network to aggregate multi-scale semantic information, which is an essential structure of spatial pyramid atrous convolution. Here, atrous convolutions with expansion rates of 1, 2, and 3 were employed to implement the multi-scale perception. Hierarchical residual connections were designed before channel concatenation to reduce the gridding effect of dilated convolutions [25]. The specific structure of MP is shown in Fig. 3. At first $F_1$, $F_2$ and $F_3$ were obtained using three parallel atrous convolutions:

$$F_j=ACon{v}_{3\times 3}^{r_j}\left(F_0\right),j=(1,2,3)$$ (5)

where $r_j$ is the dilation rate. The first four branches were then hierarchically added to obtain $A{d}_1$, $A{d}_2$ and $A{d}_3$. Finally, the multi-scale information was aggregated through channel concatenation, and the residual connection was introduced to prevent information loss. The calculation formula for the final feature maps is:

$$F_6=F_5+Concat(F_1,A{d}_1,A{d}_2,A{d}_3)$$ (6)

Fig. 3.

The structure of MP module.

2.4. Imaging system and in vivo experiments

All our photoacoustic imaging experiments were conducted using the OR-PAM system established by our research group. The details of the OR-PAM system can be found in [26]. The main features of this OR-PAM system are a tunable diode-pumped laser system (SPOT-10-200-532, ELFORLLIGHT) with 1 kHz repetition rate and an ultrasonic transducer (V214-BB-RM, OlympusNDT) with 50 MHz center frequency and 100% bandwidth. A laser beam with 532 nm wavelength was emitted to irradiate the biological tissue, and the detected photoacoustic signals from the transducer were pre-amplified by using a commercial electric amplifier (5073R, Olympus). The acquired data was digitized via a 200-MS/s data acquisition (DAQ) card (CS1422, GaGe). The 3D ultrasound volumetric dataset of the tissue was captured by two-dimensional mechanical scanning by the imaging probe, like the method described in Ref. [23]. We conducted animal experiments on mouse ears, subcutaneous hepatoma tumors, and brain glioma. All experimental animal procedures were carried out in compliance with protocols approved by the Animal Studies Committee of the Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences.

We obtained the human liver cancer cell line (Hep G2) from the American Type Culture Collection (ATCC) to establish a subcutaneous hepatoma tumor model. The obtained Hep G2 cells were cultured in DMEM medium supplemented with 10% (v/v) fetal bovine serum, 1% (v/v) penicillin, or 1% (v/v) streptomycin and incubated at 37 °C under a 5% ${\mathrm{CO}}_2$ atmosphere. The cultured Hep G2 cells $(1\times 10^6)$ were injected subcutaneously into the male Balb/c mice ear. For all the data acquisition, the optical energy per pulse was maintained at approximately 18 mJ/${\mathrm{cm}}^2$, conforming to the ANSI standard (ANSIZ136.3-2005).

Similarly, to establish an orthotopic glioma model, we cultured the mouse glioma cell line (GL261) in a DMEM medium containing 10% fetal bovine serum and 1% penicillin at 37 °C in a 5% ${\mathrm{CO}}_2$ atmosphere. In the brain glioma imaging experiment, we placed three male black rats (C57BL/6J, 6–7 week, about 20 g, Zhuhai Baishantong Biotechnology Co., Ltd.) and anesthetized them with halothane on the small animal orienting instrument. We cut the scalp and used a cranial drill to open a 6 × 6 mm cranial window in the anesthetized rats, and then injected 6 ul of GL261 cells $(1\times 10^6)$ into the right brain of the mouse at 1ul/min to a depth of 1 mm. We sealed the cranial window with film.

We monitored the brain blood vessels of mice every other day for 14 consecutive days, starting from the 0th day when the window was closed. We maintained 2% isoflurane anesthesia throughout the imaging experiment and used a temperature-controlled water bath to hold brain temperature at 37 °C. The light per pulse was kept at 130 nJ for all data collection, well below the American National Standards Institute safety standard (20 mJ/${\mathrm{cm}}^2$).

2.5. Construction of datasets

2.5.1. Data partition

Since the vessel network is distributed in whole 3D volume, data annotation in a single cross-section is difficult and prone to error. Thus, we do the data label in a sub-volume containing certain depth information.

For the standard vessel imaging of mouse ears, the acquired 3D data of size $1568\times 625\times 200$ pixels was cropped to $1280\times 512\times 200$ pixels to reduce the calculation burden while keeping the entire vessel structure intact. The cropped 3D data was cut into ten 3D volumes with each of size $256\times 256\times 200$ pixels. As shown in Fig. 4(a), a 3D volume is divided into several sub-volumes along the depth direction, with each consisting of about 3–5 cross sections. Signal features determine the size of each sub-volume. Specifically, for constructing one sub-volume, we focus on one important vascular branch and all successive depth slices belonging to the specific vascular branch are then included in the sub-volume. The aim is to have a continuous vascular network in the resulting maximum amplitude projection (MAP) image of the sub-volume. The MAP along the depth direction was carried out on each sub-volume to obtain source images ($256\times 256$ pixels), and ground truth was produced using these data. For the imaging data from mouse ear, six 3D volumes were randomly selected as training and validation sets (five for training and one for validation), while the remaining four were used as test sets. Fig. 4(c) shows the result of data division in our experiments, the blue volumes represent the training data, green volumes represent the validation data, and the yellow volumes are used as the testing data. Based on this strategy of data division, the adjacent sub-patches generated from 3D sub-volumes alone the depth direction of a volume, are uniformly assigned one of the training, validation or testing datasets. Therefore, in the depth direction, there will be no situation where adjacent patches are respectively assigned to the training set and the test set (these patches may have similar signal distribution). Although this situation may occur in the $x$–$y$ direction, it can be seen from the MAP images shown in Fig. 4(e) that adjacent patches in this direction generally have different blood vessel signal distributions, not result in data leakage in network optimization.

Fig. 4.

The overall process of constructing datasets obtained by OR-PAM. (a) Illustration of processing 3D vascular data; (b) The flowchart of a semi-automatic method of generating ground truth for microvascular images; (c) Division result for normal vessel data; (d) Division result for tumor vessel data; (e) The MAP images of normal- and tumor-vessel imaging. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

For the mice tumor imaging, the acquired 3D data of size $990\times 1760\times 98$ pixels was cropped to $768\times 1536\times 98$ pixels. The cropped 3D data was then divided into 18 3D volumes, with each volume of size $256\times 256\times 98$ pixels. Fourteen 3D volumes were selected for network training and validation (12 for training and 2 for validation), and the remaining four were used for the test set. A method same to the normal vessel data was used to construct the tumor vessel dataset as well, and the corresponding division result of tumor data in this work is shown in Fig. 4(d).

Data augmentation methods such as cropping, flipping, rotating, and transposing were employed to increase the number of data pairs. Finally, the normal vessel dataset (named NV-ORPAM) contains 660 data pairs: 518 for training, 58 for validation, and 84 for testing. There are 1168 data pairs in the tumor dataset (named TV-ORPAM): 928, 104, and 136 pairs for training, validation, and testing data, respectively.

2.5.2. Generation of ground truth

Generating the ground truth of microvessels is complex and time-consuming. To facilitate and speed up the annotation process, we developed a semi-automatic annotation method to generate ground truth, as shown in Fig. 4(b). The main idea behind this method is to produce initial ground truth using traditional image processing techniques. We employed adaptive histogram equalization and k-means clustering to obtain the coarsely segmented results. The canny operator was then used on the coarsely segmented results to extract the vessel edge and obtain the edge mask. The mask and the segmented vessel image were multiplied by each other to decrease the distorted vessel thickness of the k-means method. Subsequently, the small connected components were removed on this result, and finally, the ground truth was obtained by fine manual modification.

Notably, most of the discontinuous segments that look like noise or artifacts in raw photoacoustic images could be real signals originating from capillaries or microvessels. Specifically, when the capillaries are too small in diameter, red blood cells can only pass through one by one, resulting in dot-like signal points in the image under high-resolution OR-PAM. However, nearby points can be connected to form the trajectories of the imaged blood vessels, which is the criteria we use to differentiate vessels from dot-like noises. In addition, blood vessels are distributed throughout the 3D space with different shapes and sizes. The blood vessel signals originating from the focal area of the transducer have larger amplitudes, but they attenuate quickly in the out-of-focus area. Therefore, in MAP (maximum amplitude projection along depth, MAP) images, they appear as discrete signal segments rather than artifacts. While generating ground truth, we did not manually connect discontinuous signal segments. However, to facilitate the use of data and further augmentation of the data for further research, we have included the raw photoacoustic data in the datasets of NV-ORPAM and TV-ORPAM. A Supplementary Video-1 on time-frame OR-PAM images is provided to display these signal characteristics of microvessels visually. Some noises appearing as bright spots do exist in photoacoustic images used in our work. When analyzed, these bright spots appear at the same spot in several consecutive depth slices and then disappear in subsequent slices. Such behavior is intrinsic to the signal originating from pores or melanin in biological tissues and can be considered noise for our segmentation study. Characterization of these noises on two datasets can be found in Supplementary Video-2 and Video-3.

In summary, while generating ground truth, discontinuous signal segments and points connecting vessel trajectories are marked as blood vessels, and small bright spots are marked as noises. Furthermore, we removed small connected components with less than ten pixels in the ground truth and segmentation results. This modification prevented the impact of isolated noises that were mislabeled or misclassified as vessels on the model. Still, in the process, it removed small amounts of discontinuous signal belonging to the capillary.

3. Experimental results

All our programs run on a PC with two Intel Xeon Gold processors and are equipped with two NVIDIA Tesla P100 Cards (16 GB memory). DL model was implemented in Python-installed Tensorflow 2.2. In the training process of WSA-MP-Net, the binary cross entropy was employed as a loss function, Adam was selected as an optimizer, the epoch was set to 100, and the initial learning rate was set at 1e−3. We set the learning rate to decrease by 10x when the validation loss was higher than the minimum validation loss in three consecutive iterations. The gamma factors in the pre-processing stage were set to 0.7 and 0.5 for the normal vasculature and tumor vascular datasets, respectively, and they both were set to 0.7 in the WSA module.

3.1. In vivo experiments

To demonstrate the performance of the proposed WSA-MP-Net for microvascular segmentation in OR-PAM images, we conducted experiments on two datasets comprising normal and tumor vessel images. In addition, we also conducted experiments to compare our results against traditional region growing with the Hessian-matrix enhancement method (GR+HM) [13] and two advanced DL models of Hy-Net [22] and HM-2DCE-Net [23]. In reproducing previous DL models, the respective data augmentation operations as discussed in Ref. [22] and Ref. [23] were used. The pre-and post-processing methods proposed in this work were only employed for our network. All results are shown in Fig. 5. The first two columns show results for normal vessel images, and the last two are for tumor vessel images. The first two rows show raw images with their ground truth, and from the third to sixth row, the segmented binary vessel images using the GR+HM method, Hy-Net, HM-2DCE-Net, and our proposed WSA-MP-Net, respectively, are shown.

Fig. 5.

In vivo experiments on normal and tumor vessel datasets from OR-PAM. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

The GR+HM method extracts most vascular signals without data labeling, and vessel continuity is also good. However, since choosing an appropriate scale parameter adaptive to different blood vessel sizes is difficult, it results in vascular morphology deformation, e.g., the diameter of the thick vessel gets larger, and it does not exhibit an advantage for small vessel extraction. Different from GR+HM, Hy-Net and HM-2DCE-Net methods can segment all thick vessels with high accuracy but have shortcomings in extracting small vessels, i.e., only part of the small vessels was recognized, as shown in the fourth and fifth rows of Fig. 5. Compared to these existing methods, the results from our proposed WSA-MP-Net improved significantly. The accuracy of thick vessel extraction improved, and more continuous small vessels were recognized. According to our analysis, the WSA module extracts more capillaries, and the Hessian matrix enhancement improves their continuity. In addition, the MP module can extract different features from signals of various sizes. To clearly illustrate the segmentation effects of different methods, we selected a region of interest (ROI) in each segmented image patch and generated a fusion image with the ground truth. In this fusion image, red represents false negative pixels, and green represents false positive signals. The morphology distortion of thick vessels using GR+HM can be seen in the enlarged sub-images of ROIs. Using our method, no distortions of thick vessels are seen in the enlarged ROI, and more small vessels are extracted. To illustrate the convergence of our DL model during training, the curves of training loss and validation loss for normal and tumor blood vessel datasets are shown in Supplementary Fig. 1. At the end of training, both the training loss and the validation loss curves show stability, and the loss values are relatively small, indicating good convergence in model optimization. In summary, experiments verified that our proposed DL model has superior performance compared to previous works. More analysis of WSA-MP-Net results is presented in the following subsections.

To quantitatively evaluate the performance of our proposed DL model, five statistical parameters were introduced: accuracy (Acc), sensitivity (Sen), Dice, intersection over union (IoU) and area under the ROC curve (AUC). These statistical parameters were computed for all methods and two datasets, as shown in Table 1 and Fig. 6. Compared to previous works, our proposed WSA-MP-Net significantly improved in all statistical evaluations. Our method achieves about 1.3% to 25.9% improvement in overall statistical parameters, further verifying the advantages of our proposed method for microvascular extraction.

Fig. 6.

Quantitative comparisons among different segmentation methods.

Table 1.

Quantitative comparisons among different methods on normal and tumor vessel datasets.

Dataset	Methods	Acc	Sen	Dice	IoU	AUC
NV-ORPAM	GR+HM	0.9138 ± 0.0204	0.6892 ± 0.1036	0.7637 ± 0.0506	0.6198 ± 0.0661	0.8292 ± 0.0425
	Hy-Net	0.9401 ± 0.0232	0.7788 ± 0.0654	0.8445 ± 0.0401	0.7325 ± 0.0613	0.8808 ± 0.0330
	HM-2DCE-Net	0.9300 ± 0.0168	0.7593 ± 0.0568	0.8208 ± 0.0235	0.6812 ± 0.0326	0.8676 ± 0.0112
	Proposed	0.9525 ± 0.0182	0.8680 ± 0.0283	0.8851 ± 0.0232	0.7945 ± 0.0379	0.9213 ± 0.0174
TV-ORPAM	GR+HM	0.8440 ± 0.0168	0.8107 ± 0.0568	0.7952 ± 0.0235	0.6605 ± 0.0326	0.8446 ± 0.0112
	Hy-Net	0.9176 ± 0.0056	0.8063 ± 0.0380	0.8738 ± 0.0187	0.7761 ± 0.0291	0.8922 ± 0.0149
	HM-2DCE-Net	0.8840 ± 0.0128	0.7737 ± 0.0608	0.8240 ± 0.0393	0.7021 ± 0.0553	0.8583 ± 0.0268
	Proposed	0.9348 ± 0.0068	0.8812 ± 0.0064	0.9059 ± 0.0077	0.8282 ± 0.0128	0.9229 ± 0.0054

To comprehensively evaluate the execution efficiency of each component of our method, we generated 15 image patches of three different sizes from two datasets, containing 5 data samples of each type. Supplementary Table 1 lists the time for all the modules of our network: the semi-automatic ground truth generation method, the pre-processing module, the network prediction process, and the post-processing module at three data scales. All the performances were obtained on the same platform where the program ran.

3.2. Ablation study

To demonstrate the effectiveness of the proposed WSA and MP modules, we performed ablation experiments on baseline (U-Net), baseline+WSA, baseline+MP, and our proposed model. All experiments were performed with pre-processing and post-processing modules for a fair comparison. The segmented results of four 3D patches from two datasets are shown in Fig. 7. The baseline method segments bright signals reliably but fails to extract weak signals. With WSA, more microvessels are recognized (see the images in the fourth column) because the network can focus on strong and weak signals while suppressing irrelevant background regions. The baseline+MP uses atrous convolutions with different expansion rates to extract multi-scale signal features and thus can extract more capillaries and accurate structure of big vessels. By observation, our proposed model achieves the best results in terms of accuracy of vessel extraction, completeness, and continuity of small vessels. More detailed information can be found in the regions indicated by rectangular boxes in Fig. 7. The network convergence of different DL models was also considered. Their loss curves in the training process are shown in Fig. 8. These curves show the advantages of our method for model optimization, i.e., achieving minimum loss values under the same objective function.

Fig. 7.

In vivo experiments on normal and tumor vessel datasets from OR-PAM.

Fig. 8.

Loss curves of ablation experiments on two datasets.

The quantitative results of five statistical parameters are shown in Fig. 9. As seen in this Figure, accuracy, sensitivity, Dice, IoU, and AUC are all better for our dedicated DL model than other models. These results confirm the effectiveness of adding WSA and MP modules to the baseline.

Fig. 9.

Quantitative comparisons for different DL models in ablation study.

3.3. Discussion on attention modules

In this section, we present comparative experimental results between WSA and SAM, and they were all performed based on baseline+MP with pre-processing and post-processing modules. The straightforward implementation of the SAM module results in the loss of weak signals, as shown in Fig. 10, thus resulting in them being misclassified as background (Fig. 10(C) and (G)). In the fourth-column images, which correspond to the WSA module, more capillaries are extracted (Fig. 10(D) and (H)). Theoretically, in our WSA model, the weak signal feature maps are separated from the strong signals, and these separated weak signals are normalized to strong signals, which results in enhanced weak signal maps. The network then focuses on both weak and strong signals for segmentation. The advantages of WSA for recognizing small vessels can be visualized in the regions indicated by the rectangular boxes. Five statistical indicators were computed to quantitatively compare the segmented results using SAM and WSA modules, as shown in Table 2. Compared to SAM, statistical indicators for WSA, i.e., Acc, Sen, Dice, IoU and AUC, were increased by 1.4%, 10.2%, 4.6%, 8.0%, and 4.2%, respectively, for the normal vessel dataset. Correspondingly, the tumor blood vessel dataset increased by 0.2%, 2.2%, 0.5%, 1.0%, and 0.6% respectively. These results prove that better performance can be achieved using our proposed DL model with WSA.

Fig. 10.

Comparative results between SAM and WSA module with baseline+MP as the fundamental network. (A, E) Raw images of normal vessel and tumor vessel; (B, F) Ground truth of raw images; (C, G) Segmentation results of baseline+MP+SAM; (D, H) Segmentation results of baseline+MP+WSA.

Table 2.

Quantitative comparisons between SAM and WSA modules.

Dataset	Methods	Acc	Sen	Dice	IoU	AUC
NV-ORPAM	SAM	0.9398 ± 0.0259	0.7875 ± 0.0764	0.8464 ± 0.0428	0.7355 ± 0.0658	0.8843 ± 0.0385
	WSA module	0.9525 ± 0.0182	0.8680 ± 0.0283	0.8851 ± 0.0232	0.7945 ± 0.0379	0.9213 ± 0.0174
TV-ORPAM	SAM	0.9326 ± 0.0054	0.8622 ± 0.0237	0.9010 ± 0.0105	0.8200 ± 0.0174	0.9165 ± 0.0082
	WSA module	0.9348 ± 0.0068	0.8812 ± 0.0064	0.9059 ± 0.0077	0.8282 ± 0.0128	0.9229 ± 0.0054

3.4. Analysis on pre-processing and post-processing

Here, we compared the performance of WSA-MP-Net with and without pre-processing and post-processing modules to illustrate the influence of these modules on the final segmented results. The comparison results are shown in Fig. 11. From this figure, it can be seen that the network applying pre-processing or post-processing, independently, improves the microvascular segmentation. Thus, when the network uses both pre-processing and post-processing and applies to the data, the network achieves the best segmentation results. The completeness and continuity of small vessels is also improved by adding these modules, as can be seen clearly in the selected sub-regions. However, the appropriate scale parameters must be selected for Hessian-matrix enhancement to avoid merging of two adjacent blood vessels into one. The quantitative parameters of the four experiments are shown in Fig. 12, and it can be seen from this figure that the network using pre-processing and post-processing outperforms the network that does not use them for both normal and tumor data.

Fig. 11.

Experimental results with and without the pre- and post-processing.

Fig. 12.

Quantitative comparisons with and without pre- and post-processing.

3.5. Application research on monitoring tumor angiogenesis

To demonstrate the generality of the WSA-MP-Net for other applications, we conducted brain glioma imaging experiments on mice to monitor the growing process of tumor vessels. The DL model optimized using the TV-ORPAM dataset is directly used to predict the vascular network in the brain glioma tumor region. To observe the tumor growth in rats, three original photoacoustic images of the same rat were acquired and segmented using the trained model at the 0th, 4th, and 10th days after tumor implantation, as shown in the first column of Fig. 13. The second column shows the enlarged tumor region represented by the rectangular boxes in the first column. The columns from third to sixth show the results from GR+HM, Hy-Net, HM-2DCE-Net, and our proposed WSA-MP-Net, respectively. Observation in the tumor region in original images shows that many new blood vessels appear during the growing process of glioma, and microvascular density and curvature increase accordingly. All extracted vessel networks by GR+HM and DL methods in this figure reflect the changes in the vessel morphology. Although the GR+HM method could extract most of the new capillaries, it failed to balance between the extraction of small vessels and large vessels, i.e., either the recognition of small vessels was not good enough (discontinuous or lost) or the shape of large vessels was deformed (see the third column). The Hy-Net and HM-2DCE-Net could segment all thick vessels, but their ability to extract small vessels is low, and many new small vessels were lost in the segmented results (see the fourth and fifth columns). Compared to the previous methods, our proposed WSA-MP-Net could extract most of the continuous new capillaries while keeping the morphology of large vessels intact. In summary, this experiment on brain glioma fully proves the generality of the WSA-MP-Net for other applications.

Fig. 13.

Application experiments of different methods on the glioma dataset.

To better demonstrate the generality of the proposed WSA-MP-Net on different imaging datasets, we conducted vessel segmentation experiments on DRIVE, a public dataset of retinal fundus images. Experimental results are provided in Supplementary Fig. 2 and Supplementary Table 2.

4. Discussion and conclusion

The proposed novel WSA-MP-Net promises to improve microvascular segmentation in photoacoustic images while guaranteeing the accuracy of thick-vessel extraction. Moreover, the pre-and post-processing modules further improve this DL model’s performance in extracting more continuous small vessels. The developed semi-automatic algorithm for rapidly generating accurate and reliable ground truth was also very useful in generating GT quickly. Specifically, the semi-automatic approach helps to solve the difficulty of annotating microvascular with complex structures, such as tumor vessels. Experimental results showed the superiority of WSA-MP-Net over existing deep learning methods. We verified that monitoring tumor angiogenesis is practical using our method and proves the model’s generalization ability, i.e., it can be expanded to other applications. In conclusion, the method proposed in this paper can effectively segment microvessels in photoacoustic images from OR-PAM, assist in studying the occurrence and development mechanism of vascular-related diseases, and contribute to the early diagnosis of cancers.

Our study has limitations: (1) The semi-automatic annotation algorithm generates ground truth and is not an ideal annotation. Some parts of the capillary network in ground truth are still discontinuous, and some very small signals are not labeled or are mislabeled. Our DL network achieved good performance even under such a scenario due to the good generality and robustness of the model. (2) Although our method can extract more weak/small signals from photoacoustic images, some capillaries are still not recognized, and segmenting continuous small vessels still needs further improvement.

To extend our work further, several issues still need to be addressed, and they are discussed below.

(1) Our network model has good blood vessel segmentation capabilities but at the cost of increased training time and prediction time of the WSA module due to the increase in the number of computations needed for each feature map. Supplementary Table 3 lists the network parameters and time efficiency of different DL models considering the 128 × 128 patch size as an example. Compared to the other deep learning models, the network proposed in this article requires fewer training parameters and occupies less memory space. Although the prediction time is longer, it can meet the clinical application needs for a relatively small field of view in low-end systems. Improving the configuration of the computing platform could reduce execution time significantly. In the future, we will explore optimizing the algorithms for high performance and transforming modules into a lightweight network to increase computational speed without degrading the model segmentation performance.

(2) Although our network is trained on the MAP data pairs, it can be used to predict the photoacoustic vessel images on a single depth slice. Multiple vessel-detected consecutive slices can be used to construct the 3D vascular network. Two examples of reconstructed 3D images from two sub-volumes belonging to normal-vessel and tumor-vessel datasets are shown in Supplementary Video-4 and Video-5, respectively. The videos show that our network demonstrates good 3D vascular reconstruction capabilities, and vascular branches are continuous or easy to trace along the blood vessel direction.

(3) U-Net and its some variations have been widely used in medical imaging and are mainly used for organ segmentation and lesion segmentation. Currently, the multi-branch DL framework based on U-Net is the state-of-the-art method for extracting the vessels in the fundus imaging [27], [28], [29]. Our proposed WSA-MP-Net can be applied for the same application and could potentially achieve higher-precision extraction of all vessels in the OR-PAM fundus imaging, which is our future work.

Funding

This work was supported by National Key Research and Development Program of China (2022YFE0132300, 2023YFF0715300, 2023YFC2411700, 2020YFA0908800); National Natural Science Foundation of China (NSFC) Grant (82122034, 82327805, 81927807); Chinese Academy of Sciences Grant (Youth Innovation Promotion Association 2019352); Shenzhen Science and Technology Innovation Grant (JCYJ20220818101403008); Key Laboratory of Biomedical Imaging Science and System, Chinese Academy of Sciences; Guangdong Provincial Key Laboratory of Biomedical Optical Imaging (2020B121201010); Shenzhen Key Laboratory for Molecular Imaging (ZDSY20130401165820357); Natural Science Foundation of Shandong Province (ZR2020MF105); Innovation Improvement Project for Technology-based Enterprises of Shandong Province (2023TSGC0095).

CRediT authorship contribution statement

Jing Meng: Writing – review & editing, Writing – original draft, Methodology, Conceptualization. Jialing Yu: Writing – original draft, Validation, Software, Methodology. Zhifeng Wu: Visualization, Resources, Investigation, Data curation. Fei Ma: Writing – review & editing, Visualization, Investigation. Yuanke Zhang: Writing – review & editing, Visualization, Investigation. Chengbo Liu: Supervision, Project administration, Funding acquisition.

Declaration of competing interest

The authors declare no conflicts of interest.

Biographies

Jing Meng is a professor at Qufu Normal University (QFNU). Prior to joining QFNU, she received her Ph.D. degree in computer engineering from Soochow University in 2006. She engaged in photoacoustic imaging as a research fellow at the Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences from 2011 to 2013, and engaged in medical imaging processing and privacy protection at Georgia State University from 2019 to 2020. Her research on photoacoustic and medical image analysis has been supported by NSFC , local governments, and industry, and published more than 30 related academic papers including Optics Express, Optics Letters and Biomedical Optics Express. Her current research interests include photoacoustic imaging, image processing and analysis, reconstruction algorithms, and deep learning.

Jialing Yu was born in Weihai, Shandong Province. She is currently a post-graduate student in the Department of Computer Science at Qufu Normal University located in Rizhao. In 2021, she received her bachelor’s degree in Software Engineering from Cyberspace Security College of Qufu Normal University, mainly studying software development and testing. In the same year, she began to engage in research in the field of imaging and her current research focuses on medical image processing and optical-resolution photoacoustic microcopy imaging, specifically through some traditional methods or deep learning to extract microvessels from photoacoustic images to achieve high-precision segmentation of blood vessels.

Zhifeng Wu was born in Yiyang, Hunan Province. He graduated from China University of Mining and Technology in 2022 with a bachelor’s degree in mechanical engineering. In the same year, he was admitted as a master’s student jointly trained by Southern University of Science and Technology and Shenzhen University of Technology, and joined the Shenzhen Institute of Advanced Technology to engage in research on photoacoustic microscopy imaging technology and its biomedical applications. Specifically, it is to conduct research at the biological disease model level by developing new imaging systems to provide imaging support for disease treatment and clinical application.

Fei Ma is an associate professor with the School of Computer Science, Qufu Normal University, China. He received his Ph.D. degree in pattern recognition and intelligence system from Wuhan University, Wuhan, China, in 2019. From 2013 to 2014, he was a visiting scholar in Wuhan University. He has published more than 10 scientific papers in the international journals, including IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, Neurocomputing, MULTIMEDIA AND TOOL APPLICATIONS, Biomedical Signal Processing and Control and Journal of Biophotonics. He has served as a reviewer of several academic journals. His current research interests include computer vision, deep learning, medical image processing and machine learning.

Yuanke Zhang is a professor at the School of Computer Science, Qufu Normal University. He received his Ph.D. in Computer Application Technology at Xidian University in June 2011. From June 2016 to March 2021, he was a postdoctoral researcher at the School of Biomedical Engineering, Fourth Military Medical University and School of Biomedical Engineering, Southern Medical University. His main research areas are intelligent imaging and analysis of medical images, including medical CT image reconstruction theory and methods, intelligent CT image generation methods, medical image segmentation and recognition. He has published more than 30 related academic papers including top journals such as IEEE TMI, IEEE TCI, MP, and PMB.

Chengbo Liu is a Professor at Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences. He received his Ph.D. and Bachelor degrees from Xi’an Jiaotong University, each in 2012 in Biophysics and 2007 in Biomedical Engineering. During 2009 to 2011, he was a visiting scholar at Duke University, working on tissue spectroscopy for early cancer diagnosis. His current research interest focuses on multi-scale photoacoustic imaging instrumentation and translation. He is a senior member of OSA, SPIE and IEEE. He serves as an editorial board member of Photoacoustics journal, and as the deputy secretary of the Biomedical Optics Society in China.

Appendix A. Supplementary data

The following is the Supplementary material related to this article.

Video S1

Time - frame video of OR - PAM images.

Video S2

Noise display of normal blood vessels.

Video S3

Noise display of tumor blood vessels.

Video S4

3 D reconstruction on normal vessel.

Video S5

3 D reconstruction on tumor vessel.

MMC S6

Supplementary figures and tables.

Data availability

The datasets used in this paper are available at https://doi.org/10.17632/dp5jgrkd6k.1.