Probability analysis of axillary lymph node metastasis in breast cancer patients using particle space-time distribution model

Abstract

The possibility of axillary lymph node metastasis differs in different breast cancer patients and is the strongest prognostic indicator in breast cancer. The existing studies mainly explored the relationship of axillary ultrasound imaging and axillary lymph node metastasis, without exploring whether ultrasound imaging of breast tumour can affect and perform axillary lymph node prediction. Therefore, this Letter proposes a novel particle space-time distribution model to find the correlation between contrast-enhanced ultrasonography of breast tumour and axillary lymphatic metastasis. Starting from the imaging principle of dynamic contrast-enhanced ultrasonography, the particle space-time distribution model not only comprises space-time features of contrast-enhanced ultrasonography with an encoder–decoder network, but also the flow field information of microbubble particles is integrated into the space-time features that better serves the metastasis prediction by enhancing the particle distribution information. Extensive experiments on real patients have demonstrated that dynamic contrast-enhanced ultrasonography of breast tumour can be used to predict the probability of lymphatic metastasis. This conclusion can be interpretable from the clinical and pathological perspectives.

1. Introduction

Axillary lymph nodes (ALNs) are the earliest affected sites of breast cancer metastasis [1]. Correct evaluation of ALN metastases before surgery has important clinical value for the choice of breast cancer surgery, estimation of prognosis and the development of adjuvant treatment [2]. The traditional method is to use axillary lymph node dissection (ALND) and postoperative pathology to evaluate axillary metastases. However, ALND surgery is not only over treatment for breast cancer patients without ALN metastases, but also allows patients to suffer the complications of the operation, which seriously affects their quality of life [3]. Therefore, pre-operative probability analysis of ALN metastasis using contrast-enhanced ultrasonography will allow patients without ALN metastases to be free of ALND, reducing the complications.

Today, ultrasound has been applied to all aspects of breast examination, and inspection techniques are constantly improving. As a kind of functional imaging, contrast-enhanced ultrasound is a method of injecting microbubble particles (contrast agent) into the human body through the vein, and real-time and continuous observation of the whole process of circulating intra-tumour particles in the lesion [4]. Therefore, contrast-enhanced ultrasound can reflect the perfusion and microenvironment information of the tumour, which has obvious advantages over traditional ultrasound examination.

Related researches have been proposed to explore the relationship between ALN metastasis and ultrasound features. On the one hand, some existing research mainly uses traditional image features such as time-grey curve and average grey scale for contrast-enhanced ultrasound image analysis [5, 6] without considering the imaging principle of dynamic contrast-enhanced ultrasonography or information of microbubble particles. On the other hand, existing studies mainly explored the relationship of axillary ultrasound imaging and ALN metastasis [7, 8]. However, from pathological view, the perfusion information of breast tumour reflects the invasive ability of the tumour to a certain extent, and is a direct factor affecting the ALN metastasis. Therefore, this Letter explores whether contrast-enhanced ultrasound of breast tumour can affect and perform ALN metastasis prediction.

Specifically, the main uniqueness of our Letter includes: (i) the patients with different grade of breast tumour and tumour microenvironment has different manifestations of lymphatic metastasis [9]. In our knowledge, this Letter first explores that dynamic contrast-enhanced ultrasonography of breast tumour can be used for predict ALN metastasis quantitatively by using a novel model; (ii) starting from the imaging principle of dynamic contrast-enhanced ultrasonography, the novel particle space-time distribution model not only comprises space-time features of contrast-enhanced ultrasonography with an encoder–decoder network, but also the flow field information of microbubble particles is integrated into the space-time features that better serves the metastasis prediction by enhancing the particle distribution information; (iii) the conclusion that ultrasound imaging of breast tumour does affect predicting the ALN metastasis, which can be interpretable from the clinical and pathological perspectives. The clinical explanations of this conclusion are also given in this Letter. The quantitative conclusion will make researchers pay more attention to the contrast-enhanced ultrasonography of breast tumour for analysing ALN metastasis.

2. Method

2.1. Dataset

This study was approved by the Nanjing Drum Tower Hospital. Between August 2016 and August 2018, all 162 breast cancer patients who were eligible for an ALND with pathological examination to evaluate axillary metastases as the ground-truth label of metastasis were included in this study. Mean age was 56 years (range 32–75 years). Each patient underwent dynamic contrast-enhanced ultrasonography imaging of the region of breast tumour. Firstly, region of interest (ROI) of breast tumour is drawn manually by the professional ultrasonologist (Fig. 1b). The ROI was placed selectively in the area of the most rapid and strongest enhancement. Areas of calcifications and necrosis should be avoided. Secondly, the processed dynamic contrast-enhanced ultrasonography images are a series of rectangular images which contains ROI of breast tumour with black background to fill (Fig. 1c) and the image size is 100*128*128. As shown in Fig. 1a and c, contrast-enhanced ultrasonography can finish real-time and continuous observation of the whole process of circulating intra-tumour particles to reflect tumour microenvironment of the breast tumour. Fig. 1d shows the image structure of microbubble particles in contrast-enhanced ultrasonography. Therefore, the input of the particle space-time distribution model is a series of 100 image frames $X_t=\{X_1,X_2,\dots ,X_{100}\}$ with size 128 $\times$ 128.

Fig. 1.

Image frames of contrast-enhanced ultrasonography

a Image frames of original contrast-enhanced ultrasonography,

b Image frame of injecting microbubble particles (the green curve denotes the ROI for breast tumour),

c Image frames of contrast-enhanced ultrasonography after ROI extraction,

d Image structure of microbubble particles in contrast-enhanced ultrasonography

We represent the nth patient images of contrast-enhanced ultrasonography as a 3-tuple $\left\{X_n^s,X_n^p,l_n\right\}$, where $X_n^s={\sum }_{t=1}^T{X}_{n,t}^s\in {\mathcal{R}}^{ds}(T=100)$ denotes the image frames in space-time domain and $X_n^p={\sum }_{t=1}^{T/2}{X}_{n,t}^p\in {\mathcal{R}}^{dp}$ denotes the image frames in particle distribution field. $l_n$ is the corresponding ground-truth label of ALN metastasis. $F_n^s=\mathrm{\Psi }\{X_n^s\}$ and $F_n^p=\psi \{X_n^p\}$ are the feature analysers of $X_n^s$ and $X_n^p$.

2.2. Feature analysers of space-time domain

The pipeline of the feature analysers of space-time domain $F_n^s=\mathrm{\Psi }\{X_n^s\}$ is shown in Fig. 2. We implemented convolutional neural network (ConvNet) which contains alternating convolutional, pooling and fully-connected (FC) layers to learn features from image frames $X_n^s$ in space-time domain. The ConvNet has a multi-layer perception with hidden feature size of 128, 64, 32 and 16, then max-pooling over the resulting features over the image frame followed by the Relu layer and FC layer, leading to feature of size 1024. To further capture the long-range temporal dynamics, the long short term memory (LSTM) for temporal information is utilised.

Fig. 2.

Pipeline of the feature analysers of space-time domain

2.3. Feature analysers of particle distribution domain

The contrast-enhanced ultrasonography of breast records the whole process of circulating particles in the breast organs. Starting from this imaging principle, flow field information of microbubble particles is developed into the proposed particle space-time distribution model (see Fig. 3). Firstly, we introduced how to acquire the image frames in particle distribution field $X_n^p$, which describe the displacement of particle patch. The input image $X_{n,t}^s$ is divided into small patch $P_i^s$ with size of $5\times 5$ (Fig. 3). The pipeline of flow field information acquisition of contrast enhanced ultrasound is similar to technology of particle imaging velocimetry [10]. This technology is based on the measurement of image patch similarity.

Fig. 3.

Pipeline of the feature analysers of particle distribution domain

Statistical similarity of two patches $I_a$ and $I_b$ from the previous and current images is used to find the average particle displacement of the patch. Considering that ultrasound image quality is poor and noise is complicated, the similarity measure algorithm is modified by combining with the ultrasonic image noise model. Research shows that the signal output by the ultrasonic transducer array element in the ultrasonic probe is near plural-like noise [11], therefore, the ultrasonic multiplicative speckle noise model is $z\left(x\right)=\mu (y)+\mu (y{)}^{\gamma }\eta (x)$, here, $z(x)$ is observed image greyscale, and $\mu (y)$ is real greyscale. $\eta (x)$ is a Gaussian noise with mean of zeros and variance of ${\sigma }^2$, $\eta (x)\sim \mathcal{N}(0,{\sigma }^2)$ and the noise of ultrasound images can be well restored when $\gamma =0.5$. Therefore, for each pixel of the images, it can be obtained according to $p(z(x)|\mu (y))\sim \mathcal{N}(\mu (y{)}^{2\gamma },{\sigma }^2)$, when measuring the degree of matching of two patches $I_a$ and $I_b$, the two blocks contain p pixels. Then the overall similarity is equal to the product of the probability density of each corresponding pixel in the two patches. Finally, the statistical similarity of two patches $I_a$ and $I_b$ is calculated by correlation calculation [12].

The patches from two consecutive image frames are used to calculate statistical similarity with each other, pixel by pixel. This statistical similarity produces the signal peak that identifies the common displacement between two frames. The velocity of the patches can be calculated by dividing the common displacement with the time delay between two frames. The flow field image or particle distribution field $X_n^p$ over the whole image is obtained by repeating the statistical similarity calculation for each patch over the two consecutive image frames. Then, we implemented ConvNet and LSTM to finish feature analysis of particle distribution domain.

2.4. Particle space-time distribution model

To fully exploit the feature relationships between the features in space-time domain $F_n^s$ and particle distribution domain $F_n^p$, we proposed a regularised framework based feature fusion method (Fig. 4). In the fusion process, we impose a structural $l_{21}$ norm to explore the relations of the features. The optimisation problem of the particle space-time distribution model is proposed as

$$\underset{W}{min}\mathcal{L}+{\lambda }_1\mathrm{\Phi }\left(W\right)+\frac{{\lambda }_2}{2}{W}_{2,1}^E$$ (1)

Fig. 4.

Pipeline of the fusion process

Here W represents the weights of the other layer in the feature analysers, $W^E=\left[W_n^s,W_n^p\right]\in {\mathbb{R}}^{P\times D}$ represents the stacked weights for the fusion layer. $\mathcal{L}={\sum }_{i=1}^N\mathrm{\wp }\left(X_n^s,X_n^p\right)-l_i^2$, $\mathrm{\wp }$ denotes the non-linear function approximated by the neural network.

• (i) The other layer: Since there are no non-smooth regularisations for other layers, we compute their gradients directly and then update the weight matrix with gradient descent as in [13]. Let $G_l$ represent the gradients of ${\mathit{W}}_l$, the weight matrix of the lth layer is updated as

  $${\mathit{W}}_l={\mathit{W}}_l-\eta G_l$$ (2)
• (ii) The feature fusion process: To update the weights for the ith iteration, a proximal operator is implemented as

  $${\left(W^E\right)}^{\left(i\right)}=\mathrm{Pro}{\mathrm{x}}_q({(W^E)}^{\left(i\right)}-\mathrm{\nabla }p({(W^E)}^{\left(i\right)}))$$ (3)

  where $\mathrm{Pro}{\mathrm{x}}_q\left(W\right)=\mathrm{argmi}{\mathrm{n}}_{V}W-V+q\left(V\right)$. Note that q here is ${\ell }_{2,1}$ norm, and thus the proximal operator can be derived as

  $$W_r^E=\left(1-\frac{{\lambda }_2}{U_{r2}}\right)U_r$$ (4)

  where $U_r=\max \{V_r,0\}\cdot \mathrm{sign}[V_r]$, and $W_r,U_r,V_r$ represents the rth row of matrix $\mathit{W}$, $\mathit{U}$ and $\mathit{V}$, respectively. The overall training process of the model is shown in Fig. 5.

Fig. 5.

Algorithm 1: training the particle space-time distribution model

3. Experiment and results

3.1. Implementation details

The ConvNet contains four convolution layers with kernel size 3*3. The LSTM layer contains 512 hidden neurons for the first layer and 256 units for the second layer. To learn the optimal weights, we follow the procedures described in Algorithm 1 (see Fig. 5) using the TensorFlow Toolbox. We use an adaptive-moment-estimation with a batch size of 1, a learning rate of 0.01 and epochs of 200. ${\lambda }_1$ is fixed to $2\times {10}^{-5}$ for preventing over-fitting. ${\lambda }_2$ is selected by three-fold cross-validation.

In the experiment, the training set is comprised of 120 patients of breast tumour including 86 patients with ALN metastasis, and 34 patients without ALN metastasis. The testing set consists of 42 patients including 24 patients with ALN metastasis, and 18 patients without ALN metastasis. Each patient contains the contrast-enhanced ultrasonography images of breast tumour and the ground-truth label of ALN metastasis from ALND with pathological examination. The used ultrasound data is the common contrast-enhanced ultrasound of breast tumour, which is the basic scanning examination, therefore, the general clinical workflow will not be changed.

3.2. Results

To explore whether dynamic contrast-enhanced ultrasonography of breast tumour can affect and perform ALN prediction, we finished the prediction of ALN metastasis by using the proposed prediction model. We evaluated the prediction result with some index: the average precision (AP), classification accuracy (Acc) and the average area under the receiver operating characteristic curve (AUC) values. We compared three models: particle space-time distribution model with both space-time domain feature and feature of particle distribution domain (Model_combined); model with feature of space-time domain (Model_space-time) and model with the feature of particle distribution domain (Model_particle).

Fig. 6a shows the ROC curves and area under the ROC curve (AUC value) obtained by applying different prediction models, respectively. It reveals that our particle space-time distribution model finishes the best results. Therefore, the proposed model is effective by incorporating the space-time features of dynamic contrast-enhanced ultrasonography and the flow field information of microbubble particles that better serves the metastasis prediction by enhancing the particle distribution information. More comprehensively, we give the AP, ACC and AUC values of different models on the test set in Table 1. It clearly demonstrates that the proposed model makes a big contribution by using new feature fusion and extraction. As shown in Fig. 6b, in our particle space-time distribution model, the mean probability (score) that a patient with positive ALN metastasis is predicted to be a negative ALN metastasis type is 0.68 [95% confidence interval (CI): 0.54–0.73], a patient with negative ALN metastasis is predicted to be a positive ALN metastasis is 0.34 [95% confidence interval (CI): 0.19–0.41] in testing data. Our method took computation time of about 2.31 s to process a sequence of contrast-enhanced ultrasound of breast tumour on an average.

Fig. 6.

The prediction results of ALN metastasis by using the proposed prediction model

a ROC curves and AUC values of different models,

b Particle space-time distribution model scores for patients with and without ALN metastasis

Table 1.

Performance of different models on the testing data

Models	AP	ACC	AUC
Model_combined	0.574	0.797	0.732
Model_space-time	0.513	0.705	0.655
Model_particle	0.521	0.731	0.686
I3D	0.568	0.763	0.702

To further validate our method, on the one hand, we added a four-fold cross-validation experiment. Table 2 shows the results of our cross-validation result. During the cross-validation experiments, the average values of the AUC, AP and Acc during the cross-validation experiments achieved 0.725, 0.569 and 0.788, respectively. These results further validate our proposed method for prediction of ALN metastasis. On the other hand, we finished the comparison experiment between our method and with the standard video classification method of Two-Stream I3D [14]. The comparison results in Table 1 validate that our proposed method outperformed the I3D model with higher prediction accuracy.

Table 2.

Cross-validation accuracy results of our method

	Average	Min	Max
AUC	0.725	0.698	0.742
AP	0.569	0.558	0.592
Acc	0.788	0.752	0.801

3.3. Clinical explanation of the results

Experimental results on real patients demonstrated that ultrasonography information of breast tumour needs to be considered for predicting the probability of ALN metastasis. This result is interpretable from the clinical and pathological perspectives. On the one hand, studies have shown that higher grade of breast tumour detected in ultra-sonography information are more prone to cause tumour cell drainage via lymphatic duct, increasing the probability of lymphatic metastasis [15]. On the other hand, contrast-enhanced ultrasound of breast tumour reveals the whole process of vessel perfusion and nourishing information of the cancer. Vascular growth factor and micro-vessel density of tumour are closely related to ALN metastasis in breast cancer because neovascularisation is a necessary condition for tumour growth, local invasion and distant metastasis [16]. Therefore, the higher the rate of ALN metastasis is, the more vessels nourished detected in ultrasonography of breast cancer tumours are.

4. Conclusion

It is quite difficult for clinicians to predict ALN metastases from ultrasound images directly [17]. In this Letter, we finish probability analysis of ALN metastasis from contrast-enhanced ultrasonography of breast tumour by using the deep particle space-time distribution model. Therefore, our main contribution is using the proposed prediction model to verify the connection between dynamic contrast-enhanced ultrasonography of breast tumour and ALN metastasis quantitatively. This conclusion can be interpretable from the clinical and pathological perspectives. Therefore, different from the current studies which only focus on the prediction of lymph node metastasis with axillary ultrasound, the next step of our work is collecting the dynamic contrast-enhanced ultrasonography of breast and axillary ultrasound, and finishing the prediction of ALN metastasis with higher accuracy by using these two kinds of ultrasound images together.

6. Funding and declaration of interests

This work was supported by National Nature Science Foundation of China grants (61901214), National Key Research and Development Program of China (2018YFC2001600,2018YFC2001602), Fundamental Research Funds for the Central Universities (NJ2019010).