Morphological active contour without edge-based model for real-time and non-rigid uterine fibroid tracking in HIFU treatment

Abstract

High-intensity focused ultrasound (HIFU) therapy represents an image-guided and non-invasive surgical approach to treat uterine fibroid. During the HIFU operation, it is challenging to obtain the real-time and accurate lesion contour automatically in ultrasound (US) video. The current intraoperative image processing is completed manually or semi-automatic. In this Letter, the authors propose a morphological active contour without an edge-based model to obtain accurate real-time and non-rigid US lesion contour. Firstly, a targeted image pre-processing procedure is applied to reduce the influence of inadequate image quality. Then, an improved morphological contour detection method with a customised morphological kernel is harnessed to solve the low signal-to-noise ratio of HIFU US images and obtain an accurate non-rigid lesion contour. A more reasonable lesion tracking procedure is proposed to improve tracking accuracy especially in the case of large displacement and incomplete lesion area. The entire framework is accelerated by the GPU to achieve a high frame rate. Finally, a non-rigid, real-time and accurate lesion contouring for intraoperative US video is provided to the doctor. The proposed procedure could reach a speed of more than 30 frames per second in general computer and a Dice similarity coefficient of 90.67% and Intersection over Union of 90.14%.

1. Introduction

The minimally invasive or non-invasive treatment for uterine fibroids, such as magnetic resonance-guided high-intensity focused ultrasound (HIFU) therapy, has been rapidly evolving in recent years due to its smaller incision and better prognosis. In non-invasive treatments, especially in HIFU treatment, medical images are the primary basis for diagnosis and operation. In the intraoperative stage, ultrasound (US) video is a non-invasive and real-time imaging approach, which can detect the uterine fibroids and the surrounding soft tissues. During the operation, the doctor controls the movement of a HIFU probe based on US video. The focus point of HIFU should be concentrated in the central part of the lesion to achieve the best therapy performance. However, US images are always with low resolution, low contrast, low signal-to-noise ratio (SNR) and noises, especially during HIFU treatment. Moreover, the HIFU probe is located in the water sink. Compared with the couplant used in a conventional US scanning, water will cause considerable noise during propagation and a worse image SNR. Because the contour of the uterine fibroids is always quite fuzzy in the US video during significant intraoperative tissue deformation, the current HIFU operation can only be performed by the doctors who have lots of experience. The purpose of treatment can be achieved with an intraoperative focus, which is concentrated in the central part of the lesion. Therefore, the contour of the HIFU US image lesion is required to be ∼5 mm. Therefore, a high-precision, automatic and real-time uterine fibroid tracking technique based on US video is needed to help more doctors to better identify the lesion.

In HIFU therapy, fast lesion tracking in US images is a key research point, and many methods for real-time US video processing have been reported or applied in different scenes. Pernot et al. [1] used a triangulation method and three echoing signals. The results showed that a moving lesion could be located accurately by detecting the location of a single point but lacking contour deformation. Oliveira et al. [2] applied a US sensor to estimate the motion of a moving organ. However, sensor-based methods cannot locate the tumour. Kim et al. [3] achieved the goal of real-time tracking the organ and the tumour in US images in real-time, based on a preoperative 3D deformable organ model experiment. This method is essentially a registration of US and MR images, rather than delineating the lesion area in intraoperative US. Liao et al. [4] used an adaptive localised region and edge-based active contour model to achieve US image segmentation in HIFU therapy but not a real-time method. Compared with the method above, we proposed a real-time and non-rigid US lesion tracking pipeline based on morphological active contour without edge-based model and targeted procedures for the HIFU US video.

2. Method

2.1. Overview

Considering the main problems in HIFU US lesion tracking is the lack of real-time processing and low image SNR, our framework consists of three main steps. First, a targeted preprocessing procedure is used to remove noise and other disturbances in the HIFU US image. Second, we proposed a new morphological kernel using an improved morphological active contour without edges (MACWEs) [5] method to obtain the contour of the intraoperative lesion area in real-time. Furthermore, two special procedures including large deformation detection and incomplete area processing are used to better guarantee the accuracy and the robust of the contour tracking in some special clinical cases. The whole framework is accelerated by the GPU to obtain a high tracking frame rate.

2.2. MACWE-based model for non-rigid deformation lesion tracking

The focused US probe of the HIFU device is mounted in a water sink that causes an echo in the US image. Compared to the conventional US scanning using couplant, the HIFU US probe cannot be placed close to the patient's skin, which results in more noise in the US video. Therefore, proper preprocessing is necessary. To address the main speckle noise in the image produced by the echoes, we use a morphological close operation instead of conventional image filtering. To improve the image contrast and reduce disturbance, especially in the lesion area, we use an efficient linear conversion to highlight the lesion and remove the bladder.

The three main factors for the doctors to judge the lesion and therapy plan are the relative position between organs, the regional greyscale and the regional shape. Based on the MACWE model, we propose an improved real-time tracking method to specifically target the lesion area in HIFU US images. Generally, the MACWE model-based image segmentation method requires accurate initialisation of the contour. In the current HIFU treatment device, the HIFU device is equipped with external calibration systems. When the patient is lying above the HIFU device during treatment, the HIFU device can obtain the accurate location of the current US image corresponding to the patient's body $P_U^p$. Similarly, the position of the human body corresponding to the preoperative MR image $P_M^p$ can be obtained. The segmentation result of the preoperative MR can be accurately located on the lesion in the US image based on $P_U^p$ and $P_M^p$ as the initial contour. Thus, we need to mainly consider the shape and greyscale characteristics of the lesion.

The pre-processed HIFU US image has a more pronounced contrast than the original image, but the speckles in the image still have a large impact on the lesion contour judgment. In previous research [5], morphological operators have been proven to exhibit infinitesimal behaviour, such as partial differential equations, and a complex morphological operator was proposed to smooth the implicit hypersurfaces of the region. Therefore, we propose a process for the extraction of lesions in US.

For a given preprocessed US frame $I_u^{n\left(p\right)}$, the current hypersurface in $I_u^{n\left(p\right)}$ is defined as the level set 1/2 of a binary embedding function $u^n:\mathbb{Z}\to \left\{0,1\right\}$ based on the initialised contour ${\mathcal{C}}_{\mathrm{init}}^n$. However, the values of image greyscale parameters including v, ${\lambda }_1$, ${\lambda }_2$ and $\mu$ need to be set manually in the traditional algorithm, which is not suitable for real-time and automatic contour transformation. Thus, we set these parameters as unite values, and the algorithm is represented as

$$\begin{array}{rl}{u}^{n+(1/2)}\left(\mathit{x}\right)& =\left\{\begin{array}{cc}1,& \mathrm{if}\left|▽u^n\right|((I_u^{n\left(p\right)}-c_1{)}^2-(I_u^{n\left(p\right)}-c_2{)}^2)\left(\mathit{x}\right)<0\\ 0,& \mathrm{if}\left|▽u^n\right|((I_u^{n\left(p\right)}-c_1{)}^2-(I_u^{n\left(p\right)}-c_2{)}^2)\left(\mathit{x}\right)>0\\ u^n\left(\mathit{x}\right),& \mathrm{else}\end{array}\right.\\ u^{n+1}\left(\mathit{x}\right)& =(\left(SI\circ IS\right)u^{n+(1/2)})\left(\mathit{x}\right)\end{array}$$ (1)

where $SI\circ IS$ is the compound morphological operations including erosion and dilation. The values $c_1$ and $c_2$ are the mean of the values of $I_u^{n\left(p\right)}$ inside and outside the contour.

In the conventional algorithm, the traditional morphological operations in $SI\circ IS$ are comprised of four discrete segments ${\mathcal{K}}_3=\left\{k_3^1,k_3^2,k_3^3,k_3^4\right\}$, which have three pixels of lengths that involve all shapes of the edge (Fig. 1). However, in HIFU US videos, the uterine fibroids have some internal structures that cause greyscale differences. Therefore, we replace the ${\mathcal{K}}_3$ morphological operator by one proposed large discrete segment ${\mathcal{K}}_{15}=\{k_{15}\}$ of 15 pixels in length. For larger-sized operators, smaller interference structures in the image can be removed during morphological operations. The operator ${\mathcal{K}}_{15}$ is shown in Fig. 1. This larger structure can better solve the problem over-deformation of ${\mathcal{K}}_3$.

Fig. 1.

Conventional morphological kernel ${\mathcal{K}}_3$ and the proposed targeted morphological kernel ${\mathcal{K}}_{15}$

2.3. Large displacement detection and incomplete area processing

During the HIFU operation, a large range of movement (>5 cm) of the lesion area in the US image is encountered occasionally. Although we use the MACWE method with a specific operator to track the contour, we still cannot track the lesions in time when large displacement occurs. Therefore, we use a mutual information method to detect large displacements in US videos. We perform mutual information (MI) detection on the original US image every ten frames, and the formula [6] for the current frame is

$$\mathrm{MI}(I_u^n,I_u^{n-10})=\sum _{{\left(\mathit{x}\right)}^{\mathit{n}-5}}\sum _{{(\mathit{x})}^{\mathit{n}}}p(I_u^n,I_u^{n-10})\log \left(\frac{p(I_u^n,I_u^{n-10})}{p(I_u^n)p(I_u^{n-10})}\right)$$ (2)

where the $p(I_u^n,I_u^{n-10})$ is the joint distribution function of $I_u^n$ and $I_u^{n-10}$. The $p(I_u^n)$ and $p(I_u^{n-10})$ are marginal density functions of $I_u^n$ and $I_u^{n-10}$.

Template matching is performed on the current frame after deformation tracking, and a driving force is applied to the contour. Once a large displacement has been detected, the template matching method based on normalised cross-correlation (NCC) [7] is harnessed to find the best matching position with the initial contour in a limited range. The matching range $\left[a,b\right]$ is selected based on the empirical displacement of the lesion between frames. The NCC for image I and template u as random variables with samples $I_i,u_i,i=a,\dots ,b$ is defined as

$$\mathcal{P}\left(\mathit{I},\mathit{u}\right)=E\left[\left(\frac{\mathit{I}-E\left[\mathit{I}\right]}{\sqrt{\mathrm{var}\left(\mathit{I}\right)}}\right)\left(\frac{u-E\left[\mathit{u}\right]}{\sqrt{\mathrm{var}\left(\mathit{u}\right)}}\right)\right]$$ (3)

where $E\left[\mathit{x}\right]$ and $\mathrm{var}\left(\mathit{x}\right)$ denote the empirical mean and variance for vectors $\mathit{x}\in {\mathbb{R}}^{b-a}$. After the $\mathcal{P}$ is confirmed, the vector from $\mathcal{P}$ to the centroid of contour ${\mathcal{C}}^n$ is considered as a moving force $\mathit{m}$ with a parameter to $\alpha$ next frame.

In a large number of clinical cases, the lesion area in the HIFU US image is incomplete because of continuously tissue deformation, which will result in a significant error, which we call it ‘overflow’ in the contours produced by the MACWE method. In order to solve this error, we added an incomplete area processing procedure in the generation a level set from the image gradient. This procedure is to detect the proportion of the level set result. If most are positive (inclined gradients) in the level set, the contours are essentially aligned with the real lesion area. Finally, the entire algorithm is defined as follows:

$$\begin{array}{rl}{u}^{n+(1/4)}\left(\mathit{x}\right)& =\left\{\begin{array}{cc}1,& \mathrm{if}\left|▽u^n\right|((I_u^{n\left(p\right)}-c_1{)}^2-(I_u^{n\left(p\right)}-c_2{)}^2)\left(\mathit{x}\right)>0\\ 0,& \mathrm{if}\left|▽u^n\right|((I_u^{n\left(p\right)}-c_1{)}^2-(I_u^{n\left(p\right)}-c_2{)}^2)\left(\mathit{x}\right)<0\\ u^n\left(\mathit{x}\right),& \mathrm{else}\end{array}\right.\\ u^{n+(2/4)}\left(\mathit{x}\right)& =\left(\left(SI\circ IS\right)u^{n+(1/4)}\right)\left(\mathit{x}\right),\\ u^{n+(3/4)}\left(\mathit{x}\right)& =\left\{\begin{array}{cc}{u}^{n+(2/4)}\left(\mathit{x}\right),& \mathrm{if}\sum \left|▽u^n\right|((I_u^{n\left(p\right)}-c_1{)}^2-(I_u^{n\left(p\right)}-c_2{)}^2)\\ & \left(\mathit{x}\right)<0\\ u^n\left(\mathit{x}\right),& \mathrm{else}\end{array}\right.\\ u^{n+1}\left(\mathit{x}\right)& =\left\{\begin{array}{cc}{u}^{n+(3/4)}\left(\mathit{x}+\alpha \mathit{m}\right),& \mathrm{if}\mathrm{MI}>t\\ u^{n+(3/4)}\left(\mathit{x}\right),& \mathrm{else}\end{array}\right.\end{array}$$ (4)

In the entire algorithm flow, including morphological operations, template matching and mutual information, all these calculations are matrix operations and suitable for GPU calculation. We implement these calculations in GPU by processing each pixel in the US image with one corresponding GPU thread. Parallel computing speed is triple or quadruple improved compared with CPU calculation and meets the real-time requirement.

3. Experiments and results

In the experiment, the performance of the proposed produces and methods was evaluated by different US cases and processes. We mainly evaluated the performance of the calculating efficiency, accuracy of real-time lesion tracking procedures for clinical HIFU guidance. Totally ten randomly selected clinical and typical cases were gathered for the experiments. The computing platform included a CPU (Intel (R) Core (TM) i7-4790K) and a GPU (NVIDIA GeForce 1080 Ti). The corresponding US videos with HIFU device information are provided by National Engineering Research Center of Ultrasound Medicine (JC200/300 Haifu treatment system, Haifu, China).

3.1. Performance of real-time non-rigid US lesion tracking

The accuracy and efficiency experiments were carried out to evaluate the improved MACWE algorithm for lesion deformation in US video. One 20 s US video was manually labelled by an experienced doctor (set as ground truth) to quantitative evaluate the accuracy of the proposed method. Furthermore, other several typical clinical cases were used to evaluate the intuitionistic performance of the proposed method.

The comparison of lesion tracking using the ${\mathcal{K}}_{15}$ and ${\mathcal{K}}_3$ was shown in Fig. 2. The contours acquired by ${\mathcal{K}}_{15}$ were more robust to interference structures in HIFU US images. In contrast, the contours acquired by ${\mathcal{K}}_3$ were disturbed by small structures and have large errors compared with the ground truth. In the first two seconds, the results of the two methods were similar because the initial contours were the same. When large deformation occurred (Frame#240–#540), the proposed ${\mathcal{K}}_{15}$ has better sensitivity to the major structures. Thus, as shown in Table 1, the results of Dice similarity coefficient (DSC) [8], Hausdorff distance (HD) [9] in mm and Intersection over Union (IoU) of the ${\mathcal{K}}_{15}$ were better. Moreover, since the number of iterations was reduced in the proposed algorithm, the calculation speed of ${\mathcal{K}}_{15}$ is faster than that of ${\mathcal{K}}_3$, which resulted in the higher FPS of video processing. A comparison of time consumption between ${\mathcal{K}}_{15}$ and ${\mathcal{K}}_3$ in one iteration is shown in Table 2.

Fig. 2.

20 s US video of HIFU treatment was manually labelled by doctors (red contours). The blue contours were generated by the proposed ${\mathcal{K}}_{15}$ morphological ACWE and the yellow contours were generated by ${\mathcal{K}}_3$ morphological ACWE in same US frames

Table 1.

Evaluation results of the proposed ${\mathcal{K}}_{15}$ and ${\mathcal{K}}_3$ in DSC, HD (in mm) and IoU compared with the ground truth

Frame	Kernels
	${\mathcal{K}}_{15}$ DSC	${\mathcal{K}}_3$ DSC	${\mathcal{K}}_{15}$ HD	${\mathcal{K}}_3$ HD	${\mathcal{K}}_{15}$ IoU	${\mathcal{K}}_3$ IoU
#0	0.95	0.95	1.40	1.40	0.93	0.93
#100	0.93	0.92	1.75	3.50	0.89	0.91
#200	0.92	0.87	2.45	3.15	0.88	0.83
#300	0.92	0.78	1.50	3.15	0.92	0.80
#400	0.90	0.76	1.75	3.85	0.89	0.80
#500	0.92	0.71	1.75	5.25	0.92	0.81
#600	0.93	0.79	1.75	4.2	0.88	0.83

Table 2.

Comparison of time consumption in one iteration between the proposed ${\mathcal{K}}_{15}$ and ${\mathcal{K}}_3$

Computing platform	CPU, ms	GPU, ms
${\mathcal{K}}_{15}$	20	8
${\mathcal{K}}_3$	68	16

We integrated the proposed methods in the commercial HIFU treatment system (JC200/300 Haifu treatment system, Haifu, China). The surgeon performed the surgery with the intraoperative image we provided. A total of randomly selected eight clinical trials were performed. The experimental results were shown in Fig. 3. The proposed method can accurately identify the lesion area in cases of different shapes, sizes and locations.

Fig. 3.

Experimental results of typical clinical cases of in different shapes, sizes and positions

3.2. Performance of large displacement detection and incomplete area processing produces

The qualitative evaluation result and the effect of large displacement detection are shown in Figs. 4. In this case, a large displacement occurred in the lesion area within 5 s. The proposed large displacement detection procedure detected 8 times during this large movement event (the region in a rectangle). Two of these results (orange rectangle) were identified as invalid matches because they were out of matching range. The moving force obtained by the template matching moved the contour to the detected position (the blue contour in (a)). Compared with the deviated contour without moving force (red contour in (b)), the contour position remains unchanged during the fast and large movement event, which is completely inconsistent with the lesion region.

Fig. 4.

Continuous frames in US video with a fast and large movement of the lesion area. A total of eight frames were identified as having a large displacement. The contours with large displacement detection (a) of the lesion area are more accurately than the contours without large displacement detection (b)

The qualitative evaluation of the efficiency of incomplete area processing is shown in Fig. 5. A significant boundary missing occurred in the lesion area in the upper left of the lesion area. When the lesion area was incomplete in the US image, the contour would result in a significant structural error using the conventional MACWE method.

Fig. 5.

Sample US case with incomplete lesion area and evaluation results of different methods

In the incomplete position, the region outside the lesion region has a high similar greyscale feature with the interior area of the lesion region. This results in a structural error in the contour of the incomplete position when using the conventional MACWE method. In the level set generated by the image, the gradient value is positive (white) when the edge reaches the part where the gradient changes sharply. Correspondingly, the value of the partial level set whose gradient change is not obvious is negative (black). This value feature was used in the proposed incomplete area processing to determine whether the contour has outlined the lesion area or not. In general, the contour is considered as right when most of the level set is positive in the current frame. With this procedure, a better result was obtained, as shown in Fig. 5.

4. Discussion and conclusion

In this Letter, we present a MACWE-based method to achieve real-time non-rigid mapping of US image lesions in HIFU therapy. We proposed a targeted morphological operator and several procedures to solve clinical problems including large movement and incomplete lesion region in a HIFU US video. Quantitative and qualitative experiments indicated that the proposed method and improvement could accurately obtain the lesion area in the HIFU US image and overcome clinical disadvantages. Evaluation results show that our method can meet clinical needs. Moreover, the use of the GPU also makes the framework to reach a high FPS with more than 30 FPS.

In future works, we will consider using artificial intelligence to initialise the contour instead of using the current MR image initialisation method.