Multi-needle Detection in 3D Ultrasound Images Using Unsupervised Order-graph Regularized Sparse Dictionary Learning

Abstract

Accurate and automatic multi-needle detection in three-dimensional (3D) ultrasound (US) is a key step of treatment planning for US-guided brachytherapy. However, most current studies are concentrated on single-needle detection by only using a small number of images with a needle, regardless of the massive database of US images without needles. In this paper, we propose a workflow for multi-needle detection by considering the images without needles as auxiliary. Concretely, we train position-specific dictionaries on 3D overlapping patches of auxiliary images, where we develop an enhanced sparse dictionary learning method by integrating spatial continuity of 3D US, dubbed order-graph regularized dictionary learning. Using the learned dictionaries, target images are reconstructed to obtain residual pixels which are then clustered in every slice to yield centers. With the obtained centers, regions of interest (ROIs) are constructed via seeking cylinders. Finally, we detect needles by using the random sample consensus algorithm per ROI and then locate the tips by finding the sharp intensity drops along the detected axis for every needle. Extensive experiments were conducted on a phantom dataset and a prostate dataset of 70/21 patients without/with needles. Visualization and quantitative results show the effectiveness of our proposed workflow. Specifically, our method can correctly detect 95% of needles with a tip location error of 1.01 mm on the prostate dataset. This technique provides accurate multi-needle detection for US-guided HDR prostate brachytherapy, facilitating the clinical workflow.

I. INTRODUCTION

MANY minimally invasive procedures require inserting a needle to access a target site inside the patient’s body, avoiding the requirement of making large incisions. Accurate placement of the needle is vital to interventions, as incorrect needle insertion can lead to a failure of the procedure and even cause some complications. Ultrasound (US) images are broadly adopted to visualize and guide the interventions because of its low cost, non-invasiveness, and safety. However, needle detection in US images remains a challenging problem due to the low signal-to-noise ratio and image artifacts in US imaging [1]–[3]. To achieve accurate needle localization, many techniques have been developed, principally classified into mathematical theory based-methods [4], [5] and learning based-methods [6], [7].

Most mathematical theory based-methods are developed by modeling the needle as a line or a cylinder in a region of interest (ROI), among which intensity threshold, line filter and physical property are incorporated to enhance the visibility of suspected needle [1], [4]. Uherčík et al. used a threshold method to separate needle voxels from the background and then designed a needle fitting model by using a random sample consensus (RANSAC) algorithm [4]. This approach was further improved by Zhao et al., where a line filter was used to automatically identify ROI, and then a RANSAC with Kalman filter was used in the ROI to localize a needle [8]. Novotny et al. used principal component analysis to obtain the major axis of an inserted instrument and the linear least-squares fit to localize the center [9]. The needle reflection pattern of US waves was modeled by Daoud et al. for needle trajectory detection [5]. To obtain a better fit, Zhou et al. combined a novel line representation model with the 3D Hough transform to detect a straight needle [10]. Furthermore, several enhancements and normalization processes were introduced to identify ROI in the work of Pourtaherian et al. [11]. Recently, Daoud et al. proposed a hybrid camera- and US-based method to localize and track a needle using a 3D curvilinear US probe [12]. However, these approaches focus on the images from the target patient, regardless of cross-patient anatomical similarity which can provide helpful information for needle detection.

Learning based-method is usually to train a machine learning model to localize and segment the needle on US images [6], [7]. Geralders et al. used a multilayer perceptron network to classify the image pixels into a needle or background where the real labels were given [2]. Beigi et al. trained a probabilistic support vector machine using temporal features for pixel classification and computed the probability map as well for localizing needle with Hough transform [6]. Pourtaherian et al. proposed two deep convolutional neural networks (CNN) based methods to identify needle voxels from background and echogenic structures, e.g., bones and muscular tissues [7]. They also presented a dilated-CNNs based method to detect needles in phased-array 3D US volumes [13]. Mwikirize et al. trained a combination of a fully convolutional network and a region-based CNN in the transfer paradigm for needle detection in 2D US images [14]. However, the current learning-based methods often suffer from several shortcomings: 1) The used models’ lack of the consideration of the priors on needles. 2) The requirement of ground truths, causing expensive and time-consuming manual labeling cost. 3) The small volume of needle images that are often insufficient for training a sophisticated machine learning model.

Recently, a few approaches have been proposed to combine mathematical theory-based method and learning-based method, where image contrast between a needle and the background is first enhanced by a mathematical model, followed by training a classifier to identify needle pixel/voxel. For instance, Uherčík et al. proposed a three-step method for surgical tool localization in US images, including line filtering, a voxel classifier and the RANSAC method [15]. Mwikirize et al. used the split Bregman method to augment needle tip and then trained a deep-learning model to identify needle tip [16]. Since a hybrid scheme could benefit from both strategies, in this paper, we consider solving the problem of needle detection in this fashion. Unlike most of the current detection methods concentrated on a single needle, we propose a new workflow for multi-needle detection, which is label-free, low-cost, and practical against the sample-scares situation. Besides, we define the US images with needles as the “target” and the images without needles as the “auxiliary”. The major contributions of our work include:

1. A general multi-needle detection workflow, which aims at detecting all needles simultaneously while considering the interactions between the inserted needles. This is the first attempt at developing a solution to this key clinical application, e.g., prostate brachytherapy [17].

2. A useful technique for needle enhancement, dubbed order-graph regularized dictionary learning (ORDL). ORDL extends the classical dictionary learning model to integrate the spatial continuity of 3D US images [18], [19].

3. The introduction of a knowledge-transforming method by dictionary learning, also known as self-taught learning [20]. This method is capable of learning intrinsic features from auxiliary images to rebuild the background of target images.

4. A needle fitting method that iteratively performs RANSAC in overlapping ROIs. In each iteration, the inliers are used to reconstruct a needle and then removed, while the outliers are kept to detect other needles.

5. The evaluation of our approach using 3D US images dataset from our prostate high-dose-rate (HDR) brachytherapy. The results of visualizations and quantitative evaluations show that the proposed approach is very auspicious.

The rest of this paper is organized as follows. In Section II, we formulate the needle detection problem in mathematics, and briefly review sparse dictionary learning. Section III introduces the proposed workflow and its detailed steps, including the ORDL algorithm and a needle-fitting algorithm. Then we present the implementation details and the experimental results in Section IV and Section V, respectively. Section VI contains discussions, and Section VII finally concludes this paper.

II. The Problem And Sparse Dictionary Learning

In this section, we mathematically formalize the problem of multi-needle detection based on the clinical situation and prior knowledge of physical properties. We then briefly review the technique of sparse dictionary learning (SDL), also known as sparse coding [21], which is a promising method for transfer learning against this small-sample-size situation [20].

A. The Multi-Needle Detection Problem

During the most minimally invasive procedures in the clinic, US images provide the visualization needed to aid needle localization, as shown in Fig. 1(left) [1]. Intuitively, needles can be modeled as cylinders with curvature and have relatively high US image intensities. We explicitly model the needle detection task with a noise term as Problem 1 in mathematics.

Fig. 1.

Example slices from 3D US images of the prostate. The left picture is an image with needles, while the right one is an image without needles. The cyan short lines present the rulers of US images.

Problem 1 (Needle Detection).

Let v be the coordinates of an arbitrary voxel in an observed image Y. As shown in Eq. (1), we consider Y (v) to be composed of three components: needle N(v), tissue Ŷ(v) and noise E(v). The goal of needle detection is to extract N(v) from Y (v).

$$\mathbf{\text{Y}}(\mathbf{\text{v}})-E(\mathbf{\text{v}})=\widehat{\mathbf{\text{Y}}}(\mathbf{\text{v}})+N(\mathbf{\text{v}}).$$ (1)

Note that v is a vector for a 2D or 3D image. Alternatively, we can write Eq. (1) into matrix notation [22], i.e., Y – E = Ŷ + N, where one matrix element corresponds to a v. When matrix N contains one needle, Problem 1 is known as the single-needle detection that has been extensively studied [1]. If N has more than one needle, Problem 1 is here defined as the multi-needle detection that has not been well investigated. It’s been claimed that multi-needle detection might be solved with multiple single-needle detections in multiple ROIs, but it fails to consider the interactions between needles and an automatic ROI method [23]. Our study is focused on multi-needle detection in 3D US images, which is generalizable, useful and challenging.

As stated in Eq. (1), we can achieve N via Y – Ŷ – E to solve Problem 1. Alternatively, our problem is transformed into the extraction of Ŷ and E from Y. It could be solved by many offthe-shelf learning techniques, such as classification models and matrix factorization [24]. However, those methods are mostly limited by small-sample-size issues and/or lack off-line models for a timely manner. Fortunately, an extensive amount of US images without needles has been accumulated, shown in Fig. 1 (right), which are taken on the same tissue with the US image in Fig. 1 (left). In this identical situation, there has the following commonly used image prior:

Assumption 1.

Given sufficient images from the specific tissue denoted by S, let Φ be all the features contained in S. Let Ψ be all the features of target images from an identical situation with S. Then we can derive the two following formulas:

$$\Psi \subseteq \Phi \text{\hspace{0.17em}and\hspace{0.17em}}\#\left\{\Psi \right\}<<\#\left\{\Phi \right\},$$

where #{} is to count the elements in a collection.

Under Assumption 1, we propose to learn the latent features from the auxiliary US images and then employ these features to reconstruct the background of target US images, as well as the image artifacts. The residual difference between target images and their reconstructed versions can be naturally assumed to be the inserted needles since needle pattern is not learned by those latent features. In this paper, we adopt the popular technique of sparse dictionary learning [21] to learn the latent features from auxiliary images without needles, regardless of the small effect from the tiny-needle insertions on image feature pattern.

B. Sparse Dictionary Learning Model

Sparse dictionary learning (SDL) aims to learn overcomplete features, i.e., dictionary so that both given data and unseen data can be sparsely represented on the resultant dictionary [25]. Let $\mathbf{\text{D}}\in {ℝ}^{\mathbf{\text{D}}\times m}$ be a dictionary with per column d_l being an atom. Let $\mathbf{\text{X}}\in {ℝ}^{m\times n}$ be the sparse codes, which is the representation matrix of the data $\mathbf{\text{Y}}\in {ℝ}^{\mathbf{\text{D}}\times n}$ . The column x_i of X is corresponding to Y’s column y_i (i.e., data point). SDL is usually formulated as:

$$\underset{\mathbf{\text{D}},\mathbf{\text{X}}}{\text{min}}\Vert \mathbf{\text{Y}}-\mathbf{\text{D}}\mathbf{\text{X}}{\Vert }_F^2+\lambda \sum _{i=1}^n{\Vert {\mathbf{\text{x}}}_i\Vert }_p\text{\hspace{0.17em}s.t.\hspace{0.17em}}{\Vert {\mathbf{\text{d}}}_l\Vert }_2^2\le 1,l=1,2,\dots ,m$$ (2)

where F is Frobenius norm, λ > 0 is a regularization parameter, p = {0, 1} indicating λ₀ pseudo-norm and λ₁ norm [21]. The imposed constraint is to ensure (2) has a unique solution. The problem (2) is usually solved by iterative optimization between D and X [25], [26]. Over the past two decades, SDL has attracted great attention and produced many variations, such as low-rank sparse coding [27] and convolutional dictionary learning [28].

SDL has two major strengths of our practical problem. First, SDL is more effective among machine learning methods with a small-size dataset because of the various regularization terms for application priors [28]. Second, SDL is a useful method for transform learning, which is there dubbed self-taught learning [20], [29]. In self-taught learning, SDL could learn the common features from unlabeled similar side-data to boost the solution to the target problem, e.g. learning from the sufficient images of landscape to distinguish elephant from rhino [20]. In this paper, we use SDL to learn common features from auxiliary images in order to rebuild the background and artifacts of the target images. In addition, observed from the 3D US images, we have another useful assumption, as expressed below:

Assumption 2.

Given a 3D US images Y, let y_i, yj be two slices from Y, and x_i, xj be their codes over the new features. If y_i and yj are adjacent slices, i.e.,| i – j |= 1 , then ‖x_i – x_j ‖ is small.

Due to the independent coding process and the overcomplete dictionary, the formulation (2) violates Assumption 2 and thus loses spatial continuity. Gao et al. presented Laplacian sparse coding (LSC) [30] and Yankelevsky et al. proposed dual graph dictionary learning (graph²DL) [31] to preserve the surrounding locality. However, our dataset is composed of multiple ordered image groups from multiple patients. To this end, we design a new formulation of SDL to encode Assumption 2.

III. The Proposed Approach

The proposed workflow of needle detection mainly consists of preprocessing, our ORDL for needle reconstruction, iterative RANSAC and result refinement, as shown in Fig. 2.

Fig. 2.

The workflow of the proposed approach for multi-needle detection. The bounding boxes of size 60 mm × 50 mm are used for image cropping.

A. Image Preprocessing

We conduct the following operations for preprocessing all US images, with the aims of high performance and efficiency:

a). Image cropping.

All original slices are cropped referred to the rulers into a box of 60 × 50 (mm), where the US rulers can be found in Fig. 1 and this bounding box in the center of image slice can empirically cover the prostrate for every slice. There are three main reasons for cropping: 1) the text information can produce fake needles; 2) the noise away from needles can result in more cluster centers; 3) the unconcerned region increases the computation cost. Our approach is robust to the bounding box because dictionary modeling and needle localization are both based on position-specific image patch groups and ROIs.

b). Pixel intensity mapping.

We map all pixel intensities by the exponential function to highlight the suspected regions that could possibly have needles and enhance the contrast against the background. The used exponential function can cause noise enhancement, but the contrast between needles and noise is also certainly enlarged. In addition, image noises are dealt with in dictionary modeling, ROIs selection, and final refinements.

c). 3D image patching.

We decompose all US images into 3D small-size patches with overlaps, as shown in Fig. 2 (3D image patches). Then, every patch is reshaped into a data vector for learning or reconstruction. The patches at the identical position of all given slices are poured into a data group. We finally train a position-specific dictionary on each patch group, utilizing the proposed dictionary learning model. The motivations behind patching are three-fold: 1) needles are distributed locally; 2) the small-size patch can increase the difference between the sparse codes of adjacent slices, while our approach enforces them into the same codes so as to reduce the speckle noise; 3) the position specific dictionary learning is efficient in the way of a parallel computing. In addition, we use the overlapped patches to consider the local voxel smoothness.

B. Order-Graph Regularized Sparse Dictionary Learning

With Assumptions 1 and 2, we present our novel SDL model by combining an order graph to solve Problem 1. The detailed block diagram of the ORDL procedure can be found in Section SI of the supplementary material.

1). Motivation:

Our motivation of order graph is from the spatial priors of 3D US images. Fig. 3 exhibits a 2D US profile, where both needles and tissues are continuous, while noises are discrete, in spatial distribution. As a result, we integrate this spatial continuity into SDL to enhance needle and reduce noise, which can be reached by a graph regularization [19]. However, the traditional graph that relies on Euclidean distance suffers from the heavy noise in US images. We design an order graph that organizes the spatial continuity and then combine the graph in dictionary learning to consider the continuous structure more. In addition, the speckle noise is also sparsely distributed as shown in Fig. 3.

Fig. 3.

A 2D needle-directional slice of the cropped 3D US image with needles. The yellow lines are the inserted needles including shafts and tips [32]; the red arrows are noise points; the cyan arrows show the spatial continuity in tissues.

2). Order Graph Construction:

We construct an order graph to encode the spatial continuity of 3D US images in Assumption 2. Since the continuity lies in the images from a patient and is broken down between patients, we construct the order graph per patient as follows. Denote the order graph for the k-th patient by G = (Y^(k), E^(k)|W^(k)), where Y^(k) is the collection of position-specific patches, E^(k) includes the collection of edges among those patches, and W^(k) indicates the related adjacency matrix. We construct W^(k) as follows:

$${\mathbf{\text{W}}}^{(k)}=\left[\begin{array}{ccccc}1& 1& 0& 0& \cdots \\ 1& 1& 1& 0& \cdots \\ 0& 1& 1& 1& \cdots \\ 0& 0& 1& 1& \cdots \\ ⋮& ⋮& ⋮& ⋮& \ddots \end{array}\right]$$ (3)

where if w_ij equals 1, when y_i and yj are adjacent slices; otherwise w_ij equals 0. The problem of solving for a X that follows Assumption 2 can be reformatted as:

$$\underset{\mathbf{\text{X}}}{\text{min}}\frac{1}{2}\sum _{k=1}^K\sum _{i=1,j=1}^{n_k}{w}_{ij}^{(k)}{\Vert {\mathbf{\text{x}}}_i-{\mathbf{\text{x}}}_j\Vert }_2^2$$ (4)

where K is the number of patients, and n_k indicates the number of images in the k-th patient. Eq. (2) is usually used in manifold learning [33] and also adopted as a smooth term [19].

In this paper, we adopt this order graph to highlight the continuity pattern of the 3D US images in dictionary learning. As a result, the learned dictionary could adequately reconstruct the continuous tissues, while reducing both US noises and artifacts that have no continuity in spatiality (i.e., z-axis).

3). Objective Problem:

Our goal is to learn Ŷ and E from the auxiliary images, using Assumption 1. In this case, the matrix N = 0 in Problem 1, so it can be simplified as Y = Ŷ + E . Our ORDL is an approach that reformulates Eq. (2) and then combines Eq. (4), as following:

$$\begin{gathered} \underset{\mathbf{\text{D}},\mathbf{\text{X}},\mathbf{\text{E}}}{\text{min}}\Vert \mathbf{\text{Y}}-\mathbf{\text{D}}\mathbf{\text{X}}-\mathbf{\text{E}}{\Vert }_F^2+\lambda \sum _{i=1}^n{\Vert {\mathbf{\text{x}}}_i\Vert }_p+\alpha \Vert \mathbf{\text{E}}{\Vert }_1 \\ +\frac{\beta }{2}\sum _{k=1}^K\sum _{i=1,j=1}^{n_k}{w}_{ij}^{(k)}{\Vert {\mathbf{\text{x}}}_i-{\mathbf{\text{x}}}_j\Vert }_2^2\text{\hspace{0.17em}s.t.\hspace{0.17em}}{\Vert {\mathbf{\text{d}}}_l\Vert }_2^2\le 1 \end{gathered}$$ (5)

where α > 0 is the noise intensity factor and β > 0 is a scaler of the weight of the graph. In Eq. (5), the first term aims to ensure image fidelity; the second term is to enhance sparsity on image codes; the third term is to impose E as a Laplacian distribution for the discrete noise; the final term aims to integrate the spatial continuity of 3D US images.

Intuitively, ORDL can achieve better performance than SDL, since the former approximates the latter with tricky α and β. In this paper, we adopt the optimization framework of Augmented Lagrange Multiplier (ALM) [34] to pursue a steady solution of the proposed objective problem (5).

4). Objective Optimization With ALM Method:

We introduce the constraints Z = X, and then rewrite (5) into the following augmented Lagrangian function:

$$\begin{gathered} L(\mathbf{\text{D}},\mathbf{\text{X}},\mathbf{\text{Z}},\mathbf{\text{E}},\mathbf{\text{M}})=\Vert \mathbf{\text{Y}}-\mathbf{\text{D}}\mathbf{\text{Z}}-\mathbf{\text{E}}{\Vert }_F^2+\lambda \sum _{i=1}^n{\Vert {\mathbf{\text{x}}}_i\Vert }_p+\alpha \Vert \mathbf{\text{E}}{\Vert }_1 \\ +\frac{\beta }{2}\sum _{k=1}^K\sum _{i=1,j=1}^{n_k}{w}_{ij}^{(k)}{\Vert {\mathbf{\text{x}}}_i-{\mathbf{\text{x}}}_j\Vert }_2^2 \\ +tr\left({\mathbf{\text{M}}}^T(\mathbf{\text{Z}}-\mathbf{\text{X}})\right)+\frac{\mu }{2}\Vert \mathbf{\text{Z}}-\mathbf{\text{X}}{\Vert }_F^2 \end{gathered}$$ (6)

where M is Lagrange multiplier and μ > 0 is penalty parameter. Note that we ignore the constraints on dictionary, since it can be easy to tackle [25]. The problem (6) can be optimized using the ALM algorithm [35], briefly shown as follows.

a) Updating E is to minimize the objective function:

$$L_1=\Vert \mathbf{\text{Y}}-\mathbf{\text{D}}\mathbf{\text{Z}}-\mathbf{\text{E}}{\Vert }_F^2+\alpha \Vert \mathbf{\text{E}}{\Vert }_1,$$ (7)

which has the following closed-form solution:

$$\mathbf{\text{E}}=S_{\alpha /2}(\mathbf{\text{D}}\mathbf{\text{Z}}-\mathbf{\text{Y}}),$$ (8)

where S_ε(·) is the soft-thresholding operator [36].

b) By fixing others, updating Z is equivalent to minimizing the following objective function:

$$L_2={\Vert \left[\begin{array}{c}\mathbf{\text{Y}}-\mathbf{\text{E}}\\ (\sqrt{\mu }/\sqrt{2})(\mathbf{\text{X}}-\mathbf{\text{M}}/\mu )\end{array}\right]-\left[\begin{array}{c}\mathbf{\text{D}}\\ (\sqrt{\mu }/\sqrt{2})\mathbf{\text{I}}\end{array}\right]\mathbf{\text{Z}}\Vert }_F^2$$ (9)

where I is an identity matrix. So, Z has a closed-form solution:

$$\mathbf{\text{Z}}={\left[\begin{array}{c}\mathbf{\text{D}}+{10}^{-6}\mathbf{\text{I}}\\ (\sqrt{\mu }/\sqrt{2})\mathbf{\text{I}}\end{array}\right]}^{-1}\left[\begin{array}{c}\mathbf{\text{Y}}-\mathbf{\text{E}}\\ \sqrt{\mu }\mathbf{\text{X}}/\sqrt{2}-\mathbf{\text{M}}/\sqrt{2\mu }\end{array}\right].$$ (10)

c) By fixing others, updating E is equivalent to solving the following minimization problem:

$$\begin{gathered} \underset{\mathbf{\text{X}}}{\text{min}}\frac{\mu }{2}\Vert \mathbf{\text{Z}}-\mathbf{\text{X}}+\mathbf{\text{M}}/\mu {\Vert }_F^2+\lambda \sum _{i=1}^n{\Vert {\mathbf{\text{x}}}_i\Vert }_p \\ +\frac{\beta }{2}\sum _{k=1}^K\sum _{i=1,j=1}^{n_k}{w}_{ij}^{(k)}{\Vert {\mathbf{\text{x}}}_i-{\mathbf{\text{x}}}_j\Vert }_2^2. \end{gathered}$$ (11)

Let Q = Z + M / μ and H = R - W where

$$\mathbf{\text{R}}=\mathit{\text{diag}}\left(\sum _{j=1}^n{w}_{1j},\sum _{j=1}^n{w}_{2j},\cdots ,\sum _{j=1}^n{w}_{nj}\right).$$ (12)

Note that we leave out the superscript of w in (12). Thereby, the problem (11) can be recast into:

$$\underset{\mathbf{\text{X}}}{\text{min}}\frac{\mu }{2}\Vert \mathbf{\text{Q}}-\mathbf{\text{X}}{\Vert }_F^2+\lambda \sum _{i=1}^n{\Vert {\mathbf{\text{x}}}_i\Vert }_p+\beta \sum _{k=1}^K\sum _{i=1}^{n_k}{h}_{ij}{\mathbf{\text{x}}}_i^T{\mathbf{\text{x}}}_j.$$ (13)

We here adopt the block coordinate descent to seek the optimal solution to (13) [35]. Concretely, we individually obtain each x_i through solving the following sub-problem of

$$\underset{{\mathbf{\text{x}}}_i}{\text{min}}\frac{\mu }{2}{\Vert {\mathbf{\text{q}}}_i-{\mathbf{\text{x}}}_i\Vert }_2^2+\lambda {\Vert {\mathbf{\text{x}}}_i\Vert }_p+\beta h_{ii}\left({\mathbf{\text{x}}}_i^T{\mathbf{\text{x}}}_j+2{\mathbf{\text{x}}}_i^T{\mathbf{\text{u}}}_i\right),$$ (14)

where q_i is the column verctor of Q and

$${\mathbf{\text{u}}}_i=\frac{1}{h_{ii}}\sum _{k=1}^K\sum _{j=1,j\ne i}^{n_k}{h}_{ij}{\mathbf{\text{x}}}_j.$$ (15)

Since u_i is a constant for solving x_i, (14) is finally equivalent to the following minimization problem:

$$\underset{{\mathbf{\text{x}}}_i}{\text{min}}\frac{1}{2}{\Vert \left[\begin{array}{c}\sqrt{\mu }{\mathbf{\text{q}}}_i\\ -\sqrt{2\beta h_{ii}}{\mathbf{\text{u}}}_i\end{array}\right]-\left[\begin{array}{c}\sqrt{\mu }\mathbf{\text{I}}\\ \sqrt{2\beta h_{ii}}\mathbf{\text{I}}\end{array}\right]{\mathbf{\text{x}}}_i\Vert }_2^2+\lambda {\Vert {\mathbf{\text{x}}}_i\Vert }_p,$$ (16)

which can be efficiently solved by hard-thresholding operation for p = 0 and soft-thresholding operation for p = 1 [37].

d) In order to update D, we solve the following problem:

$$\underset{\mathbf{\text{D}}}{\text{min}}\Vert \mathbf{\text{V}}-\mathbf{\text{D}}\mathbf{\text{Z}}{\Vert }_F^2\text{\hspace{0.17em}s.t.\hspace{0.17em}}{\Vert {\mathbf{\text{d}}}_l\Vert }_2^2\le 1,$$ (17)

where V = Y - E. Since (17) is convex on D, we here obtain the optimal soluation from its Lagrange dual problem:

$$\underset{\Lambda >0}{\text{min}}\text{tr}\left(\mathbf{\text{V}}{\mathbf{\text{X}}}^T{\left(\mathbf{\text{X}}{\mathbf{\text{X}}}^T+\Lambda \right)}^{-1}\mathbf{\text{X}}{\mathbf{\text{V}}}^T+\Lambda \right),$$ (18)

which can be optimized by conjugate gradient [35]. Thereby we can achieve D as follows:

$$\mathbf{\text{D}}=\mathbf{\text{V}}{\mathbf{\text{X}}}^T{\left(\mathbf{\text{X}}\mathbf{\text{X}}+{\Lambda }^{*}\right)}^{-1},$$ (19)

where Λ* is the optimal solution to the problem (18).

e) Updating M is achieved by the following:

$$\mathbf{\text{M}}=\mathbf{\text{M}}+\mu (\mathbf{\text{Z}}-\mathbf{\text{X}}).$$ (20)

The primary procedure of solving ORDL problem by ALM is described in Algorithm 1 and is illustrated in Fig. 3.

Algorithm 1.

Solving ORDL by ALM

Input: US images Y, dictionary size m, parameters α, β.

Output: D.

Initialization: E =Z =X =0,D by random matrix.

While not converged

1. Update E by (8);

2. Update Z by (10);

3. Update X by solving (16) for each column x;

4. Update D by solving (18) and (19);

5. Update M by (20);

6. Check convergence: let t be the iteration, if

‖Y−DX−E‖/‖Y‖ < 1e – 6 and

max{‖Xt−Xt−1‖,‖Zt−Zt−1‖,‖Et−Et−1‖} < 0.01

then stop.

End while

5). Needle Reconstruction:

Since the dictionary D is trained on the auxiliary images, the needles in target images are less likely to be reconstructed on D. Yet D could perform well in rebuilding Ŷ (as shown in Eq. (1)). Hence, the needles N can be obtained from the difference between target images and their reconstructions. Wherein, we use the following theory to pursuit better performance:

Theory 1.

Given sufficient auxiliary images, the noise in target images can be sparsely reconstructed on the noise matrix E that is learned from the identically distributed noises.

Proof : This theory can be easily derived from the theory given for the sparse self-expressiveness property [38].λ

With Theory 1, we reconstruct the target images Y using the augmented dictionary of $\widehat{\mathbf{\text{D}}}=[\mathbf{\text{D}}\mathbf{\text{E}}]$ by:

$$\underset{\mathbf{\text{X}}}{\text{min}}\Vert \mathbf{\text{Y}}-\widehat{\mathbf{\text{D}}}\widehat{\mathbf{\text{X}}}{\Vert }_F^2+\lambda \sum _{i=1}^{n_i}{\Vert {\widehat{\mathbf{\text{x}}}}_i\Vert }_p+\beta \sum _{i=1,j=1}^{n_i}{\widehat{w}}_{ij}{\Vert {\widehat{\mathbf{\text{x}}}}_i-{\widehat{\mathbf{\text{x}}}}_j\Vert }_2^2$$ (21)

where n_t is the number of patches from the target images and ŵ_ij is the weights corresponding to the target images. The problem (21) can be reduced into the standard sparse coding [35]:

$$\underset{\widehat{\mathbf{\text{X}}}}{\text{min}}\sum _{i=1}^{n_t}{\Vert \left[\begin{array}{c}{\widehat{\mathbf{\text{y}}}}_i\\ -\sqrt{\beta {\widehat{h}}_{ii}}{\widehat{\mathbf{\text{u}}}}_i\end{array}\right]-\left[\begin{array}{c}\widehat{\mathbf{\text{D}}}\\ \sqrt{\beta {\widehat{h}}_{ii}}\mathbf{\text{I}}\end{array}\right]{\widehat{\mathbf{\text{x}}}}_i\Vert }_2^2+{\Vert {\widehat{\mathbf{\text{x}}}}_i\Vert }_P$$ (22)

where ĥ and û are computed using (13) and (15) respectively. In addition, we initialize $\widehat{\mathbf{\text{X}}}$ the by least-square solution. To reduce computation cost for either training or online-use, we set p = 0 and employ the orthogonal matching pursuit (OMP) algorithm with obtaining L-sparsity in sparse code [21]. Finally, the needle images N can be easily obtained by:

$$\mathbf{\text{N}}=\mathbf{\text{Y}}-\widehat{\mathbf{\text{D}}}{\widehat{\mathbf{\text{X}}}}^{*},$$ (23)

where ${\widehat{\mathbf{\text{X}}}}^{*}$ is the optimal solution to the problem of (22).

C. ROI Based RANSAC

With the resultant needle images N, we present the process of identifying needles from a large number of pixels. In general, we identify the centers of needles in all slices of a patient using cluster algorithm, and then divide those centers into overlapped regions, followed by performing RANSAC algorithm in every region with iterative correction on suspected centers. The detailed block diagram of this procedure can be found in Section SI of the supplementary material.

1). Region of Interest:

There is a large number of pixels with high intensity in each slice of N, although sparse coding in (22) has diminished most of pixels. We can denote by $V\subseteq {ℝ}^2$ the set of coordinates of pixels in a slice, and I (V ) their corresponding intensities. Since the needle pixels usually have high intensities, we here apply a threshold T_θ to reduce computation cost:

$$V_N=\left\{v\in V:I(v)\ge T_{\theta }\right\}$$ (24)

where θ is the percentage of pixels with low intensities. For each slice, we then utilize a cluster algorithm on V_N to yield the cluster centers V_C ⊆ V_N. The right side of Fig. 2 shows several clustering results in needle images, where those cluster centers are highlighted using red points.

In order to extract each needle with continue linear structure, we partition V_C into multiple overlapped regions based on the following ideal assumption:

Assumption 3.

Denote by V_b all the actual centers that yielded by all needles in the first slice. Then, we have:

$$V_b\subseteq V_C^{(1)}$$

Where $V_C^{(1)}$ includes all cluster centers in the first slice.

However, some needles may lose their first traces resulting from cluster algorithm. Thus, we seek the set of leading centers in the first n₀ slices to cover all needles. Algorithm 2 shows the major steps of identifying all regions of interest (ROIs), where an ROI is defined by the cylinder that contains more than N cluster centers on N different slice layers.

2). Iterative ROI-RANSAC:

From Algorithm 2, we obtain a set of ROIs, where each ROI may have a needle. With the ROIs, multi-needle detection can be reduced into single-needle detection which has been studied in extensive reports [1]–[5]. In single-needle detection, RANSAC algorithm is usually adopted to fit the line or curve needle.

Unfortunately, the resultant ROIs from Algorithm 2 are often overlapped because of the mutual interference in multi-needle detection. In order to address this issue, we introduce a method of iterative ROI-RANSAC. That is, in each iteration, RANSAC partitions the centers in ROI into inliers and outliers. Then, our method removes the inliers from all V_C while keeps the outliers for detecting other needles. Algorithm 3 summarizes the major steps of the designed iterative ROI-RANSAC.

D. Result Refining and Tip Detection

Although our approach considers the robustness, it might be overwhelmed by noise and artifacts. For optimal performance, we introduce the following refinement operations.

a). Center combination.

The cluster method often gives rise to redundant centers such that some centers are close to each other in the same slice. In this paper, we combine those centers by the mean of their positions and then employ the obtained mean as a new center point. As shown in Fig. 2, there are many redundant centers produced for a needle.

Algorithm 2.

Seeking ROIs

Input: $V_C^{(i)}$, i = 1,…, nt; needle diameter r0, rate ρ; and n0

Output: ROI set Ω.

Initialization: r = r0, t = 1, C = null, ρ = 1.2, N = 4.

While t < n0 + 1

1. Randomly chose a center a from $V_C^{(t)}$ and k = t + 1;

2. While k < nt

3.  Find all centers c that meets:

$C_k=\left\{\mathbf{\text{c}}\in V_C^{(k)}:\mathbf{\text{D}}is(\mathbf{\text{c}},\mathbf{\text{a}})\le r\right\}$

4.  If Ck = null, then r = ρ × r0.

5.  Else C = C + Ck, r = r0;

6.  k = k+ 1;

7. End while

8. If #{C} > N, then Ω = Ω + C;

9. t = t + 1.

End while

Algorithm 3.

Fitting Needles by Iterative ROI-RANSAC

Input: ROIs Ω, the number of ROI T.

Output: needle set Φ.

Initialization: t = 1.

While t < T

1. Find the ROI C that contains the most centers: C ⊆ Ω;

2. Perform RANSAC in C [4];

3. Obtain the inlier set Ct;

4. Add as a needle: Φ = Φ + Ct;

5. Remove the determined centers: Ω = Ω − Ct;

6. t = t + 1.

End while

b). Needle center interpolation.

We use the resulting model of needle to make up the imperceptible centers. That is, we use the intersections between the needle models and those slices as the missed centers. The example results can be found in Fig. 2.

c). Incorrect needle removal.

In the near field close to the US probe, the tissue often yields high-intensity artifacts, usually leading to incorrect detections. Thus, we remove those detected needles where most of the cluster centers lie near the needle tip.

Finally, we identify needle tips according to the following two empirical conditions: a) the pixel intensities along the fitted needle model drop sharply at the tips; b) the intensities of the nearby pixels on both sides of tips have consistent trends, i.e., no sharp changes. Fig. 2 exhibits the detected needle tips from a patient, where the red triangle highlights the tip location.

IV. Implementation Details

A. Method Details and Parameter Settings

We present all the details to specify the proposed approach in experiments. In preprocessing, we crop all images into the size 384 × 576, and then partition all the 3D US images into small patches of size 8 × 8 × 3 with overlapped size 4 × 4 × 2. In ORDL, we train the dictionary which has m atoms and sparsity L = 5. In needle modeling, we set θ = 0.99 in (24), and then use k-means algorithm to seek k = 20.¹ cluster centers, followed by using a straight line in RANSAC with the default setting. In Algorithm 2, we adopt the Euclidean distance and set n₀ = 5 and r₀ = 1.6.² In Algorithm 3, we let T = 20. In refinement, we consider two centers are close, if the Euclidean distance is less than the used needle diameter r₀. Note that the setting may vary with different practical situations. In addition, all codes are implemented in MATLAB 2018b.³ on a personal computer with a CPU.

B. Evaluation Metrics

In this paper, we use three metrics to quantitatively evaluate the results from the multi-needle detection methods, where the manual positions are provided as ground truth. Besides, we also confirm the results using the corresponding computed tomography (CT) images for needle tips and needle numbers.

In 3D US images, let (X(x), Y (x), Z(x)) be the coordinates for a spatial point $\mathbf{\text{x}}\in {ℝ}^3$. Then, the tip localization error for each needle is here measured along z-axis by:

$${\epsilon }_{tip}=|Z\left({\mathbf{\text{x}}}_{tip}\right)-Z\left({\widehat{\mathbf{\text{x}}}}_{tip}\right)|$$

where ${\widehat{\mathbf{\text{x}}}}_{tip}$ is the corresponding actual tip. The shaft localization error per needle is calculated by the average distance:

$${\epsilon }_{\text{shaft\hspace{0.17em}}}=\frac{1}{s}\sum _{i=1}^s{\left({\left(X\left({\mathbf{\text{X}}}_i\right)-\mathbf{\text{X}}\left({\widehat{\mathbf{\text{X}}}}_i\right)\right)}^2+{\left(\mathbf{\text{Y}}\left({\mathbf{\text{X}}}_i\right)-\mathbf{\text{Y}}\left({\widehat{\mathbf{\text{X}}}}_i\right)\right)}^2\right)}^{\frac{1}{2}}$$

where s is the number of shared slices by detected needle and its corresponding manual needle. In order to obtain overall evaluation, the incorrect detection number is evaluated by:

$$N_{\text{trincorr}}=\#\left\{\epsilon \mid {\epsilon }_{\text{shagt\hspace{0.17em}}}>r_0/2\Vert {\epsilon }_{\text{trip\hspace{0.17em}}}>\sigma \right\}$$

where σ is the error tolerance for needle tip. N_incorr counts the detected needles out of the tolerance errors. Note that we generally reported their average values for multiple needles.

In addition, we exhibit experimental results in terms of the three styles: 1) visualization of 2D slices and 3D needles, 2) the cumulative distributions of the number of correct needles for increasing tolerable error, 3) the overall quantitative results on needle shaft error, tip location error and correction rate.

C. Imaging Protocol for Our Data

In experiment, the used US images were produced from our institution. Those US images were obtained with the Hitachi Hi Vision Avius (Model: EZU-MT30-S1, Hitachi Medical Corporation, Japan) ultrasound system equipped with a transrectal probe (Model: EUP-U533). Ultrasound B-mode images were acquired using the same settings: 7.5-MHz center frequency, 17 frames per second, thermal index < 0.4, mechanical index 0.4, and 65-dB dynamic range. On the other hand, 12–19 (depending on patient prostate size) Nucletron ProGuide Sharp 5F needles were placed under TRUS guidance. Needle length is 240 mm and its outer diameter is about 1.6 mm. The ultrasound examination was performed by an experienced physicist.

V. Experimental Results

In this section, we present the experimental results using US data of a prostate phantom and US images from prostate cancer brachytherapy. Using our multi-needle detection workflow, we compare with: k-means by removing ORDL; k–means + KSVD by replacing ORDL with KSVD [25]; k–means+LSC by replacing with LSC [30]; and k–means + graph²DL by replacing with graph²DL [31]. For clarity, we mark the proposed method by k–means + ORDL. Note that LSC could degenerate into the classical sparse coding model [22]. Besides, we set the regularization parameters in graph²DL to be greater than zero, so as to prevent graph²DL from degenerating into KSVD.

A. Experiments on Phantom Images

We scanned a phantom with 16 inserted needles to create a phantom-specific 3D US dataset. For this dataset, we scanned a no-needle image of size 1024 × 768 × 195 for training, and after needle insertion a needle image of size 1024 × 768 × 71 for test, where the axial resolution is 0.12 mm and the slice thickness is 0.5 mm. Besides, we trained the dictionary with 256 atoms with L = 5 in KSVD; λ = 0.1, β = 1e – 3 for LSC with 5 nearest neighborhood graph; T = 5, α = 0, μ = 0, β = 1e-2 for graph²SC with 5 nearest neighborhood graph; and α = 0.1, β = 0.2 for ORDL. Note that the above parameters were set by referring to their respective literatures [25], [30], [31], and the codes were all obtained from their respective authors.

1). Visualization of Results:

Fig. 4 shows the results from the clustering stage using the five methods. It is observed that the compared methods produce many incorrect center locations labeled by those yellow circles. Sparse dictionary learning models can help reduce some noises, but barely benefits from the nearest neighborhood graph in our problem. Note that we selected small regularization parameters for decent results. The reason is that the neighborhood structure suffers from noise and artifacts [39]. In contrast, the centers from k-means+ORDL are all close to ground truths, where the difficult needles highlighted by triangles are also detected. The success of our method might be rooted in that the needles are continuous through multiple slices, while noise exists in a single slice or a few adjacent slices.

Fig. 4.

The clustering results shown in the 1st, 15th, 30th, 45th, and 60th slice in 3D US phantom images. The first, second, third, fourth, and fifth row respectively corresponds to k-means, k-means+KSVD, k-means+LSC, k-means+graph²DL, and k-means+ ORDL. The yellow circles indicate the incorrect locations. The cyan triangles show that the needles could be correctly detected only by k-means+ORDL.

Fig. 5 depicts the final detection results after refinement and tip detection. As shown, k-means results in six false needles and misses one needle; k-means+KSVD yields three false needles and two missed needles;k-means+LSC yields two false needles and two missed needles; k-means+graph²DL leads to two false needles and two missed needles; while k-means+ORDL detects all 16 needles, including the difficult needle highlighted by the black star, shown in Fig. 5(e).

Fig. 5.

The detection results on phantom. Manually detected needles are shown in (f). The black arrows indicate the wrong detections and the star is the correct detection only by k-means+ORDL.

2). Quantitative Evaluation:

In order to achieve quantitative results, we first computed the shaft localization error for each needle, and then found out the incorrect detections, followed by calculating the tip localization error. Note that we reported the errors computed on the correct detections with r₀ = 1.6 mm and σ = 3.0 mm (i.e., 6 slices).

Fig. 6 shows the distribution of the shaft localization errors and the tip localization errors among correctly detected needles. The curve intersection means that the related methods correctly detect the same number of needles under tolerance errors, while they have different error distributions. For example, there is an intersection between k-means+KSVD,k-means+graph²DL and k-means+ORDL at 0.4 in Fig. 6(left), meaning that all of them detect 12 needles at ε_{shaf t} = 0.4. But k-means+ORDL detects the 12 needles at ε_{shaf t} = 0.3, which is much stronger than others. On the other hand, the saturation in Fig. 6 shows that the compared methods cannot detect more needles than k-means. The reason is that traditional SDL models fail to consider the speckle noise that is learned by E in our objective (5), while traditional graph regularizations ignore (even disorder) the spatial continuity that is a key consideration in our method. Finally, k-means+ORDL delivers better results at all error tolerances consistently.

Fig. 6.

The cumulative distributions of shaft localization error (left) and tip localization error (right) on the phantom dataset.

Table I summarizes overall detection results, where N_all is the number of detected needles using various methods; AVG. ε_{shaf t} and AVG. ε_tip respectively correspond to the average localization shaft errors and the average tip errors with reference to the correct detections. We also computed p-values via two-sample t-tests for shaft errors (k-means+ORDL vs. other methods). The quartiles for tip errors are listed in the last row. From Table I, we could arrive at: k-means+ORDL detects all 16 needles with the smallest AVG. ε_tip, thanks to the order-graph regularization; k-means+graph²DL achieves the best at AVG. ε_{shaf t}; and both k-means+LSC and k-means+graph²DL give the relatively small interquartile range. Noteworthy, k-means+ORDL detects all 16 needles while other methods only detect 13 needles.

Table I.

Evaluation results in terms of various metrics on phantom data. The notation [q1, q2, q3] is the quartiles

Metrics	k-means	k-means + KSVD	k-means + LSC	k-means + graph2DL	k-means + ORDL
Nincorr (Nall)	6 (19)	3 (16)	2 (15)	2 (15)	0 (16)
AVG. εshaft [mm]	0.41±0.17	0.25±0.13	0.27±0.16	0.24±0.11	0.27±0.21
P-value. εshaft[vs. ORDL]	< 0.001	0.074	0.633	0.018	--
AVG. εtip [mm]	1.58±0.89	1.04±1.16	1.11±0.98	1.08±0.89	0.78±0.72
[ql, q2, q3]. εtip [mm]	[0.87, 1.50, 2.13]	[0.00, 0.50, 2.00]	[0.38, 1.00, 1.63]	[0.38, 1.00, 1.50]	[0.00, 0.75, 1.50]

B. Experiment on Prostate Images

We gathered 3D US images of prostate cancer brachytherapy patients and created a prostate dataset. This dataset contains a training set of 1024 × 768 × n_i of 70 patients without needles and a test set of 1024×768×n j of 21 patients with needles, where n_i and n j depend on patients. The axial resolution of the images is 0.12 mm, and the slice thickness is 1 mm or 2 mm for different patients. Fig. 1 shows two example images.

We trained a dictionary with 512 atoms on auxiliary images and then tested on the patients with needles one-by-one, whereL = 5 for K-SVD; λ = 0.1, β = 1e – 3 for LSC with 5 nearest neighborhood graph; T = 5, α = 0, μ = 0, β = 5e-2 for graph²SC with 5 nearest neighborhood graph; and α = 0.1, β = 0.2 for ORDL. For evaluation, we set r₀ = 1.6 mm and σ = 6.0 mm.

1). Results from One Patient:

The 3D US images of the first patient includes n j = 29 slices with the lateral resolution of 2 mm. There are in total 16 needles used in therapy. Fig. 7 shows the visualization results of the detected needles in 3D images, while Fig. 8 exhibits the final results on 2D slices of original images. Fig. 9 depicts the distributions of the detection errors. Finally, Table II summarizes the quantitative metrics, where both AVG. ε_{shaf t} and AVG. ε_tip is computed on the correct detections. From these results, it can be seen that k-means+ORDL correctly detects 15 out of all 16 needles, and consistently achieves smaller errors at all tolerations than other comparison methods.

Fig. 7.

The results on the first patient using the five methods. Manual needles are shown in (f). The black arrows indicate the wrong detections and the black star indicates the difficult needle.

Fig. 8.

The final detection results shown in the 1^st, 7^th, 14^th, 21^st, and 28^th slice in 3D US images of the first prostate patient. The first, second, third, fourth, and fifth row respectively corresponds to k-means, k-means+KSVD, k-means+LSC, k-means+graph2DL, and k-means+ORDL. The yellow circles are incorrect locations. The cyan triangles show that the needles are correctly detected only by k-means+ORDL.

Fig. 9.

The cumulative distributions on the first prostate patient. The right is tip localization error for all detected needles, while the left is shaft localization error just for the correct detections.

Table II.

Quantitative Results On The First Patient’s Prostate Images

Metrics	k-means	k-means + KSVD	k-means + LSC	k-means + graph2DL	k-means + ORDL
Nincorr (Nall)	3 (13)	4 (16)	3 (14)	4 (15)	1 (16)
AVG. εshaft [mm]	0.32±0.25	0.23±0.12	0.23±0.08	0.26±0.09	0.16±0.11
P-value. εshaft[vs. ORDL]	< 0.001	0.022	0.038	0.009	--
AVG. εtip [mm]	1.00±1.94	0.67±0.98	1.27±1.85	0.91±1.38	0.53±1.12
[ql, q2, q3]. εtip [mm]	[0.00, 0.00, 2.00]	[0.00, 0.00, 2.00]	[0.00, 0.00, 2.00]	[0.00, 0.00, 2.00]	[0.00, 0.00, 0.00]

2). Results of All Patients:

The prostate dataset has 21 patients with 318 needles in total for this test. Fig. 10 exhibits the distributions of the evaluation errors yielded from the various methods. As shown in Fig. 10, k-means+ORDL can correctly detect more needles than other compared methods at any tolerate errors on both needle tip and shaft. Table III summarizes the overall evaluation results. From the results, our method detects (312–10) / 318 ≈ 95% of needles with the average shaft location error of about 0.19 mm and the average tip location error of about 1.01 mm. The p-values of the two-sample t-tests between ORDL and other compared methods demonstrate that the superiority of our k-means+ORDL is statistically significant at 5% significance level.

Fig. 10.

The cumulative distributions of shaft localization error (left) and tip localization error (right) on the prostate data.

Table III.

Quantitative Results On The Prostate Data With In Total 318 Needles

Metrics	k-means	k-means + KSVD	k-means + LSC	k-means + graph2DL	k-means + ORDL
Nincorr (Nall)	42 (286)	36 (304)	39 (296)	29 (301)	10 (312)
AVG. εshaft [mm]	0.30±0.17	0.26±0.15	0.25±0.13	0.23±0.11	0.19±0.13
P-value. εshaft[vs. ORDL]	< 0.001	0.001	0.007	0.009	--
AVG. εtip [mm]	1.62±2.06	1.51±2.05	1.24±1.70	1.43±1.76	1.01±1.74
[ql, q2, q3]. εtip [mm]	[0.00, 0.00, 2.00]	[0.00, 0.00, 3.00]	[0.00, 0.00, 2.00]	[0.00, 1.00, 2.00]	[0.00, 0.00, 2.00]
P-value. εtip[vs. ORDL]	< 0.001	0.001	0.018	0.004	--

C. Observations and Conclusions

From the experimental results, we can arrive at the following observations and conclusions:

1. From the above results, we can arrive at that the k-means method can work on problem 1 and k-means+KSVD is better, while both k-means+LSC and k-means+graph²DL cannot lead to significant improvements compared with k-means+KSVD. Finally, the proposed k-means+ORDL method obtains the best performance in terms of various evaluation metrics.

2. The designed workflow in Fig. 2 is effective. Our method benefits from knowledge transferring by the learned dictionary from auxiliary images. The proposed method also benefits from the designed order-graph that collects the spatial continuity in 3D US images, while the traditional regularizations suffer from artifact and noise and disorder the spatial slice order.

3. Experimental results verify the feasibility of using the identical-source side-images for multi-needle detection. This knowledge-transferring method can also be extended into other practical applications, where the available data is insufficient or expensive to obtain.

All in all, the presented approach not only achieves a high accuracy for multi-needle detection, but also attempts a route to solve the small-sample-size problem. It can also be adopted to produce more labeled samples so as to help supervised learning in the applications on medical image analysis.

VI. Discussion

In order to clarify our workflow, its settings, advantages and disadvantages, and several considerations in clinical practice are discussed in the following.

A. Uncertainty in Our Workflow

The uncertainties of our workflow come from: the k-means algorithm and the RANSAC algorithm. The k-means requires the randomly initialized centers, while the RANSAC randomly selects needle points for fitting. In study, we chose the cluster result having the least within-cluster sums of point-to-centroid distances from 20 runs and chose the line with the smallest error from 100 RANSAC iterations. We independently evaluated the results from 5 runs, and exhibited the results in Section SII of the supplementary material. Those results show that all mentioned methods suffer from some effects from the randomness, while k-means+ORDL is more consistent in multiple runs.

B. Effects of Method Parameters

We probed the regularization parameters and the dictionary size to investigate their effects on detection results. Those parameters respectively vary from m ∈ {64, 128, 256, 512, 1024}, α ∈ {0.01, 0.1, 0.5, 1.0, 10} and β ∈ {0.1, 0.2, 0.5, 1.0, 10}. All results can be found in Section SIII of the supplementary material. The results show that increasing the dictionary size could improve detection performance, while both α and β have wide ranges in (0, 1] to achieve a good performance.

In addition, in order to select the three parameters in clinical applications, one can perform the genetic algorithm [40] or the grid search [19], [41] to search a set of fine parameters from the suggested ranges. In general, the dictionary atoms can be set to one or two times the patch size. For α ∈ (0, 1], one can adopt relatively small value for heavy noise in images and relatively large value for slight noise. For β ∈ (0, 1], one can make several attempts to realize its selection. More simply, one might set the suggested settings or values in the suggested ranges, since our method is not sensitive to the parameters in the wide ranges.

Another issue of concern is how we selected the final patch size, 8 × 8 × 3 in experiments. We tested the patch sizes of 4 × 4 × 3 and 16 × 16 × 3 with overlap being 2 × 2 × 2 and 8 × 8 × 2. The results can be found in Section III of the supplementary material. From the results, a large patch reduces detection performance, while a small patch requires more time. Actually, the selection of patch size is based on our assumption that two adjacent slices should have similar sparse codes but yield different codes due to noise and artifacts. With this assumption, ORDL encourages adjacent patches among a patch group into similar codes to reduce noise. Wherein, patch group is composed of all ordered slice patches at the same position from all 21 patients’ images.

C. Processing Time in Usage

The processing time is a key factor in clinical planning. We summarized the execute time of tackling each patient using our proposed method, and reported the results in Section SIV of the supplementary material. It takes about 451.2±21.6 seconds per patient on a Xeon E5–2630 CPU. We performed the proposed method on a NVIDIA GeForce RTX 2080Ti GPU to run those patch groups in more parallel manners. The execution time is reduced to about 37.6 ± 4.8 seconds, shown in Section SIV of the supplementary material. Although ORDL can be also written in parallel to reduce the computation cost [42], our method has the limitation for real-time needle tracking. This study aims to use the proposed workflow for speeding up treatment planning in HDR prostate brachytherapy, where an experienced physicist usually takes about 15–20 minutes on needle digitization.

D. Comparison with Other Popular Techniques

There are many state-of-the-art techniques proposed in image processing, such as robust matrix factorization (RMF) [24], convolutional sparse coding (CSC) [43] and deep convolutional neural networks (CNN) [44].

Comparison with RMF.

RMF is good at image diagnosing and image painting [24], while it must be re-executed for a new image. ORDL can be used on a new patient through the learned offline-learned dictionary and hence causes lower computation cost than RMF, which is an important clinical consideration.

Comparison with CSC.

CSC is a new sparse coding variant that is to address the problem of losing the consistency of pixels in overlapped patches [43]. The dictionary from CSC reduces the shifted versions of same features and thus is more consistent in data codes. This propriety can help to solve our Problem 1. In addition, the work of Vardan et al. shows that the forward pass of the CNN is in fact the thresholding pursuit serving the multi-layer CSC model [28]. We will attempt CSC in future work.

Comparison with CNN.

We made an attempt on the fashion U-Net [45] instead of ORDL, and evaluated on the 21 patients via 5-fold cross validation. With the supervised learning model, k-means+U-Net detects 94.3% of needles with the average shaft error of 0.17 ± 0.07 mm and the average tip error of 1.12 ± 1.44 mm. Statistical tests manifest no significantly differences between k-means+U-net and k-means+ORDL on ε_{shaf t} and ε_tip (p-value: 0.092 and 0.373). Therefore, our method produces comparable results with UNet, while ORDL does not require manual needles. Making more attempts on other CNN models for needle detection are our future work. More results of U-Net can be found in Section SV of the supplementary material.

E. Needle Modeling

The proposed workflow adopts the RANSAC algorithm with fitting a linear model. The assumption that objective needle is approximated as a line is inspired by their CT observations. In general, the needle shaft is often bent and moves away from a straight line [32]. We tested a two-order model or a three-order model instead of the line model. The results can be found in Section SVI of the supplementary material. The results show the two-order model obtains a good result on shaft location but weak on tip location, while the three-order model seems too complicated for the needle-point sets. The effective models are the considerations in our future work [4], [32].

F. Testing Our Assumption 1

To verify Assumption 1, we trained the dictionary by ORDL on the phantom images and then detected the needles on the patient images. The results show the dictionary from phantom produces more incorrect localizations and misses more needles than the dictionary learned from patient images. This might be due to that the phantom images cannot provide sufficient image features to represent the patient images. More details can be found in Section SVII of the supplementary material.

G. Effects of Previous Needle Paths and Tissue Deformation

The US-guided prostate needle insertion often causes needle deviations for the long needle path. As a consequence, the mis-insertion and withdrawal may perturb tissue by creating “tunnels”. However, the “tunnels” might be filled with surrounding tissue quickly due to the very small needle diameter. Moreover, the artifacts in the tissue might appear to be slightly darker in US images. In addition, tissue deformations caused by needle insertions may have effects on the proposed method. This limitation will be considered in our future work. More discussions can be found in Section SVIII of the supplementary material.

H. Use of Our Approach in Clinical Practice

There are two advantages of the proposed method in practical clinics. On the one hand, our method can be cheaply trained in an unsupervised way, thus avoiding a tedious and expensive manual work for ground truth. On the other hand, our method provides an approach to learn from side images and obtains a comparable performance with the supervised U-Net.

In a prostate brachytherapy, this technique can automatically provide accurate needle localizations on US images so as to avoid extra CT scans. Usually, an experienced physicist takes about 15–20 minutes to digitize the inserted needles for treatment planning. This technique can reduce time and treatment cost, and avoid transferring the patient to CT scans. The details of US-guided HDR prostate brachytherapy can be found in Section IX of the supplementary material.

VII. Conclusion

This paper concentrated on the seldom studied problem of multi-needle detection. We formulated the issue in mathematics and then proposed a workflow to manage it. The workflow is composed of simple preprocessing, ORDL from available side images, target image reconstruction, and needle rebuilding via iterative RANSAC algorithm in ROIs. For evaluation, we verified the proposed and comparison methods on the phantom and prostrate data from brachytherapy, using several quantitative metrics. Experimental results manifest that our workflow is effective for multi-needle detection and the presented ORDL can enhance the performance in comparison with the traditional dictionary learning, even the supervised U-net. In addition, the discussions on the ORDL parameters show that the proposed method is easy to be set up. The worth highlighting is that the proposed unsupervised workflow is label-free, cost-cheap, and data-available. In the future, with more patient images, we will consider more attempts on using deep CNNs to pursuit higher performance, meanwhile considering more clinical practices.