Patch-Based Label Fusion for Automatic Multi-Atlas-Based Prostate Segmentation in MR Images

Abstract

In this paper, we propose a 3D multi-atlas-based prostate segmentation method for MR images, which utilizes patch-based label fusion strategy. The atlases with the most similar appearance are selected to serve as the best subjects in the label fusion. A local patch-based atlas fusion is performed using voxel weighting based on anatomical signature. This segmentation technique was validated with a clinical study of 13 patients and its accuracy was assessed using the physicians’ manual segmentations (gold standard). Dice volumetric overlapping was used to quantify the difference between the automatic and manual segmentation. In summary, we have developed a new prostate MR segmentation approach based on nonlocal patch-based label fusion, demonstrated its clinical feasibility, and validated its accuracy with manual segmentations.

1. INTRODUCTION

Prostate cancer is the second leading cause of cancer death for USA male populations [1]. Prostate MRI image segmentation has been an area of intense research due to the increased use of MRI as a modality for the clinical workup of prostate cancer, e.g. diagnosis and treatment planning [2–4]. Segmentation is useful for various tasks: to accurately localize prostate boundaries for radiotherapy [5], perform volume estimation to track disease progression [6], to initialize multi-modal registration algorithms [7, 8], and to obtain the region of interest for computer-aided detection of prostate cancer [9]. Currently, physicians’ manual segmentation is the gold standard in the clinic. However, manual segmentation of the prostate is time consuming and often subject to inter- and intra-observer variation.

Accurate prostate segmentation for MRI data can be challenging due to the image noise, inter-patient anatomical differences, and the similar intensities of the prostate and surrounding tissues (e.g., the bladder). To overcome these challenges, several segmentation techniques have been proposed. Deformable models (DMs) have been particularly popular, especially active shape models (ASMs) and level-sets. For example, a DM framework was proposed for 3D prostate segmentation from T2-MRI, in which the DM evolution was controlled by voxel intensity and a statistical shape model [10]. Zhu et al. [11] proposed a hybrid 2D/3D ASM-based methodology for 3D MRI prostate segmentation. Ghose et al. [12] proposed a similar approach that aligned T2-MRI data, and then used an active appearance model (AAM), an extension of ASM, guided by appearance and shape information to segment the prostate. Also, Gao et al. [13] aligned MR images before segmenting the prostate using a level-set guided by appearance information and a learned shape prior. In addition, McClure et al. [14] utilized a nonnegative matrix factorization (NMF) feature fusion method to create a more robust model for guiding the evolution of a 3D level-set deformable model for MRI DWI prostate segmentation. A probabilistic anatomical atlas was used by Martin et al. [15] to constrain a DM-based framework for segmenting the prostate from 3D T2-MR images. In [16] a multi-feature landmark-free active appearance model was presented for segmenting 2-D medical images. These methods yield good results but cannot handle medical images in a 3D manner. Statistical-based techniques have also been utilized for segmenting prostates from MRI data. For example, a probabilistic graph-cut-based framework for 3D T2-MRI prostate segmentation based on a probabilistic atlas was proposed in [17]. Firjany et al. [18] proposed a Markov random field (MRF) image model for 2D dynamic contrast enhanced (DCE)-MRI prostate segmentation that combined a graph cut approach with a prior shape model of the prostate and the visual appearance of the prostate image, modeled using a linear combination of discrete Gaussians (LCDG). This method was later extended in [19] to allow for 3D prostate segmentation from DCE-MRI data. A maximum a posteriori (MAP)-based framework that performed automated 3D MRI prostate segmentation using an MRF model and statistical shape information was proposed in [20]. Also, an atlas-based segmentation approach was presented to extract the prostate from MR images based on averaging the best atlases that match the image to be segmented [21]. In [22] another technique was proposed to use an automated atlas approach to segment the prostate region based on a selective and iterative method for performance level estimation (SIMPLE) based alignment technique. In addition to DMs and statistical-based techniques, several other methods have been proposed to segment the prostate from MR images. A semi-automated edge detection technique in [23] was proposed for MRI prostate segmentation based on a static wavelet transform [24] to locate the prostate edges. Also, a semi-automated approach is proposed to use a prostate shape prior to detect the contour in each slice and then refined them to form a 3D prostate surface [25]. Additionally, random walk classification was used for MRI prostate segmentation in [26].

In this study we propose to integrate similar appearance-specific atlases and patch-based voxel weighting into label propagation framework to automatically segment prostate from MR images. This approach has 2 distinctive strengths: 1) Instead of performing the fusion of nonlinearly deformed template structures, the proposed method achieves the labeling of each voxel individually by comparing its surrounding patch with patches in training subjects in which the labels of the central voxels are known. When the patch under study resembles a patch in the training subjects, their central voxels are considered to belong to the same structure, and this training patch is used to estimate the final label. 2) Contrary to classical majority voting schemes that give the same weight to all the samples, the nonlocal means scheme enables the robust distinction of the most similar samples according to their local patch-based anatomical features. Finally, a patch-based weighting is used to perform a pixel-based aggregation of the labels ensuring the independency of the votes.

2. METHODS

Figure 1 shows a schematic flow chart of the automated multi-atlas segmentation of the prostate for MR images using patch-based label fusion. After an initial preprocessing for all images, all of the atlases are non-rigidly registered to a target image. The resulting transformation is then used to propagate the anatomical structure labels of the atlas into the space of the target image. The top L similar atlases are further chosen by measuring intensity difference in the region of interest around the prostate. Finally, using voxel weighting based on patch-based anatomical signature, the label that the majority of all warped labels predict for each voxel is used for the final segmentation of the target image. The three major steps are briefly described below.

Figure 1.

Schematic flow chart of the proposed algorithm for 3D MR prostate segmentation.

2.1. Image Preprocessing and Registration

Before segmentation, pre-processing is performed for all MR images in the dataset, which includes reducing noise, inhomogeneity correction and inter-subject intensity normalization [24]. The same processing will be performed for a new patient’s images that will be segmented. Such pre-processing steps are designed to improve the accuracy of the registration and segmentation. During the alignment processing of the training set, we first select one MR image as a template, and align other MR images to the template image. And we use the corresponding transformation obtained from training image alignment to align the segmented prostates (binary mask) to the template prostate. Since the segmented prostate of each training image is available, in order to optimize the alignment of training set we again align each training image to the template image by registering the binary segmentation prostates to the template prostate. When a newly acquired MR image comes, all aligned training images in the training set are registered to this new image. The deformable registration is used to obtain the spatial deformation field between the new MR image and training images. The same transformations are applied to the segmented prostates in the training set.

2.2. Atlas Selection

Due to the significant appearance difference of the prostate in both shape and intensity, the aligned atlas dataset may contain redundant atlases which could affect the segmentation accuracy. Therefore, an atlas selection should be performed to identify these similar atlases for label fusion. In order to deal with this problem we compare the intensity difference around the rectal region and rank the atlases in the aligned dataset. Here we use the normalized sum of the squared difference (NSSD) across the initialization mask instead of the normalized mutual information (NMI) over the image [27, 28]. This strategy is chosen because NSSD is sensitive to the variations in contrast and luminance. The NSSD intensity differences between each atlas and target image are defined over a region of interest to measure local image appearance base on the L2 norm. Within the ranked database, the top L atlases with the most similar appearance (smaller NSSDs) around the prostate region are selected for final multi-atlas segmentation.

2.3. Patch-Based Label Fusion

One of the most popular multi-atlases based image segmentation methods is the nonlocal mean label propagation strategy [29], and it can be summarized as follows. Given N aligned training images and their segmentation ground labels {(I_i, G_i), i = 1, …, N}, for a new treatment image I_new, each voxel $\overrightarrow{x}$ in I_new is correlated to each voxel $\overrightarrow{y}$ in I_i with a graph weight $w_i\left(\overrightarrow{x},\overrightarrow{y}\right)$. Then, we can estimate the corresponding label for the voxel $\overrightarrow{x}$ in I_new by performing label propagation from the atlases as below

$$G\left(\overrightarrow{x}\right)=\frac{{\mathbf{\sum }}_{i=1}^N{\mathbf{\sum }}_{\overrightarrow{y}\in \Omega }{w}_i\left(\overrightarrow{x},\overrightarrow{y}\right)G_i\left(\overrightarrow{y}\right)}{{\mathbf{\sum }}_{i=1}^N{\mathbf{\sum }}_{\overrightarrow{y}\in \Omega }{w}_i\left(\overrightarrow{x},\overrightarrow{y}\right)}$$ (1)

where Ω denotes the image domain, and G denotes the prostate probability map of I_new estimated by multi-atlases based labeling. $G_i\left(\overrightarrow{y}\right)=1$ if voxel $\overrightarrow{y}$ belongs to the prostate region in I_i, and $G_i\left(\overrightarrow{y}\right)=0$ otherwise. Through its corresponding graph weight $w_i\left(\overrightarrow{x},\overrightarrow{y}\right)$, we can represent the contribution of each candidate voxel $\overrightarrow{y}$ in the training image I_i during label propagation, and we can propagate its corresponding anatomical label in the ground truth segmentation image $G_i\left(\overrightarrow{y}\right)$ to the reference voxel $\overrightarrow{x}$ in the target image with a weight $w_i\left(\overrightarrow{x},\overrightarrow{y}\right)$. Therefore, the core problem to perform label propagation is how to define the graph weight $w_i\left(\overrightarrow{x},\overrightarrow{y}\right)$ with respect to each candidate voxel $\overrightarrow{y}\in I_i$, which reflects the contribution of $\overrightarrow{y}$ during label fusion.

In [29], $w_i\left(\overrightarrow{x},\overrightarrow{y}\right)$ is determined based on the intensity patch difference between the reference voxel $\overrightarrow{x}$ and candidate voxel $\overrightarrow{y}$. Here, the graph weight is given by,

$$w_i\left(\overrightarrow{x},\overrightarrow{y}\right)=\{\begin{array}{l}\exp [-(\frac{{\overrightarrow{h}}^T\sum {}^{-1}\overrightarrow{h}}{2\tau K^2})],\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\overrightarrow{y}\in {\text{ψ}}_i(\overrightarrow{x})\\ 0,\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}otherwise\end{array}$$ (2)

where τ is the smoothing parameter, and Σ is a diagonal matrix which represents the variance in each feature dimension. ${\text{ψ}}_i\left(\overrightarrow{x}\right)$ denotes the neighborhood of voxel $\overrightarrow{x}$ in image I_i, and it is defined as the M × M × M sub-volume centered at $\overrightarrow{x}$. ${\overrightarrow{F}}_I\left(\overrightarrow{r}\right)$ denotes the features of image I at voxel $\overrightarrow{r}$. $\overrightarrow{h}=\left({\overrightarrow{F}}_{I_{new}}\left(\overrightarrow{x}\right)-{\overrightarrow{F}}_{I_i}\left(\overrightarrow{y}\right)\right)$, with ${\overrightarrow{F}}_{I_{new}}\left(\overrightarrow{x}\right)$ and ${\overrightarrow{F}}_{I_i}\left(\overrightarrow{y}\right)$ denote the patch-based signature at $\overrightarrow{x}$ of I_new and at $\overrightarrow{y}$ of I_i, respectively.

Due to the noise and anatomical complexity of prostate MR images, patch-based representation using voxel intensities alone may not be able to effectively distinguish prostate and non-prostate voxels. We use patch-based anatomical features as signature ${\overrightarrow{F}}_I\left(\overrightarrow{r}\right)$ for each voxel to characterize the prostate image appearance. Three types of images features, namely, the Gabor wavelet (GW) feature, the histogram of gradient (HOG) feature, and the local binary pattern (LBP) feature, are extracted from a small image patch centered at each voxel of each aligned training image. Gabor and HOG features can provide complementary anatomical information to each other, and LBP can capture texture information from the input image [30]. However, the graph weight $w_i\left(\overrightarrow{x},\overrightarrow{y}\right)$ in Eq. (2) may not be able to effectively identify the most representative candidate voxels in atlases to estimate the prostate probability of a reference voxel, especially when the reference voxel is located near the prostate boundary, which is the most difficult region to segment correctly. So we propose to enforce the sparsity constraint in the conventional label propagation framework to resolve this issue. More specifically, the sparse graph weights based on least absolute shrinkage and selection operator (LASSO) [31] are estimated to reconstruct the patch-based signature of each voxel $\overrightarrow{x}\in I_{new}$ by signature of neighboring voxels in the training images. To do this, firstly signatures ${\overrightarrow{F}}_{I_i}\left(\overrightarrow{y}\right)$, $\overrightarrow{y}\in {\text{ψ}}_i\left(\overrightarrow{x}\right)\left(i=1,\cdots ,N\right)$ can be organized as columns in a matrix Θ. Then, the corresponding sparse coefficient vector ${\overrightarrow{\xi }}_{\overrightarrow{x}}$ of voxel $\overrightarrow{x}$ can be estimated by solving the following optimization problem:

$${\widehat{\overrightarrow{\xi }}}_{\overrightarrow{x}}=\arg {\min }_{{\overrightarrow{\xi }}_{\overrightarrow{x}}}{‖{\overrightarrow{F}}_{I_{new}}\left(\overrightarrow{x}\right)-\Theta {\overrightarrow{\xi }}_{\overrightarrow{x}}‖}_2^2+\lambda {‖{\overrightarrow{\xi }}_{\overrightarrow{x}}‖}_1,\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}{\overrightarrow{\xi }}_{\overrightarrow{x}}\ge 0$$ (3)

The optimal solution of this equation is denoted as ${\overrightarrow{\xi }}_{\overrightarrow{x}}^{opt}$. Here we can set the graph weight $w_i\left(\overrightarrow{x},\overrightarrow{y}\right)$ to the corresponding element in ${\overrightarrow{\xi }}_{\overrightarrow{x}}^{opt}$ with respect to $\overrightarrow{y}$ in image I_i. Then, we can use the prostate probability map G estimated by Eq. (1) to localize the prostate in the new treatment image. Through enforcing the sparsity constraint, candidate voxels assigned with large graph weights are all from the prostate regions, while candidate voxels from non-prostate regions are mostly assigned with zero or very small graph weights.

3. EXPERIMENTS AND RESULTS

The proposed segmentation method was tested with MR images of 13 prostate-cancer patients. All MR data were acquired using a Siemens MR scanner and an external-body surface coil. Each 3D T2W MR data sets consisted of 256×256×176 voxels to cover the whole pelvis and the voxel size was 1.0×1.0×1.0 mm³. All prostate glands were contoured in MR images by an experienced physician. We used leave-one-out cross-validation method to evaluate the proposed segmentation algorithm. In other words, we used the 12 training images and segmented prostates as the training set and applied the proposed method to process the remaining subject. Our prostate segmentations are compared with the physicians’ manual contours. The Dice volume overlap was calculated for quantitative evaluation. Figure 2 is an example that shows the proposed segmentation method works well for 3D MR images of the prostate and achieved similar results as compared to the manual segmentation. We successfully performed the segmentation method for all enrolled patients. Figure 3 shows the Dice volume overlaps between our automatic and manual segmentations for all patients. Overall, the averaged Dice volume overlap was 89.5±2.6%, which demonstrated the accuracy of the proposed segmentation method.

Figure 2.

Comparison between the automatic and manual segmentations. (a) Axial, (b) coronal and (c) sagittal T2 MR images of the prostate. The automatic prostate segmentation is shown is yellow dashed line and the manual segmentation is shown in red dotted line.

Figure 3.

Dice volume overlaps between the automatic and manual segmentations.

4. DISCUSSION AND CONCLUSION

We report a novel 3D MR prostate segmentation method based on the patch-based label fusion framework. The atlases with the most similar global appearance are selected to serve as the best subjects in the label fusion. A local patch-based atlas fusion is performed using voxel weighting based on anatomical signature. In this study, we have demonstrated its clinical feasibility, and validated its accuracy with manual segmentations (gold standard). This segmentation technique could be a useful tool in image-guided interventions for prostate-cancer diagnosis and treatment.