Hierarchical Multi-atlas Label Fusion with Multi-scale Feature Representation and Label-specific Patch Partition

Abstract

Multi-atlas patch-based label fusion methods have been successfully used to improve segmentation accuracy in many important medical image analysis applications. In general, to achieve label fusion a single target image is first registered to several atlas images, after registration a label is assigned to each target point in the target image by determining the similarity between the underlying target image patch (centered at the target point) and the aligned image patch in each atlas image. To achieve the highest level of accuracy during the label fusion process it’s critical the chosen patch similarity measurement accurately captures the tissue/shape appearance of the anatomical structure. One major limitation of existing state-of-the-art label fusion methods is that they often apply a fixed size image patch throughout the entire label fusion procedure. Doing so may severely affect the fidelity of the patch similarity measurement, which in turn may not adequately capture complex tissue appearance patterns expressed by the anatomical structure. To address this limitation, we advance state-of-the-art by adding three new label fusion contributions: First, each image patch now characterized by a multi-scale feature representation that encodes both local and semi-local image information. Doing so will increase the accuracy of the patch-based similarity measurement. Second, to limit the possibility of the patch-based similarity measurement being wrongly guided by the presence of multiple anatomical structures in the same image patch, each atlas image patch is further partitioned into a set of label-specific partial image patches according to the existing labels. Since image information has now been semantically divided into different patterns, these new label-specific atlas patches make the label fusion process more specific and flexible. Lastly, in order to correct target points that are mislabeled during label fusion, a hierarchically approach is used to improve the label fusion results. In particular, a coarse-to-fine iterative label fusion approach is used that gradually reduces the patch size. To evaluate the accuracy of our label fusion approach, the proposed method was used to segment the hippocampus in the ADNI dataset and 7.0 tesla MR images, sub-cortical regions in LONI LBPA40 dataset, mid-brain regions in SATA dataset from MICCAI 2013 segmentation challenge, and a set of key internal gray matter structures in IXI dataset. In all experiments, the segmentation results of the proposed hierarchical label fusion method with multi-scale feature representations and label-specific atlas patches are more accurate than several well-known state-of-the-art label fusion methods.

1. Introduction

Many medical image analysis studies require an accurate segmentation of anatomical structures in order to measure structural differences across individuals or between groups (Aljabar et al., 2009; Hsu et al., 2002). For example, in connectome applications multiple brain regions, in hundreds of brain MR images, need to be automatically identified before constructing a brain connectivity network (Li et al., 2012; Liu and Ye, 2010) that describes network architecture of the human brain. Therefore, to improve segmentation accuracy the development of automatic ROI (region of interest) labeling methods have seen increased attention in the medical imaging field over the last several years (Aljabar et al., 2009; Coupe et al., 2011; Rousseau et al., 2011; Tong et al., 2012; Wang et al., 2012; Wang et al., 2011a, b; Warfield et al., 2004; Wu et al., 2014).

Multiple atlases with manually identified labels have proven to be very useful when used to detect and label ROIs in the target image that may show high structural variations in the population. The basic assumption behind multi-atlas based segmentation is the target image point should bear the same label as the atlas image point if the local tissue shape or appearance is very similar. All atlas images are required to be registered to a target image before label fusion. To alleviate possible registration errors, patch-based label fusion (Coupe et al., 2011; Rousseau et al., 2011) seeks multiple correspondence candidates using patchwise similarity measurements between the target image patch and the atlas image patches within a certain voxel neighborhood. Intuitively, if the calculated similarity measurement between a target image patch and a particular atlas image patch is very high, then the atlas label assigned to the target point is the correct one.

To accurately assess image patch similarity, the identification and selection of ideal image patches is a key component of patch-based label fusion methods. Most state-of-the-art methods simply use fixed size patches throughout the entire label fusion procedure. For example, 7×7×7 or 9×9×9 cubic patches are usually used in the literature (Coupe et al., 2011; Rousseau et al., 2011; Tong et al., 2012; Wang et al., 2012). In order to make the label fusion robust to noise, image patches are required to be sufficiently large enough to capture the intended image content. However, using a large image patch may create additional problems when labeling small anatomical structures, e.g. the patchwise similarity measurement could be dominated by other larger anatomical structures surrounding the smaller one in the image patch. In short, methods that use fixed-size patches lack of the discriminative power to characterize complex appearance patterns in the medical imaging data.

During last decade, many efforts have been made to improve the discrimination ability of image patches during label fusion. For instance, sparse dictionary learning is used in (Tong et al., 2013) to find the best feature representations prior to label fusion. Moreover, in (Wang et al., 2012) and (Wu et al., 2014) dependencies among atlas image patches have been investigated to improve labeling accuracy by iteratively inspecting incorrectly labeled patches that show similar labeling error patterns. However, these state-of-the-art approaches use patches with fixed size and therefore still suffer from this limitation.

In this paper, we address the above limitations by developing hierarchical and high-level feature representations to adequately describe image patches. We propose the following three contributions: First, a layer-wise multi-scale feature representation adaptively encodes image features at different scales for each image point in the image patch. In the proposed approach, feature representations near the center of the patch provide more detailed (fine-scale) shape or appearance information, whereas feature representations near the edge of the patch provide less detailed (coarse-scale) shape or appearance information. Second, it's very common that the structure to be segmented, e.g. the hippocampus, is surrounded by other anatomical structures in the image patch. In such cases it becomes very difficult to correctly recognize the intended structure from the surrounding ones and mislabeling is likely to occur. In computer vision, object recognition algorithms address this limitation by attempting to separate the foreground pattern from background clutter (Li et al., 2010). In light of this research, a novel label-specific patch partition technique is proposed that splits each atlas patch into a set of new complementary label-specific (or structure-specific) image patches. To handle the increased number of label-specific image patches after the proposed patch splitting strategy a group sparsity constraint is included. As result, the discriminative power of each label-specific image patch is enhanced because it only contains the image information of the corresponding anatomical structure. To the best of our knowledge, this type of representation is rarely exploited in label fusion. Third, because existing label fusion methods typically use a fixed patch size, and label the entire target image in one pass, they are not given a chance to correct possible errors. To overcome this limitation the proposed method uses an iterative label-fusion procedure. Specifically, larger image patches are used in the beginning to increase the search range, however at each iteration the labeling result is evaluated and the size of the image patch is gradually reduced. To ensure spurious artifacts do not dominate the proposed label-fusion method, a sparsity constraint is included that only allows a small number of atlas patches to participate in the label fusion process.

It should be noted that this paper is an extension of our previous work in (Wu and Shen, 2014). However, there are several differences, specifically: a group sparsity constraint is used instead of a weighting vector sparsity constraint, a more comprehensive validation of each contribution (i.e., multi-scale feature representation, label-specific patch partition, and iterative label fusion), and additional datasets are used to evaluate the performance of the proposed label fusion method.

Performance of the proposed label fusion method is compared to existing state-of-the-art patch-based labeling methods (Coupe et al., 2011; Rousseau et al., 2011) using several different datasets. Specifically, the datasets used to evaluate the proposed method are the MICCAI 2013 segmentation challenge dataset (Landman and Warfield, 2012) with 14 manually labeled ROIs in the mid-brain, the LONI LBPA40 dataset (Shattuck et al., 2008) with 54 manually labeled ROIs at sub-cortical regions, and the IXI dataset with 83 manually labeled ROIs (Hammers et al., 2003; Hammers et al., 2007). Finally, we also include hippocampus segmentation experiments using the ADNI (Alzheimer’s Disease Neuroimaging Initiative) dataset and 7.0 tesla MR images (Cho et al., 2010). For each dataset the proposed method achieves a more accurate labeling result.

The remainder of the paper is organized as follows: In Section 2 we present our novel generative probability model for label fusion, in Section 3 we evaluate its performance by comparing it with conventional patch-based methods, and in Section 4 we provide a brief conclusion.

2. Method

Given the target image T, the goal of label fusion is to automatically determine the label map L_T for the target image. We first register each atlas image, as well as the label maps, onto the target image space. We use I = {I_s|s = 1, …, N} and L = {L_s|s = 1, …, N} to denote the N registered atlases and label maps, respectively. For each target image point x (x ∈ T), all the atlas patches^‡ within a certain search neighborhood n(x), denoted as β⃑_s,y(β⃑_s,y ⊂ I_s, y ∈ n(x)), are used to compute the patchwise similarities w.r.t. the target image patch α⃑_T,x (α⃑_T,x ⊂ T). We arrange each patch, β⃑_s,y and α⃑_T,x, into a column vector. We use the tuple b = (s,y) to denote both the atlas image index s and the location of the patch center point y, respectively. Thus, each atlas image patch β⃑_s,y can now be simplified to β⃑_b(b = 1, …, Q), where Q = N × |n(x)| is the total number of atlas image patches which are used to label the center point of the target image patch α⃑_T,x. For clarity, we use only α⃑ to denote the underlying target image patch by dropping off the subscripts in α⃑_T,x.

Label fusion methods such as non-local averaging (Coupe et al., 2011; Rousseau et al., 2011), can be used to calculate the weighting vector w⃑ = [w_b]_b=1,…,Q for all atlas patches, each of which is denoted by β⃑_b. As we will explain in Section 2.2, we adopt the sparsity constraint (Liu et al., 2009a, b; Tibshirani, 1996) in our method by regarding the label fusion procedure as the problem of finding the optimal combination among a set of atlas patches {β⃑_b} for the target image patch α⃑ (Tong et al., 2012; Zhang et al., 2012):

$$\widehat{\stackrel{\rightharpoonup }{w}}=\text{arg}\underset{\stackrel{\rightharpoonup }{w}}{\text{min}}\frac{1}{2}{\Vert \stackrel{\rightharpoonup }{\alpha }-\mathit{B}\stackrel{\rightharpoonup }{w}\Vert }^2+\lambda {\Vert \stackrel{\rightharpoonup }{w}\Vert }_1,$$ (1)

where the scalar λ controls the strength of sparsity constraint and B is a matrix built by assembling all column vectors {β⃑_b} in a columnwise manner. The image patch vectors are usually required to be normalized to the unit vector before optimizing over the sparse coefficients w⃑ (Wright et al., 2009). Assuming that we have M possible labels {l₁, …, l_m, …, l_M} in the atlases, the label on target image point x can be efficiently determined by:

$${\widehat{L}}_T(x)=\underset{m=1,\dots ,M}{\text{arg max}}\sum _{b=1}^Q[w_b·\delta (L_b,l_m)],$$ (2)

where L_b denotes the label in the center point of the atlas patch β_b, and the Dirac function δ(L_b, l_m) is equal to 1 when L_b = l_m and 0 otherwise.

As we can see in Eq. 1, the image intensities in the entire image patch are used for label fusion. Since one image patch may contain more than one anatomical structure and the to-be-segmented target ROI may have a complex shape/appearance pattern, the current patch-based label fusion methods have a certain risk of being misled by the patchwise similarities computed using image patches of fixed size or scale. We address this issue by introducing the idea of adaptive scale that has the following three components. Firstly, we treat each element within the image patch differently w.r.t. the radial distance toward the patch center. Therefore, a single image patch can convey image information from multiple scales (Section 2.1). Secondly, we treat the label information within the image patch separately, instead of as a whole. Specifically, we adaptively build label-specific atlas patches by using the existing label information in the atlases (Section 2.2). Thirdly, we dynamically reduce the patch size from large to small in order to hierarchically improve the label fusion accuracy in a coarse-to-fine manner (Section 2.3).

2.1 Multi-scale Feature Representations

As demonstrated in our previous work (Wu et al., 2006), image points at different brain regions should use different image scales to precisely characterize the local anatomical information. However, in most patch-based label fusion methods, every point in the image patch contributes equally and uses just its own intensity value for computing the patchwise similarity. We overcome this limitation by allowing each point to use an adaptive scale for capturing local appearance characteristics. Specifically, we first partition the whole image patch into several nested non-overlapping layers, spreading from the center point to the boundaries of the image patch. Next, we capture the fine-scale features for the layer closest to the patch center since the label fusion procedure eventually aims at determining the label for the central point. We gradually use larger and larger scales to capture the coarse-scale information as the distance to the patch center increases. Although the image pyramid technique (Liu and Ye, 2010) can be applied for multi-scale feature representation, we choose the less computationally demanding solution of adaptively replacing the original intensity values with the convolved intensity values using different Gaussian filters.

Fig. 1 illustrates the procedure of how to integrate the multi-scale feature representation into the conventional image patch. In the following example, we use three non-overlapping layers. First, we deploy three Gaussian filters upon the original image patch separately and obtain three smoothed image patches. Then, for each element in the image patch, we replace its original intensity value with the new value in the smoothed image at the same location. In this example, we replace the intensities in the inner most layer by the convolved intensity values smoothed via a Gaussian filter with the smallest kernel (in blue, in the right side of figure). For each point in the middle layer, we use the convolved intensity value from the smoothed image patch via a Gaussian filter with the medium kernel (in red). Similarly, we use the smoothed image patch via a Gaussian filter with the largest kernel as the feature representation for the image points in the third layer (in green). In this way, the image patch is now equipped with the multi-scale feature representations, as shown in the right of Fig. 1. Hereafter, α⃑ and β⃑_b denote the image patches after replacing the original intensities with the multi-scale feature representations.

Fig. 1.

Construction of the multi-scale image patch by adaptively replacing the intensity values with the convolved intensity values via multiple Gaussian filters.

The advantage of using multi-scale feature representation in patch-based label fusion is shown in Fig. 2. Specifically, we examine the discriminative power of two target image points, designated by red ‘+’ and red ‘Δ’ in Fig. 2. For clarity, we only use one atlas image in this example (bottom left of Fig. 2). The corresponding locations of the two target image points in the atlas image are designated with blue ‘+’ and blue ‘Δ’, respectively. For each candidate point in the search neighborhood (i.e., blue dash boxes in Fig. 2), we compare the patch-wise intensity similarity w.r.t the target image point by using small-scale image patches (3 × 3 × 3), large-scale image patches (17 × 17 × 17), and our proposed multi-scale image patches, respectively. Fig. 2 (a)–(c) shows the similarity maps obtained by comparing the target image patch and each candidate atlas image patch in the search neighborhood, where bright colors indicate high similarity, and dark colors indicates low similarity.

Fig. 2.

The advantages of using the multi-scale patch representation in patch-based label fusion. Each marker (“+” or “Δ”) denotes a pair of corresponding points in the atlas and target images, respectively. We show the advantages of using the multi-scale patch representation by examining the patch-wise similarity maps for a particular target image point w.r.t. each atlas image point. Compared with the similarity maps by using a small-scale image patch (a) and a large-scale image patch (b), our multi-scale image patch (c) can identify the corresponding locations in the atlas image domain more accurately. This can be noted by the fact that the similarity maps have high values (bright red) around the corresponding location and comparatively lower values (dark blue) elsewhere.

The principle behind patch-based label fusion methods is that two image patches should bear the same label if they have similar appearances. Therefore, the benefit of our multi-scale feature representation lies in its ability to recognize more reliable correspondences than the conventional image patches. As shown in Fig. 2 (a), when using conventional small-scale image patches many image regions from obviously different anatomical structures present high similarities. This is because the appearance information in the small-scale image patch is too limited to characterize the complex anatomical structures. This has the undesirable effect of introducing misleading labels in label fusion. On the other hand, using conventional large-scale image patches can alleviate this issue by incorporating global information, but at the expense of losing discriminative power. This can be seen in Fig. 2 (b), where a conventional large-scale image patch can approximately distinguish the atlas patches nearby the corresponding locations. However, a large number of atlas image patches belonging to different anatomical structures still present high similarities when using conventional large-scale image patches. Our multi-scale image patch combines both local and global information, which leads to a more reasonable similarity map as shown in Fig. 2 (c). As we can see, our method can identify more accurate correspondences than using either small or large conventional patches. Thus, in the scenario of patch-based label fusion, the similarity map obtained by using our multi-scale image patch representation encourages assigning high weights to the true anatomical correspondences (with the correct labels) and also suppresses the atlas patches belonging to other structures (with incorrect labels).

2.2 Label-specific Atlas Patch Partition

Since atlas image patches have label information, we can partition each atlas patch into a set of new label-specific atlas patches, thus separately encoding the image information for each individual label. Given the atlas image patch β⃑_b, we use γ⃑_b to denote its associated label patch. Suppose there are M^b (0 < M^b ≤ M) different labels in γ⃑_b. Then, the proposed label-specific atlas patch set P_b consists of M^b label-specific atlas patches, i.e., ${\mathit{P}}_b=\{{\stackrel{\rightharpoonup }{p}}_b^m|m=1,\dots ,M^b\}$, where ${\stackrel{\rightharpoonup }{p}}_b^m$ is a column vector. Each element u in ${\stackrel{\rightharpoonup }{p}}_b^m$ preserve the intensity value β⃑_b(u) if and only if γ⃑_b(u) has the label l_m; otherwise, ${\stackrel{\rightharpoonup }{p}}_b^m(u)=0$. Mathematically, we have ${\stackrel{\rightharpoonup }{p}}_b^m(u)={\stackrel{\rightharpoonup }{\beta }}_b(u)·\delta ({\stackrel{\rightharpoonup }{\gamma }}_b(u),l_m)$ and ${\stackrel{\rightharpoonup }{\beta }}_b={\cup }_{m=1}^M{\stackrel{\rightharpoonup }{p}}_b^m$, where δ(․,․) is the same Dirac function as used in Eq. 2.

Fig. 3 demonstrates the construction of the label-specific atlas patch partition. For clarity, we only use the original 3 × 3 image patch in this example, instead of the above multi-scale image patch. Suppose that we have three atlas image patches and there are two labels (hippocampus and non-hippocampus) in each patch, i.e., M^b = 2 (b = 1,2,3). Next, for each atlas patch β⃑_b, we split it into two partial patches ${\stackrel{\rightharpoonup }{p}}_b^1$ and ${\stackrel{\rightharpoonup }{p}}_b^2$, which are denoted in blue (non-hippocampus) and red (hippocampus) in Fig. 3, respectively. Each label-specific atlas patch $p_b^m$ preserve the intensity value only if the element bears the label l_m. Otherwise, use zero to represent the elements with a different label in the particular partial patch $p_b^m$.

Fig. 3.

The construction of a label-specific atlas patch partition.

Note that the number of image patches increases significantly after we partition each atlas patch into the label-specific atlas patch set. Thus, we propose to use the sparsity constraint in label fusion, in order to select only a small number of label-specific atlas patch ${\stackrel{\rightharpoonup }{p}}_b^m$ for representing the target image patch α⃑. By replacing each conventional atlas patch with the label-specific atlas patches, the matrix of atlas patches B in Eq. 1 now expands to P = [P_b]_b=1,…,Q. Then, the new energy function for label fusion can be reformulated as:

$$\widehat{\stackrel{\rightharpoonup }{\xi }}={\text{arg min}}_{\stackrel{\rightharpoonup }{\xi }}\frac{1}{2}{\Vert \stackrel{\rightharpoonup }{\alpha }-\mathit{P}\stackrel{\rightharpoonup }{\xi }\Vert }^2+\lambda {\Vert \stackrel{\rightharpoonup }{\xi }\Vert }_1,\mathrm{s}.\mathrm{t}.\stackrel{\rightharpoonup }{\xi }>0,$$ (3)

where $\stackrel{\rightharpoonup }{\xi }=[{\xi }_b^m]$ is the weighting vector for each label-specific atlas patch ${\stackrel{\rightharpoonup }{p}}_b^m$ . Since the goal of Eq. 3 is to minimize the difference between the target image patch and its sparse representation of label-specific atlas patches, the padded zero values in each label-specific atlas patch ${\stackrel{\rightharpoonup }{p}}_b^m$ have no influence when optimizing Eq. 3.

Conventionally, each weight ${\xi }_b^m$ in ξ⃑ is independently treated when optimizing Eq. 3. Here, we go one step further and enforce the group sparsity constraint on ξ⃑. Obviously, there are Q non-overlapping groups of label-specific atlas patches, where each group consists of a set of label-specific atlas patches split from β⃑_b. Supposing that ${\stackrel{\rightharpoonup }{\xi }}_b={[{\xi }_b^m]}_{m=1,\dots ,M^b}$ denotes the weights for all label-specific atlas patches within the original atlas image patch β⃑_b, the new energy function with group sparsity constraint is:

$$\widehat{\stackrel{\rightharpoonup }{\xi }}={\text{arg min}}_{\stackrel{\rightharpoonup }{\xi }}\frac{1}{2}{\Vert \stackrel{\rightharpoonup }{\alpha }-\mathit{P}\stackrel{\rightharpoonup }{\xi }\Vert }^2+{\lambda }_1{\sum }_{b=1}^Q{\Vert {\stackrel{\rightharpoonup }{\xi }}_b\Vert }_2+{\lambda }_2{\Vert \stackrel{\rightharpoonup }{\xi }\Vert }_1,\mathrm{s}.\mathrm{t}.\stackrel{\rightharpoonup }{\xi }>0,$$ (4)

where λ₁ and λ₂ control the strength upon non-overlapping groups and the entire weighting vector ξ⃑, respectively. The new energy function falls into the scenario of sparse group LASSO (Friedman et al., 2010; Vincent and Hanse, 2014) which encourages sparsity not only for the entire weighting vector ξ⃑ (as reflected by the third term in Eq. 4), but also for the number of selected groups (as reflected by the second term in Eq. 4). The optimization of Eq. 4 can be efficiently solved by using the SLEP (Sparse Learning with Efficient Projection) software package (Liu et al., 2009b; Liu and Ye, 2010).

Since each ${\stackrel{\rightharpoonup }{p}}_b^m$ is only related with a particular label l_m, each element ${\xi }_b^m$ in ξ⃑ represents the probability of labeling the center point x of the target image patch α⃑ with the label l_m. Therefore, the labeling result on the target image point x can be obtained by:

$${\widehat{L}}_T(x)=\underset{m=1,\dots ,M}{\text{arg max}}\sum _{b=1}^Q{\xi }_b^m.$$ (5)

The advantage of using label-specific atlas patches is demonstrated by a toy example in Fig. 4, where we use red and blue to denote two different labels and we use numbers to represent the intensity values. For the sake of simplicity, we have only used two atlas patches in this example. Both the first atlas patch (i.e., the first column in B) and the target patch α⃑ belong to the same structure since their intensity values are in ascending order. If we estimate the weighting vector w⃑ based on the entire atlas patch by Eq. 1 (λ = 0.01), the weights for the first and second atlas patches are 0.43 and 0.49, respectively. According to Eq. 2, we have to assign the target point with the blue (incorrect) label. In our method, we first extend the matrix B to the label-specific atlas patch set P, as shown in the bottom of Fig. 4, and then solve the new weighing vector ξ⃑ by Eq. 4 (λ₁ = 0.1 and λ₂ = 0.1). According to Eq. 5, the overall weights for red and blue labels are 0.8194 (0.8094+0.0100) and 0.3329 (0.2683+0.0646), respectively. Therefore, it is straightforward to correctly assign the target point with the red label. It is worth noting that, if only using the sparsity constraint on ξ⃑, the overall weights for the red and blue labels are 0.8850 (0.8000+0.0050) and 0.8004 (0.6901+0.1103), respectively. As we can see, the vote for the red label is only slightly better than the blue label. This example demonstrates the benefits of both the label-specific patch partitions and of enforcing group sparsity.

Fig. 4.

The advantage of enforcing the group sparsity constraint in label fusion.

2.3 Hierarchical Patch-based Label Fusion

In Section 2.1, we have presented the use of multi-scale image patch to adaptively treat each element in the image patch. As observed in Fig. 2, combining global and local information can significantly increase the robustness and discriminative power of the image patch in label fusion. Along the same lines, we further propose to dynamically adjust the patch size from large to small during the label fusion procedure. The idea is to initially resort to global information (i.e., using a large patch size) to discard the misleading candidate atlas patches and then gradually use more local information (i.e., using smaller patch sizes) to refine the optimization of the energy function (in Eq. 4) based on the remaining atlas patches.

In the beginning of patch-based label fusion, we propose to use a large patch size in order to capture the global image information. Since we use the sparsity constraint for solving the weighing vector ξ⃑, only a small number of image patches are selected to represent the target image patch α⃑, since many elements in ξ⃑ are zero or almost zero. After discarding those unselected atlas patches, we can confidently reduce the patch size of those selected atlas patches and then repeat the whole label fusion procedure as described in Sections 2.1 and 2.2 by using more detailed, local features. In this way, our label fusion method can iteratively improve the labeling results in a hierarchical way.

The advantage of our hierarchical patch-based label fusion is demonstrated in Fig. 5, where we aim to determine the label of the target image point x (red cross in Fig. 5) located near the boundary of the hippocampus (with ground-truth label corresponding to hippocampus). To determine the label for this target image point x, a set of candidate atlas image patches are examined in a 15 × 15 × 15 search neighborhood (i.e., blue dash boxes). After patch pre-selection (Coupe et al., 2011), only around 2000 atlas patches are used in determining the label for target image point x. In order to explicitly show the advantage of our hierarchical patch-based label fusion scenario, we only use the original image patch, with neither multi-scale representation nor label-specific partition. Moreover, only the sparsity constraint is used to seek for the label fusion weights (i.e., Eq. 3), instead of using the group sparsity constraint. In the first iteration, the patch size is set to 11 × 11 × 11. In Fig. 5 (a), we plot the sparse coefficients after solving Eq. 3. The red and blue plots correspond to the atlas patches with hippocampus label and non-hippocampus label, respectively. It is clear that a large number of atlas image patches with non-hippocampus labels are selected to represent the target image patch, which makes the selection of the underlying label somewhat arbitrary due to the fact that the overall weight for the non-hippocampus label is nearly the same as for the hippocampus. In the second iteration, we only focus on the remaining (selected) atlas patches by discarding the unselected atlas patches (with zero coefficients). At this point, we reduce the patch size from 11 × 11 × 11 to 7 × 7 × 7, in order to resort to the local image information to refine the sparse representation. Since a lot of misleading and noisy image patches have been removed, the task of sparse representation becomes relatively easier. As shown in Fig. 5 (b), the overall weight voting for hippocampus dominates the weight for non-hippocampus. However, we still can see some large sparse coefficients for some non-hippocampus image patches. Thus, we finally repeat the same procedure with the patch size reduced to 3 × 3 × 3. It can be observed in Fig. 5 (c) that (1) only a few atlas patches are used to determine the label for the target image point x, and (2) all the selected atlas patches have the correct label w.r.t. the target image point x. It is worth noting that directly using the 3 × 3 × 3 image patches from scratch does not result in good estimations, as indicated by the plot of sparse coefficients in Fig. 5 (d). The main reason is that the appearance information from small image patches is too local to deal with the complex anatomical structures present in the brain images. On the contrary, our hierarchical label fusion framework uses the global image information to gradually remove the misleading candidate atlas patches, thus ensuring to obtain more accurate label fusion results when applying the small image patch size in the end.

Fig. 5.

The advantage of using hierarchical label fusion with dynamic patch sizes.

3. Experiments

To evaluate label performance, the proposed label fusion method is compared to several existing state-of-the-art patch-based methods using publically available neuroimaging datasets. Specifically, the non-local weighting (Nonlocal-PBM) (Coupe et al., 2011; Rousseau et al., 2011), and the recently proposed sparse patch-based labeling method (Sparse-PBM) (Tong et al., 2012; Zhang et al., 2012) are tested. To assess label accuracy, the Dice ratio is used which measures the degree of overlap between two ROIs O_1 and O_2 as follows:

$$\mathit{\text{Dice}}(O_1,O_2)=2\times \frac{|O_1\cap O_2|}{|O_1|+|O_2|},$$ (6)

where |·| means the volume of the particular ROI.

As shown in Table 1, an iterative process that uses varying configurations is implemented. In general, a configuration defines several partition layers defined within the patch. For instance, if the label fusion method initially starts with a 9×9×9 patch it will be partition into three layers. From the center point of the patch the [1,1,2] setting describes the width of each partition. For this particular setting, the width of the first two layers is 1 voxel and the width of the third layer is 2 voxels. Lastly, the [0.5, 1, 2] setting controls the kernel width that is used for smoothing. Likewise for this setting, 0.5 is the kernel width for the first layer, 1 is the kernel width for second layer, and 2 is the kernel width for the third layer. The above process is executed for two additional iterations that gradually reduce the patch size to 3×3×3. These parameters are fixed throughout all the experiments. For the other counterpart methods, we report the results using the parameters that result in the best performance. Lastly, The values of λ₁ and λ₂ are both set to 0.1 for all experiments.

Table 1.

Example multiple layer configuration

Patch size	Number of layers	Layer width	Gaussian Kernel size
9 × 9 × 9	3	[1,1,2]	[0.5,1.0,2.0]
5 × 5 × 5	2	[1,1]	[0.5,1.0]
3 × 3 × 3	1	[1]	[0.5]

The remainder of this section is organized as follows: In Section 3.1 a comprehensive evaluation of the proposed label fusion method is performed using the ADNI dataset, then in Section 3.2 the parameters are fixed and the proposed method is used to segment the hippocampus in 7.0 tesla MR images. In Section 3.3, the proposed method is evaluated using the 14 mid-brain structures in the SATA MICCAI 2013 segmentation challenge dataset, and in Section 3.4 the proposed method is evaluated using 54 manually labeled sub-cortical regions in the LONI LBPA 40 dataset (Shattuck et al., 2008). For a fair comparison, the Nonlocal-PBM and Sparse-PBM parameter settings reported in (Coupe et al., 2011) and (Zhang et al., 2012) respectively, are used.

3.1 Experimental Result of Hippocampus Labeling on the ADNI Dataset

In many neuroscience studies, accurate delineation of hippocampus is very important for quantifying the inter-subject anatomical difference and subtle intra-subject longitudinal changes, since the structural change of hippocampus is closely related with dementias, such as Alzheimer’s disease (AD). In this experiment, we randomly select 23 normal control (NC) subjects, 22 MCI (Mild Cognitive Impairment) subjects, and 21 AD subjects from the ADNI dataset^§. The following three pre-processing steps have been performed to all subject images: (1) Skull removal by a learning-based meta-algorithm (Shi et al., 2012); (2) N4-based bias field correction (Tustison et al., 2010); (3) intensity standardization to normalize the intensity range (Madabhushi and Udupa, 2006). Semi-automated hippocampal volumetry was carried out using a commercial high-dimensional brain mapping tools (Medtronic Surgical Navigation Technologies, Louisville, CO), which has been validated and compared to manual tracing of the hippocampus (Hsu et al., 2002). In this experiment, we regard the hippocampal segmentations from ADNI as the ground truth.

A leave-one-out strategy is used to compare the label performance of Nonlocal-PBM, Sparse-PBM, and proposed label fusion method. In each leave-one-out experiment, affine registration is first performed by FLIRT in the FSL toolbox (Smith et al., 2004) with 12 degrees of freedom and the default parameters (i.e., normalized mutual information similarity metric, and a search range of ±20mm in all directions). Then after the affine registration, a deformable registration is performed using the diffeomorphic Demons (Vercauteren et al., 2009) method and the default registration parameters (i.e., smoothing sigma 1.8, and iterations in low, middle, and high resolutions as 20 × 10 × 5).

To evaluate the contribution of each component in the proposed label fusion method, we compare our method with the three degraded versions of our method: Degraded_1: our method using only the multi-scale feature representation (with patch size of 9 × 9 × 9), Degraded_2: our method using only the label-specific atlas patches (with patch size of 9 × 9 × 9), and Degraded_3: our method using only the hierarchical labeling mechanism.

Using all 66 leave-one-out cases, the mean and standard deviation of the Dice ratios from th hippocampus label results are calculated and reported in Table 2. A few important observations can be made. Compared to the five other methods, the proposed label fusion method with no degradation achieves the highest Dice ratio results, obtaining approximately a 1.9% and 1.2% improvement over the Nonlocal-PBM and Sparse-PBM methods, respectively. Each component in the proposed label fusion method improves labeling accuracy as seen by the 0.6%, 0.9%, and 0.3% Dice ratio increases over Degraded_1, Degraded_2, and Degraded_3, tests respectively. The computation times by the 6 label fusion methods are also reported in the last row of Table 2. The computation environment of our experiments is 8 CPUs @ 3.0GHz and 16G RAM.

Table 2.

Dice ratio mean, standard deviation, and mean computation time results for Nonlocal-PBM, Sparse-PBM, degraded versions of the proposed method, the proposed label fusion method and when used to label the hippocampus.

	Nonlocal-PBM	Sparse-PBM	Degraded_1	Degraded_2	Degraded_3	Proposed Method
Dice Ratio	86.6±3.5	87.3±3.4	87.9±3.0	88.2±2.5	87.6±2.9	88.5±2.2
Time (sec)	75	128	136	196	511	618

Since the improvement in label fusion is usually obtained around the boundary of hippocampus, it is interesting to examine the label results at the hippocampus surface. To perform this experiment we first construct ground-truth hippocampus surface mask and an estimated hippocampus surface mask. Then the distance at each vertex between two surfaces is computed. Table 3 shows the values of the averaged surface-to-surface distance and the maximum surface-to-surface distance by Nonlocal-PBM, Sparse-PBM, Degraded_1, Degraded_2, Degraded_3, and the proposed method with no degradation. We further perform the paired t-tests upon the surface distances. We observe that all degraded methods and the proposed method with no degradation have significant improvement (p<0.05) over Nonlocal-PBM, the Degraded_1, Degraded_2, and the proposed method with no degradation have significant improvement (p<0.05) over Sparse-PBM.

Table 3.

Dice ratio mean, standard deviation, and maximum surface distance results when used to label the hippocampus (unit: mm).

	Nonlocal-PBM	Sparse-PBM	Degraded_1+*	Degraded_2+*	Degraded_3+	Proposed Method+*
Mean	0.410±0.15	0.380±0.10	0.353±0.10	0.342±0.09	0.369±0.12	0.334±0.09
Max	4.359	3.742	3.317	3.000	3.464	2.450

Symbols ‘+’and ‘*’ indicate significant improvement (p<0.05) over the Nonlocal-PBM and sparse-PBM methods.

3.2 Experimental Result on the 7.0 Tesla MR Images

With the advent of 7.0-tesla MR imaging technology (Cho et al., 2010) the achievement of high signal-to-noise ratio (SNR), as well as a dramatic increase in tissue contrast compared to the 1.5- or 3.0-tesla MR images, is possible. A visual comparison in provided in Fig. 6, which shows a typical brain image slice produced by a 7.0-tesla scanner with resolution of 0.35 × 0.35 × 0.35mm³ next to slice from a 1.5-tesla scanner with a resolution of 1 × 1 × 1mm³. These high-resolution images enable researchers to clearly observe fine brain structures with sub-milimetric precision. We believe that the 7.0-tesla MR imaging technique has the potential to become the standard technique for discovering the morphological patterns in the human brain in the near future.

Fig. 6.

The hippocampus shown by (a) 1.5-tesla and (b) 7.0-tesla MR scans. The 1.5-tesla image has been enlarged to match the size of the 7.0-tesla image for visual comparison purposes.

For the 7.0-tesla scanner (Magnetom, Siemens), an optimized multichannel radiofrequency (RF) coil and a 3D fast low-angle shot (Spoiled FLASH) sequence were utilized, with TR=50ms, TE=25ms, flip angle 10°, pixel band width 30Hz/pixel, field of view (FOV) 200mm, matrix size 512 × 576 × 60, 3/4 partial Fourier, and number of average (NEX) 1. The image resolution of the acquired images is isotropic, e.g., 0.35 × 0.35 × 0.35mm³. The hippocampi were manually segmented by neurologists (Cho et al., 2010). All images were pre-processed by the following steps: 1) inhomogeneity correction using N4 bias correction (Tustison et al., 2010); 2) intensity normalization for making image contrast and luminance consistent across all subjects (Madabhushi and Udupa, 2006); 3) affine registration to the selected template by FSL.

Using 7.0–tesla MR imaging technology, the proposed label fusion method is used to segment the hippocampus from twenty-one 7.0-tesla MR brain images. Unfortunately, existing state-of-the-art deformable image registration methods that are developed for 1.5-tesla or 3.0-tesla MR images do not perform well when used on 7.0-tesla MR images. In general, this is primarily due to the severe intensity inhomogeneity in 7.0-tesla MR images, the richer texture information in 7.0-tesla MR images (as seen in Fig. 6(b)), and that only a small segment of the brain covering the hippocampus is scanned, instead of whole brain.

Since we have the manually labeled hippocampus for each 7.0-tesla MR image, we can quantitatively measure label fusion accuracy using a leave-one-out cross validation strategy. The mean and standard deviation of the Dice ratios on hippocampus are (77.42 ± 3.44)% by Nonlocal-PBM, (79.29 ± 2.46)% by Sparse-PBM, and (82.65 ± 1.37)% by the proposed method. Furthermore, in Table 4 we list the average and maximum surface distances between the manually segmented and the automatically estimated hippocampus masks by three different label fusion methods. Fig. 7 shows the mappings of the surface distances on three typical 7.0-tesla MR images.

Table 4.

Dice ration mean, standard deviation, and maximum surface distance results found by Nonlocal-PBM, sparse-PBM, and the proposed label fusion method when used to label the hippocampus in 7.0-tesla MR image (unit: mm).

	Nonlocal-PBM	Sparse-PBM	Our method
Mean	1.91±0.41	1.43±0.32	0.86±0.16
Max	7.07	5.20	4.69

Fig. 7.

Surface distance renderings obtained by Nonlocal-PBM, Sparse-PBM and our proposed label fusion method on 7.0-tesla MR images.

3.3 Experimental Result on the SATA MICCAI 2013 Challenge Dataset

Using the SATA dataset, provided by MICCAI 2013 segmentation challenge workshop (https://masi.vuse.vanderbilt.edu/workshop2013/index.php/Main_Page), 35 training samples (atlas images and labels) as well as a collection of 12 testing images are provided. There are 14 ROIs that cover accumbens area, amygdala, caudate, hippocampus, pallidum, putamen, and thalamus on both hemispheres. Since the organizers have provided all registered atlas images to each target image to be labeled, no registration is needed for this experiment. After the proposed label fusion method generates the label results, they are submitted to the workshop organizer that returned the quantitative results shown in Table 5. The Dice ratios in all ROIs by the three label fusion methods are shown in Fig. 8. It worth noting that the proposed method (named “UNC IDEA SuperMAS”) is currently ranked the topmost label fusion method in this challenge (http://masi.vuse.vanderbilt.edu/submission/leaderboard.html).

Table 5.

Mean Dice ratio, standard deviation, median, maximum and minimum results found by Nonlocal-PBM, Sparse-PBM, and the proposed label fusion method using the SATA dataset.

	Mean	Standard Deviation	Median	Max	Min
Nonlocal-PBM	85.81	2.80	86.95	89.04	80.61
Sparse-PBM	85.94	3.25	87.09	89.28	78.27
Proposed Method	86.54	2.59	87.67	89.23	82.00

Fig. 8.

Dice ratios for each ROI obtained by Nonlocal-PBM, Sparse-PBM, and the proposed label fusion method.

3.4 Experimental Result on the LONI LPBA40 Dataset

Here we evaluate the performance of label fusion using the LONI LBPA 40 dataset (Shattuck et al., 2008) that includes 40 brain images, and each brain image has 54 manually labeled ROIs. We randomly select 20 images as atlases and another 20 as target images. To label each target image, we first apply affine registration by FLIRT in the FSL toolbox (Smith et al., 2004) with 12 degrees of freedom and the default parameters (i.e., using the normalized mutual information similarity metric, and the search range ±20 in all directions). Then after the affine registration, a deformable registration is performed using the diffeomorphic Demons (Vercauteren et al., 2009) method and the default registration parameters (i.e., using the smoothing sigma 2.0, and iterations in low, middle, and high resolutions as 20 × 10 × 5).

The Dice ratio mean and standard deviation measures for the 54 ROIs are provided in Table 6. The proposed method achieves a 3.15% and 1.51% improvement compared the Nonlocal-PBM and Sparse-PBM methods, respectively. Fig. 9 shows the Dice ratio in each ROI found by the Nonlocal-PBM (blue), Sparse-PBM (green), and the proposed method (red). The proposed label fusion method shows a significant improvement in 34 of 54 ROIs when compared to Nonlocal-PBM (‘+’ denoting significant improvement according to a paired t-test (p < 0.05)), and in 29 of 54 ROIs when compared to Sparse-PBM (‘*’ denoting significant improvement according to a paired t-test (p < 0.05)).

Table 6.

Mean Dice ratio and standard deviation results found by Nonlocal-PBM, Sparse-PBM, and the proposed method using the LONI LPBA40 dataset.

	Nonlocal-PBM	Sparse-PBM	Proposed Method
Mean and standard deviation	78.31±3.52	79.95±3.38	81.46±2.25

Fig. 9.

Dice ratio for each ROI found by Nonlocal-PBM (blue), Sparse-PBM (green), and the proposed label fusion method (red). Symbols ‘⁺’ and ‘*’ indicate significant improvement (p<0.05) with respect to Nonlocal-PBM and sparse-PBM, respectively.

3.5 Comparison with other State-of-the-art Methods on the IXI Dataset

Recently, many multi-atlas based label fusion methods (Artaechevarria et al., 2009; Asman and Landman, 2012; Cardoso et al., 2013; Sabuncu et al., 2010a) have been developed to segment anatomical structures in medical images. STEPS (Similarity and Truth Estimation for Propagated Segmentations) (Cardoso et al., 2013) is one the most recent label fusion method that integrates image appearance information into the classic STAPLE algorithm (Warfield et al., 2004). Specifically, STEPS has achieved better segmentation results than other existing label fusion methods including (Asman and Landman, 2011; Asman and Landman, 2012; Sabuncu et al., 2010b; Yushkevich et al., 2010).

Here we compare segmentation performance of the proposed label fusion method with STEPS, Nonlocal-PBM, and Sparse-PBM using the IXI dataset (Hammers et al., 2003; Hammers et al., 2007)^**. The IXI dataset contains 30 subjects, each with 83 manually labeled ROIs. For the sake of comparison, we report the Dice ratios for same 7 ROIs (Hippocampus, Amygdala, Caudate Nucleus, Nuc. Accumbens, Putamen, Thalamus, Globus pallidus) originally reported in (Cardoso et al., 2013). Similarly to (Cardoso et al., 2013), we run Nonlocal-PBM, Sparse-PBM, and the proposed label fusion methods on all 30 subjects using a leave-one-out cross validation strategy. Table 7 shows the mean Dice ratio value for the 7 ROIs found by the different label fusion methods under test. As we can see, the proposed method achieves the best (i.e. greatest value) Dice ratio.

Table 7.

Mean Dice ratio results found by STEPS, Nonlocal-PBM, Sparse-PBM, and the proposed method using the IXI dataset.

	STEPS	Nonlocal-PBM	Sparse-PBM	Proposed Method
Hippocampus	84.2	82.3	84.0	84.6
Amygdala	80.5	78.2	79.5	81.5
Caudate Nucleus	89.2	88.5	88.9	89.5
Nuc. Accumbens	69.5	68.9	69.1	70.6
Putamen	89.1	87.4	88.8	89.2
Thalamus	89.4	87.8	89.2	89.5
Globus Pallidus	79.8	78.1	79.5	80.3

4. Discussion

Linear vs Deformable Image Registration

In (Rousseau et al., 2011), the authors propose the strategy that combines non-local label fusion with deformable image registration. According to their conclusions, accurate correspondences derived from deformable image registration could further improve non-local label fusion performance, especially when the intensity contrast is low. Since the overall goal of our paper is to improve labeling accuracy. In light of this, the proposed label fusion method was applied after deformable registration (using diffeomorphic demons) to map the labels from the atlas images to the target image. However, after several experiments we observed some interesting label fusion results that used linear registration instead of a non-linear one. In particular, Table 8 shows the mean and standard deviation of Dice ratios when segmenting the hippocampus using a 66 leave-one-out cross validation experiment. As shown in this table, we compared the Nonlocal-PBM, Sparse-PBM, Degraded_1, Degraded_2, Degraded_3, and our full label fusion method. Furthermore, the same dataset in Section 3.1 was used with one exception: all the label fusion methods under test were executed after linear registration. Compared to the Dice ratios in Table 2, the segmentation of the proposed method are more accurate when linear registration is performed, and is less accurate when a deformable image registration is performed. Moreover, in hippocampus dataset, label fusion results after deformable image registration are more accurate than after linear registration (87.9% by linear registration vs 88.5% by deformable registration), but at the expense of longer computational time (i.e., 20 minutes by linear registration vs 55 minutes by deformable registration).

Table 8.

Dice ratio mean and standard deviation results when the hippocampus is labeled using only a linear registration.

	Nonlocal-PBM	Sparse-PBM	Degraded_1	Degraded_2	Degraded_3	Our method
Dice Ratio	85.7±4.0	86.2±3.8	86.8±3.0	87.2±2.8	86.7±3.3	87.9±3.1

Overlapping vs Non-overlapping Layers in the Multi-Scale Feature Representation

In Section 2.1, the image patch was partitioned into non-overlapping layers that may present blockness problems across different layers. In order to evaluate how this potential problem effects label fusion performance the two additional tests were evaluated: Include overlapping layers with a 1 image point overlap between two layers, and increase the number of layers in each image patch. For each additional test, the label fusion method was rerun with non-overlapping layers and with overlapping layers on the hippocampus dataset. After performing a 66 leave-one-out cross validation, the mean and standard deviation of the Dice ratios achieved by the proposed label fusion method, with non-overlapping layers, and with overlapping layers, were 87.91±3.04 and 87.95±2.96, respectively. Paired t-test indicates no significant statistical difference between when non-overlapping or overlapping layers are used. However, in our implementation there is a significant difference in computation time. Specifically, the time required when overlapping is used requires significantly more time when non-overlapping layers is used.

Limitations and Future Work

In order to efficiently obtain the multi-resolution feature representation at each point, we experimentally partition the image patch into several nested non-overlapping layers and assign each layer with a pre-determined Gaussian Kernel. However, as we demonstrated in our previous work (Wu et al., 2006b), each image point should have its own best scale to describe the local characteristics of the anatomical structure. Thus, one of our future works is to develop an adaptive method to use the best image patch size and the best set of smoothing kernels for each point. To further increase the computational efficiency of the proposed method, GPU processing using the CUDA programming technique can be used to exploit parallel patch operations. Lastly, the integration of the proposed label fusion method into a open-source stand-alone software package, like MARS (Multi-Atlas Robust Segmentation) that is hosted at NITRC (http://www.nitrc.org/projects/mars), would give other researchers direct access to the software developed in this manuscript.

Finally, although we address the limitation of existing label fusion methods that use fixed size image patches, many other works are aimed at improving label fusion performance from different perspectives. For example, Ta et al (Ta et al., 2014) introduced a new patch-based method using the ‘PatchMatch’ algorithm that provides competitive segmentation accuracy in near real-time. Results showed that their label fusion method can segment the hippocampus from MR images in less than 1 second. From the application point of view, the non-local based method has been adapted to multiple medical imaging studies, such as intracranial cavity extraction (Eskildsen et al., 2012; Manjón et al., 2014) and extraction of hippocampus structural features for early detection of AD (Coupé et al., 2012a; Coupé et al., 2012b). In our future work, we plan to evaluate our proposed label fusion method in other imaging-based studies (Chen et al., 2009; Liu et al., 2012; Verma et al., 2005).

5. Conclusion

In this paper, new techniques are used to improve multi-atlas patch-based label fusion performance. Specifically, each atlas patch is assigned a multi-scale feature representation; atlas image patches are partitioned into several label-specific patches based on existing label information; and a hierarchical label fusion mechanism that iteratively improves the labeling result by gradually reducing patch size. Label fusion performance is evaluated using the ADNI dataset, 7.0-tesla MR image dataset, SATA MICCAI 2013 segmentation challenge dataset, LPBA40, and IXI dataset. Compared to publically available state-of-the-art label fusion methods, the proposed method has demonstrated the best label performance for each dataset. Lastly, it is worth noting that the proposed method has achieved the highest ranking in the SATA segmentation challenge.

Research Highlight.

1. Integrate the multi-scale feature representation into conventional image patch in label fusion.

2. Adaptively treat multiple anatomical structures within the image patch.

3. Hierarchically improve the label fusion accuracy by dynamically changing the patch size.