Consistent Multi-Atlas Hippocampus Segmentation for Longitudinal MR Brain Images with Temporal Sparse Representation

Abstract

In this paper, we propose a novel multi-atlas based longitudinal label fusion method with temporal sparse representation technique to segment hippocampi at all time points simultaneously. First, we use groupwise longitudinal registration to simultaneously (1) estimate a group-mean image of a subject image sequence and (2) register its all time-point images to the estimated group-mean image consistently over time. Then, by registering all atlases with the group-mean image, we can align all atlases longitudinally consistently to each time point of the subject image sequence. Finally, we propose a longitudinal label fusion method to propagate all atlas labels to the subject image sequence by simultaneously labeling a set of temporally-corresponded voxels with a temporal consistency constraint on sparse representation. Experimental results demonstrate that our proposed method can achieve more accurate and consistent hippocampus segmentation than the state-of-the-art counterpart methods.

1 Introduction

The hippocampus plays a crucial role in memory and spatial navigation function of brain [1]. The structural change of hippocampus over time is highly related to many neurodegenerative diseases, such as Alzheimer's disease (AD). As a characteristic feature of AD, hippocampal atrophy is considered as a potential biomarker for the diagnosis and assessment of AD in magnetic resonance (MR) imaging based neuro-science studies [2, 3]. In order to measure the hippocampal atrophy over time, accurate quantization of hippocampal volumes from serial structural three-dimensional (3D) MR images is required. To this end, it is important to accurately and temporally consistently segment four-dimensional (3D+t) hippocampus from longitudinal structural MR images.

Many automatic segmentation methods have been proposed to segment 3D hippocampus independently from MR images of different time points [4–8]. In this case, hippocampus segmentation from longitudinal MR images is decomposed into a series of separate hippocampus segmentation from each 3D MR image. However, due to various reasons such as variant noises and hippocampal tissue contrast in the acquired longitudinal MR images, 3D segmentation methods, when applied to longitudinal MR images, have limited temporal consistency for segmented longitudinal hippocampi. To improve segmentation consistency from longitudinal MR images, several longitudinal segmentation methods have been proposed [9–12]. Wolz et al. [9] proposed a 4D graph-cut based method to simultaneously segment longitudinal MR images. By using a 4D graph to represent longitudinal MR data, the method segmented MR images at all time points by solving the min-cut/max-flow problem on the 4D graph. Chincarini et al. [12] presented a hippocampal segmentation method by integrating longitudinal information. They implemented longitudinal analysis with four progressive steps, and addressed the impact of these steps on longitudinal performance of hippocampal volume measurements for early detection of AD. However, due to large variance of noise level and intensity bias field across different time points in the longitudinal MR images, consistent hippocampus segmentation of serial MR images remains a challenging problem.

Accordingly, in this paper, we propose a 3D+t hippocampus segmentation method for longitudinal MR brain images, by integrating temporal sparse representation within the multi-atlas patch-based label fusion framework. First, we use the groupwise longitudinal image registration toolbox (GLIRT) [13] to simultaneously (1) estimate a subject-specific group-mean image and (2) register all time-point images of subject image sequence consistently to the estimated group-mean image. Then, by registering all atlas images to the estimated group-mean image, we can align all atlases longitudinally consistently to each time point of the subject image sequence. Thus, given the temporal correspondence in subject image sequence, we can form a 3D+t image patch at each location of subject brain. Then, we can use temporal sparse representation technique to simultaneously determine labels for all time points of the subject image sequence by propagating labels from all aligned atlas images. Experimental results on both simulated and real longitudinal MR images demonstrate that our proposed method can achieve more accurate and consistent hippocampus segmentation than the state-of-the-art multi-atlas label fusion methods, which often apply label fusion for each time point independently.

2 Method

Before describing our method in detail, we first introduce some mathematical descriptions. First, a longitudinal subject image sequence (also namely 3D+t subject image) is denoted as ${\{{\mathit{T}\text{'}}_t\}}_{t=1}^N$, where T′_t is a 3D image at time point t(t ∈ {1, …, N}). Then, M 3D atlas images are denoted as I′⁽¹⁾, …, I′^(M) with their corresponding hippocampal label images denoted as L′⁽¹⁾, …,L′^(M). So, our goal is to segment 3D+t subject image, i.e., to automatically estimate a hippocampal label image sequence ${\{{\mathit{L}\text{'}}_t\}}_{t=1}^N$ corresponding to the 3D+t subject image ${\{{\mathit{T}\text{'}}_t\}}_{t=1}^N$ as illustrated in Fig. 1, where L′_t denotes a 3D subject label image at time point t(t ∈ {1, …, N}).

Fig. 1.

Schematic diagram of the proposed 3D+t hippocampus segmentation method.

2.1 Temporal Sparse Representation

Unbiased Groupwise Registration

The first step in our method is to estimate temporal correspondences along different time points of the subject image sequence. Here, we use GLIRT [13] to achieve this goal. Unlike other pairwise registration methods, which need choose a reference template, the groupwise registration is free of template selection, and can thus build an unbiased subject-specific group-mean image and simultaneously register all time-point images to this group-mean image. Assuming ${\{{\phi \text{'}}_t\}}_{t=1}^N$ as the deformation fields for N time points, the original 3D+t subject image ${\{{\mathit{T}\text{'}}_t\}}_{t=1}^N$ can be transformed to the group-mean image space as ${\{{\mathit{T}\text{'}}_t\}}_{t=1}^N$, which is illustrated in Fig. 1. Here, ${\mathit{T}}_t={\phi \text{'}}_t({\mathit{T}\text{'}}_t).$.

Furthermore, all 3D atlas images are registered to the estimated group-mean image, first by affine registration [14] with 12 degrees of freedom and then by deformable registration [15]. By following the same estimated affine transformation matrix and deformation field, the corresponding 3D hippocampal label image of each atlas can be also registered to the estimated group-mean image space. As shown in Fig. 1, I⁽¹⁾, …, I^(M) and L⁽¹⁾, …, L^(M) denote the M aligned 3D atlas images and their corresponding aligned 3D label images in the estimated group-mean image space, respectively.

Temporal Sparse Patch-Based Representation

For each voxel x in the group-mean image, its 3D+t subject patch can be extracted as ${\{{\mathit{\alpha }}_{(\mathit{x},t)}\}}_{t=1}^N$, where α_(x,t) represents a 3D patch centered at voxel x of the aligned 3D+t subject image at time point t. Let n(x) denote a spatial neighborhood of x. All candidate atlas patches within the search neighborhood n(x) of the aligned atlas images ${\{{\mathit{I}}^{(m)}\}}_{m=1}^M$ are denoted as { ${\mathit{\beta }}_{\mathit{z}}^{(m)}|\mathit{z}\in \mathit{n}(\mathit{x})$, m = 1, …, M}, along with their corresponding center voxel labels { ${\mathit{l}}_{\mathit{z}}^{(m)}|\mathit{z}\in \mathit{n}(\mathit{x})$, m = 1, …, M}. The total number of atlas patches { ${\mathit{\beta }}_{\mathit{z}}^{(m)}|\mathit{z}\in \mathit{n}(\mathit{x})$, m = 1, …, M} used to label the 3D+t subject patch ${\{{\mathit{\alpha }}_{(\mathit{x},t)}\}}_{t=1}^N$ is Q = M × |n(x)|, where |n(x)| denotes the cardinality of n(x).

After rearranging ${\mathit{\beta }}_{\mathit{z}}^{(m)}$ and α_(x,t) into a d-dimensional column vector ${\mathit{b}}_{\mathit{z}}^{(m)}$ and a_(x,t), respectively, where d is the number of voxels in each 3D patch, our temporal sparse representation can be formulated as a problem of finding optimal sparse representation for the 3D+t subject patch vector ${\{{\mathit{a}}_{(\mathit{x},t)}\}}_{t=1}^N$ by using all atlas patch vectors ${\{{\mathit{b}}_{\mathit{z}}^{(m)}\}}_{m=1}^M$ as follows [6]:

$${\{{\widehat{\mathit{w}}}_{(\mathit{x},t)}\}}_{t=1}^N=\underset{{\{{\mathit{w}}_{(\mathit{x},t)}\}}_{t=1}^N}{arg\; min}\{\frac{1}{2}{\sum }_{t=1}^N{\Vert {\mathit{B}}_{\mathit{x}}{\mathit{w}}_{(\mathit{x},t)}-{\mathit{a}}_{(\mathit{x},t)}\Vert }_2^2+{\lambda }_1{\sum }_{t=1}^N{\Vert {\mathit{w}}_{(\mathit{x},t)}\Vert }_1+{\lambda }_2{\sum }_{t=1}^{N-1}{\Vert {\mathit{w}}_{(\mathit{x},t)}-{\mathit{w}}_{(\mathit{x},t+1)}\Vert }_1\}s.t.w_{(\mathit{x},t)}\ge 0,\forall t=1,\dots ,N$$ (1)

where B_x ∈ ℛ^d×Q denotes a dictionary matrix constructed by arranging ${\{{\mathit{b}}_{\mathit{z}}^{(m)}\}}_{m=1}^M$ column by column, w_(x,t) ∈ ℛ^Q denotes a weight vector by arranging all non-negative weights { $w_{(\mathit{x},t),\mathit{z}}^{(m)}|w_{(\mathit{x},t),\mathit{z}}^{(m)}\ge 0$, z ∈ n(x), m = 1, …, M} into a column vector, and $w_{(\mathit{x},t),\mathit{z}}^{(m)}$ is the representation coefficient of the patch vector ${\mathit{b}}_{\mathit{z}}^{(m)}$ of the m-th atlas image in constructing the patch vector a_(x,t) of the subject image at time point t. The first term in Eq. (1) is the reconstruction discrepancy. The second term, which is equivalent to the ℓ₁-norm, enforces sparsity in w_(x,t). The third term is the temporal fused smoothness term, used to constrain the temporal consistency of two successive sparse representation vectors (w_(x,t) and w_(x,t+1)).λ₁ and λ₂ are the two weighting parameters used to balance the contributions from the second and third terms. The objective function in Eq. (1) can be solved by the fast proximal gradient method [16].

2.2 Multi-Atlas Based Label Fusion with Temporal Sparse Representation

Once the temporal sparse code ${\{{\widehat{\mathit{w}}}_{(\mathit{x},t)}\}}_{t=1}^N$ is estimated by solving the optimization problem in Eq. (1), the label at the voxel (x, t) of the aligned subject image T_t can be obtained by the multi-atlas based label fusion method by combining the center voxel labels { $l_{\mathit{z}}^{(m)}|\mathit{z}\in \mathit{n}(\mathit{x})$, m = 1, …, M} using the estimated temporal sparse code ${\{{\widehat{\mathit{w}}}_{(\mathit{x},t)}\}}_{t=1}^N$.

By following the same order of the dictionary matrix B_x, { $l_{\mathit{z}}^{(m)}|\mathit{z}\in \mathit{n}(\mathit{x})$, m = 1, …, M} is constructed as a label vector l_x(l_x ∈ ℛ^1×Q). Supposing there are P possible labels {L₁, …, L_p, …,L_P} in the atlases, the label at the voxel (x, t) of the aligned subject image T_t can be determined by:

$$\begin{array}{cc}{\widehat{L}}_{(\mathit{x},t)}=arg{min}_{L_p,p=1,\dots ,P}\{{\sum }_{j=1}^Q{\widehat{w}}_{(\mathit{x},t),j}\cdot \delta (l_{\mathit{x},j},L_p)\},& \forall t=1,\dots ,N\end{array}$$ (2)

where ŵ_(x,t),j and l_x,j are the j-th components of ŵ_(x,t) and l_x, respectively, and the function δ{l_x,j, L_p) is equal to 1 if l_x,j = L_p and 0 otherwise.

After determining the label L̂_(x,t) at each voxel (x, t) of aligned 3D+t subject image ${\{{\mathit{T}}_t\}}_{t=1}^N$, the aligned 3D+t subject label image ${\{{\mathit{L}}_t\}}_{t=1}^N$ in the group-mean image space can be obtained. Then, we can obtain the 3D+t subject label image ${\{{\mathit{L}\text{'}}_t\}}_{t=1}^N$ corresponding to the input 3D+t subject image ${\{{\mathit{T}\text{'}}_t\}}_{t=1}^N$ by transforming ${\{{\mathit{L}}_t\}}_{t=1}^N$ back to the subject image space by following ${\mathit{L}\text{'}}_t={\phi }_t^{\text{'}-1}({\mathit{L}}_t)$, where ${\phi }_t^{\text{'}-1}$ is the inverse transformation field of ${\phi }_t^{\text{'}}$.

3 Experimental Results

In this section, we evaluate our proposed longitudinal (3D+t) hippocampus segmentation method on both simulated and real longitudinal MR brain image datasets. Specifically, 10 subjects with simulated atrophy in hippocampi, and 12 subjects with each subject having three MR images acquired at three time points in the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (http://www.adni-info.org/) are used, respectively. All the images in both datasets are the T1-weighted MR images, which are processed to have the same size and same resolution of 256 × 256 × 256 and 1 × 1 × 1 mm³, respectively. The proposed method is compared to two state-of-the-art label fusion methods, namely nonlocal patch-based method (Non-local) [5] and sparse patch-based method (SPBM) [6]. To be fair, a similar unbiased groupwise registration is applied to the two comparison methods. That is, subject image is registered to the estimated group-mean image of subject image sequence by GLIRT, and all atlases are registered to the estimated group-mean image sequentially by affine registration and deformable registration. The final segmentations are evaluated in each subject's own space. In the following experiments, the patch size and search neighborhood size are both set to 3 × 3 × 3, and the regularization coefficients λ₁ and λ₂ are both set to 0.001.

3.1 Experiments on Simulated Dataset

In the simulation experiments, 10 subjects with manual hippocampal labels at year 1 are used as the baseline data (t = 1), and then used to simulate atrophy on the hippocampi. Each subject is simulated with three longitudinal MR images, where images at year 2 (t = 2) and year 3 (t = 3) are generated by an atrophy simulation model [17] to ensure shrinking hippocampal volumes along the temporal dimension. Thus, ten sets of simulated longitudinal data with about 5% of annual hippocampal volume shrinking are obtained. The total simulated atrophy rate of hippocampus in three years is 9:15%.

A leave-one-out strategy on the total 10 simulated subjects is adopted to compare the segmentation performances of Non-local, SPBM, and our proposed method. Specifically, in each leave-one-out experiment, one subject is selected as 3D+t subject image, and the rest 9 subjects are selected as 3D atlases (27 atlases in total). For both Non-local and SPBM methods, three time-point images in the 3D+t subject image are segmented independently. The mean and standard deviation of Dice ratios are shown in Table 1. We can observe that our proposed method receives significant improvement over both Non-local and SPBM methods in terms of Dice ratio according to the paired t-test (p < 0:05). Thus, our proposed method achieves the best segmentation accuracy.

Table 1.

Mean and standard deviation of Dice ratio in hippocampus segmentation on simulated atrophy data (Unit: %).

Method	Left hippo	Right hippo	Overall
Non-local	81.27 ± 1.98	80.08 ± 2.82	80.69 ± 2.12
SPBM	80.80 ± 1.71	79.55 ± 2.43	80.19 ± 1.76
Proposed	82.71 ± 1.51*	81.86 ± 2.44*	82.30 ± 1.76*

^*Indicates significant improvement over Non-local and SPBM methods (p < 0.05)

Figure 2(a) shows the curves of the longitudinal loss of overall hippocampus volume in a typical subject by Non-local, SPBM, and our method. Figure 2(b) shows the averaged loss of overall hippocampus volume estimated by these three methods. The final estimated mean and standard deviation of loss of overall hippocampus volume is 11:84% ± 1:63% by Non-local, 12:31% ± 1:57% by SPBM, and 10:87% ± 1:21% by the proposed method. We can see that our proposed method is the closest to the ground truth, and also the most consistent in measuring longitudinal hippocampal volume changes, due to the use of temporal consistency constraint on sparse representation of multi-atlas based label fusion.

Fig. 2.

Demonstration of loss of overall hippocampus volume through 3 time points. (a) Loss of longitudinal hippocampus volume for a typical subject, and (b) average loss of longitudinal hippocampus volumes for all subjects.

3.2 Experiments on Real Dataset

In the real experiments, we randomly select 12 subjects with each having 3 time points (baseline, 6 and 12 months) from ADNI dataset. The hippocampi of these MR images have been manually labeled, which are regarded as ground truth.

We also adopt a leave-one-out strategy on the total 12 real subjects for experiments of hippocampus segmentation. The mean and standard deviation of Dice ratios and also the average symmetric surface distance (ASSD) of the hippocampus segmentation results by Non-local, SPBM, and our proposed method are shown in Table 2. We can observe that our proposed method achieves significant improvement over Non-local and SPBM methods in terms of both Dice ratio and ASSD according to the pair t-test (p < 0.05). Figure 3 shows the surface distances of the left hippocampus for a typical subject, between manual segmentations and automatic segmentations by Non-local, SPBM, and our proposed method. It is obvious that our proposed 3D+t hippocampus segmentation method achieves the best segmentation performance.

Table 2.

Mean and standard deviation of Dice ratios and the average symmetric surface distance (ASSD) for automatic segmentations by Non-local, SPBM, and our proposed method.

	Method	Left hippo	Right hippo	Overall
Dice ratio (%)	Non-local	83.53 ± 2.99	82.37 ± 4.02	82.93 ± 2.97
	SPBM	82.32 ± 3.83	82.32 ± 2.64	82.31 ± 2.87
	Proposed	84.85 ± 2.30*	84.29 ± 2.34*	84.55 ± 2.22*
ASSD (mm)	Non-local	0.497 ± 0.072	0.524 ± 0.098	0.512 ± 0.067
	SPBM	0.520 ± 0.096	0.509 ± 0.057	0.515 ± 0.066
	Proposed	0.455 ± 0.049*	0.476 ± 0.063*	0.466 ± 0.053*

^*Indicates significant improvement over Non-local and SPBM methods (p < 0.05)

Fig. 3.

Visualization of surface distances (in mm) for hippocampus segmentation results by three methods.

4 Conclusion

In this paper, we proposed an integrated temporal sparse representation and multi-atlas patch-based label fusion method for longitudinal (3D+t) hippocampus segmentation in the longitudinal MR images. To make the registration at different time points consistent to the subsequent data analysis, we registered the 3D+t subject image and all atlases to the group-mean image of 3D+t subject image by using GLIRT. Moreover, to respect the smooth change of longitudinal structure (i.e., hippocampus), we added a temporal fused smoothness term to the objective function of sparse representation, for enforcing small difference between two successive sparse representation vectors from adjacent time points. Experimental results demonstrated the improved segmentation accuracy and longitudinal consistency by our proposed method, compared to both Non-local and SPBM methods.