Multi-Object Model-based Multi-Atlas Segmentation for Rodent Brains using Dense Discrete Correspondences

Abstract

The delineation of rodent brain structures is challenging due to low-contrast multiple cortical and subcortical organs that are closely interfacing to each other. Atlas-based segmentation has been widely employed due to its ability to delineate multiple organs at the same time via image registration. The use of multiple atlases and subsequent label fusion techniques has further improved the robustness and accuracy of atlas-based segmentation. However, the accuracy of atlas-based segmentation is still prone to registration errors; for example, the segmentation of in vivo MR images can be less accurate and robust against image artifacts than the segmentation of post mortem images. In order to improve the accuracy and robustness of atlas-based segmentation, we propose a multi-object, model-based, multi-atlas segmentation method. We first establish spatial correspondences across atlases using a set of dense pseudo-landmark particles. We build a multi-object point distribution model using those particles in order to capture inter- and intra-subject variation among brain structures. The segmentation is obtained by fitting the model into a subject image, followed by label fusion process. Our result shows that the proposed method resulted in greater accuracy than comparable segmentation methods, including a widely used ANTs registration tool.

1. INTRODUCTION

Atlas-based segmentation has advantages over model-based segmentation in that a pre-existing label map can be directly propagated to obtain a new segmentation label via the result of image registration. The underlying image registration process computes the voxel-wise correspondence between the atlas and the target image based on local intensity similarities. However, the dependency on the underlying image registration process can limit the performance of atlas-based segmentation as well, for example, producing less accurate segmentation due to registration errors. The cause of registration errors vary including low-contrast boundaries, inaccurate initial alignment, and biases introduced by the atlas used. In order to improve accuracy and robustness of atlas-based segmentation and avoid such concerns, several methods have been proposed including group-wise image registration and multi-atlas based segmentation.

A recently proposed non-rigid group-wise image registration method[7] employed the set of freely moving particles as a sense set of landmarks to drive B-spline based transformations. Inspired by surface-based correspondence algorithm[4], the method minimized an information theoretic objective function[14] in which each particle is forced to move towards locally similar area, yet the ensemble entropy of particles is optimized to capture the entire regions of interest. Later, the group-wise image registration method was extended to multi-label atlas-based segmentation that performed separate registration for each structural label presented in the atlas[8]. Because separate registration for each region improves the initial alignment per region, their method showed higher volume overlap ratio than the group-wise registration based segmentation method. However, the separate registration increased computational cost that was repeated per region.

We propose to combine model-based segmentation into the multi-atlas segmentation framework based on the group-wise image registration method. The proposed model-based multi-atlas segmentation method takes a hybrid approach that combines multi-object model-based segmentation along with multi-atlas segmentation approach. Based on the particle-guided group-wise registration, we employ multilevel component analysis (MCA) in order to capture the variation of particles among subjects. MCA proposed by [15] is an extension of principal component analysis for hierarchical recursive structures and is used in order to capture between-group variation and within-group variation separately using principal vectors and their weights. In the proposed multi-object model, the displacement and orientation of an object are captured by its bounding box using eight points, and its shape is described by the particles inside the box.

This paper is organized as follows. First, we review the particle-guided image registration as background and its extension to multi-label segmentation. MCA is then introduced followed by the description of the model construction for the registration. The final segmentation is created by label fusion of multiple intensity patches at corresponding location via each particle. The result is evaluated in terms of volume overlap ratio and surface distance measures with respect to the group-wise registration parameters such as the number of particles and the number of atlases.

2. METHOD

The proposed segmentation method develops a key idea that is based mainly on the particle representation utilized in the previously proposed group-wise image registration method[8]. A particle-based group-wise correspondence method allows the construction of a statistical multi-object model based on the set of discrete particles. The proposed segmentation method performs segmentation by fitting the model into a subject image followed by a label fusion process.

2.1 Multi-object Structure in Rodent Brain

Considering that a rodent brain consists of identifiable multiple structures that include, for example, the neocortex, the fimbria, the hippocampus, or the thalamus, we expect that those structures coincide across images when appropriate correspondence is computed. In terms of corresponding particles, if a particle p_i of subject i is located within the region of structure A in one image, the corresponding particle p_j of subject j should be located within the region of A as well. In the original group-wise registration, the initial particles were sampled from the entire image without constraints. However, if an atlas contains the information of the structures of interest in the form of a label map, we can preassign a membership to structures for each particle by sampling initial particles separately for each structure that is found in the atlas. The rodent brain actually consists of several smaller sub-structures, each of which may have its unique variation as well as variation that are correlated with those of other structures. Therefore, we aggregated smaller structures into larger four regions as shown in Figure 1.

Figure 1.

The multi-level structure of the statistical shape and appearance model. (a) An image is interpreted as the aggregation of multiple objects, and each object may have multiple sub-structures. Based on multilevel component analysis, the idea is to decompose the data into a within-individual and a between-individual component. (b) The rodent brain consists of four large structures. Since these structures are anatomically loosely connected as separate structures, their variation may be better captured when they are described as separate between-individual geometric objects.

Introducing the structural membership for each particle, a new constraint is imposed while optimizing particles to ensure that particles remain within the expected structural regions. In other words, the shape of a region must be maintained in a proper form during the optimization. It is very challenging to solely control the boundary via particle interaction terms. Thus, in order to solve this problem, we employ the use of a multi-object model that is based on multilevel component analysis (MCA), which was proposed by Timmerman[15] first. MCA is an extension of principal component analysis for hierarchical recursive structures and is used in order to capture between-group variation and within-group variation separately using principal vectors and their weights. In this model, the displacement and orientation of an object are captured by its bounding box using eight points, and its shape is described by the particles inside the box.

2.2 Multilevel Component Analysis

The main idea of MCA is to decompose the data into a within-group and a between-group component, which is analogous to the concept of analysis of variance (ANOVA). In one-way ANOVA, the fundamental technique is a partitioning of the total sum of squares (SS) into components that are related to the effects used in the model [5]. For example, the model for simplified ANOVA with one type of treatment at different treatment levels is expressed as equation (1):

$$S{S}_{Total}=S{S}_{Error}+S{S}_{Treatments}.$$ (1)

Suppose that a data sample, for example a point coordinate component, is represented as x_ijk , where i, j, and k denote the index of an item, a variable, and a group respectively, can be decomposed into a within-group and a between-group term, such as

$$x_{ijk}={\stackrel{‒}{x}}_j+({\stackrel{‒}{x}}_{jk}-{\stackrel{‒}{x}}_j)+(x_{ijk}-{\stackrel{‒}{x}}_{jk}),$$ (2)

where ${\stackrel{‒}{x}}_j$ is the grand mean $\frac{1}{N}{\sum }_{k=1}^K{\sum }_{i=1}^{K_k}{x}_{ijk}\text{and}{\stackrel{‒}{x}}_{jk}$ is a within-group mean $\frac{1}{K_k}{\sum }_{i=1}^{K_k}{x}_{ijk}$. In Equation (2), $({\stackrel{‒}{x}}_{jk}-{\stackrel{‒}{x}}_j)$ is the residual of the group k from the grand mean, and $(x_{ijk}-{\stackrel{‒}{x}}_{jk})$ is the residual of the item i from the group k.

Let X_k be a K_k × J data matrix belonging to the group k where K_k is the number of data items in the group k. Analogous to PCA, which models a data using the linear combination of principal vectors, the multi-level model X_k implies:

$$X_k=1_{K_k}{m}^T+1_{K_k}{d}_B^T{\Phi }_B^T+D_{w,k}{\Phi }_W^T+E_k,$$ (2)

where m is a J × 1 vector that represents the overall mean; 1^Kk denotes a K_k × 1 vector with each element equal to one; d_B,k stands for a R_B × 1 vector containing the R_B between the component scores of group k; Φ_B represents J × R_B matrix of the between-group principal components, D_W,k is a K_k × R_W matrix of the R_W within the component scores of group k, Φ_W denotes a J × R_W matrix of the within principal components for the group k and, finally, where E_k is a K_k × J matrix of residuals for the group k.

2.3 MCA Model Construction

MCA allows an object to be represented as

$${\mathbf{A}}_i=\stackrel{‒}{A}+{\varphi }_W{d}_{W,i}+{\varphi }_B{d}_{B,i},$$ (4)

where A_i is a vector of an object representation, Φ_W and Φ_B are the principal vectors for within-group and between-group variations. Suppose that I number of subjects, each contains K number of groups, which will describe an individual substructure (e.g., the hippocampus or the neocortex).

This multilevel representation addresses here the affine variation of structures using a bounding box representation, which represents the orientation and anisotropic scales of the structure with eight-point coordinates. Figure 2 on the next page shows a two-dimensional example for the shape model constructed in the proposed method. Thus, in the segmentation method proposed here, the within-group variation Φ_W captures the variations of the bounding box representations of object k, and the variations of the bounding box representation of objects, Φ_B, contains information about the bounding box variations among the mean bounding boxes of the objects.

Figure 2.

The proposed multi-object shape model using bounding boxes. Subject samples are illustrated using two-dimensional bounding boxes as examples. A rodent brain is decomposed into four objects: the olfactory bulb, the cortex and subcortical structures, the cerebellum, and the brain stem. The within-group variation explains the deformation of the olfactory bulb shown in the bottom row. The right column shows the between-group variation that captures the relationship between subcomponents with respect to the first principal mode. The shape of a valid object is modeled using the linear combination of two-level principal modes of variation.

Suppose that an array A of the size IK × 24 represents all the objects in a training set. The principal modes of variation Φ_B and Φ_W are obtained by performing PCA of particular decompositions of A. First, the within-group parameters are computed over all the mean-centered sub-matrices A_k of size I × J , where A_k is the partition of the matrix A belonging to group k. This decomposition is given by: ${\mathbf{A}}_{c,k}={\mathbf{A}}_k-1_I{m}_k^T$, where m_k is a J × 1 vector that describes the mean induced by the matrix X_k over the bounding box variables, which is 24 real numbers here. Let A_c be the matrix that results from the vertical concatenation of matrices A_c,k . The principal vectors Φ_W is composed of the eigen-vectors of the covariance matrix that is related to A_c. The variance of the weights d_W,k , which limits the deformation of the within-group model is determined by the eigen-values of the covariance matrix that are related to A_c.

The between-group parameters Φ_B and d_B,k are computed in a similar manner. First, subtract the grand mean from X and let $\stackrel{‒}{X}$ be such a matrix. Denoting ${\stackrel{‒}{X}}_k$, which is the partition of the matrix $\stackrel{‒}{X}$ that belongs to group k, consider the K vectors ${\stackrel{‒}{m}}_k$, each of which represents the mean of the associated matrix ${\stackrel{‒}{X}}_k$. These vectors characterize the between-group differences. Therefore, the matrix that results from the vertical concatenation of the vectors ${\stackrel{‒}{m}}_k$ allows the matrix Φ_B of the between-group principal components to be retrieved.

Once the model is constructed, the model can be fitted into the subject image to be segmented, followed by label fusion to generate the segmentation result.

2.3.1 The Model Fitting

Assume a set of arrays of particles A = {A₁, A₂, … , A_Ns }, where A_i is a set of particles {p₁, p₂, … , p_Np} that describes the shape of a rodent brain. The statistical shape model of A includes the array of averaged particles Ā and the principal modes of variation Φ_B and Φ_W and allows A_i(∈ A) to be represented as the linear combination of Φ_B and Φ_K such that

$$A_i=\stackrel{‒}{A}+{\Phi }_B{d}_B+{\Phi }_W{d}_W,$$ (5)

where d_B and d_W are the coefficients of the principal vectors. The segmentation of a subject image I is performed by fitting a model instance A′ into I by optimizing a cost function that measures the goodness of fit between A′ and I. The model instance A′ is created initially by the duplication of $\stackrel{‒}{A}$ and then is placed into the region of interest Ω ⊂ I, which is the region of the entire object, using the initial alignment process. This initial alignment is important in order to avoid a suboptimal local minimum during the optimization.

In order to compute the goodness of fit, the local intensity patch of the image I near particle p_i is compared to local intensity patches of atlas images I at corresponding positions. The patch-based appearance similarity between p ∈ A′ and p_j ∈ A_j is computed following as

$$w_j(\text{p},{\text{p}}_j)=f\left(\frac{\sum _{\text{x}\in P_I\left(\mathbf{p}\right),\mathbf{y}\in P_{I_i}\left({\mathbf{p}}_j\right)}{(I\left(\mathbf{x}\right)-I\left(\mathbf{y}\right))}^2}{2N\beta {\widehat{\sigma }}^2}\right).$$ (6)

Using w_j , the cost function with parameters of d_B and d_W is written as expressed in equation (7):

$$F(d_B,d_W)=\sum _{{\mathbf{p}}_i\in A^{\prime }}\frac{\sum _{j=1}^{N_a}\parallel \mathbf{p}-{\mathbf{p}}_j{\parallel }_{2}w(\mathbf{p},{\mathbf{p}}_j)}{\sum _{j=1}^{N_a}\parallel \mathbf{p}-{\mathbf{p}}_j{\parallel }_2}.$$ (7)

The goodness of fit function F can be optimized using a gradient descent optimizer or a non-differential optimizer such as BOBYQA (Bound Optimization BY Quadratic Approximation), which is a numerical optimization algorithm by Powell[11].

The fitting process ensures that the particles in A′ can be located at corresponding positions across all training atlases. The final label L of the subject is determined by the non-local patch-based segmentation method by Rousseau[12] such that

$$\forall {\mathbf{x}}_i{\mathbf{x}}_j\in \Omega ,L\left(\text{x}\right)=\frac{\sum _{j=1}^{N_a}\sum _{{\mathbf{x}}_j\in T\left({\mathbf{x}}_j\right)}\rho ({\mathbf{x}}_j,{\mathbf{p}}_j)\sum _{\mathbf{y}\in N\left({\mathbf{x}}_j\right)}{w}_j(\mathbf{x},\mathbf{y}){\mathcal{ℒ}}_j\left(\mathbf{y}\right)}{\sum _{j=1}^{N_s}\sum _{{\mathbf{x}}_j\in T\left({\mathbf{x}}_j\right)}\rho ({\mathbf{x}}_j,{\mathbf{p}}_j)\sum _{y\in N\left({\mathbf{x}}_j\right)}{w}_j(\mathbf{x},\mathbf{y})},$$ (8)

where N (x) is the neighborhood of y and x_j is a corresponding location at subject j estimated using particles p_i, which are k closest to x_i, and p_i’s corresponding particles p_j . Since this process leads to fuzzy labeling at x, the indicator function for label k at x can be used to determine a hard labeling by taking the maximum of each vector L(x) as a threshold such that

$$h_k\left(\mathbf{x}\right)=\{\begin{array}{cc}1,\hfill & \text{if}{l}_k\left(\mathbf{x}\right)=\parallel L\left(\mathbf{x}\right){\parallel }_{\infty }\hfill \\ 0,\hfill & \text{otherwises}\hfill \end{array}\phantom{\}}$$ (9)

3. RESULTS

The performance of the proposed segmentation method was evaluated in comparison to manually created brain structure labels (ground truth), the Dice coefficient (DSC) and the average surface distance using various experimental settings. For the dataset, the Brookhaven atlas database [9] was used in this study. The Brookhaven atlas database is publicly available and frequently cited in rodent brain segmentation papers [2, 7, 10]. The Brookhaven atlas consists of 10 adult male C57BL/6J mouse brain images derived from T2*-weighted 3D MRM images acquired on a 17.6T magnet. With 20 segmented structures, the C57BL/6J 3D digital brain atlas database offers individual brain atlases produced by single atlas-based segmentation followed by manual corrections.

3.1 Evaluation of Segmentation Accuracy

The proposed registration method was compared with other four non-rigid registration methods. First, the stack-entropy based group-wise image registration method reported in [3] was used. This method employs a similar entropy-based similarity metric as well as B-spline transformation so that the effects of particle-based registration can be directly compared. In order to compare the performance with a pair-wise image registration method, the B-spline based non-rigid registration method packaged in BRAINSFit [6] and the original free form deformation non-rigid registration reported in [13], which are the most popular B-spline transformation based image registrations, were used. In addition to these, Advanced Normalization Tools [1], which is practically the most widely used diffeomorphic image registration software, was compared to the aforementioned procedures utilized.

Before the registration experiment was performed, each subject was first preprocessed with the N3 intensity bias correction tool [16] in order to correct inhomogeneous intensity bias across an MR image and followed by the histogram matching and intensity rescaling processes across all subjects. The histogram matching normalizes intensity unit and therefore allows direct intensity comparison across subject MR images. Following the intensity normalization pre-processing, each subject image was then aligned towards a chosen template with the second intensity momentum and subsequently by the rigid registration process using the squared sum of difference metric. All the aligned subjects were manually quality assured to prevent potential initial alignment errors.

Table 1 on the next page shows the raw values for all the methods’ results in terms of each measurement and each region of interest. Figure 3 on page 8 shows bar graphs that compare the DSC ratio and the average surface distance with respect to the regions of interests. In (a), the proposed method exhibits taller peaks than most of the other methods as well as in most of the regions of interest. The ANTs tool exhibits the second tallest peaks, except for Anterior Commissure. Although the non-local patch-based method does not involve any non-rigid transformation, the method performs better than the pair-wise B-spline registration method, which can make non-local patch-based segmentation methods promising for computational efficiency.

Table 1.

The average distance, the maximum distance, and the DSC were computed for each region with respect to the three compared methods: pair-wise B-spline registration-based segmentation with majority voting, ANTS tool registration with majority voting, and non-local patch-based segmentation method.

	Pair-wise B-spline with Majority Voting			ANTS with Majority Voting
	Avg. Dist.	Max Dist.	DSC	Avg. Dist.	Max Dist.	DSC
Hippocampus	0.055	0.507	82.307	0.033	0.447	88.680
External1Capsule(Right1EC)	0.053	2.093	54.440	0.036	1.237	66.370
Caudate,1Putamen,1Globus1Pallidus	0.057	0.346	81.438	0.038	0.245	87.060
Anterior1Commissure	0.103	1.159	36.449	0.080	1.416	57.308
Substantia1Nigra	0.052	0.289	70.413	0.027	0.173	82.900
Internal1Capsule	0.047	0.315	59.986	0.027	0.566	74.370
Thalamus	0.045	0.234	89.556	0.027	0.200	93.310
Cerebellum	0.091	0.805	82.944	0.085	0.583	84.076
Superior1Colliculus	0.055	0.363	81.926	0.037	0.316	87.290
Third1Ventricle	0.069	0.959	51.771	0.072	1.616	60.440
Hypothalamus	0.050	0.297	84.638	0.033	0.223	89.430
Inferior1Colliculus	0.070	0.353	70.409	0.050	0.300	77.790
Central1Gray	0.046	0.228	81.153	0.030	0.141	87.130
Neocortex	0.074	0.662	81.199	0.052	0.424	86.460
Amygdala	0.083	0.490	71.058	0.073	1.063	73.888
Olfactory1Bulb	0.115	8.928	77.410	0.099	8.930	79.060
Brainstem	0.140	1.482	81.441	0.164	2.385	81.320
Rest1of1Midbrain	0.061	0.680	74.637	0.051	0.616	78.940
Rest1of1Forebrain	0.053	0.370	78.776	0.039	0.245	84.160
Fimbria	0.040	0.293	64.622	0.021	0.583	79.620
(a) registration-based segmentation
	Proposed Method			NonBlocal PatchBbased Segmentation
	Avg. Dist.	Max Dist.	DSC	Avg. Dist.	Max Dist.	DSC
Hippocampus	0.033	0.447	88.790	0.044	0.574	87.780
External1Capsule(Right1EC)	0.028	0.849	69.760	0.039	2.706	63.439
Caudate,1Putamen,1Globus1Pallidus	0.036	0.245	88.000	0.046	0.300	83.043
Anterior1Commissure	0.065	1.086	47.810	0.105	1.757	34.210
Substantia1Nigra	0.028	0.141	82.570	0.036	0.198	75.275
Internal1Capsule	0.024	0.224	76.750	0.031	0.350	65.921
Thalamus	0.025	0.200	93.810	0.040	0.208	90.435
Cerebellum	0.060	0.436	88.030	0.086	0.900	82.025
Superior1Colliculus	0.038	0.322	86.910	0.064	0.398	82.510
Third1Ventricle	0.027	0.735	70.680	0.044	0.898	62.102
Hypothalamus	0.034	0.224	89.250	0.049	0.249	83.219
Inferior1Colliculus	0.053	0.283	76.860	0.065	0.321	73.897
Central1Gray	0.033	0.173	85.900	0.041	0.213	81.191
Neocortex	0.054	0.412	86.100	0.069	0.809	84.912
Amygdala	0.059	0.374	78.400	0.084	0.444	72.439
Olfactory1Bulb	0.091	8.940	82.330	0.103	8.902	81.561
Brainstem	0.070	0.762	88.010	0.149	2.446	78.449
Rest1of1Midbrain	0.050	0.656	79.170	0.055	0.801	76.701
Rest1of1Forebrain	0.033	0.310	86.280	0.056	0.356	83.767
Fimbria	0.020	0.211	79.760	0.030	0.249	70.645
(b) patch-based segmentation

Figure 3.

Comparison of the DSC ratio and the average surface distance. In (a), the ANTs tool is shown to be superior to the other methods with respect to the DSC ratio in the segmentation of Anterior Commissure (AC). Interestingly, however, the proposed method shows the lower average surface distance in (b), which implies the ANTs tool possibly produced over-segmentation of the long, thin shaped AC. Excluding the AC, the proposed method shows better or at least comparable performance compared to the ANTs tool and outperformed the other two methods. In contrast, the non-local patch-based method performed worse in terms of the segmentation of the AC but better in almost every other region than the pair-wise B-spline method.

3.2 Evaluation of the Statistical Shape Model

Analysis of the variability decomposition is important, because it is a measure to assess if the statistical model is relevant and fits the data. In contrast to conventional PCA, the variance is determined using two terms: SSD_tot, the sum of the squared differences between the structures and the overall mean. Similar to ANOVA, SSD_tot is written as

$$SS{D}_{tot}=\sum _{j,k}({\stackrel{‒}{x}}_{jk}-{\stackrel{‒}{x}}_j^2)+\sum _{i,j,k}{(x_{ijk}-{\stackrel{‒}{x}}_{jk})}^2,$$ (10)

which is equivalent to SSD_tot = SSD_between +SSD_within. Using the equation Equation (10), it is now possible to determine the partition of the total variability according to a within-group term and a between-group term. This evaluation is performed on the bounding box parameters. Globally, the results indicate that the magnitudes of the within-group (27.23%) and the between-group variability (72.77%) are sufficiently large to satisfy this multi-level component model as compared to a standard PCA model.

Computing the proportion of explained variance according to the number of principal components provides information about ways to select the MCA components. Usually, the number of principal components is chosen with respect to a given percentage of the total variance explained by the components. Hence, the within-group and the between-group sub-models have to be distinguished. The results for this specific modeling reveal that for a given percentage of total variance, the within-group sub-model requires more principal components than the between-group sub-model, despite the fact that 72.27% of the model is between-group information. For example, let us assume that our multilevel model should capture 95% of the total variance. In this case, the within-group sub-model requires eight principal components whereas the between-group sub-model is characterized by only four principal components.

4. CONCLUSION

In this study, we have proposed a novel model- and patch-based segmentation method that utilizes the multiobject shape model as well as label fusion. In order to represent multiple objects, the particles were grouped according to multiple regions of interest. This particle representation specifically allowed the parameter-free multi-object shape model. Using this shape model built on multiple atlases, the proposed segmentation method was able to incorporate the shape variation of the training set for the purpose of segmentation. The actual segmentation of a novel subject image was performed by fitting the model onto the subject image, followed by searching the most similar patches and fusing them into a single label.

The proposed segmentation method can be adapted for further applications. Cortical thickness analysis is definitely a good choice for the proposed segmentation method. Rodent brain MR skull stripping is another possible application, which remains a challenging mandatory preprocessing step that always requires manual quality control.