LATEST: Local AdapTivE and Sequential Training for Tissue Segmentation of Isointense Infant Brain MR Images

Abstract

Accurate segmentation of isointense infant (~6 months of age) brain MRIs is of great importance, however, a very challenging task, due to extremely low tissue contrast caused by ongoing myelination processes. In this work, we propose a novel learning method based on Local AdapTivE and Sequential Training (LATEST) for segmentation. Specifically, random forest technique is employed to train a local classifier (a single decision tree) for each voxel in the common space based on the neighboring training samples from atlases. Then, for each given voxel, all trained nearby individual classifiers (decision trees) are grouped together to form a forest. Moreover, the estimated probabilities are further used as additional source images to train the next set of local classifiers for refining tissue classification. By iteratively training the subsequent classifiers based on the updated tissue probability maps, a sequence of local classifiers can be built for accurate tissue segmentation.

1 Introduction

The first year of life is the most dynamic phase of the postnatal human brain development, with rapid tissue growth and development of a wide range of cognitive and motor functions. Accurate tissue segmentation of infant brain MR images into white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF) in this phase is of great importance for studying both normal and abnormal early brain development [1]. However, the segmentation of infant brain MRI is very challenging, due to reduced tissue contrast [2], increased noise [3], severe partial volume effect [4], and the ongoing white matter myelination [2, 5]. Especially, due to the ongoing myelination, at around 6 months of age, which is often referred to as isointense phase [6], the infant brain image appears isointense and exhibits the extremely low tissue contrast in both T1- and T2-weighted MR images, thus posing significant challenges for automatic tissue segmentation.

Although many methods have been proposed for infant brain MR image segmentation, most of them focused on segmentation of neonatal brain images in the infantile phase (≤ 5 months) [2, 4, 5, 7, 8], where images have the relatively distinguishable contrast between WM and GM in T2-weighted MR images. In contrast, there are only few works [9–11] focusing on segmentation of isointense (at ~6 months of age) infant brain images. In recent work [9], the convolutional neural networks (CNN) method [9] was proposed to segment the isointense infant brain images; however, it can only be applied to 2D image slices, instead of 3D data. In the work of [10], a learning-based method was proposed to integrate information from both multimodality images and the tentatively-estimated tissue probability maps for infant brain image segmentation. Specifically, random forest [12] was first used to train a multi-class tissue classifier based on the training subjects with multiple imaging modalities. This trained classifier provided initial tissue probability maps for each training subject. Inspired by the auto-context model [13, 14], the estimated tissue probability maps were further used as additional input images to train the next classifier, by combining the high-level multi-class context features (calculated from the estimated tissue probability maps) with the appearance features (calculated from multi-modality images) for refining tissue classification. By iteratively training the subsequent classifiers based on the updated tissue probability maps, a sequence of classifiers were obtained for tissue segmentation. However, the classifiers were trained globally, i.e., the training samples extracted from the entire atlases were mixed for training, and the same classifiers were applied for every voxel. As demonstrated in [15, 16], local spatially-adaptive classifiers can significantly improve the performance of global classifiers. However, only one-layer classifiers were trained in [16, 17]. Inspired by both of these works [10, 16, 17], we propose to train spatially-adaptive sequential classifiers for segmentation of isointense infant brain MR images, by taking advantage of both sequential [10] and spatially-adaptive [16, 17] training. Specifically, all the atlases are first registered into a common template space. Then, for each voxel in the common space, an individual tree is trained via random forest based on the spatially-neighboring training samples from the aligned atlases. Then, for each given voxel, all the nearby trained individual trees are grouped together to form a forest for estimating its tissue probabilities. The estimated probabilities are further used as additional source images to train the next-layer local classifiers, by combining the high-level multi-class context features (calculated from the previously estimated tissue probability maps) with the appearance features (calculated from multi-modality MR images) for refining tissue classification. Finally, these sequential and spatially-adaptive classifiers can be built for accurate tissue segmentation.

2 Method

2.1 Dataset and Image Preprocessing

T1- and T2-weighted MR images of 20 infants were acquired on a Siemens head-only 3T scanners with a circular polarised head coil. During the scan, infants were asleep, unsedated, and fitted with ear protection, with their heads secured in a vacuum-fixation device. T1-weighted MR images were acquired with 160 sagittal slices using parameters: TR/TE=2000/3ms and resolution=1×1×1 mm³. T2-weighted MR images were obtained with 160 sagittal slices using parameters: TR/TE=4000/299ms and resolution=1×1×1 mm³. For image preprocessing, T2 images were linearly aligned onto their corresponding T1 images. Afterwards, standard image preprocessing steps were performed before tissue segmentation, including intensity inhomogeneity correction [18] and histogram matching.

Accurate manual segmentation is of great importance for training classifiers in the learning-based segmentation methods. Due to low contrast and huge number of voxels in brain images, manual segmentation is very difficult and time-consuming. Hence, to generate reliable manual segmentations for isointense infant subjects, we first obtained automatic segmentation results using a publicly available software iBEAT (http://www.nitrc.org/projects/ibeat/). Then, based on these obtained automatic segmentation results, manual editing was carefully performed by an experienced neuroradiologist to correct errors by using ITK-SNAP [19]. Manual editing of each subject took approximately 3 hours. The intra-rater reliability (3 repeats) for WM, GM and CSF is 0.932, 0.931, and 0.960, respectively, in terms of Dice ratio. These 20 images with their edited tissue segmentation maps were used as multiple infant brain atlases.

2.2 Local AdapTivE and Sequential Training (LATEST)

In this paper, random forest [12] is adopted as a multiclass classifier to produce a tissue probability map for each tissue type (i.e., WM, GM, and CSF) by voxel-wise classification. As a supervised learning method, our method consists of training and testing stages. In the training stage, all the atlases are first linearly registered to a common template space. Then, an individual tree is independently trained at each voxel in the common space. Specifically, for a given voxel (i.e., a red point in Fig. 1(a)), all its nearby samples in a specified neighborhood (i.e., blue square) are used together as training samples. In the testing stage, to estimate tissue probability at a given voxel (i.e., a red point in Fig. 1(b)), all the neighboring trained individual trees (i.e., within red square) are grouped together to form a forest for classification. To deal with the challenges of low tissue contrast, inspired by [10], an auto-context model [13, 14] is further adopted to iteratively refine the tissue probability maps, by including context information calculated from the previously-estimated tissue probability maps. By iteratively training local trees with random forest and auto-context model on both the multi-modalities (T1 and T2) and the updated tissue probability maps, we can train a sequence of local classifiers for infant brain segmentation. All these training steps are detailed below.

• Step 1: Registration to a common template space. In the training stage, all the atlases are linearly registered to a common template space. Their corresponding tissue labels are also warped into the common space. In the testing stage, the target image is similarly registered to the same common space. Therefore, the rough correspondences between atlases and target image are established, based on which the trained sequential and spatially-adaptive classifiers can be mapped into the target image for testing.

• Step 2: Extraction of appearance and context features. For each voxel in the common space, all its nearby samples from each aligned atlas are randomly selected as training samples. Then, we extract various features from each selected training sample for training the classifiers. Specifically, we extract appearance features from multi-modality MR images. Based on these extracted appearance features, we train the first-layer local classifier (with a single decision tree) for each voxel. Then, all the neighboring trained individual local classifiers are grouped together to form a forest for classification of each training sample, thus providing the initial tissue probability maps for each training atlas. Inspired by the auto-context model [14], we further extract context features from these initial tissue probability maps. Note that these context features are used to coordinate segmentations across different parts of multi-modality images, which have been shown effective in both computer vision and medical image analysis fields [20–23].

• Step 3: Training of random-forest based local classifiers. We train a local classifier to learn the complex relationship between local appearance/context features and the corresponding (manual ground-truth) tissue label on each selected training sample. Although many advanced classifiers have been developed, herein we adopt random forest [12] because of 1) its effectiveness in handling a large number of training data with high dimensionality and also 2) its fast speed in testing. Besides, random forest also allows us to explore a large number of image features to select the most suitable ones for accurate classification [24].

• Step 4: Repeating Steps 2 and 3 until convergence. Note that we train our local classifiers in a serial manner. Specifically, based on the local classifier trained in Step 3, all the neighboring individual trees are grouped together to form a forest for classification of each training voxel. By visiting each training voxel, we can update the tissue probability maps for each training atlas. Then, according to Step 2, we extract context features from the updated tissue probability maps, and further employ them along with the appearance features to train the next-layer local classifier for each voxel in the common space. Eventually, we will train and obtain sequential and spatially-adaptive classifiers for infant brain segmentation.

Fig. 1.

Illustration of training stage (a) and testing stage (b). In (a), the blue region denotes a neighborhood, within which all the voxels are used as training samples for the center red point. In (b), the red region denotes a neighborhood, where all trained individual trees are grouped together to form a forest in the testing stage. Note that the blue and red regions in (a) and (b) can have different sizes.

Given a new target image, the testing stage is similar to the training stage. Specifically, the target image is first registered to the same common space. Then, for each voxel in the target image, the corresponding trained sequential and spatially-adaptive classifiers will be identified and used for classification. In the first iteration, three tissue probability maps (for WM, GM and CSF) are estimated by the first-layer spatially-adaptive local forest, using only the image appearance features calculated from target multi-modality images. In the later iterations, the tissue probability maps estimated from the previous iteration are also fed into the next-layer classifier for refinement. An example is shown in Fig. 3, in which the input images are T1 and T2 images. In a local region indicated by a red square, its tissue probability maps estimated by sequentially local forests are gradually improved with iterations and become more and more accurate. It is worth noting that the result of first iteration can be regarded as the result obtained using only local classifiers in [17].

Fig. 3.

The estimated probability maps by the proposed sequentially local forests along the iterations (#1-#3). #1 can also be considered as the result by using only local classifiers [17]. The last column shows the result by LINKS which uses sequentially global classifiers [10].

3 Experimental Results

We have evaluated the proposed method on 20 isointense infant subjects using leave-one-out cross-validation. The manual segmentation for each subject is considered as the “ground truth” for quantitative comparison. In our implementation, we use 3D random Haar-like features to compute both image appearance features and multi-class context features. Also, for each voxel, its neighborhood is defined as a 3D cube with size of 15×15×15 for training, and 5×5×5 for testing. For each tissue type, we select 2,000 training samples from the neighborhood for each training atlas. Then, for each training sample with the patch size of 7×7×7, 10,000 random Haar-like features are extracted from all source images, i.e., T1- and T2-weighted MR images and also three probability maps of WM, GM, and CSF. As mentioned, for each voxel in the common space, we train one individual decision tree with conservative parameters setting, such as we stop the tree growth with maximal depth 50, or with a minimum of 8 sample numbers in each leaf node. We set the maximal iteration as 3 since we find the performance typically increases dramatically in the first 3 iterations, and then gradually after 3 iterations. In the following, we compare mainly with [10], since it achieves the state-of-the-art segmentation results on the 6-month infant brain MRIs.

Fig. 3 shows the estimated probability maps by the proposed sequentially local classifiers in 3 iterations (#1-#3). As mentioned before, the classifiers in the first iteration of the proposed method can be considered as the local (one-step) classifiers (namely random forest based label fusion, RFLF) proposed in [17], and the corresponding results are shown in the third column (#1). Due to the absence of sequential training based on the intermediate tissue probabilities, the results by local (one-step) classifiers are noisy. The last column shows the result by sequential (global) classifiers (namely LINKS) proposed in [10]. Although the result by LINKS is free of noise, it is not accurate due to missing of local details. By contrast, the result by our proposed sequentially local classifiers is more accurate via visual observation, compared with the results by both local (one-step) classifiers [17] and sequential (global) classifiers [10].

Fig. 2 shows the Dice ratios of WM, GM and CSF by sequentially applying the learned classifiers. It can be seen that the Dice ratios are improved with the iterations and become stable after a few iterations (i.e., 5 iterations), as reflected by the reduced standard deviation. These results demonstrate the importance of using iterative training.

Fig. 2.

Changes of Dice ratios of WM, GM and CSF with respect to the iteration number.

In the following, we will make comparisons with (a) local (one-step) classifiers [17], (b) sequential (global) classifiers [10], and (c) manual segmentations. Fig. 4 shows the segmentation results for a typical isointense infant subject by different methods. The first row shows the original T1 and T2 MRIs with manual segmentation. The second, third and fourth rows show the segmentation results by local (one-step) classifiers [17], sequential (global) classifiers [10], and our proposed sequentially local classifiers, respectively. For a fair comparison, we have trained sequential (global) classifiers [10] with optimised parameters on the warped atlases. As we can see, the result by the proposed method is more accurate than all other methods, particularly for the places indicated by dashed circles. Last two columns of Fig. 4 also show the segmented WM/GM rendering results (along with zoomed views) by different methods. As we can see, our result is more consistent with manual segmentations. Then, we further employ Dice ratios to evaluate the performances of different methods on 20 subjects, as given in Table 1. Besides Dice ratio, we also measured the modified Hausdorff distance (MHD), which is defined as the 95^th percentile Hausdorff distance. It can be clearly seen that our proposed method outperforms all other methods.

Fig. 4.

Comparison of segmentation results on (a) a typical subject by (b) local (one-step) classifiers (RFLF [17]), (c) sequential (global) classifiers (LINKS [10]), and (d) our sequential & local classifiers (Proposed). The first row shows the T1 and T2 MR images and also manual segmentation. From left to right, the figure shows estimated the probabilities of WM, GM, and CSF, entire brain segmentation, 3D rendering, and zoomed 3D rendering, by three methods.

Table 1.

Average Dice ratios (in percentage) and MHD by 3 different methods on 20 isointense infant images. The bold indicates that the results by the proposed method are significantly better than others (p-value < 0.001).

Method		RFLF [17]: local (one-step)	LINKS [10]: sequential (global)	Proposed: sequential & local
Dice ratio	WM	84.6±0.76	88.7±0.43	91.4±0.33
	GM	85.9±0.88	87.1±0.66	90.5±0.45
	CSF	88.2±0.41	89.3±0.34	93.4±0.36
MHD	WM/GM	2.57±0.69	1.59±0.25	1.24±0.28
	GM/CSF	3.81±0.78	1.89±0.33	1.50±0.21

4 Discussion and Conclusion

We have presented a novel learning method based on local adaptive and sequential training (LATEST) for tissue segmentation of isointense infant brain MR images. The key idea is to train spatially-adaptive and sequential classifiers. Specifically, a first-layer local classifier was first trained for each voxel in the common space. The classification results from neighboring classifiers are average to generate intermediate tissue probability maps, which were further used as additional source images to train the next-layer classifier. By iteratively training the subsequent classifiers based on the updated tissue probability maps, a sequence of local classifiers were built for accurate tissue segmentation. Experimental results on 20 isointense infant subjects show that the proposed method achieves better performance than the state-of-the-art methods.

It is worth noting that the neighborhood size is important for training. In our experiments, we found the performance dropped when using larger size of neighborhood. This is mainly because the use of larger neighborhood complicates the classification problem by including more irrelevant samples into training set. This observation also shows the importance of training local classifiers for accurate segmentation.

There are still some limitations for the proposed work. First, the proposed work cannot guarantee a correct topology result. Thus the post processing such as topology correction may be needed. Second, during the training, more samples should be selected from these incorrectly labeled voxels. Third, the Haar-like features are selected from one scale (patch size), which is not optimal since different structures have different scales.