THALAMIC PARCELLATION FROM MULTI-MODAL DATA USING RANDOM FOREST LEARNING

Abstract

The thalamus sub-cortical gray matter structure consists of contiguous nuclei, each individually responsible for communication between various cerebral cortex and midbrain regions. These nuclei are differentially affected in neurodegenerative diseases such as multiple sclerosis and Alzheimer’s. However thalamic parcellation of the nuclei, manual or automatic, is difficult given the limited contrast in any particular magnetic resonance (MR) modality. Several groups have had qualitative success differentiating nuclei based on spatial location and fiber orientation information in diffusion tensor imaging (DTI). In this paper, we extend these principles by combining these discriminating dimensions with structural MR and derived information, and by building random forest learners on the resultant multi-modal features. In training, we form a multi-dimensional feature per voxel, which we associate with a nucleus classification from a manual rater. Learners are trained to differentiate thalamus from background and thalamic nuclei from other nuclei. These learners inform the external forces of a multiple object level set model. Our cross-validated quantitative results on a set of twenty subjects show the efficacy and reproducibility of our results.

1. INTRODUCTION

The thalamus is a symmetric sub-cortical gray matter structure located between the cerebral cortex and midbrain. It plays a central role in the processing and relay of signals involving both higher cognitive functions and the regulation of consciousness and sleep [1]. The thalamus is composed of neuronal clusters called nuclei, each individually responsible for communication between various cerebral cortex and midbrain regions [2]. Thalamic nuclear changes have been associated with multiple human functional disorders, including Parkinson’s disease, multiple sclerosis [3], and Alzheimer’s disease [4], with the affected nuclei a target of surgical treatment [5]. Given the potential clinical applicability, automatic parcellation of the thalamic nuclei has become a common goal in the literature.

While thalamic nuclei present minimal contrast in conventional MRI (with few notable exceptions [6, 7]), diffusion tensor imaging (DTI) shows more potential. DTI measures the directional diffusivity of water molecules at each voxel, and records this information in the tensor matrix. With their distinct tract connectivities and cytoarchitectures [2], the nuclei may be distinguishable via inhomogeneity in this tensor field image or its derivatives.

Several groups have had qualitative success differentiating nuclei based on spatial location and fiber orientation information in DTI. Behrens et al.[8] used probabilistic tractography and manually labelled each thalamus voxel according to the most probable cortical or midbrain connection and an established histological atlas of the nuclei [2]. Apart from connectivity, others have shown qualitative results based on optimizing regional tensor homogeneity using level sets [9], mean-shift clustering [10], and watershed segmentation on the image of local inhomogeneity in the tensor [11].

Other articles have shown quantitative results. Wiegell et al.[12] combined spatial and tensor information per voxel, using a Mahalanobis distance for location and a Frobenius norm on tensor differences in k-means clustering. Ziyan et al.[13] used normalized graph cuts to divide the thalamus into putative nuclei, also showing that the principal eigenvector (PEV) of the tensor was a strong distinguishing feature. In both works however, the putative nuclei are manually labelled or inherit from a manual parcellation the label of the nucleus they are compared against. Ye et al.[14] recently proposed the use of a multiple object level set method called MGDM [15]. The external forces driving the optimization were derived from the Knutsson edge map, which measures the speed of change in orientation of the PEV (see Sec. 2). While they compared their nuclear parcellations against those of a manual rater, their initialization depended on the rater’s thalamus boundary and they had only a small number of cases.

In this paper, we propose automatic thalamus parcellation using random forest learners [16] trained on manually delineated cases. We integrate the many potentially discriminating features used in prior work, including spatial coordinates, the Knutsson map, and other DTI-based and structural MRI information. In training, we form a multi-dimensional feature per voxel, which we associate with a nucleus label from a manual rater. We then use random forest classification to discriminate thalamus from background and thalamic nuclei from each other, using all the multi-contrast data at our disposal. The resulting learners, when applied to a target case, inform the external forces of the MGDM parcellation method used in [14]. We perform a quantitative study with bagged crossvalidation of twenty cases, showing a generalizable method that, unlike [14], does not require prior information about the target image. We follow with details of our method, results, and discussion of strengths, weaknesses and future directions.

2. METHODS

In this section we describe three key components of our automatic thalamus parcellation scheme: the Knutsson mapping of the principal eigenmode, which informs both our classification and the manual parcellations we train from and test against; our training of random forest classifiers on multimodal data; and target thalamus parcellation using MGDM.

2.1. Mapping the PEV into Knutsson Space

The principal eigenvector is the unit vector associated with the largest eigenvalue of the diffusion tensor. Since opposing vectors in Cartesian coordinates should represent the same orientation, there is a sign ambiguity in defining a difference measure between PEV’s [13]. Fan et al.[17] recently addressed this ambiguity using the Knutsson [18] map, defined as M : ℝ³ → ℝ⁵ for the unit vector [x, y, z]:

$$M(x,y,z)=(x^2-y^2,2xy,2xz,2yz,\frac{1}{\sqrt{3}}(2z^2-x^2-y^2\left)\right)$$ (1)

In the Knutsson space, two opposing Cartesian vectors are the same. The Knutsson edge map ∥G∥_F is the Frobenius norm (read: vector magnitude) of the 5 × 3 local gradient matrix, computed using finite differences. High intensities in the edge map, where the transformed PEV changes quickly, are considered to be boundaries between nuclei or nuclear groups.

Manual raters first use the fractional anisotropy image (FA) to find the thalamus boundary, then use these Knutsson cues to delineate nuclear structures that we identify as the anterior, medialdorsal, and pulvinar nuclei [17]. A fourth structure we call the ventral group is the complement of these structures (see Fig. 1). The lateral and medial geniculate nuclei extend inferior from the pulvinar.

Fig. 1.

Left: axial brain slice with thalamus highlighted. Middle: closeup of left thalamus in fractional anisotropy (FA). Right: left thalamus nuclear delineation from the Knutsson edgemap image (Sec. 2.1): anterior nucleus [yellow], medialdorsal [red], ventral group [blue], and pulvinar [orange].

We use these reproducible manual delineations as a ground truth for our training and testing, and we use the 5D tuple of Eqn. 1 in addition to ∥G∥_F as part of our random forest classification.

2.2. Random Forest Learning

Since its introduction by Breiman [16], random forest machine learning has become very popular due in part to its speed of convergence and generalizability, with a number of tools implementing variations [19]. The inputs to the random forest classifier are the set of observations and their associated class labels, where each observation consists of a multidimensional feature vector acquired at a single voxel, including the MRI, DTI, and derived spatial and orientation information. For the number of trees in the forest, a random selection of the observations is used to construct a decision tree, where the decision at each node of a tree—in this work—is based on minimizing the misclassification rate with respect to a subset of the features available. The output is a tree ensemble that, given a new observation, returns a putative class label for that observation and classification or membership scores for that and the other (less likely) class labels. There are of course numerous parameters that affect tree and forest construction (see [20] for a rich tutorial).

With co-registered modalities available, we sample 12 features at every voxel: the normalized MP-RAGE value, FA, mean diffusivity (MD), the spatial location (separately, x, y, and z), the five Knutsson dimensions M(PEV), and the Knutsson edge map ∥G∥_F. Fig. 2 shows an example vector-valued image. At each voxel, the sampled feature vector is associated with the thalamic nuclear label for that voxel (or background) provided by the manual rater. After random forest learning on one subject, the resulting tree ensemble can then be applied to another subject, with the classification scores per voxel driving the MGDM parcellation.

Fig. 2.

An example vector-valued image, showing MP-RAGE, FA, MD, ∥G∥_F, and the five Knutsson dimensions. Spatial location modalities not shown.

2.3. Nuclear Parcellation with MGDM

MGDM is a multiple object level-set segmentation framework that guarantees object relationships and prevents overlaps and gaps [15]. MGDM is thus applicable to thalamus parcellation, where nuclei are contiguous with one another without gaps (see Fig. 1). MGDM also enables the use of boundary-specific forces, where a particular force might apply only on the boundaries between designated objects (e.g., the boundary shared by the ventral nucleus and background). Within the level-set framework, this allows for regional appearance modeling, which has proven successful relative to global-scale appearance [21]. In this work, we are the first to take advantage of MGDM’s expressiveness in defining specific nucleus-nucleus and nucleus-background boundary forces.

For each training subject, we generate seven random forest tree ensembles using subsets of the vector-valued image and manual delineation. First, the nucleus-nucleus tree ensemble is constructed using only the nucleus voxels (and therefore not background). In the target case, this ensemble produces a membership image for each nucleus, and these images inform the respective nucleus-nucleus boundaries (see Fig. 4). Then a separate tree ensemble is constructed for each of the six nucleus-background boundaries. The voxels sampled for such an ensemble include those identified with the particular nucleus and those background voxels nearest to that nucleus, leading to a two class ensemble. Fig. 3 shows the relative importance of the features in example ensembles.

Fig. 4.

Target nucleus-nucleus and nucleus-background membership images with manual contours overlaid. From left: ventral-other nuclei, ventral-background, pulvinar-other.

Fig. 3.

Variable importance for example nucleus-nucleus (left) and nucleus-background tree ensembles. Note the associated target membership images in Fig. 4.

In testing a particular subject, we apply the ensemble classifiers from the training cases to that subject’s data, combining the associated membership scores. Since we do not know the manual rater’s thalamus boundary, we do not mask the resulting score images, nor do we register these images as in [14]. Fig. 4 shows example resulting score images with the manual boundary overlaid, showing agreement at the relevant boundaries. The initialization of MGDM consists of a labelling of the image that is essentially the mode of the labels at each voxel coordinate across the training cases.

3. RESULTS

We processed 22 image sets from an ongoing ataxia study. The MP-RAGE and DTI images were obtained on a 3T MR scanner (Intera, Philips Medical Systems, Netherlands) and co-registered. The DTI were acquired using a multi-slice, single shot EPI sequence. Each sequence had one B₀ image and used 30 gradient directions. The data were resampled to 0.828125 mm isotropic [14].

We trained nucleus-nucleus and nucleus-background ensemble classifiers for each subject as in Sec. 2.3. As shown in Fig. 3, the spatial and Knutsson dimensions are most discriminative among nuclei, as is expected from the literature. For example, note in Fig. 1 that the pulvinar, ventral/medialdorsal, and anterior are broadly separated along the anterior-posterior direction. However, also note how other modalities, such as the MP-RAGE and FA, are relatively more important in distinguishing nuclei from background.

It should be noted here that all images were cropped to within ten voxels of the manually delineated thalamus. While this amounts to limited prior information about the target in this study, there are atlas-based registration schemes that can provide this modest accuracy [22].

We then performed ten splits each with five training and the remaining test. In that each case was in training for between one and three of those splits, each of the images was parcellated seven to nine times. We computed the common Dice volume overlap coefficient over all nuclei and all splits, with the pooled statistics reported in Table 1.

Table 1.

Dice results for the six defined nuclei.

Nucleus	Mean Dice	Median Dice
Anterior	0.576 ± 0.146	0.593
MedialDorsal	0.641 ± 0.142	0.664
Ventral Group	0.833 ± 0.074	0.838
Pulvinar	0.711 ± 0.102	0.725
Lateral Gen	0.394 ± 0.202	0.405
Medial Gen	0.489 ± 0.244	0.515

Results on all the groups were much better than the inter-rater variability reported for a limited study in [17]. For example, while the automatic result on the anterior nucleus agrees with the manual parcellation at .576, manual raters’ agreement with themselves is reported at .324. Also note that while the results on the lateral and medial geniculates are poor, those nuclei are also exceedingly small, potentially only two or three voxels in the original DTI resolution, and thus difficult to resolve. In fact, the pulvinar and ventral group are often ~80% of the delineated thalamus volume in this study. Finally, we note that on these larger nuclei our results are comparable to those of [14], but without requiring extensive prior knowledge of the target’s thalamic boundary.

4. DISCUSSION

Our long-term goal is the large scale study of thalamic neuropathology using automated methods. In this paper we have extended thalamic parcellation in important ways that place us closer to that goal. We have used random forests classification to integrate multiple sources of potentially discriminating information (as reported in the literature) into an established thalamic parcellation scheme using MGDM. We have also limited prior knowledge of the target in computing our automatic results. Finally, this has allowed us to conduct a true validation experiment testing an automated method’s ability to reproduce a manual rater’s result. However, there are several concerns to address as we move forward.

We will first look to improve our accuracy and precision on the smaller structures, by training more specific (nucleus-everything) random forest learners and pooling results over a larger number of training cases. We will also integrate other discriminating contexual features in addition to spatial coordinate, like connectivity information at each voxel, as in [8].

Second, in addition to the Knutsson space of [17], there are several other tensor and PEV dissimilarity measures [13, 11]. We will compare these measures on synthetic and real data to determine if they imply related image structures.

Continuing in this line, we must understand how true to reality our nuclei are, whether manual or automatic. We use manual parcellations in training that we assume are correct, despite high inter-rater variability in the only study. We must further evaluate the reproducibility of these Knutsson-implied image structures and their underlying physiology, potentially through the use of 7T images or registered histological data.