Registration-Free Infant Cortical Surface Parcellation using Deep Convolutional Neural Networks

Abstract

Automatic parcellation of infant cortical surfaces into anatomical regions of interest (ROIs) is of great importance in brain structural and functional analysis. Conventional cortical surface parcellation methods suffer from two main issues: 1) Cortical surface registration is needed for establishing the atlas-to-individual correspondences; 2) The mapping from cortical shape to the parcellation labels requires designing of specific hand-crafted features. To address these issues, in this paper, we propose a novel cortical surface parcellation method, which is free of surface registration and designing of hand-crafted features, based on deep convolutional neural network (DCNN). Our main idea is to formulate surface parcellation as a patch-wise classification problem. Briefly, we use DCNN to train a classifier, whose inputs are the local cortical surface patches with multi-channel cortical shape descriptors such as mean curvature, sulcal depth, and average convexity; while the outputs are the parcellation label probabilities of cortical vertices. To enable effective convolutional operation on the surface data, we project each spherical surface patch onto its intrinsic tangent plane by a geodesic-distance-preserving mapping. Then, after classification, we further adopt the graph cuts method to improve spatial consistency of the parcellation. We have validated our method based on 90 neonatal cortical surfaces with manual parcellations, showing superior accuracy and efficiency of our proposed method.

1. Introduction

The highly folded human cerebral cortex can be parcellated into many neurobio-logically meaningful regions [1], which are the foundation of many brain structural and functional studies. However, manual parcellation is extremely time-consuming, difficult to reproduce, and expertise-dependent. Therefore, automatic methods for cortical surface parcellation is highly desired. Previously, several methods have been proposed [1-6]. However, these methods have two main issues: 1) Cortical surface registration is needed to establish the atlas-to-individual cortical correspondences. 2) The mapping from cortical shape to the parcellation labels requires designing of specific hand-crafted features. However, the infant brain undergoes a rapid expansion, which is also highly heterogeneous across subjects; This raises challenges for surface registration and designing of hand-crafted features.

To address these issues, for the first time, we explore to build a highly nonlinear mapping from the cortical shape domain to the parcellation label domain, by leveraging the strong representation ability of the deep convolutional neural network (D-CNN). To cope with the heterogeneous expansion patterns across subjects, as well as to reduce the amount of required data for effective training, we use a patch-wise training strategy. To extend the convolution operation from the Euclidean space to the cortical surface mesh in Riemannian space, we further project the spherical cortical surface patch to its tangent plane by a geodesic-distance-preserving mapping, which can be done efficiently by leveraging the 2D topological nature of the cerebral cortex. Benefited from the strong representation power of DCNN, our method is free of registration and free of designing hand-crafted features. After patch-wise classification, we further use the graph cuts method to improve the spatial consistency of the parcellation. To validate our proposed method, we manually parcellated 90 term-born neonatal cortical surfaces by following the FreeSurfer parcellation protocols [2], using an in-house developed toolkit. Comparisons between our automatic parcellations and the manual parcellations showed the effectiveness and efficiency of our method. The main contribution of this paper lies in twofold: 1) We propose an automatic infant cortical surface parcellation method based on D-CNN, working on the surface mesh. 2) Our method requires no surface registration and designing of hand-crafted features.

2. Method

2.1. Materials and Image Processing

A total of 90 term-born neonates were recruited. Their T2-weighted brain MR images were collected using a Siemens head-only 3T scanner. All images were processed using a standard pipeline [7], which includes: a) intensity inhomogeneity correction; b) skull stripping; c) cerebellum and brain stem removal; d) tissue segmentation; e) masking and filling non-cortical structures; f) separation of the left and right hemispheres. Then, the topologically correct and geometrically accurate inner and outer cortical surfaces were reconstructed using a topology-preserving deformable surface method based on tissue segmentation [8]. Each cortical surface was represented by a surface mesh. Each vertex on the mesh was coded with 3 shape descriptors, i.e., the mean curvature, sulcal depth, and average convexity. These three shape descriptors reflect local geometric shape of the cortical surface, which have been used for cortical surface parcellation [2]. After that, we followed the FreeSurfer parcellation protocols [2] to manually label each cortical surface by an in-house toolkit. The rationality of using FreeSurfer folding-based parcellation protocols lies in the fact that all major gyral and sulcal patterns of the cortex are established at term birth and are stable during postnatal brain development [9].

2.2. Convolution on Cortical Surface

Our main idea is to train a DCNN model to build the mapping from the cortical shape domain to the parcellation label domain. However, unlike in the Euclidean space, there is no straightforward convolution operation for surface mesh, which sits in the Riemannian space and has no consistent neighborhood definition. Literally, two strategies have been proposed to extend the convolution operation to the surface mesh [10], including a) convolution in other domains, e.g., the spectral domain obtained by the graph Laplacian [11]; b) projecting the original surface to a certain intrinsic space, e.g., the tangent space (which is an Euclidean space with consistent neighborhood definition [12]). In this paper, we adopted the later strategy, i.e., projecting each surface patch to its local tangent plane by preserving the geodesic distance.

However, there are two challenges: a) the geodesic distance is scale dependent, while the infant brain size varies dramatically across individuals; and b) the geodesic distance computation for each vertex on cortical surface is time consuming. To address these issues, we leverage the 2D topological nature of the cerebral cortex. First, the inner cortical surface was inflated to a standard sphere by minimizing geometric distortion between original cortical surface and its spherical representation [13], which also normalizes the size variation. Then, on the spherical surface, the geodesic distance is homogenous everywhere. Therefore, the positions of two equal-size (measured in geodesic distance) spherical patches only differ each other in a simple rotation, indicating that we can obtain consistent spherical patches at all vertices by simple rotations.

The projection rule is illustrated in Fig. 1(a). For a vertex V, given a point P_i on the tangent plane, we need to find the corresponding point P_i on the spherical surface. Given P_i, we can obtain: a) the distance r from V to P_i; b) the angle θ between the vector $\underset{V{P}_i}{\to }$ and the positive x-axis. Then, on the spherical surface, we can locate $P_i^{\prime }$ by preserving: a) the geodesic distance r (i.e., the arc length on sphere) from V to $P_i^{\prime }$ b) the angle θ between two great circles. One great circle is inscribed to the vector $\underset{V{P}_i}{\to }$, while the other is inscribed to the x-axis, as illustrated in Fig. 1(a).

Fig. 1:

Projection of a spherical surface patch to its tangent plane by preserving the geodesic distance and angle. (a) Illustration of projection rule. P_i and $P_i^{\prime }$ are the corresponding points on the local tangent plane of V and spherical surface. They have the same geodesic distance r to V and the same angle θ. In the tangent plane, r is the distance from V to P_i, and θ is the angle between $\underset{V{P}_i}{\to }$ and the positive x-axis. On the spherical surface, r is the geodesic distance from V to $P_i^{\prime }$, and θ is the angle between two great circles, who are inscribed to the vector $\underset{V{P}_i}{\to }$ and the positive x-axis, respectively. (b)-(d) The spherical surfaces color-coded by the mean curvature, sulcal depth, and average convexity (top row), respectively; and their corresponding intrinsic patches at vertex V (bottom row).

With this projection rule, for each vertex V on the spherical surface, we can sample an Euclidean local patch {P_i(V), i = 1,…, K²} on the tangent plane using the Cartesian coordinates, with i indicating the point index and K² as the point number for a K × K patch. Of note, the sampled Euclidean patch in Cartesian coordinates can allow direct application of the conventional convolution definition. Therefore, we can define the convolution operation for the spherical patch $\{P_i^{\prime }(V),i=1,\dots ,K^2\}$ on the spherical surface based on the one-to-one correspondence between P_i(V) and $P_i^{\prime }(V)$, using the sampled Euclidean patch as a bridge. Herein, we name the spherical patch $\{P_i^{\prime }(V),i=1,\dots ,K^2\}$ as the intrinsic patch at vertex V for better specificity and clarity. The top row of Fig. 1(b)-(d) shows the spherical cortical surfaces color-coded by the mean curvature, sulcal depth, and average convexity, respectively. While the bottom row of Fig. 1(b)-(d) shows the corresponding intrinsic patches at vertex V.

Once we located the intrinsic patch positions for a vertex V₁, i.e., $\{P_i^{\prime }(V),i=1,\dots ,K^2\}$, then for any other vertex on the spherical surface, e.g., V₂, we can simply rotate $\{P_i^{\prime }(V_1)\}$ to V₂ to obtain the positions $\{P_i^{\prime }(V_2)\}$ of the intrinsic patch at V₂. As mentioned before, this is because the geodesic distance on the sphere is homogeneous. Thus, we only need to construct the intrinsic patch for one time, which dramatically improves the efficiency of our method.

It is worth noting that different subjects may have different sphere tessellation. Therefore, we use the same tessellation to resample each spherical cortical surface. Through this strategy, we have the consistent intrinsic patches across different subjects. Note that our projection is different from the way in [12] in twofold: a) unlike the arbitrary angular direction at different vertices, our local intrinsic patches at different vertices share the consistent angular direction, since they are obtained based on the rotation; b) the projection only needs to conduct one time, which is much more efficient.

2.3. DCNN Architecture

After extending the convolution operation on the cortical surface, we use the DCNN to train the parcellation classifier. For each cortical surface, we have 3 shape descriptors, which reflect the folding patterns of the cortical surface in different views. Basically, the mean curvature measures the cortical folding in a fine view, and the average convexity measures the cortical folding in a coarse view. The sulcal depth measures the cortical folding by combining both the coarse and fine views. They thus provide complementary shape information. We include all of them into a 3-channel DCNN, with its architecture shown in Fig. 2. Each channel first independently performs the convolution and max-pooling operations. Then, all three channels are flattened and connected to the fully connected layer. Finally, the soft-max layer outputs probability of each vertex belonging to each parcellation label.

Fig. 2:

The 3-channel DCNN architecture for cortical surface parcellation.

2.4. Improving Spatial Consistency with Graph Cuts

Once the DCNN classifier is trained, for each vertex on a testing surface, we can extract the intrinsic patch and then apply the trained classifier for parcellation. However, since each vertex is classified independently without considering the spatial consistency, thus possibly producing spatially inconsistent labels. To improve the parcellation, we further use the graph cuts method to explicitly impose the spatial consistency. Specifically, based on the manual parcellation protocol, we know that the manual parcellations tend to split two cortical regions at the highly bent sulcal fundi [2]. Therefore, we explicitly formulate parcellation as a cost minimization procedure, i.e., E = E_d + λE_s. Here, E_d is the data fitting term, E_s is the smoothness term, and λ is a weight used to balance them. The data fitting term is defined as: E_d = −∑_V logp_V(l_V), where p_V(l_V) is the probability of assigning vertex V as label l_V, which is obtained from the DCNN output. And l_V = 1,…, 36 corresponds to 36 labels defined in the parcellation protocols [2]. The smoothness term is defined as: $E_s={\sum }_{V^{\ast }\in {\mathcal{N}}_V}{\mathcal{C}}_{V,V^{\ast }}(l_V,l_{V^{\ast }})$, where vertex V* is the direct neighbor of V, and ${\mathcal{C}}_{V,V^{\ast }}(l_V,l_{V^{\ast }})$ is the cost to label vertex V as l_V and also label vertex V* as l_V*. This cost can be defined as:

$${\mathcal{C}}_{V,V}\ast (l_V,l_V\ast )=\frac{1+\overrightarrow{n_V}\cdot \overrightarrow{n_V\ast }}{2}\times \frac{e^{-\mid H_V\mid }+e^{-\mid H_V\ast \mid }}{2}\times (1-\delta (l_V,l_V\ast ))$$ (1)

Herein, $\underset{n}{\to }$ denotes the unit normal direction of a vertex, and H denotes the mean curvature of a vertex. δ(l_V, l_V*) is the Dirac delta function: if l_V = l_V*, δ(l_V, l_V*) = 1; otherwise, δ(l_V, l_V*) = 0. This cost definition adaptively encourages the label smoothness based on local cortical geometry. At the highly-bent cortical area (e.g., the sulcal fundi), two vertices V and V* having the different labels generally have quite different normal directions and also large curvature magnitudes, therefore both the first and second terms in Eq. (1) is are small. On the other hand, if V and V* have the same label, they generally have similar normal directions and large curvature magnitudes, therefore only the second term in Eq. (1) is small. However, if two vertices are on the flat cortical area, i.e., their normal directions are generally similar and their curvature magnitudes are close to 0, then both the first and second terms in Eq. (1) are close to 1. The minimization of the above cost function can be efficiently solved using the graph cuts method [14].

3. Experiments

To evaluate our method, we use a 3-fold cross-validation on 90 neonatal subjects. Specifically, the dataset is equally partitioned into 3 groups in a random manner. At each fold, two groups of data are used to train the network, while the other group is used for testing. The process is repeated until all groups have been used for testing. During the training, we augment the data by randomly rotating each training patch to improve the model generalization ability. We empirically set the patch size as 35 × 35. In the graph cuts method, we set the parameter λ as 1. To assess the performance, we use Dice ratio to measure the overlap of the manual parcellation and the automatic parcellation.

For comparison, we adopt a popular multi-atlas based parcellation method (using majority voting). Specifically, we treat the training surfaces with manual parcellations as multiple atlases and use surface registration to propagate the parcellations of all atlases onto each testing subject. Then, we use majority voting to determine the final parcellation on testing subject. Herein, we use spherical demons [15] for registration, which has been well validated in cortical surfaces registration.

Fig. 3 shows parcellation results using different methods. Specifically, Fig. 3(a) shows the manual parcellations. Fig. 3(b)-(d) show the parcellation results using (b) multi-atlas with majority voting, (c) DCNN without graph cuts, and (d) DCNN with graph cuts, respectively. From this figure, we can see that the parcellation results of DCNN with graph cuts are highly consistent with manual parcellations.

Fig. 3:

Visual comparison of cortical parcellation results using different methods. (a) Manual parcellation; (b) Multi-atlas method with majority voting; (c) DCNN without graph cuts; (d) DCNN with graph cuts.

Fig. 4 further shows the quantitative comparison of parcellation results using different methods. From this figure, we can see that DCNN with graph cuts achieves the best performance. On average, the multi-atlas method with majority voting achieves average Dice ratio 84.54 ± 0.08 %. DCNN without graph cuts achieves the average Dice ratio 86.18±0.06 %. And, DCNN with graph cuts achieves the average Dice ratio 87.06 ± 0.06 %. Compared to the multi-atlas method with majority voting, DCNN with graph cuts achieves better results in 34 out of 36 ROIs.

Fig. 4:

Quantitative comparison of different parcellation methods for each ROI.

Of note, our method is registration free, thus much more efficient than the multi-atlas method. Specifically, for DCNN based parcellation, once the classifier is trained, the parcellation can be accomplished in less than a minute in a general PC. While, for the multi-atlas method, the computation time is much longer. For example, for the parcellation with 60 atlases, the computation time is around 2-3 hours, since all 60 atlases need to be registered onto the testing subject one-by-one.

4. Conclusion

In this paper, we propose a registration free method for infant cortical surface parcellation, by using DCNN to learn the mapping from cortical shape domain to the parcellation label domain in a patch-wise manner. Our proposed method has been validated on 90 neonatal cortical surfaces. Both visual and quantitative comparisons show the effectiveness and efficiency of our method. In future, we will test our proposed method on more datasets and also make it publically available.