A Hybrid Multishape Learning Framework for Longitudinal Prediction of Cortical Surfaces and Fiber Tracts Using Neonatal Data

Abstract

Dramatic changes of the human brain during the first year of postnatal development are poorly understood due to their multifold complexity. In this paper, we present the first attempt to jointly predict, using neonatal data, the dynamic growth pattern of brain cortical surfaces (collection of 3D triangular faces) and fiber tracts (collection of 3D lines). These two entities are modeled jointly as a multishape (a set of interlinked shapes). We propose a hybrid learning-based multishape prediction framework that captures both the diffeomorphic evolution of the cortical surfaces and the non-diffeomorphic growth of fiber tracts. In particular, we learn a set of geometric and dynamic cortical features and fiber connectivity features that characterize the relationships between cortical surfaces and fibers at different timepoints (0, 3, 6, and 9 months of age). Given a new neonatal multishape at 0 month of age, we hierarchically predict, at 3, 6 and 9 months, the postnatal cortical surfaces vertex-by-vertex along with fibers connected to adjacent faces to these vertices. This is achieved using a new fiber-to-face metric that quantifies the similarity between multishapes. For validation, we propose several evaluation metrics to thoroughly assess the performance of our framework. The results confirm that our framework yields good prediction accuracy of complex neonatal multishape development within a few seconds.

1 Introduction

Knowledge about postnatal brain development fuels our understanding of cognition, actions, sensation, perception, decision, and thought. From a modeling perspective, one could see the developing brain as characterized by complex and dynamic interactions of multiple shapes, comprising highly folded cortical surfaces and white matter fiber tracts that are evolving rapidly due to myelination. Developing models that accurately capture the spatiotemporal growth of a specific multishape (here, tract and cortical surface) can help the investigation of brain development and improve the diagnosis of several neurodeveopmental and psychiatric illnesses that are rooted in early infancy [1].

Recently, the generic varifold metric tailored to measure multidimensional shapes (e.g., a set of landmarks, a set of lines, and surfaces) was introduced in [2]. It has been used for population-based multishape atlas reconstruction of subcortical surfaces and fiber tracts [3,4]. However, the evaluation and hence the utility of these methods are limited. First, they were tested on simple deep brain structures (e.g., caudate) and specific fiber tracts (e.g., those connecting the cortical surface to the caudate) [3]. Second, they were tested on adult patients, where the inter- and intra-subject multishape variability is not as large as that in postnatal development (Fig. 1). More recently, Gori et al. presented a double-diffeomorphism strategy to jointly estimate a cortical surface and fiber-bundle template for both adult control and patient populations. The idea of double-diffeomorphism nicely accounts for the possibility of having fibers connecting to a specific cortical region in one subject and then ‘switching’ to another cortical region for another subject. However, when modeling subject-specific multishape development in infants, one would not expect the fibers to change their connecting spots on the cortical surface. Furthermore, these fiber tracts undergo fundamental topological changes, especially for the fiber tracts which bifurcate, branch out and multiply with myelination after birth. This is a non-diffeomorphic growth behavior, which contrasts the more stable diffeomorphic fiber deformation in older children and adults. For instance, Li et al. recently found that cortical fiber density is regionally heterogeneous and increases dramatically in the first year [1]. Put together, these facts present key challenges for predicting subject-specific postnatal brain multishape development, solely from the neonatal multishape. To the best of our knowledge, this is a problem that has not been addressed.

Fig. 1. Training steps of hybrid multishape prediction framework for one training subject.

(Top row) Estimate the baseline cortical surface diffeomorphic deformation trajectory through the diffeomorphism ϕ using [6]. (Middle row) Whole-brain deterministic tractography to estimate diffusion fiber tracts {F_i} at each acquisition timepoint. The red box demonstrates the non-diffeomorphic nature of fiber growth. (Bottom row) Non-diffeomorphic projection using ${\pi }^{A_i}$ of training longitudinal fibers tracts on the estimated longitudinal mean atlas {A_i}.

Noting the limited works targeting the prediction of subject-specific postnatal cortical shape development from a single timepoint [5], we propose in this work the first learning-based multishape prediction framework from neonatal cortex and fibers. The proposed framework comprises training and testing stages. In the training stage, for each infant, we learn from the training subjects (1) the geometric features (surface vertices), (2) the dynamic features of the baseline cortical surface development (smooth and invertible evolution trajectories), and (3) the fiber-to-face connectivity features via projections on an empirical longitudinal cortical surface atlas. In the testing stage, for a new neonatal multishape, we hierarchically select the best learned features that simultaneously predict the triangular faces on the cortical surface (or meshes) and the fibers traversing them at all training timepoints (in our case, 3, 6 and 9 months of age) based on cortical shape topographic properties and a novel fiber-face selection criterion.

Our proposed method has several advantages. First, it is not only restricted to predicting the cortical surface growth as in [5]. Second, it does not require the computationally expensive process of registering or regressing out thousands of fibers to establish tract-to-tract correspondence for prediction, which is less likely to be achieved using a conventional diffeomorphic multishape registration setting as in [2]. Third, it relies on the diffeomorphic cortical surface deformation trajectory, which is less complex and more accurate to estimate than for developing fibers, to guide fiber prediction. More importantly, this enables us to account for fiber connectivity changes and the appearance of ‘new’ fibers with different topologies. Ultimately, we present a new metric for jointly predicting both diffeomorphic surface evolution and non-diffeomorphic fiber growth within the multishape, thus making our approach hybrid.

2 Hybrid Longitudinal Surface-Fiber Evolution Modeling (Training Stage)

In this section, we present the advanced mathematical tools that mold our work. As a preliminary step, we embed the multishape (both fibers and cortical surface) into the varifold space. The multidirectional varifold-based surface representation will be used to estimate the diffeomorphic cortical growth [6], whereas the conventional unidirectional varifold-based fiber representation will be part of the proposed non-diffeomorphic fiber selection criterion for prediction [2].

Surface and Fiber Tract Representation Using Respectively Multidirectional and Unidirectional Varifold Metrics

The varifold metric measures the rich geometry of any shape with dimension d > 0 by the way the shape integrates a square-integrable 3D vector field ω ∈ W through convolutions based on a reproducing kernel K^W [2,5]. In this case, measuring a surface S as a varifold is defined as an integration of a testing vector field ω ∈ W along its nonoriented normal vectors n and principal curvature direction. More simply, measuring a fiber F as a varifold refers to the mathematical operation of integrating ω along the fiber nonoriented tangent vectors $\tau :F=\int \omega {(x)}^t\tau (x)\mathit{dx}$. In this context, W is defined as a Reproducing Kernel of Hilbert Space (RKHS) with a Gaussian kernel $K^W(x,y)=\exp (-{\mid x-y\mid }^2)/{\sigma }_W^2$. The kernel decays at a rate σ_W, which defines the scale under which geometric details will be overlooked when converting a shape into a varifold. Hence, any discrete shape embedded in the varifold space W* is a summation of local discrete measurements, each encoding the interaction of the shape at a local scale with a vector field ω [2,6].

Diffeomorphic Geodesic Longitudinal Surface Regression for Extracting Geometric and Dynamic Features

To longitudinally deform a source varifold surface S₀ observed at t₀ into a set of target varifold surfaces {S₁, … , S_N} respectively observed at {t₁, … , t_N}, we adopt the Hamiltonian formulation setting as described in [2,5,6] to estimate a diffeomorphism ϕ(x, t), t ∈ [0, 1], which is fully parametarized by a set of control points c_k and their attached initial deformation momenta α_k. The initial momenta fully guide the geodesic shooting of S₀ onto subsequent surfaces and are estimated along with the control points through minimizing the following energy functional using a conjugate gradient descent [5]: $E=\frac{1}{2}{\int }_0^1{\mid {\upsilon }_t\mid }_V^2\mathit{dt}+\gamma {\sum }_{j\in \{1,\dots ,N\}}{\Vert S_j-\varphi (S_0,t_j)\Vert }_{W\ast }^2$, with γ denoting the trade-off between the deformation smoothness term and the fidelity to data term, respectively. The velocity field ν_t belongs to a RKHS V, with a Gaussian kernel K_V decaying at rate σ_V, and is defined at a location x and timepoint t in terms of convolutions as: $\upsilon (x,t)={\sum }_{k=1}^{N_c}{K}_V(x,c_k(t)){\alpha }_k(t)$, with N_c as the number of the estimated control points. This allows to set vertex-to-vertex correspondence across subjects and timepoints. For prediction, we define the set of geometric features V as the set of all vertices positions x belonging to baseline training surfaces and the dynamic features as their corresponding evolution trajectories ϕ (x, t).

Estimation of Non-diffeomorphic Longitudinal Fiber-to-Face Connectivity Features Using Multi-projections on Spatiotemporal Atlases

Since we aim to predict the multishape growth from a single timepoint, we estimate a set of spatiotemporal surface atlases {A₀, … , A_N} by averaging the shapes of the training surfaces at each timepoint to help guide the prediction process (Fig. 1). Note that all these atlases are in correspondence with all subjects and across all acquisition timepoints. Then, to define the fiber-to-face connectivity features that capture the non-diffeomorphic growth of neonatal fibers, for each ensemble of fibers F_i from a training subject at t_i, we introduce the surjective projection function ${\pi }^{A_i}(F_i)$ to project it onto the corresponding surface atlas A_i. Specifically, for a fiber line f ∈ F_i with two extremities f¹ and f², we perform: $f^k\mapsto {\pi }^{A_i}(f^k)=\xi$, where k ∈ {1, 2} and ξ denotes a face in A_i. In turn, this allows us to identify for each training subject the connectivity features for each face in the atlas A_i at a specific timepoint t_i as the set of proximal fibers that hit it or are ‘connected’ to it (noted as F_i(ξ)) (Fig. 1). To define the connectivity features from all training subjects, we independently project the set of fibers for each training subject on the atlas. Hence, each atlas face stores for each training subject a set of connecting fibers through this process of multi-projections onto a fixed atlas.

3 Longitudinal Multishape Prediction Algorithm from Baseline (Testing Stage)

In the prediction stage, we first warp all baseline training surfaces onto the baseline cortical surface of a testing subject. Then, in the common space, we estimate the baseline testing fiber tracts using deterministic whole-brain tractography. Because of the non-diffeomorphic nature of neonatal fibers growth, we avoid to diffeomophically regress out fibers as for surfaces for prediction; instead, we explore the fiber-cortex relationship (or connectivity) to guide the fiber prediction. Hence, we introduce the following fiber-face selection criterion.

Fiber-face Selection Criterion

We define a distance between two faces ξ and ξ′ with respectively F(ξ) = { f₁, … , f_N} and $F({\xi }^{\prime })=\left\{f_1^{\prime },\dots ,f_{N^{\prime }}^{\prime }\right\}$ the set of fibers that ‘connect’ to it as follows: $d(\xi ,{\xi }^{\prime })=d_{\mathit{shape}}(\xi ,{\xi }^{\prime })+d_{\mathit{termini}}(\xi ,{\xi }^{\prime })+d_{\mathit{connectivity}}(\xi ,{\xi }^{\prime })$. The first term measures the overall shape difference between fibers attached to faces ξ and ξ′ using the varifold metric as: $d_{\mathit{shape}}(\xi ,{\xi }^{\prime })=\mid \frac{1}{N}{\sum }_{k=1}^N{\Vert f_k\Vert }_{W\ast }-\frac{1}{N^{\prime }}{\sum }_{j=1}^{N^{\prime }}{\Vert {f^{\prime }}_j\Vert }_{W\ast }\mid$. The second term quantifies the geometric closeness between the fiber termini positions $d_{\mathit{termini}}(\xi ,{\xi }^{\prime })=\frac{1}{2}({\mid \frac{1}{N}{\sum }_{k=1}^N{f}_k^1-\frac{1}{N^{\prime }}{\sum }_{j=1}^{N^{\prime }}{f}_j^1\mid }_2+{\mid \frac{1}{N}{\sum }_{k=1}^N{f}_k^2-\frac{1}{N^{\prime }}{\sum }_{j=1}^{N^{\prime }}{f}_j^2\mid }_2)$. And the third term computes the difference between the number of fibers attached to respectively ξ and ξ′ (with η = 0.01 for normalization): $d_{\mathit{connectivity}}(\xi ,{\xi }^{\prime })=\eta \mid N-N^{\prime }\mid$. This criterion defines a distance between two faces in terms of their ‘attached’ fiber characterstics in shape, geometric proximity, and connectivity.

Postnatal Multishape Prediction Algorithm

Algorithm 1 presents the key steps for multishape prediction based on the learned geometric, dynamic and connectivity features. Briefly, for surface prediction, we use a surface topography-based metric similar to the one introduced in [5] to hierarchically identify the closest baseline training vertices, which falls within an ε-distance from the baseline atlas A₀, to a baseline testing vertex. Specifically, we propose to reconstruct a testing baseline surface S̃₀ from training baseline surfaces using the following nested steps: (a) selecting a set of geometrically closest training vertices to the testing vertex, (b) selecting a subset of these vertices that have most similar normal directions to the normal vector at the testing vertex, and (c) selecting another subset of vertices marked in (b) that additionally share the same maximum principal curvature sign. As for fiber prediction, we first aim to reconstruct the baseline testing fibers F̃₀ using training fibers. To do so, we project the fibers of the testing subject onto the baseline atlas A₀, hence estimating the testing connectivity features. Then, for each vertex μ in the reconstructed baseline surface S̃₀, we use the fiber-face selection criterion to first mark the most similar corresponding training face in fiber properties to the testing face, then add its connecting fibers to F̃₀. Note that this uses the baseline atlas A₀ as a proxy since each of its faces stores the set of its connecting fibers from all training subjects. Ultimately, for each marked training face, we trace its diffeomorphic deformation using ϕ, while retrieving the set of its connecting fibers at different acquisition timepoints t_i, thereby estimating F̃_i.

Algorithm 1. Hybrid longitudinal multishape evolution prediction from baseline
1:	INPUTS:
	The longitudinal mean atlases Ai, the set of training baseline vertices V, the baseline testing multishape M0 = (S0, F0), and ${\pi }^{A_0}$ (F0).
2:	Initialize S̃i ← Ai and F̃i ={} for i∈ {0, …, N}.
3:	Initialize ε as the mean distance between S0 and A0 plus its standard deviation.
4:	for every vertex μ in the reconstructed baseline shape S̃0 do
5:	if its 3D position x is located outside the ε — neighborhood from S0 then
	Update x using a hierarchically surface topography-based metric.
	⋆ For each unchecked adjacent face ξ to μ, use the fiber-face selection criterion to identify the most similar corresponding training face in fiber properties to the testing face. Mark this face as ‘checked’.
	Retrieve the dynamic feature for μ as S̃i(x) = ø (x, ti) at each timepoint.
	Retrieve the spatiotemporal connectivity features for the selected deforming training face (set of fibers Fi(ø(ξ, ti)) that hit ø(ξ, ti) at timepoint ti), then F̃i = F̃i ∪ Fi(ø(ξ, ti)).
6:	else
	Implement ⋆ while using projections of both training and testing fibers on A0.
7:	end if
8:	end for
9:	OUTPUT:
	Set of predicted multishapes {M̃i = (S̃i, F̃i)} at timepoints ti.

4 Experiments and Discussion

Dataset and Parameter Setting

We use leave-one-out cross-validation to evaluate the proposed framework using data of 10 left and right cortical hemispheres from 5 infants, each with longitudinal diffusion and structural MR images acquired at around birth, 3, 6, and 9 months of age. For varifold surface and fiber representation, we set σ_W = 5 for the shape kernel K_W, σ_V = 30 for the deformation kernel K_V, and γ = 0.001 for the energy E as explained in [6]. Streamline tractography [7] was used to estimate the fibers inside each cortical surface at each timepoint.

Evaluation Metrics

For surface evaluation, we use both Dice index, which quantifies the face-to-face cortical overlap between two surfaces S and S′ as the ratio $\frac{2S\cap S^{\prime }}{S\cup S}$, and the symmetric Euclidean distance. For fiber prediction evaluation, we introduce three metrics: (1) Global mismatch (%). This represents the percentage of faces with attached fibers while the corresponding predicted faces had no fibers and vice versa. (2) Mean varifold difference. For a pair of faces both with traversing fibers, we use the varifold metric to measure a face-wise discrepancy between the ground truth and predicted fibers F and F̃ connected to two surfaces S and $\stackrel{\sim }{S}:\frac{1}{N_S}{\sum }_{i=1}^{N_S}\mid {\Vert F^{{\xi }_i}\Vert }_{W\ast }-{\Vert {\stackrel{\sim }{F}}^{{\xi }_i}\Vert }_{W\ast }\mid$, with N_S denoting the number of faces in S, and ξ_i a face in S. (3) Fiber mismatch per face. This metric represents the average number of mismatched fibers per face across surface faces that are hit by either predicted or ground truth fibers or both. We also evaluate the joint prediction accuracy for both surface and tracts using a unified varifold difference metric: $\frac{1}{N_S}{\sum }_{i=1}^{N_S}\mid {\Vert F^{{\xi }_i}\Vert }_{W\ast }-{\Vert {\stackrel{\sim }{F}}^{{\xi }_i}\Vert }_{W\ast }\mid +\mid {\Vert S_i\Vert }_{W\ast }-{\Vert {\stackrel{\sim }{S}}_i\Vert }_{W\ast }\mid$.

Multishape Prediction Evaluation

Despite the small size of our dataset and its large variability in cortical shape and fiber tracts, our framework led to very promising results as summarized in Table 1. Since this is the first work to predict developing cortical fibers, we compared our prediction error with the error of the observable baseline multishape reconstruction from the baseline ground truth multishape, which is very low (0 month in Table 1). We notice that the prediction accuracy generally decreases from 3 to 9 months compared to the baseline reconstruction from the ground truth, with a slight potential improvement at 6 months. Notably, the global mismatch for the predicted fibers peaks at 3 months. This is quite expected since the training fibers at around 3 months are largely variable due to the rapidly developing myelination. Moreover, the proposed rich fiber-face selection criterion generated better prediction results compared to using symmetric Euclidean distance as a similarity metric between fibers for face-fiber selection. Indeed, mean fiber mismatch per face dropped from 1.76 to 1.64 and mean varifold value from 19.98 to 18.83 when using our metric. Figure 2 shows a good overall overlap between ground truth and predicted fibers for a representative testing subject. The red-blue fiber mismatch regions can be explained by a large variability in the training fiber data as well as the use of inconsistent subject-specific tractography in the temporal domain. Additionally, we locally evaluated the accuracy of our prediction method in 35 anatomical cortical regions (Fig. 3), which showed a spatially-varying prediction accuracy that generally decreased with time. Nonetheless, it still fitted into a promising range of prediction values for each evaluation metric (e.g., ~3 mismatched fibers per face). For the cortical surface, the prediction mainly dropped in highly folded and buried cortical regions such as the insular cortex. On the other hand, the prediction error of the overall shape of the predicted fiber tracts compared with the ground truth tracts, quantified using the varifold distance, reached its apex in the paracentral lobule, the posterior cingulate cortex and the precentral gyrus. This can be explained by large variability in the shape of the fibers connected to these regions. For potentially similar reasons, the mean face-wise mismatch was below 15 % in most cortical regions, except for the anterior and posterior cingulate cortices, and the insular cortex. These regions were also affected by largest values of mean fiber mismatch per face (which generally remained below 5).

Table 1.

Surface (S) and fiber (F) prediction accuracy evaluation averaged across 10 cortical hemispheres.

	0 month	3 months	6 months	9 months
Global mismatch % (F)	15.40 ± 2.31	20.40 ± 3.68	19.50 ± 1.26	19.87 ± 1.24
Mean varifold difference (F)	18.83 ± 4.39	21.80 ± 3.22	21.22 ± 4.27	23.83 ± 4.59
Fiber mismatch per face (F)	1.64 ± 0.63	3.09 ± 1.14	2.76 ± 0.44	3.15 ± 0.30
Mean Dice index (S)	1	0.81 ± 0.03	0.81 ± 0.03	0.77 ± 0.02
Mean Euclidean distance in mm (S)	0.45 ± 0.07	0.68 ± 0.09	0.91 ± 0.14	1.08 ± 0.14
Unified varifold difference (S + F)	50.43	55.2	56.42	60.23

The baseline multishape reconstruction error (in bold) is considered as a ‘reference’ in assessing the performance of our prediction framework.

Fig. 2. Multishape prediction for a representative subject.

The blue multishape represents the ground truth while the one in red represents the predicted multishape. The reconstructed baseline multishape (S̃₀, F̃₀) is used as guidance for multishape prediction at late timepoints and as a reference for evaluation.

Fig. 3.

Multishape prediction evaluation in 35 anatomical inflated cortical regions.

5 Conclusion

We proposed the first hybrid developing multishape prediction model that captured well both the diffeomorphic cortical shape deformation and non-diffeomorphic fiber tracts growth. Our method leveraged on exploring the fiber-surface relationship through multi-projections of fiber termini on the corresponding surface. Our prediction results are promising and we hope that in the light of this work more attention will be drawn to solving this challenging problem. Eventually, building an accurate and fast multishape prediction model can also help predict structural brain connectivity of axonal wiring during early postnatal stages. One way to improve our work is to develop a non-diffeomorphic longitudinally consistent brain tractography algorithm as a preprocessing step – which, to our knowledge, is still not tailored to handle developing 3D fiber tracts.