A Flexible Graphical Model for Multi-modal Parcellation of the Cortex

Abstract

Advances in neuroimaging have provided a tremendous amount of in-vivo information on the brain’s organisation. Its anatomy and cortical organisation can be investigated from the point of view of several imaging modalities, many of which have been studied for mapping functionally specialised cortical areas. There is strong evidence that a single modality is not sufficient to fully identify the brain’s cortical organisation. Combining multiple modalities in the same parcellation task has the potential to provide more accurate and robust subdivisions of the cortex. Nonetheless, existing brain parcellation methods are typically developed and tested on single modalities using a specific type of information. In this paper, we propose Graph-based Multi-modal Parcellation (GraMPa), an iterative framework designed to handle the large variety of available input modalities to tackle the multi-modal parcellation task. At each iteration, we compute a set of parcellations from different modalities and fuse them based on their local reliabilities. The fused parcellation is used to initialise the next iteration, forcing the parcellations to converge towards a set of mutually informed modality specific parcellations, where correspondences are established. We explore two different multi-modal configurations for group-wise parcellation using resting-state fMRI, diffusion MRI tractography, myelin maps and task fMRI. Quantitative and qualitative results on the Human Connectome Project database show that integrating multi-modal information yields a stronger agreement with well established atlases and more robust connectivity networks that provide a better representation of the population.

1. Introduction

In-vivo neuroimaging and its recent advances have significantly contributed towards a thorough understanding of the brain’s organisation. The brain’s anatomy and cortical organisation can be investigated from the point of view of several sources of information: functional and diffusion Magnetic Resonance Imaging (fMRI and dMRI respectively) have allowed to infer the brain’s structural and functional connectivity, while cortical folding or myelination patterns can be extracted from structural MRI. In particular, dMRI, fMRI and myelin maps have been largely studied for mapping functionally specialised cortical areas (Glasser et al., 2013; Craddock et al., 2012; Moreno-Dominguez et al., 2014), an objective which has been prominent for over a century (Zilles and Amunts, 2010).

There is strong evidence that a single modality is not sufficient to fully identify the brain’s cortical mapping (Eickhoff et al., 2015). Indeed, cortical areas are believed to be defined by their microstructure, their connectivity and their function (Passingham et al., 2002). Because of this, a specific modality might not allow identification of the boundaries of all cortical areas. An accurate delineation of all cortical areas requires multiple modalities to exploit their complementarity and confirm the existence of certain boundaries. Yet, most existing brain parcellation methods are typically developed and tested on a specific type of information. Several popular parcellations have been derived from cortical folding (Tzourio-Mazoyer et al., 2002; Destrieux et al., 2010) or cytoarchitecture (Brodmann and Garey, 2005). More recently, connectivity-driven parcellations, in particular from resting state fMRI (rs-fMRI), have attracted a growing interest (Baldassano et al., 2015; Arslan et al., 2015; Blumensath et al., 2013; Cohen et al., 2008; Shen et al., 2013; Gordon et al., 2016; Thomas Yeo et al., 2011). The idea is to regroup vertices on the cortical surface based on how similar their connectivity profiles are. This is linked to the fact that parcellation can also be approached from a dimensionality reduction point of view for the study of brain connectivity networks. An essential step for the construction of these networks is the definition of the network nodes, which is typically done using parcellation techniques, where each parcel corresponds to a node. Connectivity-driven parcellations are expected to provide more accurate nodes than anatomical or random parcellations as they are derived directly from the connectivity data (Sporns, 2011).

In addition to the fact that a single modality cannot provide accurate cortical areas, mono-modal approaches are plagued by modality specific noise and biases which can significantly decrease the performance of parcellation algorithms. For instance, myelin maps only provide information on a subset of the cortex (highly myelinated regions), while diffusion MRI is a very indirect measurement of structural connectivity which is sensitive to the tractography algorithm used to recover the white matter fibres (Maier-Hein et al., 2016). dMRI is also subject to a gyral bias and geometrical issues associated with tractography algorithms. For these reasons, such algorithms tend to terminate fibres in gyri over sulci and align parcel boundaries with cortical folding (Van Essen et al., 2014). In contrast to this, rs-fMRI has the potential to provide reliable information across the cortex, but the modality has a poor signal to noise ratio (SNR) which can affect the accuracy and reproducibility of parcel boundaries. While the influence of noise can be strongly reduced when doing analysis on large groups, biases in the data (e.g. dropout in fMRI susceptibility regions) will constitute another important issue. Combining multiple modalities in the same parcellation task has the potential to provide more accurate and robust parcellations, since the different modalities are intrinsically related.

Nonetheless, only a few methods have tackled the problem of combining multiple modalities in the parcellation task. A popular approach aims to combine structural and functional connectivity so as to construct a multi-modal connectivity matrix, typically with the aim of constructing less noisy fMRI connectivity matrices using dMRI information (Ng et al., 2012; Venkataraman et al., 2012). Assuming fMRI is directly dependent on structural connectivity, fMRI connections are only considered valid if they are supported by a physical connection measured by tractography. One of the main flaws of this approach is the fact that tractography itself can be unreliable and particularly prone to false positives (Maier-Hein et al., 2016), but also to false negatives. As a result, the functional connections estimated from structural support could be biased and inaccurate due to tractography errors. Furthermore, working directly on creating multi-modal connectivity matrices does not allow considering other types of information (e.g. myelin maps) for the construction of a multi-modal parcellation. Glasser et al. (2016) approached the problem differently and generated a group-wise parcellation using a semi-automated approach where an algorithm placed parcel boundaries based on expert decisions using aligned rs-fMRI, task fMRI, myelin maps, cortical thickness and topographically organised functional connectivity. Parcel boundaries were delineated if they were consistent across at least two modalities. This group level segmentation was then transferred to the single subject level using a classifier.

Markov/Conditional Random Fields (M/CRFs) offer a way of constructing very tunable models which is beneficial for the problem of parcellation in several aspects: control over the level of parcel smoothness, flexible model design, easy incorporation of prior knowledge and natural extension to multi-modal or group-wise analysis. MRFs have been used for a plethora of image processing applications, including image segmentation (Boykov and Funka-Lea, 2006), registration (Glocker et al., 2008) or image denoising (Geman and Geman, 1984). Through the MRF formulation, parcellation is cast as a labelling problem, where each label corresponds to a specific parcel and is to be assigned to a set of nodes in a graph representing brain geometry. In our setting, the vertices of the cortical surface mesh correspond to the nodes of the MRF model. This labelling problem is solved by minimising the MRF energy which comprises unary and pairwise terms. The unary terms describe the likelihood of assigning a node to a specific parcel (i.e. label) while pairwise terms model the interactions between neighbouring nodes and typically act as smoothing priors. A significant advantage of MRF models is that they do not make any assumption on the input data. In other words, several modalities can be considered and processed within the same framework. The use of MRFs for rs-fMRI driven parcellation has recently been the subject of several publications (Lashkari et al., 2010; Ryali et al., 2013; Honnorat et al., 2015; Parisot et al., 2015, 2016b). The proposed methods describe the likelihood of assigning a node to a specific parcel as the correlation to the parcel’s average connectivity profile. Ryali et al. (2013) proposed an EM-like approach tailored for fMRI data that iteratively estimated the parcellation using graph cuts (Boykov and Funka-Lea, 2006) and estimated the unary cost’s parameters based on the current parcellation status. Honnorat et al. (2015) introduced the notion of parcel centres which are associated with a representative connectivity profile of the parcel. The MRF unary cost describes the correlation of a node’s profile with the parcel centre’s profile. This approach considers all nodes as potential parcel centres, which can be computationally expensive and subject to noise. A connectedness prior is introduced in the form of a star shape prior. Both approaches determine the number of parcels using label costs that estimate the number of necessary labels given a penalty. Despite the appeal of inferring the number of parcels from the data, this setting boils down to replacing the intuitive choice of the number of parcels with a different parameter of unknown impact.

In Parisot et al. (2015, 2016b), we introduced Graph-based Multi-modal Parcellation (GraMPa), an iterative MRF framework designed to handle the large variety of available multi-modal data: (1) We propose a set of modality specific unary cost functions that allow parcellating the brain according to different modalities and their properties. This allows to construct modality specific parcellations that can be compared without biases introduced by the use of different parcellation methods. (2) We extend the proposed framework to the context of multi-modal parcellation through the introduction of a multi-modal merging step. At each iteration, we obtain a set of parcellations from different modalities and fuse them based on prior knowledge of the modalities’ reliabilities and their interactions. The fused parcellation is used to initialise the next iteration, forcing the parcellations to converge towards a set of coherent yet modality specific parcellations. This provides a framework that allows 1) to directly compare different modalities, 2) to construct a multimodal parcellation and 3) to increase the robustness of mono-modal parcellations through the introduction of additional and complementary information.

In Parisot et al. (2015, 2016b), we focused on parcellation at the single subject level. In this paper, we provide a more general and detailed formulation of the model and investigate the impact of the proposed method on group-level analysis. This allows to explore multiple multi-modal associations and to compare obtained parcellations to well established atlases. In particular, we investigate the use of task activation maps as an additional source of information, which are too noisy to be used at the single subject level. We evaluate the ability of our framework to parcellate based on different types of inputs and exploit this property to provide an experimental set-up that quantitatively evaluates multi-modal agreements. Additionally, we evaluate the impact of integrating multiple modalities on the delineation of cortical areas, as well as from the point of view of network analysis. Our experiments on data from the Human Connectome Project (HCP) database using rs-fMRI, dMRI, myelin maps and task activation maps show that GraMPa yields a stronger agreement between modalities and more robust connectivity networks that provide a better representation of the population.

2. Material and Methods

The proposed iterative multi-modal model is illustrated in Fig. 1. In this section, we first introduce the mono-modal setting where one modality is parcellated using an MRF unary cost tailored for this specific input data. We then introduce the multi-modal extension, which is designed as a step that fuses information from multiple modalities. Finally, we describe the methodological details for our evaluation set-up.

Figure 1:

Overview of the GraMPa method. Each iteration updates an initial parcellation into a set of modality specific parcellations using an MRF model. The parcellations are then merged based on the modalities’ relative influences into a multi-modal parcellation, which initialises the next iteration.

We introduce below a set of notations that will be used throughout this paper. We aim to parcellate the brain’s cortical surface into a set of K distinct subregions, where K is known a-priori. We represent the cortical surface as a triangular mesh $\mathcal{S}=\{\mathcal{V},\mathcal{E}\},$ where each vertex $\mathrm{v}\in \mathcal{V}$ is to be assigned to a specific parcel. N_v is the number of vertices and ε corresponds to the set of edges of the mesh. χ is a feature matrix that can be defined as the output of dMRI tractography algorithms or as fMRI time series and χ(v) is the row of the feature matrix at vertex v.

2.1. Dataset and Preprocessing

Our dataset comprises two sets of 100 unrelated subjects randomly selected from the S500 release (November 2014) of the Human Connectome Project (HCP) database. The first set (54 female, 46 male adults, aged 22–35) is used as our main database. The second set (50 female, 50 male adults, aged 22–35) is used for reproducibility purposes, to evaluate whether similar results can be obtained from one group to the next. Both sets of 100 subjects are pre-processed using the same strategy.

The structural, diffusion and functional data have been preprocessed following the HCPs minimal preprocessing pipelines (Glasser et al., 2013). The cortical surface mesh S consists of 32k nodes with a 2mm spacing. All cortices are registered to a common reference space using sulcal depth information (Robinson et al., 2014).

The diffusion MR images have been acquired using a multi-shell approach, with three shells at b-values 1000, 2000, and 3000 s/mm² and 90 gradient directions per shell (Sotiropoulos et al., 2013). We obtain the tractography matrix using FSL’s bedpostX and probtrackX methods (Behrens et al., 2007; Jbabdi et al., (2012). The former estimates the orientation of the fibres passing through each voxel of the brain volume while the second performs probabilistic tractography based on the estimated fibre orientations. The probabilistic tracking is performed on the native mesh (before registration) representing the gray/white matter interface. We fitted three fibre compartments per voxel and 5000 streamlines are seeded from each of the white matter surface vertices and terminated on the pial surface. We use a step length of 0.3. The obtained tractography matrix records the number of streamlines that reached the rest of the mesh. To reduce the bias towards short range connections inherent to tractography, we compute the element-wise log transform of the tractography matrix (Jbabdi et al., 2009; Moreno-Dominguez et al., 2014; Parisot et al., 2016a) which reduces the dynamic range of fibre counts.

We perform rs-fMRI driven parcellation using timeseries from a 30 minutes acquisition. The 4D rs-fMRI volumes are projected onto the cortical surfaces which associates each mesh vertex with a rs-fMRI timeseries. The dataset was preprocessed and denoised (using the HCP’s ICA + FIX approach (Salimi-Khorshidi et al., 2014)) by the HCP structural and functional minimal preprocessing pipelines (Glasser et al., 2013). Subsequently, we temporally normalise each timeseries to zero-mean and unit-variance.

The remaining modalities considered in this paper (myelin maps, sulcal depth maps, task fMRI and cytoarchitectural map) are obtained as part of the HCP’s S500 release. Myelin maps are computed as the ratio of T1 and T2 structural MRI intensities. Probabilistic cytoarchitectural maps (Fischl et al., 2008; Zilles and Amunts, 2010; Amunts et al., 2007) were registered using freesurfer to each individual’s cortical surface and binarised throughout the HCP pipeline.

Group average data is obtained as follows: myelin maps are obtained by averaging all subjects’ maps. Similarly, the population average structural connectivity matrix is computed by averaging all the subjects’ tractography matrices. The average cytoarchitectural map, illustrated in Fig. 11, is obtained by majority voting from the registered individual maps, while we compute task fMRI group-level activation maps using FSL’s standard tools (FEAT) that use general linear modelling to construct activation maps (Barch et al., 2013). The analysis is carried out across sessions (single subject activation maps) and then across subjects (group-wise activation map). Task based parcellations are generated using the Z-statistics maps. Average rs-fMRI pseudo-timeseries are computed using FSL’s incremental group PCA (MIGP) (Smith et al., 2014), which can efficiently approximate the PCA output applied to the concatenation of all subjects’ timeseries using computationally tractable resources. We use a dimensionality of 2300 eigenmaps, which correspond to the concatenation of same day scans. Connectivity networks are then obtained by computing the Pearson’s correlation coefficient between the pseudo time-series.

Figure 11:

Illustration of the cytoarchitectural areas that are being matched with our parcellations.

2.2. Modality specific Markov Random Field Model

We cast the parcellation task as a labelling problem on the mesh S. Each surface vertex $\nu \in \mathcal{V}$ is to be assigned a label $l_{\mathrm{v}}\in 〚1,K〛,$ where each label corresponds to a parcel assignment.

2.2.1. Model

A labelling (corresponding to a parcellation) can be estimated by minimising the energy of an MRF defined by:

$$E_{MRF}=\sum _{\mathrm{v}\in \mathcal{V}}{U}_{\mathrm{v}}\left(l_{\mathrm{v}}\right)+\sum _{\mathrm{v}\in \mathcal{V}}\sum _{\mathrm{w}\in \mathcal{N}(\mathrm{v})}{U}_{\mathrm{v},\mathrm{w}}\left(l_{\mathrm{v}},l_{\mathrm{w}}\right)$$ (1)

Here $\mathcal{N}(v)$ represents the neighbouring vertices of v, defined from the structure of the triangular cortical surface mesh S. The unary term $U_v\left(l_v\right)$ describes the likelihood of assigning a vertex to a specific parcel. In order to define parcel specific costs, we associate each parcel with a parcel centre c. The parcel centre associates each parcel with a characteristic that uniquely defines it and corresponds to a representative vertex which is associated with an average feature $\chi (c).$ The definition of parcel attributes and their corresponding modality specific unary costs are described in Sec. 2.2.2. The pairwise term $U_{v,w}\left(l_v,l_w\right)$ acts as a smoothing prior. Its role is to describe the pairwise interactions between neighbouring vertices. It is designed here as a Potts model, introducing a penalty β when neighbouring nodes are assigned different labels.

$$U_{\mathrm{v},\mathrm{w}}\left(l_{\mathrm{v}},l_{\mathrm{w}}\right)=\{\begin{array}{ll}0\hfill & \text{if}{l}_{\mathrm{v}}=l_{\mathrm{w}}\hfill \\ \beta \hfill & \text{otherwise}\hfill \end{array}$$ (2)

Intuitively, β controls the smoothness the parcel boundaries and their contiguity. A low value of β will yield more precise but also noisier boundaries as well as discontinuous parcels. A high β value can excessively smooth parcel boundaries and yield large and fewer parcels than what is specified by parameter K (number of sought parcels). We provide an example of the influence of β on rs-fMRI driven parcellations on visual and quantitative results in the supplementary material.

Adopting an iterative approach allows to identify relevant parcel centres. Given an initial parcellation, it is iteratively updated by alternating between two steps: 1) updating the parcel centre based on the current parcellation, 2) parcellating the brain through MRF optimisation of the energy introduced in Eq. 1.

2.2.2. Modality-specific cost functions

The strength of adopting an MRF model is the possibility to perform parcellation with different modalities by simply changing the unary cost. We design a set of unary cost-functions, each adapted to a specific type of modality. Each cost-function is associated to a specific parcel centre definition. We consider three different types of input data: 1) High SNR connectivity data, where each surface vertex is associated to brain connectivity information with limited amount of noise. An example is to associate each vertex with a high resolution dMRI tractography profile. 2) Low SNR connectivity data will typically be fMRI data (task or resting state), where each vertex is associated with fMRI timeseries. The low SNR requires a different, more robust cost function than the one considered in case 1). 3) Non connectivity data associates each vertex with data projected on the cortical surface and requires a different approach to parcellation. This data does not contain connectivity information per se but characterises the similarity/dissimilarity of neighbouring nodes.

High SNR connectivity data.

A schematic representation of this unary cost is shown in Fig. 2a. The parcel centre is defined as a vertex $\mathrm{v}\in \mathcal{V}.$ Considering the parcellation at iteration t, the parcel centre c¹ is defined as the vertex within parcel l that has the highest similarity to the rest of the parcel. The unary cost $U_v\left({\mathrm{l}}_{\mathcal{v}}\right)$ is then defined as follows:

$$U_{\mathrm{v}}\left(l_{\mathrm{v}}\right)=1-\rho \left(\chi (v),\chi \left(c^l\right)\right)$$ (3)

Where ρ(.,.)is a measure of similarity between the two rows of the connectivity matrix χ. Here, we define ρ as the Pearson’s correlation coefficient, a very common measure of similarity for connectivity data. This type of unary cost has been used successfully in Parisot et al. (2015) for dMRI based parcellation. This cost has the advantage of being very simple, however, it relies on a single node to evaluate the likelihood of a parcel assignment which can make it sensitive to noise. As a result, it is well suited to robust data such as high resolution dMRI connectivity matrices, but can be too sensitive for noisier data such as fMRI.

Figure 2:

Illustration of the modality specific unary costs for high (a) and low (a) SNR connectivity data and (c) non connectivity data.

Low SNR connectivity data.

We design a more robust cost for noisier data such as fMRI time series. A schematic representation of this unary cost is shown in Fig. 2b. The parcel centre is again defined as the vertex with the highest similarity to the rest of the parcel. However, it is assigned an average connectivity profile $\chi \left(c^l\right)$ that is the average time series within a neighbourhood $\mathcal{N}$ around $c^1:$

$$\chi \left({\text{c}}^{\mathrm{l}}\right)=\sum _{\text{w}\in \mathcal{N}\left({\text{c}}^{\mathrm{l}}\right)}\frac{1}{\left|\mathcal{N}\left({\text{c}}^{\mathrm{l}}\right)\right|}\chi (\text{w})$$ (4)

$\mathcal{N}\left({\mathrm{c}}^1\right)$ is defined as the set of N closest vertices to c¹ that belong to parcel l. Closeness is determined by computing the shortest path on the surface mesh S, where edges are weighted by the similarity between vertices. The unary cost is then defined following Eq. 3:

$$U_v\left(l_{\mathrm{v}}\right)=1-\rho \left(\chi (v),\chi \left(c^{\mathrm{l}}\right)\right)$$ (5)

Increasing the value of N makes the unary cost more sensitive to the current parcellation (and therefore to initialisation), while decreasing N is more sensitive to noise. In the context of rs-fMRI driven parcellation, $\chi (w)$corresponds to the computed pseudo-timeseries as described in Sec. 2.1. A similar type of unary cost was used in Honnorat et al. (2015), where N was set as the number of vertices within the parcel. While using functional connectivity networks as connectivity profiles would provide more robust parcellations, it would significantly slow down the parcellation due to the correlation operation. A similar approach can be used for group averages with low SNR (e.g. non registered fMRI data) by computing the average correlation between the vertex to label and the N closest nodes around the parcel centre. Figure 6 shows the reduced influence of noise using this type of unary cost.

Figure 6:

Parcellation results using the low and high SNR unary costs for connectivity data (dMRI and rs-fMRI) for 50 (top row) and 150 parcels (bottom row). (a,c) shows parcel assignments obtained by simply minimising the unary cost over all potential labels without MRF smoothing constraint. (b,d) are the corresponding parcellations obtained after optimisation. Each colour corresponds to a parcel. The middle column shows how the high SNR unary cost fails for rs-fMRI data.

Non connectivity data.

The last type of input data we consider is non connectivity data projected on the cortical surface. Considering HCP data, myelin, sulcal depth or task activation maps are typical examples. In this setting, we can take inspiration from MRF-based image segmentation methods. We define the parcel centre as the geometric centre of a parcel, which we obtain by erosion. Each parcel centre c¹ is assigned the average vertex value within the parcel l. The unary cost, as illustrated in Fig. 2c is the shortest path on the cortical surface mesh S between the centre and the vertex v. The edges are weighted by the absolute difference between the values of the parcel centre and the vertices s_ialong the path:

$$U_v\left(l_v\right)=\sum _{s_i,v}\left|D\left(s_i\right)-\overline{D}\left(c^{\mathrm{l}}\right)\right|$$ (6)

Where D(v) is the non connectivity data value at vertex v and $\overline{D}\left(c^{\text{l}}\right)$ is the average value within parcel 1 associated with parcel centre c¹. Computing the absolute difference between the vertices’ values and the parcel centre (i.e. not using shortest paths) is also an option, but it does not yield spatially contiguous parcels.

Using the same framework allows to construct a set of parcellations for different modalities in a straightforward way and investigate their similarities. However, each parcellation converges independently and the different parcels do not have direct correspondences between modalities. In addition, each parcellation is affected by modality specific shortcomings: for instance, fMRI’s low SNR, dMRI’s biases with gyri and short range connections or the lack of information provided by myelin maps in large parts of the cortex.

In order to address these shortcomings, we extend the proposed framework to the multi-modal setting where the different modalities are forced to converge towards a set of mutually informed modality specific parcellations, where correspondences are established. This is done by initialising each iteration with a multi-modal parcellation obtained by fusing the individual parcellations.

2.3. Multi-modal Integration

Multi-modal integration is carried out through the introduction of an additional step in the iterative approach described in Sec. 2.2. At each iteration t, we now consider that the parcellation update is carried out in parallel for a set of N_mod modalities. Each modality mod is then associated with an MRF labelling solution ${\text{I}}^{\text{mod}}$ obtained by optimising Eq. 1 with the appropriate unary cost.

We then introduce a merging step, where the set of modality-specific parcellations are fused into a joint parcellation. The main objective is to increase the robustness of the mono-modal parcellation exploiting the similarities between modalities, while at the same time relying on more stable modalities when others are too noisy and/or uninformative.

2.3.1. Merging Model

To this end, we model the merging problem as a new MRF optimisation problem so as to allow flexibility with respect to the modalities considered, their interaction and the quality of the data.

$$\underset{1}{\arg \min }{E}^m(1)=\sum _{\mathrm{v}\in \mathcal{V}}{U}_{\mathrm{v}}^m\left(l_{\mathrm{v}}\right)+\sum _{\mathrm{v}\in \mathcal{V}}\sum _{w\in \mathcal{N}(\mathrm{v})}{U}_{\mathrm{v},w}\left(l_{\mathrm{v}},l_w\right)$$ (7)

Here, the pairwise cost is again a Potts model as described in Eq. 2. MRFs allow this flexibility in the design of the unary cost function $U_{\mathrm{v}}^m\left(l_{\mathrm{v}}\right),$ which describes here the merging criteria.

We associate each modality with a reliability map α^mod that describes how reliable and informative it is locally. A low reliability could be linked to noise in the data, or shortcomings inherent to specific modality processing (the gyral bias of dMRI tractography for instance). We propose to define the unary cost based on the different modalities labellings ${\text{I}}^{\text{mod}}$ and their reliability maps:

$$U_{\mathrm{v}}\left(1_{\mathrm{v}}\right)=\underset{{}_{mod}}{\min }\left(1-{\alpha }^{mod}(\mathrm{v})\delta \left(1_{\mathrm{v}},l_{\mathrm{v}}^{mod}\right)\right)$$ (8)

Here, δ is the Kronecker delta function. This unary cost exploits the reliability maps and ensures that labels selected by a given modality axe given a better or worse likelihood based on the modality’s local reliability. This set-up allows one modality to have more influence than the others on the joint parcellation if it is deemed very reliable. A typical example of locally reliable data would be pre-existing atlases such as cyto-architectural maps.

2.3.2. Modality-specific merging reliability maps

In this section, we detail the design of a set of modality specific weights. We aim to merge modalities in a way that acknowledges the shortcomings and assets of each modality. Diffusion and resting-state functional MRI are the only modalities considered here that have the potential to provide information across the whole cortex, while modalities such as task fMRI and myelin maps are only informative locally.

Resting-state fMRI.

Assessing the reliability of functional MRI data is a very challenging task. The main objective is to differentiate signal from noise, and in particular to identify which regions are dominated by noise. Intuitively, comparing different rs-fMRI acquisitions could provide this kind of information. However, the temporal dynamics of rs-fMRI signals remain unclear and could introduce differences between two acquisitions that do not correspond to noise. Furthermore, regions that are heavily connected to the rest of the brain (i.e. provide more information) could contain more noise than poorly connected regions (containing very little information and signal). As a result, while comparing two acquisitions may provide an indication of local reliability, this measurement may not be directly comparable between brain regions. We therefore associate rs-fMRI data with a fixed cost C that covers the whole brain so as to avoid losing information by inaccurate noise estimations. In this set-up, rs-fMRI is defined as the default modality that will drive the multi-modal parcellation when others are deemed not reliable.

Diffusion MRI.

Many challenges associated with dMRI and tractography data could be considered and incorporated in the reliability maps. Here, we propose to focus on one specific shortcoming of the dMRI processing pipeline that has been observed to strongly influence parcellation boundaries. This major issue associated with tractography algorithms is referred to as the gyral bias (Van Essen et al., 2014): tractography streamlines terminate preferentially within gyri instead of sulci. On top of that, several geometrical effects of tractography tend to cause a preponderance of connectivity transitions to lie on gyral crowns. This has a strong influence on the boundaries of dMRI driven parcellations that tend to align with cortical folding. In order to identify regions affected by such biases, we consider probabilistic tractography and the corresponding connectivity matrix χ, where each entry $\chi (v,w)$ counts the number of streamlines sent from vertex v that reach vertex w. The obtained matrix is not symmetric, one of the main reasons being the gyral bias. A column of the matrix will count the number of times a specific vertex v is reached, a number which will be higher for vertices located in gyri. On the other hand, a row of the matrix estimates which vertices can be reached from v and will not be sensitive to whether the location of v is in gyri or sulci. As a result, we can estimate which vertices are influenced by the gyral bias by comparing rows and columns of χ

$${\alpha }^{dMRI}(v)=\frac{{\sum }_{w\in \mathcal{V}}\chi (v,w)}{{\sum }_{w\in \mathcal{V}}\chi (w,v)}$$ (9)

This reduces the influence of vertices where fibres tend to terminate, i.e vertices which influence the positioning of parcel boundaries. As a result, dMRI parcellation boundaries affected by this bias will have little influence on the boundaries of the merged multi-modal parcellation. In these regions, dMRI essentially acts as a smoothing prior, providing high resolution information on which vertices should be regrouped in the same parcel based on similar connectivity.

Non connectivity data.

The main modalities of this type which we consider are myelin maps, task-fMRI activation maps and cortical thickness maps. These modalities provide information that only cover a subset of the cortical surface, where sharp variations are observed (e.g. strong activation or myelination). In order to ensure that the multi-modal parcellation aligns with these sharp transitions, the influence of non-connectivity data is designed to be proportional to the smoothed gradient maps of the non connectivity data. Gradient maps are computed using the HCP’s workbench¹, which uses a regression between the positions of the vertices (unfolded onto a tangent plane) and their values. The obtained gradient maps are subsequently smoothed on the surface using a Gaussian kernel with sigma set to 8. As a result, a high reliability is associated with strong gradients (i.e. strong evidence of a transition between function and/or structure), while uninformative regions will have no influence.

The computed weights, as defined above, are defined in very different ranges and scales. In order to ensure that the reliability weights of all modalities are directly comparable, all modality specific weights are rescaled between 0 and 1. To further increase comparability, the histograms of the weights’ values are equalised to match a normal distribution with mean zero and standard deviation 1.

2.3.3. Merging modalities with fusion moves

Despite the plethora of existing MRF optimisation methods, minimising multi-label MRF energies remains a difficult task with no optimality guarantees (Lempitsky et al., 2010). The concept of fusion moves, introduced in Lempitsky et al. (2010) casts the MRF optimisation problem as the task of combining an ensemble of suboptimal solutions. This is done through a set of iterative simple binary MRF sub-problems that will iteratively update the global solution with respect to one suboptimal solution. Fusion moves have the advantage of being very fast and scalable, and often yield globally optimal solutions to multi-label problems. Fusion move optimisation is particularly useful for our problem: the mono-modal parcellations can be seen as a set of suboptimal solutions to the multi-modal problem.

Each binary optimisation is called a fusion move and considers two suboptimal labellings ${\mathrm{l}}_i\text{and}{\mathrm{l}}_t^c\in {\mathcal{L}}^{\mathcal{V}}$, where ${\mathrm{l}}_i^c$ is the combined labelling obtained at the current fusion move t. We seek to label the cortical mesh $\mathcal{M}$ with a binary label $\mathrm{b}\in {\{0,1\}}^{\mathcal{V}}$ so as to update ${\mathrm{l}}^{\text{c}}\text{as}{l}_{t+1}^{\text{c}}(\mathbf{\text{b}})={\text{l}}_i°(1-\mathbf{\text{b}})+{\text{l}}_t^{\text{c}}°\mathbf{\text{b}},$ where 0 is the Hadamard product. The fusion move is carried out by minimising the binary MRF energy $E^b(\mathrm{b})=E^m\left(l^c(\mathrm{b})\right)$ which can be done with simple MRF optimisation techniques. The next fusion move considers a new modality and the current combined parcellation 1^c as the suboptimal labellings to merge. Fusion moves are repeated for all modalities until the combined labelling is no longer updated. Once the joint parcellation is obtained, it is used to initialise the next iteration of the multi-modal parcellation scheme. This forces the multiple modalities to converge together towards a set of modality specific yet coherent parcellations.

2.4. Evaluation: quantitative measures

The evaluation of brain parcellation tasks is a challenge in itself since there is no ground truth to compare to. While there have been accepted boundaries for certain regions of the brain, there is still a lot of uncertainty regarding the organisation of the brain as a whole, and how accurately a parcellation driven by one modality should match a given cortical area. In addition, a parcellation method yielding very accurate cortical boundaries may not be the best suited for representing brain connectivity networks, as functional/structural organisation at coarse resolutions may differ from cortical organisation. The accuracy of MRF based parcellation methods as a means of generating homogeneous parcellations from connectivity data (fMRI and dMRI) has been investigated and demonstrated in Parisot et al. (2016b, 2015); Honnorat et al. (2015).

Here, we focus our evaluation on the impact of the multi-modal integration, particularly how its flexibility can be exploited, and how it can improve upon the mono-modal parcellation. To this end, we focus on 1) the flexibility and reproducibility of the model (e.g. the ability to use different modalities), 2) network analysis, 3) the impact of dimensionality reduction and 4) the ability to delineate cortical areas (e.g. comparison to delineated cyto-architectural regions and other modalities). The flexibility of the model is explored by generating parcellations from broad set of modalities (dMRI, rs-fMRI, task fMRI, myelin and cortical folding), including multiple multi-modal combinations. The parcellations are compared across modalities and their quality is visually assessed. Reproducibility is assessed by computing parcellations from two different groups of 100 subjects.

2.4.1. Network analysis

We evaluate the impact of multi-modal integration on functional and structural network construction. This is assessed by 1) comparing the networks constructed using two different rs-fMRI sessions and 2) evaluating how well the group average represents the population by comparing all individual connectivity networks to their group average, similarly to what we proposed in Parisot et al. (2016a). The similarity between connectivity networks that we compute is the sum of absolute differences (SAD) between network edges, which can be seen as a graph edit distance (Raj et al., 2010) when nodes correspondences have been established and, as a result, no edit operations need to be considered for the node elements.

We expect an accurate and robust parcellation to yield similar networks between sessions, as well as an average parcellation that accurately summarises the features shared by the population.

For a given parcellation, rs-fMRI connectivity networks are computed by averaging the timeseries of all vertices in each parcel. The connectivity matrix is then obtained by computing the partial correlation between the timeseries of all pairs of nodes and applying the r-to-z Fisher transform to normalise and allow for accurate comparisons. Due to the fact that parcellation resolutions can slightly vary across methods (the method can yield empty parcels), the SAD between rs-fMRI matrix edges is normalised by the matrix size.

dMRI networks are computed from the probabilistic tractography matrix by summing the number of fibres that reach each parcel. Considering two parcels P_i and P_j, we account for the varying parcel sizes by normalising the fibre count as (Hagmann et al., 2008) :

$$w_{i,j}=\frac{2}{S_i+S_j}\sum _{v_i\in P_i}\sum _{v_j\in P_j}\chi \left(v_i,v_j\right)$$ (10)

where S_i and S_j are the sizes of parcels P_i and P_j respectively.

The average dMRI network is constructed by averaging all structural networks. The matrices are then normalised to sum to 1 to make the comparisons less sensitive to matrix size and the variability in fibre count. The average rs-fMRI network is constructed following the method described in (Salvador, 2004). We first compute an average connectivity matrix by averaging the partial correlation matrices of all subjects. We then identify non-zero partial correlation edges by testing the null hypothesis that the average partial correlation is zero for a given edge. This is done using one-sample t-tests for each edge of the individual partial correlation matrices using the False Discovery Rate approach to correct for multiple comparisons.

2.4.2. Comparison between modalities and parcellations

We evaluate the agreement between a parcellation and another modality (e.g. myelin or task activation maps) by generating a coarse parcellation from the considered modality using the proposed mono-modal MRF model. The parcels of both parcellations are quantitatively compared and matched using the measure of spatial overlap proposed in Bohland et al. (2009). This measure is non symmetric, and evaluates the proportion of one region i that is contained in another region j. We refer to r_i and r_j as the ensembles describing the vertices that belong to regions i and j respectively. The similarity measure P_ij is then defined as:

$$P_{ij}=\frac{\left|r_i\cap r_j\right|}{\left|r_j\right|}$$ (11)

where |r| is the number of voxels in ensemble r. A symmetric measure is also defined as $O=\sqrt{P_{ij}{P}_{ji}}.$

We match the two parcellations by selecting the parcels that have the highest similarity scores O. It should be noted that several parcels can be matched to the same one and therefore merged into a larger parcel. We then compute the Dice Similarity Coefficient (DSC) between the matched parcels. This is more flexible than the commonly used overlap methods as it does not search for perfect overlap, but also for inclusion of a parcel in another. This approach is particularly relevant when comparing parcellations with a very different number of parcels.

Coarse parcellations (e.g. the cytoarchitectural map) are also compared to the GraMPa parcellation following this approach. Overlap/DSC based measures with parcel merging have been very popular for the evaluation of parcellation schemes (Eickhoff et al., 2015; Arslan et al., 2017) and are therefore used here to facilitate comparison with other methods. Nonetheless, it should be noted that the parcel merging step could artificially improve the values of the computed DSC.

2.4.3. Dimensionality reduction quality

Last but not least, we evaluate the quality of our parcels with respect to the underlying data they are computed from. Our objective is to demonstrate how multi-modal parcellations can be used reduce the overall error when the same parcellation is used to reduce the dimensionality of a set of modalities. We aim to obtain parcellations that represent the dataset as well as possible.

A parcels average value should be as close as possible to all the values of the vertices within the parcel. For each modality and a given parcellation, we evaluate how well the data can be approximated when all values within the same parcel are represented by the average parcel value.

This is done by computing, for each modality, a merged low dimensional matrix where the values within the same parcel are averaged. For dMRI, the merged matrix of size K × K is constructed as described in Sec.2.4.1 and Eq.10. The rs-fMRI timeseries and non connectivity values (size K × dim, where dim is the dimensionality of the input data) are simply averaged within each parcel. An illustration of the construction of ${\chi }_{av}$ for dMRI data is shown in Fig. 3.

Figure 3:

Illustration of the process to construct a K × K low dimensionality representation of dMRI data, followed by a high resolution approximate matrix.

We then compute a high dimensional approximation ${\chi }_{av}$ of the original input data (dMRI tractography matrix, rs-fMRI timeseries and task activation and myelin maps) by assigning the parcel average values computed in the merged matrices to all vertices in the same parcel.

For dMRI data, we compute the Kullback-Leibler Divergence (KLD) between this matrix and the original tractography matrix $\chi$ that are normalised to be probability mass functions. The KLD measures how much information is lost by approximating the tractography matrix $\chi \text{with}{\chi }_{av}.$ A low KLD corresponds to a better parcellation with respect to dMRI data.

For rs-fMRI and non-connectivity maps, the same approach cannot be used as the data can contain important negative values and cannot be assimilated to a probability mass function as naturally. Instead, we compute the Root Mean Square Error (RMSE) between the original and approximate matrices. A low RMSE corresponds to a better parcellation with respect to the data.

3. Results

3.1. Experimental setup

In order to highlight the flexibility of our model, we compute parcellations using a set of different modalities, namely: diffusion MRI, resting-state functional MRI, myelin maps, sulcal depth and task activation maps. In particular, we show how we can use this framework to compare the different modalities, for instance to demonstrate the impact of the gyral bias on dMRI driven parcellation and to quantitatively assess the agreement of parcellations with other modalities.

We compute two different sets of group-wise multi-modal parcellations by 1) combining connectivity modalities (dMRI, rs-fMRI) and myelin (GraMPa multi-modal parcellation), and 2) combining myelin, resting state and task fMRI (GraMPa multi-modal parcellation with task). We evaluate the impact of such combinations and particularly focus on how rs-fMRI driven parcellations are impacted. In the remainder of this paper, we refer to the joint multi-modal parcellations as GraMPa multi-modal and GraMPa multi-modal with task. The corresponding modality specific parcellations are referred to as fMRI/dMRI GraMPa and fMRI GraMPa with task. The parcellations obtained from one single modality are dMRI/fMRI mono-modal.

The GraMPa parcellations are computed using rs-fMRI, dMRI and myelin maps. This set up was tested in (Parisot et al., 2016b) for single subject parcellation. The intuition behind using these three modalities is based on the fact that functional connectivity is intrinsically linked to structural connectivity, while myelin has been observed to have strong agreements with both connectivity modalities.

GraMPa with task exploits the information provided by rs-fMRI, myelin maps, and all tasks protocols provided by the HCP database. In order to avoid providing redundant information, we only select a subset of the 86 contrasts described in (Barch et al., 2013): the 2BK, 2BK-0BK, BODY-AVG, FACE-AVG, PLACE-AVG and TOOL-AVG contrasts of the WORKING MEMORY protocol; the PUNISH-REWARD contrast of the GAMBLING protocol; the LF-AVG, LH-AVG, RF-AVG, RH-AVG and T-AVG contrasts of the MOTOR protocol, the MATH and MATH-STORY contrasts of the LANGUAGE protocol; the RANDOM and RANDOM-TOM contrasts of the SOCIAL protocol; the MATCH and MATCH-REL contrasts of the RELATIONAL protocol; and the FACES and FACES-SHAPES contrasts of the EMOTION protocol.

Evaluation is performed on the multi-modal GraMPa and GraMPa with task parcellations (obtained after merging of all modalities) and the corresponding rs-fMRI and dMRI driven parcellations (obtained via the multi-modal iterative process, i.e. rs-fMRI driven parcellation informed by multi-modal data for centre definition). These parcellations are compared to the random initialisation (so as to compare results to chance), and the mono-modal parcellation using solely rs-fMRI data to evaluate the impact of multi-modal information.

For visualisation purposes, results for dMRI and rs-fMRI parcellations are reported on different graphs with corresponding axes. The merging reliability maps as well as the corresponding computed for dMRI, myelin and two example task contrasts are shown in Fig. 4. We can see that the dMRI weights align with cortical folding, with the regions with the lower weights being on the gyri.

Figure 4:

Merging reliability maps and corresponding maps obtained from dMRI, myelin and task activation maps. Higher values correspond to giving a higher influence to the modality.

3.2. Implementation

The method is implemented using a coarse to fine multi-resolution approach. The intuition behind the use of multiple resolutions is that the lower resolutions are less influenced by noise and allow a more efficient exploration of the search space for parcel centres updates. The cortical mesh is resampled into four different resolutions comprising 4k, 8k, 16k and 32k nodes, where the finest resolution corresponds to the original cortical surface mesh. The data projected on the cortical surface is resampled using the adaptive barycentric approach proposed in the HCP pipelines (Glasser et al., 2013). Quantitative results are computed for the left and right hemispheres separately and then averaged, with the exception of rs-fMRI connectivity network analysis which is performed on the whole cortex.

Parcellations are initialised using random parcellations of same resolution, computed using Poisson Disc Sampling. They are computed for a range of different resolutions: 50, 75, 100, 150, 175 and 200 parcels to investigate the performance of the method at different scales. The MRF smoothing parameter β is heuristically set to 0.3 for connectivity driven and GraMPa parcellations. It is adjusted to 2 for individual task driven parcellations, and 0.1 for myelin based ones.The fusion moves pairwise cost is set to 0.2. The rs-fMRI weight is set to 0.55 for the GraMPa parcellation (matching the average values of other modalities), and 0.65 for the GraMPa with task parcellation as we only want the most relevant task and myelin features to impact the multi-modal parcellation. Increasing the fMRI weight also allows to reduce the influence of locally noisy weight maps as seen in Fig. 4c. Parameters are optimised visually, so as to obtain compact parcels with smooth boundaries. The value of β has to be adapted with respect to the unary costs’ average values, which is why different values are used for non-connectivity and connectivity data. The higher rs-fMRI weight for the GraMPa with task is linked to the fact that task and myelin provide very local information, and that they should not overpower the overall role of rs-fMRI in the parcellation scheme. Similarly, the high smoothness parameter (set to 8, as described in section 2.3.2) for non-connectivity reliability weights ensures smoothness of the merged parcellation and reduces the influence of locally noisy task activation and myelin maps.

The MRF optimisation is implemented using the openGM2 library (Andres et al., 2012) which provides a framework for easily testing multiple discrete MRF solvers. Figure 5 shows an example of parcellations obtained with different solvers. We can observe that the differences are minimal between the discrete solvers. While the parcellation remains very similar, we observe stronger differences using the continuous solver we introduced in (Parisot et al., 2015). This can be attributed to the fact that smoothness constraints are defined differently for both systems, and as a result do not provide the exact same parcel smoothness result. In this paper, we use fastPD (Komodakis and Tziritas, 2007) as a solver due to its computational efficiency. Fusion moves are optimised using QPBO (Rother et al., 2007).

Figure 5:

Visual comparison of the impact of optimisation methods on the parcellation. The red arrows highlight the main differences between the continuous optimisation and discrete ones.

3.3. Parcellation results: Model flexibility

In this section, we illustrate the flexibility of our framework by presenting parcellation results from a set of different modalities. We provide visual results of parcellations and visually assess their accuracy. In addition, we highlight the usefulness of the low SNR cost function for rs-fMRI driven parcellation.

Figure 6 shows comparative results of parcellations using dMRI, and rs-fMRI with both the high SNR and low SNR unary costs. As we can see on the left hand side, the high SNR cost function appears to be adapted to dMRI data, yielding well delineated parcels that only require a limited amount of smoothing. On the other hand, this unary cost yields very noisy results and only identifies resting-state communities in a subset of the cortex. The low SNR unary cost yields a much more detailed parcellation, while we can still observe strong similarities in parcel boundaries with the parcellation obtained from the high SNR cost.

The flexibility and accuracy of our method for using non connectivity data to parcellate the brain is illustrated in Fig. 7. We provide visual examples using myelin maps, sulcal depth and task activation contrasts. We can see that our parcel boundaries tend to align with sharp transitions. We also provide comparative visual results using the non-connectivity unary cost with and without the shortest path constraint. The simple unary cost (without shortest path) yields accurate delineations of the data with very few, but discontinuous parcels (4 or 5 parcels are sufficient for myelin and task activation maps). The shortest path constraint yields compact and contiguous parcels while still following transition boundaries. However, some homogeneous regions can be split in several parcels due to the location of the parcel centres, and more parcels are necessary to accurately parcellate the cortical surface (25 parcels are computed in Fig. 2). As a result, the simpler approach is sufficient (and can be more accurate since there are no parcel splittings) for identifying boundaries and providing a reference for inter-modality comparisons. The shortest path method is more adapted to the GraMPa multi-modal integration, where compact parcels are compared across modalities.

Figure 7:

Visual examples of parcellations on non-connectivity data. For the different considered maps, we compare the parcellations obtained with (right) and without the shortest path constraint (left).

Finally, we illustrate the impact of the gyral bias on dMRI-driven parcellation boundaries in Fig. 8. We provide in Fig. 8a a visual comparison of dMRI-driven parcellation boundaries with rs-fMRI driven ones, where we can see a much stronger agreement between dMRI and cortical folding. Quantitatively, we use a cortical folding based parcellation to compute how myelin, rs-fMRI and dMRI driven edges (100 parcels) align with gyri for all 100 subjects in our database. Each parcel of the cortical folding-driven parcellation is characterised as belonging to a gyrus or not using the average sulcal depth within each parcel. For each parcellation, we then compute the number of vertices on parcel boundaries that pass through a gyrus. As shown in Fig. 8b, this number is much higher for dMRI-driven parcellations than rs-fMRI and myelin.

Figure 8:

Illustration of the agreement between dMRI parcel boundaries and cortical folding. (a) Visual assessment of the alignment of dMRI parcel boundaries with cortical folding. The boundaries of the parcellations are superimposed to a sulcal depth map, where lighter colours correspond to gyri. This is shown alongside a rs-fMRI driven parcellation for comparative purposes. (b) Quantitative assessment of the alignment of dMRI parcel boundaries with cortical folding.

3.4. Multi-modal comparisons

In this section, we provide quantitative and qualitative evaluation of the agreement of our parcellations with other modalities. We compare our results with 1) myelin maps, 2) task activation maps, 3) cytoarchitecture, and 4) the recent semi-supervised multi-modal parcellation introduced in (Glasser et al., 2016). All overlap measures are computed using the method described in Sec. 2.4.2. Since myelin and task only provide information on a subset of the brain, we only compute overlap scores with myelin and task parcels that have high or low average values. Thresholds are set to +/ – 5 for task activation maps and 1.3 for myelin maps (average myelin map value). This avoids to bias the results with uninformative regions and allows to compare solely to highly myelinated and activated regions.

Comparisons between the multi-modal parcellations and myelin or task activation maps are expected to be in favour of our method as these parcellations axe driven by myelin and task. Such results can be seen as a validation that the merging step is accurate. It should be noted that the boundaries of the multimodal rs-fMRI parcellations are solely driven by rs-fMRI data. Only the definition of parcel centres at each iteration is driven by multi-modal information. As a result, an improvement in agreement between rs-fMRI parcellations and task or myelin supports the assumption that there is an underlying agreement between these modalities, and that the integration of this additional information yields more accurate and robust centre definitions.

We evaluate the agreement of our parcellations with task activation maps by comparing them to a coarse task-driven parcellation (4 parcels), using the non connectivity unary cost without the shortest path constraint. Results are reported in Fig. 9. We observe a very similar performance from both the multi-modal and fMRI GraMPa with task parcellations. Both parcellations outperform the mono-modal approach. On the other hand, the GraMPa configuration with dMRI and myelin does not appear to increase the agreement with task activation maps. While the rs-fMRI GraMPa parcellation has a similar performance to the mono-modal approach, the GraMPa multi-modal parcellation has a lower agreement with task. This can be attributed to the fact that dMRI parcellations have a limited agreement with task, as can be seen in Fig. 9-dMRI group plot. We can see that dMRI parcellations perform similarly to the random initialisation.

Figure 9:

Overlap between task activation maps and our parcellations. dMRI based parcellations are reported with dashed lines.

We use the same process to compare to myelin maps. As shown in Fig. 10 we observe that both task and GraMPa configurations lead to an increase in performance for the multi-modal and rs-fMRI driven parcellations. While this increase is only marginal for task, the GraMPa configuration yields an increase in overlap of up to 6%. The good performance of multi-modal rs-fMRI driven parcellations suggests an underlying agreement between rs-fMRI and myelin maps. This increase in performance is also observed for the dMRI-driven parcellation.

Figure 10:

Overlap between myelin maps and our parcellations. dMRI based parcellations are reported with dashed lines.

We then measure the overlap of our parcels with the cytoarchitectural areas using the group-level cytoarchitectural map computed from our dataset (Brodmann and Garey, 2005). We compute overlap agreements for the primary somatosensory cortex (BA 3, 1, and 2), the primary motor cortex (BA 4), the premotor cortex (BA 6), Broca’s area (BA 44, 45), the visual cortex (VI and V2), and the perirhinal cortex as shown in Fig. 11. Results are presented in Fig. 12 for all areas (Fig. 12a and 12c), and focussing on only the motor and visual cortices (Fig. 12b, 12d and 12e). While we observe a limited improvement from the GraMPa configuration over all areas, the advantage of using myelin and dMRI data is clear in the motor and visual areas (i.e. BA 1–4,6 and V1-V2) where we observe a sharp increase for the GraMPa parcellation. This increase is also observed more marginally for the individual parcellations. The poor performance of mono-modal rs-fMRI and GraMPa with task parcellations with respect to the random initialisation might mean that the fMRI modalities we have considered in this paper do not provide sufficient information to reflect the cytoarchitectural organisation. Specialised methods such as visuotopic rs-fMRI and retinotopic task fMRI may be necessary to accurately delineate the sensori-motor and early visual cortices.

Figure 12:

Overlap between cytoarchitectural areas and our parcellations.

Last but not least, we compare our parcellations to the HCP’s multi-modal parcellation, that makes use of rs-fMRI, task fMRI activation map, myelin maps, cortical thickness and topographically organised functional connectivity to generate a parcellation of the cortex comprising 180 parcels. We compute the Adjusted Rand Index (ARI) (Hubert and Arabie, 1985) between parcellations, a measure which has been used several times for parcellation evaluation (Eickhoff et al., 2015) and has the advantage of allowing to compare parcellations of different resolutions (Milligan and Cooper, 1986). Overall, as seen in Fig. 13, we observe a relatively poor agreement between the parcellations, with multi-modal rs-fMRI driven parcellations providing the best performance and improving mono-modal results. Interestingly, the GraMPa with task parcellation has the worst performance which may be linked to the fact that this parcellation can be noisier than the rest. These results might be attributed to the fact that the HCP’s parcellation is generated from a different and larger dataset (210 HCP subjects). Our dataset consists of 100 subjects, and at this scale, our group average parcellations might still be influenced by the variability of the subjects comprising the group. In particular, anatomical differences are likely to be observed and exacerbated by the integration of additional information. In addition, our dataset uses a different registration scheme between cortical surfaces than the data used in Glasser et al. (2016): we use a simpler folding based registration, while theirs is driven by multi-modal information, including rs-fMRI, which will provide sharper boundaries. A more accurate comparison of our method with this multi-modal parcellation would have been to generate a group-level parcellation from the single subject ones applied to our data-set, and to use the same registration method. Unfortunately, the data was not available at the time our data was collected and experiments were run.

Figure 13:

Overlap between the HCP’s multi-modal parcellation and our parcellations. dMRI based parcellations are reported with dashed lines.

Comparative visual examples between our parcellations and other modalities are also provided. Comparisons to myelin, cytoarchitecture and task are shown in Fig. 14, 15 and 16 respectively, where parcel boundaries are overlayed to the different maps. The improvement with respect to myelin is particularly striking for the GraMPa multi-modal parcellation, similarly to what was observed quantitatively. Despite some relatively poor quantitative results, we observe that GraMPa with task can improve the agreement with cytoarchitecture, notably in the motor area. Similarly, the GraMPa parcellation also shows some improvement with respect to task activation maps.

Figure 14:

Comparison between the average myelin maps and the mono-modal, GraMPa and GraMPa with task rs-fMRI, dMRI, myelin driven and multi-modal parcellations (100 parcels). Parcel boundaries are superimposed to the average myelin map (red: highly myelinated regions.)

Figure 15:

Comparison between the average cytoarchitectural map and the mono-modal, GraMPa and GraMPa with task rs-fMRI, dMRI, myelin driven and multi-modal parcellations (75 parcel).

Figure 16:

Comparison between the average task activation maps and the mono-modal, GraMPa and GraMPa with task rs-fMRI and multi-modal parcellations.

3.5. Reproducibility analysis

We compute the agreement between parcellations and all their corresponding modalities for a second set of 100 unrelated subjects. Overlap results for myelin maps, task activation maps, the HCP multi-modal parcellation and the cytoarchitectural map are reported in Fig. 18. The obtained results are compared to the ones obtained with the first set of 100 subjects in Fig. 9, 10, 13 and 12e.

Figure 18:

Reproducibility analysis: Adjusted Rand Index between the parcellations obtained from two different groups of 100 subjects, (a)Edge distance between networks and their average for all subjects (b) Ratio between the mono-modal distance and the other methods’ distances

The most direct comparison can be done with respect to the multi-modal parcellation, as both sets of parcellations are compared to the same atlas. We observe a very similar behaviour, both in terms of ranges as well as relative performance of methods. Comparing the overlaps with the cytoarchitectural maps is also natural, as we expect very few changes between the maps averaged over all subjects. Similarly to what is obtained with respect to the HCP parcellation, performances are very similar between groups. Overlaps with respect to myelin maps are also very similar, with a poorer performance of the fMRI mono-modal parcellation overall with respect to other modalities. Similarly to what is obtained for the first group, the GraMPa parcellations substantially outperform the other methods. Task overlaps are almost identical between groups with the exception of the GraMPa multi-modal parcellation which yields a performance equivalent to the random initialisation. Overall, we observe very similar performances between groups, which suggests that our method is robust with respect to the group studied.

Finally, we compute the Adjusted Rand Index between the parcellations computed for all groups for all methods. The obtained values are expectedly higher than for comparisons with the HCP’s multi-modal parcellation as the pairs of parcellation benefit here from using the same method and assumptions, as well as the same registration strategy (contrarily to the HCP parcellation). We observe a slightly reduced reproducibility across groups when introducing more modalities. This can be linked to the fact that the parcellations are more precisely specific to the group considered, the added complexity of the multi-modal model, or the fact that the boundaries of the mono-modal parcellations are smoother. Additionally, the reproducibility of dMRI parcellations is expectedly reduced as its influence on the final GraMPa parcellation is relatively limited compared to fMRI which is less reproducible than dMRI. Interestingly, the GraMPa, fMRI GraMPa and fMRI GraMPa with task parcellations all yield very similar performances.

3.6. Network analysis

Network consistency results comparing the distance between connectivity networks and their corresponding average are shown in Fig. 19 for functional networks and Fig. 20 for structural networks. The similarity between the functional networks computed from different rs-fMRI sessions is shown in Fig. 21. To better visualise the different performances between rs-fMRI methods, we also plot the ratio of the distances computed for a given method and the fMRI mono-modal parcellation. We observe that multi-modal integration generally improves the robustness of connectivity networks. For functional networks, the best results are obtained for the GraMPa with task configuration where we consistently obtain lower distances. We obtain statistically significant results (p < 0.05) for 50, 75 and 200 parcels for the distance to average, and for 50 and 200 parcels for the distance between rs-fMRI sessions. The improvement is less prominent for the GraMPa configuration: we only obtain significant results for 50 parcels and the distance to average. In addition, the rs-fMRI GraMPa parcellation yields slightly worse results at higher resolutions. Interestingly, for both configurations, we obtain the best performance for the multi-modal parcellation with respect to their rs-fMRI driven counterpart. Finally, we obtain consistently better structural networks from the GraMPa dMRI parcellation, while the multi-modal one yields worse results.

Figure 19:

similarity between the functional networks and their corresponding average networks.

Figure 20:

Similarity between the structural networks and their corresponding average networks.

Figure 21:

Similarity between two functional networks obtained from two different rs-fMRI sessions. (a) Edge diBtance between the two sessions’ net-work for all Subjects, (b) Ratio between the mono-modal distance and the other methods’ distances

3.7. Dimensionality reduction quality

We compute the KLD and RMSE for all computed parcellations and modalities. KLD results with respect to the dMRI tractography matrix are reported in Fig. 22. To increase clarity, we only report results related to parcellations that comprised dMRI data. Unsurprisingly, the best results are obtained for the mono-modal dMRI parcellation since the remaining methods are attempting at reducing the influence of certain regions. We still obtain a very similar performance for the dMRI GraMPa parcellation which decreases at high resolutions. Interestingly, the multi-modal parcellation yields a poor performance (on par with the initialisation), which may be linked to the reduced influence of dMRI on the joint parcellation.

Figure 22:

Kullback Liebler Divergence between the original tractography matrix and the merged one for dMRI driven parcellations

We report RMSE values for myelin maps, task activation maps and fMRI pseudo-timeseries in Fig. 23a, 23c and 23b respectively. For myelin maps, results for the initial random parcellation are not shown for increased clarity, as they were in a very different range (i.e. approx 0.03 w.r.t to 0.005 for other modalities.).

Figure 23:

Impact of parcellation based dimeruionality reduction: error and loss of recorutruction with respect to the different modalities.

We observe that the multi-modal parcellation RMSE/KLD are not outperforming the mono-modal RMSE/KLD of the investigated modality, but outperform the mono-modal parcellations RMSE/KLD corresponding to different modalities. This suggests that integrating multiple modalities does not alter the precision of the method, and yields, for each modality involved, parcellations that are as accurate as the corresponding mono-modal ones.

4. Discussion

In this paper, we proposed a flexible graphical model for automatic parcellation of the cortex using multiple sources of information. The proposed method can parcellate the brain using connectivity (rs-fMRI, dMRI) as well as non connectivity data projected on the cortical surface (e.g. myelin maps). We provide a multi-modal extension to this model, where our iterative approach computes modality-specific coherent parcellations as well as a multi-modal parcellation that merges modalities based on their reliabilities.

We compute two sets of multi-modal parcellations, by merging connectivity information (rs-fMRI and dMRI) and merging rs-fMRI with task fMRI. We evaluate the impact of integrating multiple modalities in the parcellation framework and more specifically compare the results obtained from the two different configurations. Our results show that multi-modal parcellations yield more accurate brain delineations with respect to well established atlases. We also observe an increase in performance for the rs-fMRI and dMRI parcellations when they are informed by multiple modalities. Last but not least, our method provides a framework for quantitatively comparing parcellations to other modalities, using coarse parcellations as a reference.

An important observation is that we obtain different performances depending on the modalities considered in the multi-modal framework. On one hand, the task based configuration appears to yield a better representation of the functional organisation, since both the agreement with task activation maps and the robustness of constructed connectivity networks increase. On the other hand, merging rs-fMRI, dMRI and myelin maps yields parcellations that provide a better representation of microstructure (i.e. a better agreement with cytoarchitecture and myelination). From a dimensionality reduction point of view, we observe that the best performances with respect to all modalities are obtained from the fMRI parcellations derived from the multi-modal scheme. This suggests that there is no unique parcellation or parcellation method that is optimally adapted to all tasks (e.g. cortical delineations and network construction). A similar observation was made in Arslan et al. (2017), where certain parcellation methods provided more accurate cortical delineations while other performed better with respect to dimensionality reduction tasks. One may need to adapt the parcellation scheme to the task at hand. For example cortical delineation requires precise boundaries and is likely to require manual inspection by experts. Several cortical areas (e.g. V1/V2 ) have already been well defined, and the work of Glasser et al. (2016) is an important step towards identifying a ground truth parcellation. Fully automated methods are more adapted to the dimensionality reduction problem. While our method shows promising results towards identifying cortical areas, its flexibility and multi-modal aspect makes it well adapted to data-driven parcellations for dimensionality reduction, for example to build a common parcellation suitable for multiple modalities.

While it might be interesting to consider merging all modalities together, the overall performance is likely to decrease as we would attempt to integrate too much information that can be contradictory and therefore cannot be appropriately resolved in a single parcellation. This effect can be observed for myelin maps. Despite the fact that they are integrated in both multi-modal configurations, we obtain a much better agreement using rs-fMRI and dMRI than using multiple task contrasts. The influence of myelin is considerably reduced in the task based multi-modal parcellation. Another possible interpretation is that we are currently using Z-statistics task activation maps. A better performance might be obtained using effect size maps, which are expected to reflect the biological boundaries within a population more accurately. Similarly, the data used in this paper relies on a registration based on cortical folding. This may impact the quality of the parcellations (e.g. misalignments between modalities, blurring in areas in which areal boundaries and folds are not well correlated). More robust results might be obtained using an areal feature-based alignment.

Regarding our network analysis experiments, it is interesting to see that we generally observe the best performance from the multi-modal parcellation rather than the purely rs-fMRI driven one. One can infer from this that some functional transitions that can be identified from other modalities cannot be identified robustly from rs-fMRI mono-modal data. Our results also show that the influence of multi-modal information is reduced compared to what is observed for the task of brain mapping. This is to be expected as the boundaries of the parcels have a smaller influence for this kind of task. The impact of parcellation on network analysis has the potential to be more important in the context of diseased brains, especially in the presence of lesions which can considerably affect the brain’s organisation.

One particular advantage of the method is its flexibility with respect to the input data and design of the merging step. This flexibility can be particularly relevant when working on pathological data, where the quality fMRI or dMRI processing could be affected. In such settings, exploiting multiple sources of information has the potential to provide a clearer interpretation of how a disease affects the brain. Furthermore, flexibility is essential for clinical data where some modalities may be missing from the dataset.

It is important to keep in mind that comparisons between different modalities have to be considered carefully. All modalities considered utilise an ensemble of different processing steps in which multiple errors and wrong assumptions could have been made. The cytoarchitectural map is a particularly interesting case. It has been constructed from a limited set of subjects (Fischl et al., 2008) and then mapped to our dataset through a series of projections and registrations. More specifically, we focus our evaluation on the motor and visual cortices due to potential errors in registration. The cytoarchitectural atlas is mapped onto each subject using cortical folding based registration, which is optimised for the motor and visual cortices where folding is more consistent across subjects. As a result, areas located outside these regions have the potential to be misaligned. The high performance of the random parcellation with respect to data-driven parcellations suggests that this is the case, or that connectivity locally disagrees with cytoarchitecture. Another point is the use of overlap based measures, which can be affected by the size and shape of the parcels. While this bias is reduced due to the fact that we are using a common framework to carry out comparisons, it could still affect the performance of noisier parcellations such as the multi-modal ones.

Several extensions and improvements to the model could be considered. One option could be to adapt the model for group-wise parcellation, where each subject would correspond to a modality and the merged parcellation would be the group-level representation. We could obtain a set of coherent single-subject parcellations while preserving inter-subject variability. Here, the merging step could be decided based on group consistency measurements rather than reliability weights. A similar extension could be carried out for longitudinal data, which could be particularly interesting for brain development studies. We propose a fully automated model, which we believe is an important option to have considering neuroanatomists are not necessarily available for such tasks. Nonetheless, the parcellation quality could be largely increased by allowing user feedback, both in terms of reliability weights and parcel boundaries. Our model flexibility also allows an easy extension to a semi-automated setting, allowing neuroanatomists to provide feedback during our iterative and weight computation processes.

The reliability weights that we have proposed in this paper are relatively simple and aim at demonstrating the potential of the method. More accurate merging criteria could be learned from the data, or designed from prior knowledge on the agreement between modalities. In particular, the weights could be optimised for a specific task, for example with the objective of maximising the parcels’ overlap with known areas or other modalities. This can be optimised using the quantitative measures of overlap provided within our framework. Another option would be to compute data-driven reliability weights by computing parcellation uncertainties using the MRF’s min-marginal energies obtained from each optimisation (Kohli and Torr, 2006). This could be particularly relevant for the construction of a more robust rs-fMRI parcellation, which could be done using our framework with multiple rs-fMRI acquisitions. In such setting, a consistency measurement also has the potential to provide more accurate parcellations.

Finally, the MRF model itself could benefit from integrating the pairwise similarities between nodes within the smoothing pairwise term. This could provide more information and help define the parcel boundaries more precisely. Another interesting extension would be to investigate the impact of using fully connected CRFs on the obtained parcellations (Krähenbühl and Koltun, 2011).

Figure 17:

Reproducibility analysis: overlap between our parcellations and the different modalities for a second group. dMRI based parcellations are reported with dashed lines.