Evaluation of functional MRI-based human brain parcellation: a review

Abstract

Brain parcellations play a crucial role in the analysis of brain imaging data sets, as they can significantly affect the outcome of the analysis. In recent years, several novel approaches for constructing MRI-based brain parcellations have been developed with promising results. In the absence of ground truth, several evaluation approaches have been used to evaluate currently available brain parcellations. In this article, we review and critique methods used for evaluating functional brain parcellations constructed using fMRI data sets. We also describe how some of these evaluation methods have been used to estimate the optimal parcellation granularity. We provide a critical discussion of the current approach to the problem of identifying the optimal brain parcellation that is suited for a given neuroimaging study. We argue that the criteria for an optimal brain parcellation must depend on the application the parcellation is intended for. We describe a teleological approach to the evaluation of brain parcellations, where brain parcellations are evaluated in different contexts and optimal brain parcellations for each context are identified separately. We conclude by discussing several directions for further research that would result in improved evaluation strategies.

INTRODUCTION

A brain parcellation is a spatial division of the brain into regions with distinct functional roles (1, 2). Traditionally, brain regions are defined as locations in the brain that share similarity in one or more properties including cyto- or myelo-architecture, their pattern of connectivity to other locations, the functional activity they exhibit, and their topographical representation (2). Although historically brain parcellations were constructed based on histological properties of the cortex, over the past two decades several in vivo brain parcellations using functional magnetic resonance imaging (fMRI) have been constructed. These functional parcellations group voxels with similar blood-oxygen-level-dependent (BOLD) signal based on the assumption that voxels with similar activity serve the same functional role. The resultant parcellation is therefore expected to partition the brain into functionally relevant regions. Functional brain parcellations have now become an indispensable tool for analyzing various brain imaging data sets acquired using fMRI (3–10) and diffusion-weighted imaging (DWI) (11). One of the most common applications of brain parcellations is to calculate functional and structural connectivity between different brain regions. Functional connectivity is calculated as the correlation coefficient between the average time series of voxels within each region. Structural connectivity is calculated as the number of white matter fibers connecting voxels within different regions. Calculating connectivity between regions instead of single voxels increases the signal to noise ratio (12). Region-wise connectivity profiles are studied in a variety of contexts such as how connectivity profiles vary across individuals (5), how much they can predict behavior (3, 10), how they change under different cognitive states such as sustained attention (9, 13), how they evolve across the human lifespan (4, 7), and how they are altered by various disorders such as Alzheimer’s disease (14) and Parkinson’s disease (11).

FMRI-based brain parcellations differ from each other in three major aspects. First, brain parcellations are constructed using different algorithms such as K-means, spectral clustering, and mixture models (15, 16). Second, brain parcellations are constructed at different parcellation granularities, i.e., the number of regions the brain is divided into ranging from tens to several hundreds (16). Third, parcellations have been constructed at the individual or group level (15). Group-level parcellations are constructed by combining data sets from a group of subjects to construct a single parcellation, whereas individual-level parcellations are constructed from data sets acquired from a single subject. The structure and properties of a parcellation depend on all of the aforementioned factors. Four of the most commonly used brain parcellations are shown in Fig. 1. As can be seen, location of the boundaries and shape of the regions vary substantially between parcellations. For example, the regions of the parcellation by Shen et al. (10) have more irregular shapes than regions of other parcellations at granularity levels that are roughly the same order of magnitude. On the other hand, the regions of the parcellation by Craddock et al. (17) are more circular and have a relatively uniform size distribution.

Figure 1.

Example brain parcellations (10, 17, 18, 22). Different brain parcellations are constructed using different algorithms, at different granularity levels, and using either a single individual’s data set or data sets acquired from a group of individuals. All these factors impact the structure of the resultant parcellation as is evident by comparing these example parcellations.

However, the ground truth, i.e., the exact location of boundaries between brain regions where microstructure, functional activity, connectivity pattern, or topographical representation of the brain tissue show sharp transitions is not known. In the absence of ground truth to compare against, it is not trivial to determine which brain parcellations are more accurate in demarcating the brain into functionally segregated regions. Due to this reason, current evaluation approaches compare the properties of a “hypothetical” ideal parcellation to properties of a brain parcellation that is to be evaluated (15, 18). In other words, brain parcellations are evaluated based on how well they conform to the desired characteristics of an ideal parcellation. An ideal brain parcellation is a partitioning of the brain tissue into regions that consist of voxels with highly similar functional activity or functional connectivity profiles and sharp transition in these properties across the boundaries between regions. In addition, the robustness of brain parcellations has been studied by quantifying their reproducibility across different individuals, reliability between different scans from the same individuals, and consistency between different initializations of the parcellation algorithm. Finally, brain parcellations have been evaluated on how well they capture known properties of the brain organization, such as microstructure of the brain or language laterality (19), using sources of information that were not used for the construction of the parcellation such as microstructure maps or task-evoked activity. A variety of methods and measures have been used in the literature to quantify these characteristics. A comprehensive review of existing evaluation strategies is lacking in the neuroimaging community to shed light on major themes, their purpose, and role in characterizing a parcellation.

Furthermore, it is unlikely that a ground truth parcellation as described earlier exists. Adjacent locations in the brain might differ in one property (e.g., microstructure) and not others (functional activity, connectivity pattern, and topographical representation) (1). So, brain parcellations are generally regarded as a tool to analyze imaging data sets rather than a method of uncovering some ground truth. However, existing strategies for evaluating parcellation approaches do not answer the question most relevant to researchers in the larger neuroimaging community who use brain parcellations to analyze their data sets: which brain parcellation maximally reveals the signal of interest? For example, a study interested in determining the functional connectivity between brain regions that is altered in schizophrenia needs a brain parcellation whose region-wise connectivity patterns maximally differentiate between healthy subjects and subjects with the disorder. Current parcellation evaluation approaches are not designed to answer such questions. The question of which brain parcellations are more suitable for different applications has thus remained largely unexplored.

In this manuscript, we review different evaluation approaches used in the literature and provide a taxonomy of existing approaches. The taxonomy captures the different properties of ideal brain parcellations that are studied. We then discuss how different evaluation approaches have been used to choose the optimal parcellation granularity. Finally, we propose that augmenting evaluation of brain parcellations with a teleological approach is necessary to identify appropriate brain parcellations for different applications. We argue for a context-based evaluation approach where utility of brain parcellations is assessed for different applications and optimal parcellations for different contexts are identified. We conclude by proposing three key directions for future research with the potential to improve upon the existing evaluation approaches.

METHODS OF EVALUATION

Brain parcellations constructed from fMRI data sets use data mining algorithms to group voxels with similar time series or functional connectivity profiles, assuming that resultant regions serve distinct functional roles (20). These brain parcellations have been constructed using different algorithms, at different granularities, and from data sets acquired either from a single individual or a group of individuals (15). The structure and properties of the resultant brain parcellation depend on all of these factors.

One important question is which brain parcellation more accurately delineates the brain into functionally relevant regions. In the absence of ground truth, brain parcellations have been evaluated on their effectiveness and reliability to parcellate the brain into regions of voxels with similar functional activity or connectivity profiles. The effectiveness of parcellations has been evaluated by assessing their conformity to characteristics of an ideal parcellation. Reliability, on the other hand, is assessed by systematically quantifying variation of parcellations due to stochasticity that is either inherent to the parcellation algorithm or stems from stochasticity present in the data sets used in the construction of a parcellation. In addition to these evaluation strategies, a supplementary approach is to use external sources of information, i.e., information that was not used in constructing the parcellation, for validating parcellations. Each of these evaluation strategies examines a different set of desired characteristics for brain parcellations. A taxonomy of these strategies is provided in Fig. 2. In the following, we present a description of each approach and the methods and measures that have been used in the literature.

Figure 2.

Taxonomy of evaluation approaches. We categorized evaluation approaches into three categories. Each approach evaluates a different class of parcellation properties. Effectiveness assessment is an evaluation of how well the properties of the parcellation match the properties of an ideal parcellation. Reliability assessment evaluates the reproducibility of parcellations. External validation evaluates the parcellation using other sources of information than the data set used for the construction of the parcellation.

Effectiveness Assessment

Effectiveness assessment evaluates the conformity of a parcellation to characteristics of an ideal parcellation of the brain. One characteristic of an ideal parcellation is high homogeneity within each region. Another characteristic is separation between regions, which measures how dissimilar voxels in different regions are compared with voxels in the same region. In addition, an ideal brain parcellation is one that is capable of assigning voxels to “correct” regions in synthetic data sets (assuming ground truth is available). Methods of assessment of each ideal characteristic are described in Region homogeneity, Region separation, and Supervised evaluation.

Region homogeneity.

An effective parcellation is expected to result in “homogeneous” regions, i.e., regions consisting of voxels that are highly similar to each other. Homogeneity is measured as the similarity between time series of voxels within each region or similarity of their functional connectivity profiles (15, 17, 18, 21–28). Regions consisting of highly similar voxels have higher homogeneity values than regions consisting of voxels with diverse time series or connectivity profiles. In other words, homogeneity is a measure of how purely each region consists of similar voxels. Voxels within more homogeneous regions are hence more likely to serve the same functional role, making homogeneity a desired property for an effective brain parcellation. A summary of measures used for quantifying homogeneity is provided in Table 1.

Table 1.

Homogeneity measures

Measure	Description	References
Pearson correlation coefficient (29)	Pearson correlation coefficient between BOLD time series or functional connectivity profile* (30) from voxels within a region is computed to assess homogeneity. Pairwise values are averaged across voxels within each region. Assumes a linear relationship between variables whose similarity with each other is to be quantified. It should be noted that the Pearson correlation coefficient is insensitive to the magnitude of the variables. Therefore it is suitable for measuring similarity between fMRI time series where magnitude is not relevant. However, for measuring similarity between functional connectivity profiles, magnitude is relevant. χ2 (31) is, therefore, a more suitable measure.	15, 17, 18, 23, 24, 32
Kendall’s coefficient of concordance (33)	Kendall’s coefficient of concordance between BOLD time series or functional connectivity profiles of voxels within a region is computed to assess homogeneity. This measure does not assume a linear relationship between variables whose similarity is to be measured. Measures similarity between all voxels within the same region simultaneously.	26, 32, 34
Percentage of variance explained by the first principal component (22)	Calculates the principal components across voxels within each region. The amount of variance explained by the first component is an indication of how homogeneous a given region is. Higher the amount of variance in the first component, higher is the homogeneity. Assumes a linear relationship between variables whose similarity is to be quantified.	22, 32, 35, 36

Different measures used for calculating homogeneity. From left to right, column 1: measure name; column 2: a brief description of the measure; column 3: studies that have used the measure. BOLD, blood-oxygen-level-dependent; fMRI, functional magnetic resonance imaging.

^*Functional connectivity profile of a voxel is the functional connectivity between that voxel and all other voxels in the brain, Functional connectivity between a pair of voxels is the Pearson correlation coefficient between their BOLD activity.

Voxels in spatial proximity to each other have a natural tendency to have similar functional activity and connectivity. Grouping neighboring voxels together arbitrarily can also result in regions with high homogeneity. For this reason, it is important to account for this high homogeneity. This is accomplished by comparing the homogeneity values of a given parcellation with values calculated for parcellations that arbitrarily group spatially contiguous voxels into regions without considering the similarity between their time series (15, 17, 22). It is important to ensure that the number of regions and distribution of region sizes for randomly generated parcellations are at least similar if not identical to that of the original parcellation since homogeneity values are expected to be higher for smaller regions (Fig. 3). The reason is that adding more voxels to a region increases variability and results in lower homogeneity (22). An effective parcellation method must result in a parcellation that has significantly higher homogeneity values than those of random parcellations.

Figure 3.

Homogeneity maps. Example homogeneity maps at two different granularity levels constructed using the method developed by Schaefer et al. (18). Homogeneity values of each region are shown with different colors. A: parcellation of the brain into 100 regions. Homogeneity values are higher in the occipital lobe and lower in the orbitofrontal and ventral temporal cortices. B: parcellation of the brain into 1,000 regions. Compared with A, homogeneity values are on average higher since the regions are smaller.

Homogeneity is typically averaged across all brain regions and reported as a single value. However, regional homogeneity is not uniform across the brain, i.e., different regions have different homogeneity values. One study, for example, has reported lower homogeneity values in the ventral temporal lobe and orbitofrontal cortex, possibly due to lower signal to noise ratio (SNR) in those areas (18). Due to this we argue that it is a useful practice to report homogeneity values for different regions separately, possibly in the form of a map (Fig. 3) as done by Schaefer et al. (18). Such a map makes it possible to compare different parcellation methods in more detail. Moreover, spatial variability of homogeneity values necessitates the development of homogeneity measures that are robust to variations in SNR levels, which can potentially be achieved by properly normalizing homogeneity of each region by its SNR.

Region separation.

High homogeneity is a necessary criterion for an effective parcellation, but it is not sufficient. An effective parcellation is expected to also result in well-separated regions. Voxels assigned to different regions are expected to be dissimilar compared with voxels assigned to the same region. The separation between regions is assessed using region separation criteria developed by the data science community, also known as cluster validation indices (37). Region separation criteria quantify the balance between within- and between-region similarity across voxels provided as a single index and they require the calculation of two components (37). The first component, known as cohesion, captures the similarity between voxels assigned to the same region and is conceptually similar to region homogeneity. The second component, known as separation, captures the similarity between voxels assigned to different regions. The same measures of voxel similarity used for calculating region homogeneity are also used to quantify the two components.

Several cluster-separation criteria that are used for parcellation validation are presented in Table 2. Some of the commonly used measures such as the Silhouette index and clustering quality index take values in a normalized range that facilitates interpretation of the results where the extremes represent best- and worst-case scenarios (51). The lower extreme of the range indicates no separation between regions, i.e., voxels assigned to the same region are as similar to each other as voxels assigned to different regions. The higher extreme indicates perfect separation, i.e., voxels assigned to the same region are maximally similar and voxels assigned to different regions are minimally similar.

Table 2.

Region separation criteria

Measure	Normalized Range	Description	References
Silhouette index (38)	[−1, 1]	Calculated as the difference between the average within region similarity and maximum between region similarity. Is computationally heavy and is sensitive to outliers.	15, 17, 21, 39–42
Modified Silhouette index (43)	[−1, 1]	Conceptually similar to the Silhouette index. Developed for quantifying cluster separation across both individual and group level parcellations simultaneously.	43, 44
Clustering quality index (45)	[0, 1]	Calculated as the difference between the average within region similarity and the average between region similarity. It is a less conservative measure than the Silhouette index as it uses average between region similarity instead of maximum between region similarity.	45, 46
Dunn index (47)	Not normalized	Calculated as the maximum between region similarity to the minimum within region similarity. It provides a lower bound for cluster separation. Is computationally heavy and is sensitive to outliers.	48, 49
Functional clustering index (23)	Not normalized	Conceptually similar to the Dunn index, but more robust to outliers as quantiles are used instead of minimum and maximum.	23
Davies-Bouldin index (50)	Not normalized	Calculated as the average similarity between each region and its most similar one. Unlike other measures, lower values indicate better separation between regions.	49

Different measures used to assess region separation. From left to right, column 1: measure name; column 2: normalized range if the measure is normalized. Normalization facilitates interpretation; column 3: a brief description of the measure; column 4: studies that have used the measure.

Different cluster separation criteria, while using different mathematical formulas, are found to be in general agreement with each other about how well separated and compact the grouping structure of a data set is in contexts other than brain parcellation (37). However, simulation studies, where the ground truth about the grouping structure of the data is known, have shown that the partionings of data sets with high Silhouette scores are the most similar to the true partitioning of data sets that include noisy data points and consist of groups of unequal sizes (37). Both characteristics are known to be present in fMRI data sets (32, 42, 52). Therefore, the Silhouette index is a more robust measure for cluster separation than other measures listed in Table 2. Adopting Silhouette index as the single measure used for measuring cluster separation will further facilitate direct comparison between different parcellation studies.

Similar to homogeneity assessment, region separation values must be compared with values from random parcellations whose number and size of regions match those of the parcellation under evaluation as performed in a couple of studies (15, 32), although this is not common practice among the majority of parcellation studies.

Supervised evaluation.

Supervised evaluation is comparing the parcellation under evaluation with a “standard parcellation.” Ideally, the standard parcellation would be the “true” parcellation of the brain, which is not known. Therefore, standard parcellations are constructed by creating synthetic data sets where a parcellation is first defined and simulated time series for voxels within each region are generated such that voxels within each region are assigned highly similar time series and voxels that belong to different groups have dissimilar time series. The parcellation algorithm is then applied to this synthetic data set and the resultant parcellation is compared with the synthetic standard parcellation using parcellation similarity measures listed in Table 3. The challenge with this approach is in creating synthetic time series that capture relevant aspects of BOLD activity. Multiple themes are used to construct synthetic fMRI activity. One theme in the literature is to assign a shared time series to voxels within each region, corrupted by independent additive noise. Typical time series used in the literature are sine waves at different phases (65), waveforms generated by an autoregressive process (43), waveforms generated by an autoregressive process convolved with the hemodynamic response function (66, 67), and averaged real fMRI time series from a few spatially distant regions (25). Another theme is to create synthetic regions by assigning a common phase to real fMRI time series of all voxels within cytoarchitectonically delineated regions (44) or to impose a correlation structure aimed at mimicking the spatiotemporal dynamics of the hemodynamic response onto real fMRI time series with their autocorrelation removed to generate a synthetic data set. Removal of autocorrelation is referred to as prewhitening (68, 69) and it eliminates spurious pairwise correlations that are due to autocorrelations (70). Real fMRI time series are better candidates for synthetic data sets than simulated time series as they capture the spatiotemporal structure of real data sets, e.g., spatial autocorrelation, hemodynamic response, and temporal frequency, more accurately (66). Since synthetic data sets are not deemed to fully capture the spatiotemporal structure of BOLD activity, performance of any parcellation algorithm when applied to these data sets is to be treated as a higher bound for its performance when applied to actual data sets.

Table 3.

Parcellation similarity measures

Measure	One-to-One Correspondence	Adjustment for Chance	Normalized Range	References
Percent agreement	Yes	No	[0, 1]	42
Jaccard index	Yes	No	[0, 1]	53
Dice similarity coefficient	Yes	No	[0, 1]	15, 17, 23–25, 41, 49, 54
Adjusted Rand index	No	Yes	[−1, 1]	15, 23, 55
Probabilistic Rand index	No	No	[0, 1]	44
Adjusted mutual information	No	Yes	[0, 1]	55
Normalized mutual information	No	No	[0, 1]	22, 56, 57, 58
Variation of information	No	No	Not normalized	59–64

Different measures used to compare parcellations. From left to right, column 1: measure name; column 2: whether or not the similarity measure requires a one-to-one correspondence between regions of the two parcellations that are to be compared. When one-to-one correspondence is required, the two parcellations must have the same number of regions; column 3: whether or not the measure is adjusted for chance. Chance adjustment corrects the measure for baseline values that are expected from comparing two random parcellations; column 4: normalized range of the measure if the measure is normalized. Normalization facilitates interpretation; column 5: studies that have used the measure.

Some of the parcellation similarity measures listed in Table 3, such as adjusted rand index and adjusted mutual information, correct the measure for baselines that are expected to be observed from comparing two random parcellations. Simulations have shown that adjustment for chance is not necessary when the number of voxels is more than 100 times the number of regions (51). This is typically the case for parcellations at coarser levels. When this criterion is not met, however, measures that are adjusted for chance must be used, or resultant values must be compared with values from comparing randomly generated parcellations with similar number and size of regions to the two parcellations that are compared.

Reliability Assessment

Reliability assessment is an evaluation approach that assesses how reproducible brain parcellations are. Reproducibility is typically measured as the variability of the parcellation results. Parcellations that are less variable are more reproducible and consequently deemed more reliable. Variability in parcellation results can be attributed to three factors: 1) stochasticity that is inherent to the parcellation algorithm, referred to as intrascan variability (42, 71), 2) stochasticity in the different scans of the same individual, referred to as intraindividual variability (15, 54), and 3) variability due to subject-specific idiosyncrasies, referred to as interindividual variability (15, 24, 72). Variability quantification entails comparing several parcellations produced by the same method. The more variable the results are, the lower is the degree of agreement between them as measured by parcellation similarity measures (Table 3).

In Reliability Assessment, approaches to quantifying variability from different sources are discussed. Assessment of reliability as described here is similar to the concept of “stability of clustering” in the data mining literature (73, 74). The stability of a clustering scheme refers to the consistency of the results when a clustering algorithm is applied to the same data set multiple times or when it is applied to multiple data sets sampled from the same distribution. Stability analysis is a popular method for evaluating partitioning algorithms (74, 75).

Intrascan variability.

Intrascan variability is computed to assess the impact of stochasticity inherent to the parcellation algorithm on the resultant parcellation. Intrascan variability is quantified by applying the same parcellation algorithm to the same scan multiple times and comparing the results using the parcellation similarity measures listed in Table 3 (42, 71). Intrascan variability is relevant for parcellation algorithms such as K-means and mixture models that require random initialization where initial assignment of voxels to regions is random (76). The algorithm then iteratively calculates the similarity between each voxel and voxels within each parcel and re-assigns voxels to the parcels they are most similar to in search of a solution that maximizes similarity between voxels assigned to the same region. The algorithm is initialized several times, each time resulting in a different solution. Intrascan variability is a measure of variance across these different solutions. There are several ways to construct a single parcellation from different solutions produced by different initializations. One approach is to pick the solution that maximizes some criterion such as likelihood. Another approach is to combine all solutions into a single aggregate parcellation that captures the commonality between all single solutions (76). The amount of intrascan variability is relevant information for assessing the quality of the final parcellation. Parcellation algorithms that are more sensitive to the initial condition are less likely to converge on the optimal solution for the parcellation problem and require a higher number of initializations. It is concerning that reporting intrascan variability is currently not common in the brain parcellation literature. Parcellation algorithms with higher intrascan variability can benefit from strategic initialization techniques that increase the propensity of the algorithm to find the global optimal solution although their utilization in the brain parcellation literature is underexplored (77, 78).

Intraindividual variability.

Intraindividual variability is computed to assess variability across parcellations constructed using data sets from different scans acquired from the same subject. Parcellations derived from different scans of the same subject using the same algorithm can be different if the information captured in the different scans is variable. This is driven by different sources such as scanner-induced noise (79–81), physiological noise (81, 82), menstrual cycle (83), and subject hydration at the time of scan acquisition (84–86). Additional variability is also introduced by the cognitive state of the subject, such as whether or not they were fed or caffeinated before scan acquisition (36, 87), the time of scan acquisition (88), drowsiness (72, 89, 90), sleep deprivation (91), subjects’ mood at the time of scan acquisition (92), and the cognitive tasks they performed before scan acquisition (93).

Several studies have computed intraindividual variability (15, 21, 24, 49, 54, 72, 82, 94, 95). Reported intraindividual variabilities are less than 40% for the majority of algorithms, meaning that parcellations produced from different data sets of the same subject were at least 60% in agreement with each other. Intraindividual variability changes as a function of parcellation granularity but the degree and manner of the change are highly dependent on the parcellation algorithm. For the K-means and hierarchical clustering algorithms, the difference in intraindividual variability at two different granularity levels was found to be as high as 10%, whereas for others it is negligible (<5%) (15). Intraindividual variability is also heterogeneous across the brain and is higher in the sensory-motor cortex compared with the associative cortex (24, 72). Finally, intraindividual variability reduces as the duration of data used for parcellation increases. Previous studies have reported that at least 30 min of resting-state data is required to obtain less than 15% intraindividual variability and intraindividual variability typically reaches asymptotic values less than 5% for more than 90 min of resting-state data (36, 72).

Intraindividual variability can potentially be improved by reducing variability in the pairwise correlation between voxels across different scans. One way to do so is to combine data sets collected on different days to average out the variability introduced by the cognitive state of the subject (8). Choice of preprocessing parameters (96, 97) and in particular the cut-off frequencies of the temporal filter (96) have been shown to affect the variability of pairwise correlations. It is reasonable to expect that using a preprocessing pipeline that minimizes pairwise correlation variability would also decrease the intraindividual variability of functional parcellations. Whether or not this is the case is the subject of future research.

Currently, the common way to report intraindividual variability is to report a single value that calculates the degree of agreement between two parcellations of the same individual. However, as mentioned previously, a couple of studies have shown that intraindividual variability is not homogeneous across the brain but is higher in the sensory and motor cortices (24, 72). Whether or not such variability patterns are specific to the parcellation algorithms used in these studies is not known. We recommend that intraindividual variability be reported for different locations in the brain, for example as a map (24, 72) instead of reporting a single number that captures the overall degree of agreement between different parcellations, although this is not currently the common practice. Such maps can potentially be informative about the possible sources of intraindividual variabilities and can lead to remedies for reducing them.

Interindividual variability.

Interindividual variability quantifies variations in parcellations of different individuals constructed using the same algorithm. Individual brains share a common functional organization, but the exact size, location, and arrangement of brain regions vary from subject to subject (24). A reliable parcellation is expected to capture the shared organizational structure of the brain among subjects despite individual differences. Parcellation methods with lower interindividual variability are deemed more reliable.

We note that caution must be taken in interpreting what interindividual variability captures. Interindividual variability is caused by “real” interindividual differences, nuisance intraindividual variability, and inaccuracies introduced by the parcellation pipeline, including the parcellation algorithm as well as imaging noise and inaccuracies in preprocessing of the data (79, 98). Hence, interindividual variability is not going to be zero, even if the parcellation algorithm did a perfect job. To remove the contribution of true interindividual differences and intraindividual variability, many studies have quantified the agreement between group-level parcellations (15, 18, 21, 23, 25, 28, 42, 49). Group-level parcellations are constructed by combining data sets from several individuals so that intra- and interindividual variabilities are averaged out. Intergroup variability is generally not sensitive to parcellation granularity (15, 23, 28), although it decreases as granularity increases for parcellations constructed using the spectral clustering algorithm (28).

Group-level parcellations require individual brains to be registered to a common stereotaxic space. Registration is done by aligning T1-weighted images of individual brains based on their morphology. Registering individual brains based on their functional characteristics instead of morphology, however, was found to result in a significant improvement in the reproducibility of the parcellations across different groups (19). The reason is that the brain anatomy and its function are not tightly correlated. When aligned based on anatomy, corresponding voxels across different brains are not guaranteed to be functionally similar, resulting in seemingly increased intergroup variability of the functional parcellation.

The structure of brain parcellations was shown to depend on age (23, 99). Accordingly, when comparing parcellations constructed from two groups of individuals that do not match in age, some portion of mismatch can be attributed to a true difference between the two groups, and would not reflect the reliability of the brain parcellations. To minimize the confounding effect of age, group-level parcellations that are to be compared should be constructed from groups of individuals with the same age distributions.

Measuring true interindividual variability is also of interest (24, 36, 72, 94, 98, 100). Robust estimation of interindividual variability requires minimizing the contribution of intraindividual variability to the difference between two individual level parcellations. For example, if two subjects were scanned at different times of the day, their fMRI activity is systematically impacted by their circadian rhythm (88). It is not known whether such changes can systematically alter the subjects’ individual-level parcellations and manifest as interindividual variability if not properly controlled for. To remove intraindividual variability, several scans from each subject are combined so that intraindividual variabilities are averaged out (36, 72, 79, 94, 98, 100). Alternatively, intraindividual variability can be explicitly estimated using hierarchical statistical models that can achieve comparable performance with only a single scan (24). Hierarchical statistical models estimate intrasubject variability simultaneously with interindividual variability from the data and have been shown to obviate the need for combining multiple scans.

Interindividual variability is on average greater than intraindividual variability (19, 24, 94). Unlike intraindividual variability, however, interindividual variability is higher in the associative cortex compared with the sensory-motor cortex (19, 24, 36, 49, 72, 94, 98, 100). A possible explanation for this observation is that across individuals, functional connectivity itself is more variable in the associative cortex than the sensory-motor regions (79). It has also been reported that functional connectivity profiles of voxels in parts of the brain that are linked to behavioral interindividual differences are more variable, indicating a possible link between interindividual variability of brain organization and behavior (5, 79, 101). In fact, one study has reported that interindividual variability in spatial topography of the brain regions is highly correlated with several behavioral traits (70).

External Validation

The external validation approach to evaluation uses external sources of information that were not used for the construction of the parcellation. The rationale is that a brain parcellation that accurately partitions the brain into functionally meaningful regions is expected to be predictive of known properties of the brain the parcellation method was agnostic to. External validation, in essence, assesses the generalizability of brain parcellations to also capture the organization of the brain as imposed by features that were not used for parcellation (102). The three external sources typically used for evaluation of parcellations are 1) microstructure of the brain, 2) task-evoked activity, and 3) personal characteristics. In what follows we will briefly describe each source and how it is used.

Alignment with microstructure.

Brain parcellations are evaluated on how well they align with microstructural maps of the brain. The rationale behind this approach is the observation that different characteristics of the neural tissue are highly correlated with each other (20). Several studies focusing on sensory and motor areas have shown that voxels with similar cytoarchitecture also have similar functional activity (103, 104).

Alignment with microstructure is typically performed by comparing a selected set of regions from a parcellation with regions defined based on myeloarchitecture measured in vivo (105) or Brodmann’s cytoarchitectonically defined regions (15, 18, 22, 42, 54, 72, 98). Brodmann’s map is based on microscopic features including formation of cortical layers, arrangement of cells in clusters and across cortical columns, and presence of certain cell types (106). Myeloarchitecture of the nervous tissue measured in vivo captures the myelin density in the brain tissue that varies across the cortex (105, 107). Alignment with microstructure is done both qualitatively by visual inspection (18, 22, 42), and quantitatively using parcellation similarity measures such as percent agreement (108), Dice coefficient (15, 95), and distance between boundaries (18, 54).

Comparison of functional parcellations to cyto- and myeloarchitectonically defined regions have revealed several interesting trends. First, agreement between functional parcellations and microstructurally defined regions increases as parcellation granularity increases (15). Second, brain parcellations that are based on the location of major sulci (e.g., the AAL or Desikan brain atlases) and not from functional activity are in higher agreement with cytoarchitectonically defined regions than functional parcellations (15). This observation poses an enigma since the boundaries of only a few cytoarchitectonically defined regions align with macrostructural landmarks that are used for constructing atlases such as the AAL and Deskian (109). Although these results might reflect poor performance of parcellation methods, another possible explanation is that functional parcellations and microstructurally defined regions partition the brain across different organizational boundaries. For example, in the visual cortex, functional parcellations are in alignment with the retinotopic map of the visual cortex and not its cytoarchitecture (18). Another example is a study done by Gao et al. (110) in nonhuman primates. They observed that regions delineated by parcellation algorithms in the motor cortex were in poor alignment with cytoarchitectonically defined primary and premotor regions. The reason is that both premotor and primary motor regions include a motor map of the body and the parcellation algorithm grouped voxels that are involved in controlling the same parts of the body in each region together, even though they are cytoarchitectonically distinct (110). These examples demonstrate that misalignment between parcellations based on functional activity and microstructurally defined regions is not necessarily due to inaccuracies in the parcellation but occurs because these modalities capture different characteristics of the neural tissue (1). A misalignment between functional parcellations and microstructural maps calls for further investigation before any conclusions regarding parcellation quality can be made. Although a certain degree of agreement between the two is reassuring, owing to the fact that different characteristics of the neural tissue are highly correlated (20), disagreement between them does not necessarily reflect poor parcellation performance.

The study by Gao et al. (110) also demonstrates the utility of animal models for evaluating functional parcellation methods. The main advantage of using animal models such as monkeys and rats is that the microstructure of their brains has been extensively studied using invasive histology and tracing techniques. Availability of invasive perturbation methods such as optogenetics and microstimulation also allows examination of causal functional links between brain regions and how it relates to functional connectivity captured by fMRI (111). Finally, more powerful MRI scanners [e.g., 9.4 T (112)] are available for rodents that allow for imaging functional activity at higher spatial resolutions (e.g., 0.2 mm) compared with the available technology for human imaging studies (e.g., 3-T scanners, 2 mm spatial resolution used for the Human Connectome Project). On top of the methodological advantages of animal models, the overall functional organization of animal brains appears to be consistent across species. For example, the default mode network has also been identified in rats and monkeys (113). As another example, functional connectivity patterns in the hippocampal-prefrontal network of rats are consistent with connectivity patterns observed in humans (114). These studies demonstrate that animal models offer a promising translatable approach to research on human brain parcellation. For example, functional connectivity and anatomical connectivity captured by invasive tracing in rats are shown to be tightly correlated (115). On the other hand, a detailed comparison of functional and anatomical connections in the default mode network of the monkeys has revealed that 14% of functional connections do not have corresponding anatomical connections (116). These studies demonstrate how animal models that combine invasive methods of studying the nervous tissue with fMRI can provide invaluable insight into the extent to which functional connectivity is correlated with microscale features of the brain.

Alignment with task-evoked activity.

By grouping voxels with similar BOLD activity, brain parcellation algorithms are expected to partition the brain into functionally relevant regions. As a result, task-evoked activity is expected to align with regions of the parcellation that have relevant functional roles to the task (117). It should be noted however that comparison with task-evoked activity is only possible for parts of the brain that can be robustly activated by known tasks such as the sensory and motor cortices. It cannot be assumed that a parcellation that aligns well with task-evoked activity in some parts of the brain, is also accurate in parcellating other parts of the brain for a variety of reasons such as heterogeneity of fMRI signal to noise ratio (42) and differences in granularity of task activation in different parts of the brain. For evaluation using task-evoked activity, we recommend collection of task-evoked fMRI data for a variety of tasks that activate different parts of the brain including the associative cortex, sensory-motor cortex, and subcortical structures.

One way to examine the alignment of a given parcellation with task-evoked activity is to qualitatively compare the alignment of regions of the parcellation with regions that are activated by a task, for example visually comparing the arrangement of regions located in the visual cortex with regions that are activated when subjects were presented with a visual stimulus (42). Another way is to quantitatively compute the agreement between regions of the parcellation and regions defined using task-evoked activation. One method of such quantification is to measure agreement between the two sets of regions using parcellation similarity measures (15, 36, 54, 55). Defining regions using task-evoked activation, however, requires thresholding activation z-scores and must be performed at different thresholds to ensure results are not dependent on the choice of threshold. Another approach is to measure the homogeneity of task-evoked activation for each region as the variance of z-scored task-evoked activity of voxels assigned to that region (18, 24, 72). This method obviates the need for thresholding activation z-scores.

One study has reported that alignment between the brain parcellation and task-evoked activity is task-dependent, meaning it is higher for some tasks than others (24). One interpretation for this observation is that the parcellation is more accurate in some parts of the brain than others. However, one study has shown that a functional parcellation of the brain is better aligned with preoperative functional maps estimated using invasive electrical stimulation compared with task-evoked activity measured by fMRI (94). One possible explanation for this intriguing observation is that the intraindividual variability of task-evoked activity is higher than a parcellation constructed from resting-state fMRI (94) which is in line with the observation that intraindividual variability is higher in task-evoked fMRI activity compared with resting-state fMRI activity (118). Collectively, these results suggest that alignment with task-evoked activity as an evaluation method should be augmented with examining the reliability of the task-evoked data sets in tandem. One way to minimize the contribution of brain state fluctuations to intraindividual variability and consequently misalignment between functional parcellations and task-evoked activity is to collect resting-state and task-evoked activity within the same session (119). Another potential solution is to use tasks with higher test-retest reliability such as working memory, theory of mind, emotional face matching, and monetary incentive delay tasks (118), although this method reduces the number of tasks that can be used for evaluation. Further research is needed to identify what types of tasks or data collection protocols produce task-evoked fMRI data sets that are suitable for the evaluation of brain parcellations.

Prediction of personal characteristics.

Organization of the brain is dependent on individual characteristics such as gender (120, 121), age (122), handedness (121), language laterality (121), performance in complex cognitive tasks such as working memory (123), and personality traits (124). The ability of a parcellation algorithm to capture organizational differences that predict personal characteristics indicates their utility in studying the neural substrates of different personal characteristics. To systematically quantify the predictive power of a given parcellation, some property of the parcellation, such as the size of its regions, is used as an independent variable to predict one or more personal characteristics. Predictive power is quantified using a variety of approaches such as correlation analysis (19, 24), statistical tests (94), and classification using machine learning algorithms (15, 49).

This approach of evaluation has produced promising results. Classification of subjects into male and female is done at 60% to 85% accuracy levels using both parcellation topography (49) and network topography (15) as the independent variables. Network topography is calculated for network models of the brain constructed using regions as nodes and the pairwise correlation between their time series as edges. In both cases, higher performances are achieved at higher parcellation granularities. Another personal characteristic is language laterality, which captures the asymmetry of fMRI activity across hemispheres during semantic classification tasks. Language laterality is significantly correlated (r = 0.6) with how lateralized functional connectivity between regions is (19). Laterality in the structure of the parcellation itself is also significantly different in right-handed versus left-handed individuals (94). Performance at cognitive tasks such as working memory and reading is also modestly correlated with both parcellation topography (r = 0.13) and region size (r = 0.08) (24).

Evaluating parcellations using any independent variables that require computation of pairwise functional connectivity between regions has been done using resting-state functional activity. However, recent work suggests that behavioral individual differences are amplified by functional connectivity measured using task-evoked activity compared with resting-state (6). Thus, using resting-state activity to calculate the independent variable of interest reduces its predictive power. In addition, some tasks such as gambling, working memory, and emotion processing tasks highlight interindividual differences better than others such as motor, language processing, social cognition, and relational processing tasks. Due to this, when evaluating parcellations using independent variables that require calculating region-wise functional connectivity, using task-evoked activity collected from a battery of tasks is preferred.

The list of personal characteristics that can potentially be used for the evaluation of brain parcellations is quite long. To name a few, characteristics such as age (125–127), dichotic listening (121), and numerous personality traits (e.g., see Refs. 8, 9, 70, 124, 128, and 129) have been linked to the organization of the brain. Each of these characteristics can be used for the evaluation of brain parcellations.

DETERMINING THE OPTIMAL NUMBER OF REGIONS

There is no widely agreed-upon number of regions or granularity level for parcellating the brain (Fig. 4). Parcellation granularity depends on the method of choosing the number of regions. Although several parcellation methods are capable of estimating the optimal number of regions that explains a given dataset from the data itself, other methods require the user to specify the number of regions (76). It is not known a priori what parcellation granularity describes the data better. Consequently, parcellations studies have developed two data-driven approaches to identify the optimal number of regions.

Figure 4.

Granularity of available parcellations. Different studies have parcellated the brain at different granularity levels (10, 17, 21–24, 27, 28, 32, 36, 43, 49, 53–57, 66, 94, 95, 131, 181–187). When a range of granularities was reported, the minimum and maximum granularity were used in this figure to reflect the whole range.

The first approach is to pick the number of regions that maximizes region separation. In this approach, parcellations at several different granularity levels are constructed and region separation for each of them is calculated. The granularity level that produces the highest separation between regions is chosen (43, 44, 95, 130, 131). Of the measures listed in Table 2 for measuring region separation, simulations have shown that the Silhouette and Davie–Bouldin indices are the most successful in retrieving the true number of groups in simulated noisy data sets (37).

The second approach is to pick the number of regions that maximizes reproducibility or equivalently minimizes the variability of parcellation. Minimizing intrascan (42) and interindividual (25, 44, 55, 132) variabilities are both used for choosing the optimal number of regions.

The rationale behind choosing a granularity that minimizes variability is that choosing the wrong number of regions introduces an extra source of variability. More specifically, if the number of regions is lower than the true number of groups in the data set, then several of such true groups are merged by the algorithm to construct bigger ones. Since the merge is not based on a true underlying grouping structure in the data, the choice of which of the true groups are to be merged is likely to be arbitrary. Accordingly, applying the algorithm to different samples of data or starting it from different initial conditions results in the merging of different sets of true groups and different parcellation results. Conversely, when the number of regions is higher than the true number of groups, true groups are arbitrarily split into smaller groups, and depending on the specific data set and initial conditions, the split will happen differently each time the algorithm is applied. So, if the number of regions is not optimal, resultant parcellations will be highly variable (74). However, it has been shown that the presence of certain grouping structures can cause variability to be minimal for a number of groups other than the true number (73). For example, if a data set consists of three true groups, where two of the three groups are more similar to each other, choosing to cluster the data into two groups will always result in the algorithm merging the two more similar groups together. So when choosing the number of regions by minimizing variability, the analysis must be complemented with other criteria such as region separation or other methods commonly used in the field of data sciences (for a review, see Ref. 133).

CONTEXT-BASED EVALUATION

Although the aforementioned methods for evaluating brain parcellations are necessary for characterizing different properties of brain parcellations, the brain imaging community that uses brain parcellations to analyze various types of data sets, is also concerned with which brain parcellations are better at highlighting the signal of interest in a given study. A brain parcellation is required to be of acceptable quality as assessed by the methods discussed in methods of evaluation to be deemed reliable for use, but it is not clear however, which brain parcellations are a better choice for a given application. For example, a study aimed at identifying changes in functional connectivity due to a neurological disorder needs a brain parcellation with region-wise connectivity patterns that maximally differentiate between the control group and subjects with the disorder. Available literature on brain parcellation is focused on assessing the quality of brain parcellations and is agnostic to the application the parcellation is intended for.

We argue that a teleological approach should be taken with regard to selection of brain parcellations for a given study. Brain parcellation is in nature a clustering problem and needs to be dealt with as such. In recent years, researchers in the data science community have argued that a clustering problem should not be defined independent of the application it is intended for (134). The proper evaluation approach for any clustering algorithm should similarly be geared toward the specific application of the clustering. Likewise, we argue here that brain parcellations should be evaluated in the context of their intended application along with the evaluation approaches discussed previously. Parcellation algorithm, granularity, and whether it was constructed from data sets from single or multiple subjects are relevant parameters that need to be optimized for any given application.

Furthermore, we believe building a “taxonomy of parcellation problems” (134) helps identify suitable parcellation and evaluation methods for different application categories. Such an approach has been fruitful in different branches of engineering. The taxonomy should include categories of major applications for brain parcellations. Different parcellations can then be compared and evaluated in the context of each application category. We further argue that such comparison should be performed in a data-driven manner. Data-driven approaches are more apt to deal with fMRI data sets whose structure is not fully understood (135): “In complex systems that do not lend themselves to intuitive models, data-driven modeling and hypothesis generation is key to understanding system behavior and interactions.” The choice of parcellation can be thought of as a step in preparing fMRI data for analysis, which needs to be optimized for the particular application. Optimization entails trying multiple parcellations and reporting on their performance, as done by a few studies already (5, 124, 136–140) (Fig. 5A). As more parcellations are used and tested in the context of real applications, a more complete picture of what parcellations might be more suitable for different application categories is expected to emerge (Fig. 5B). For example, in one context parcellations with low granularity levels might be optimal regardless of algorithm or the number of subjects used for construction of the parcellation, whereas in another context group level parcellations at high granularity levels constructed using a specific algorithm might be the optimal choice.

Figure 5.

The proposed teleological approach to brain parcellation evaluation. A: parcellations constructed using different algorithms, at different granularity levels, and constructed at individual- or group levels are used in the context of interest, e.g., calculating function connectivity between regions, assessing test-retest reliability of data sets, or constructing network models of the brain. The signal of interest is then quantified. For each context, the parcellation that maximizes the signal of interest (Parc*) is picked for that application. B: a toy model of a hypothetical catalog of applications. Different parcellations that are suitable for different applications (Parc*) are identified. Potentially, for studying different disorders for example, different parcellations would be used. It is possible that for some applications more than a single optimal parcellation is identified. For example in this toy model, parcellations with finer granularity levels are equally good at identifying biomarkers for depression.

Evaluating parcellations in the context of their application is currently not the norm. Multiple studies however have demonstrated the importance of the teleological approach as well as the necessity of using data-driven methods (24, 136, 139, 140). One study that best demonstrates how a data-driven teleological approach can identify the optimal parcellation for a given application, compared eight parcellations in their ability to identify associations between functional connectivity within resting-state networks of the brain and individual differences in age, poverty, and cognitive capacity (136). They quantified functional connectivity within the salience, dorsal attention, control, and default mode networks using eight different parcellations. They then calculated the strength of association between each network’s functional connectivity and age, poverty, and executive control in performing an inhibitory control task. They observed that a separate set of parcellations identified significant associations between within-network connectivity and age, poverty, and executive control. For example, a parcellation that revealed a significant association between age and within-network connectivity failed to reveal the relationship between within-network connectivity and poverty or executive control. No single parcellation revealed the relationship between all three factors and within-network connectivity. So, if the investigators of a study interested in identifying correlations between poverty and resting-state networks picked a parcellation that was not capable of revealing the association between the two, they would arrive at the erroneous conclusion that the signal in resting-state networks is not associated with poverty. These results demonstrate the importance of adopting a teleological approach to parcellation evaluation: A parcellation that is successful in revealing the signal of interest in one context, will not be the optimal parcellation in another context. In addition, making prior assumptions about which parcellations will outperform others in a given context is not straightforward as is the case in the study by Bryce et al. (136). A data-driven approach where utility of several parcellations is compared is a more powerful approach in identifying the optimal parcellation in a given context.

Another example is a study by Yu et al. (139) that examined how the image acquisition site (i.e., acquisition protocol and scanner manufacturer) impacts region-wise functional connectivity and to what degree such impact depends on the parcellation used for delineating the regions. To study the site-effect, resting-state fMRI activity from the same cohort of subjects was acquired at four different sites and region-wise functional connectivity for each site was calculated separately using three different parcellation schemes. The results showed that the percentage of connectivity values that were significantly different across sites depended heavily on the parcellation and ranged from 0.04% to 0.2%. In a context where data is collected at multiple sites, using a parcellation scheme that minimizes site-effects would increase statistical power to detect the signal of interest. However, it is difficult to even hypothesize which parcellations would minimize site-effects. A data-driven approach that tests several parcellations can identify which parcellations are the most suitable for multisite studies.

The data-driven teleological approach is useful when, as is the case in the previous examples, it is not clear which parcellation is optimal for a given context. But it also guards against potential biases when it is intuitive to make assumptions about what parcellations might be appropriate for a given application. We will discuss two studies that demonstrate the importance of examining multiple parcellations without making any prior assumptions about which parcellations are the most suitable for a given application. A study done by Kong et al. (24) compared four different parcellations in their ability to predict a comprehensive set of behavioral phenotypes that included cognition, personality, and emotion. Three of the parcellations were constructed at individual level and one was constructed at group level. The individual-level parcellations outperformed the group-level parcellation in predicting behavioral measures of cognition and emotion, but surprisingly the group-level parcellation had higher predictability for the personality measures. These results are counter intuitive since one expects individual-level parcellations to be more predictive of each individual’s unique personality traits compared with a parcellation that combines data sets from a group of subjects. This study demonstrates that making prior assumptions about what parcellations would suit a given application can reduce the detectability of the signal of interest, which in this case was prediction of personality traits. Further research is required to compare other group-level parcellations against individual-level parcellations to see if group-level parcellations in general are a better choice for predicting personality traits.

A second study demonstrating the importance of the teleological approach is from our own group, where we compared test-retest reliability of region-wise functional connectivity profiles using two different parcellations (140). The first parcellation was constructed using a clustering algorithm to group voxels with similar time series into regions. The second parcellation was a geometric parcellation constructed by grouping spatially adjacent voxels into contiguous regions and did not use functional activity for parcellation. Functional connectivity between regions of each parcellation were calculated separately for two data sets acquired at different sessions. Unexpectedly, test-retest reliability was higher for the geometric parcellation that had more circularly shaped regions. These results again defy expectation as in this case the parcellation scheme with the higher test-retest reliability for functional connectivity seemed to be the parcellation method that was oblivious to functional activity. It is possible that in the context of maximizing test-retest reliability of region-wise functional connectivity, parcellations with circularly shaped regions are the optimal choice, although further research is required to confirm this possibility.

We note that the issue of reproducibility of the results must be considered in the process of identifying the optimal parcellation. For any given context and effect of interest, if enough number of parcellations are tested, one of the parcellations is bound to outperform others in revealing the signal of interest or is likely to result in a statistically significant effect. However, the optimal parcellation might be specific to the data set used for its identification and might not generalize to other data sets for the same application, a phenomenon known as overfitting. To avoid overfitting, some form of cross validation or control for type I error must be incorporated into the process of identifying the optimal parcellation. A useful method to ensure reproducibility of the results is to use a subset of the data to identify the optimal parcellation and report its performance in identifying the signal of interest in the rest of the data. For example, if a study is interested in differentiating between a group of subjects with a neurological disorder and a control group, a subset of the subjects from each group can be used for identifying the optimal parcellation. The identified parcellation can then be used to classify the remaining subjects into one of two groups. Another method to safeguard against type I error in cases where the optimal parcellation is to be used to identify significant links between neurological signals and an effect of interest is to correct for multiple comparisons where the signal produced by each parcellation is treated as a single comparison. Although the particular approach to avoiding overfitting depends on the study design, it is a necessary to incorporate some control methods into the process of identifying the optimal brain parcellation in any given context to ensure reproducibility of the results and avoid overfitting.

To further elaborate on what we mean by the taxonomy of parcellation problems, we provide six examples of common applications for brain parcellations and how brain parcellations are or can be evaluated in each context. These examples are only provided for the sake of clarifying our point and are not a complete catalog of applications for brain parcellations.

Brain parcellations are frequently used for reducing dimensionality of MRI data sets that typically consist of thousands of voxels. Reducing dimensionality is mostly relevant in the context of interrogating connectivity profiles of the brain. Examples include identifying the unique connectivity profile of individuals (5), comparing connections in individuals with a neurological disorder to healthy controls in search of biomarkers (e.g., see Refs. 141–146), and studying the impact of drugs on brain connectivity (147). It is however computationally expensive to construct voxel-wise connectivity maps. In addition, voxel-wise maps are more susceptible to noise. Instead, connectivity profiles are typically studied between tens to hundreds of brain regions. Evaluation of a brain parcellation in this context can be done in different ways. One approach is to examine how well the region-wise connectivity patterns represent voxel-wise connectivity patterns (17, 53). Another approach is to quantify the discriminative power of different parcellations, for example to distinguish between different individuals (5) or between groups of individuals with a neurological disorder and healthy controls (138). In this context, parcellations that produce more relevant region-wise connectivity profiles are deemed more appropriate.

Modeling the brain as a network of interconnected regions has provided important insights into the functional systems of the brain in the past two decades (148). Brain networks allow for studying the complex topography of interactions between different regions using graph-theoretic measures. The definition of the regions used for construction of brain networks is provided by brain parcellations. Connections between regions are typically quantified as the functional connectivity between them. Brain networks are typically characterized using graph theoretic measures (148). It is not clear how sensitive graph theoretic measures are to the definition of regions and hence which parcellations result in brain networks that capture the critical aspects of the functional systems of the brain. Initial work has shown that graph-theoretic measures are insensitive to parcellation algorithm (15) or whether the parcellation was constructed from individual data sets or a group of individuals (15) but highly sensitive to parcellation granularity (15, 149–152). These results suggest that when constructing network models of the brain, parcellation granularity is the factor that should be optimized and the parcellation algorithm or whether the parcellation was constructed at individual level might be irrelevant.

Brain parcellations are also used in clinical settings as a diagnostic tool (153–156), or used to predict treatment response in individuals with neurological or mental disorders (137, 157, 158). In this context, parcellations are compared in terms of their predictive power for identifying patients at the early onset of the disease or for the effectiveness of treatment in different individuals. One preliminary study, for example, has used several parcellations to predict the response of patients with social anxiety disorder to treatment (137) and reported that parcellations constructed at lower granularity levels had a higher prediction performance. Although these results are preliminary, they indicate that low granularity parcellations might be the optimal parcellations in the context of predicting outcome of anxiety treatment.

Brain volume measurement, where the volume of different regions of the brain is measured and compared across different groups, is another common application for brain parcellations. Different parts of the brain are shown to atrophy as a result of various neurological disorders such as Alzheimer’s disease (159, 160), multiple sclerosis (161), Parkinson’s disease (162), and schizophrenia (163). Determining the spatial extent of atrophy is crucial in determining the brain structures impacted by the disease and developing hypotheses about the underlying mechanisms of the disorder. The common procedure is to compare the volume of different brain regions in patients to healthy controls. For such analysis, choosing a parcellation granularity that is too coarse or too fine compared with the spatial extent of atrophy would hurt the sensitivity and specificity of the analysis and consequently its statistical power, resulting in misleading conclusions (164). In this context, the parcellation with the highest sensitivity and specificity in delineating the atrophied region is preferred.

Linking behavior or neuropathology to brain organization also requires brain parcellations. In this context, different properties of the parcellation itself, for example, its granularity, are used to investigate the phenomenon of interest. A parcellation is evaluated based on how well its properties can predict different behaviors or distinguish between healthy and abnormal brains. The goal is to find the parcellation with the maximum discriminative power. For example, one study has constructed individual parcellations of the striatum and compared region separation across healthy controls and subjects with autism to find the optimal number of regions that distinguishes between the two groups (142). Other studies have compared parcellations based on how well functional connectivity maps between their delineated regions predicted personality traits (124) or cognitive state of the subject (165). These preliminary results have shown that some brain parcellations are more predictive of behavior than others and potentially provide more insight into the neural substrates of different behaviors.

Longitudinal studies also use brain parcellations to compare different properties of the brain at different time points. Longitudinal studies are used in different contexts, for example, to track changes associated with learning (166), aging (e.g., see Refs. 167 and 168), and progress or remission of a neurological or mental disorder over time (e.g., see Refs. 141, 169, and 170). One of the most common types of longitudinal analyses is to compare functional connections between brain regions at different points in time (141). In this context, it is important to choose a parcellation that produces parcellations with low intraindividual variability. It is also important for the region-wise connectivity profiles themselves to have low intraindividual variability. Otherwise, intraindividual variability can get confused with real changes caused by the factor of interest over time (e.g., aging). For this reason, it is vital to measure intraindividual variability both for the parcellation and for the connectivity profiles between its regions. For example, as already discussed one study has shown that a geometric parcellation with more circularly shaped regions produces more reproducible functional connectivity profiles than a functional parcellation constructed using hierarchical clustering (140). In addition, functional connectivity profiles are more reproducible at lower granularities.

Despite the recent advances in developing new parcellations, until these parcellations are used in various different contexts and their relative strengths and weaknesses compared and contrasted, it is not clear which parcellations would be suitable for a given application. The few studies that have compared multiple parcellations in different contexts clearly demonstrate the dependence of detectability of signals of interest on the choice of parcellation. Therefore, we believe an important step in advancing brain parcellation research is context-based evaluation. Such an approach will increase the statistical power to detect signals of interest that would otherwise be missed.

FUTURE DIRECTIONS

An implicit assumption made about the existing parcellations is that the functional organization of the brain as captured by fMRI is stationary over time. This assumption is challenged by the observation that functional connectivity profiles are not stationary during resting state (171, 172), but switch between several brain states on a time scale of tens of seconds (171–175). The number and spatial location of regions of functional parcellations change as a result of brain state (173–175). In addition, functional parcellations constructed from resting-state fMRI are different from those constructed from task-evoked activity. The structure of functional parcellations constructed from task-evoked activity also depends on the specific task (176, 177). Current evaluation methods do not factor in the dynamic nature of functional parcellations and might spuriously report lower performance for the available parcellations because the data sets used for evaluation of the parcellation were acquired at different states from the ones used for the construction of the parcellation. One potential approach to deal with the nonstationarities of fMRI data sets is to construct state-specific functional parcellations and evaluate them using fMRI data set acquired within the same state. Whether or not these results will show an improvement in the quality of existing brain parcellations is a topic to be explored in further research.

Current evaluation approaches have exclusively focused on parcellations of healthy brains. Brain parcellations are frequently used to study the mechanisms underlying different neurological disorders. If the organization of the brain is altered by the disorder, then a brain parcellation constructed from data acquired from healthy subjects is not suitable for parcellating the brains of individuals with a disorder. Comparison between parcellations constructed from data sets acquired from healthy individuals and individuals with neurological disorders, and whether using parcellations specifically constructed for different disorders is advantageous for studying these disorders is the subject of further research.

Evaluation of fMRI data sets used for construction of brain parcellations has been scarcely explored. FMRI data sets used for the construction of functional brain parcellations are preprocessed, which includes bandpass filtering the BOLD activity and spatial smoothing. However, The cut-off frequencies for the filter and the extent of spatial smoothing vary across studies. The choice of the cut-off frequencies impacts the test-retest reliability of fMRI data sets (96) and can potentially impact the reproducibility of functional parcellations as well. Spatial smoothing also impacts functional connectivity patterns in the brain (e.g., see Ref. 178) and if optimized for, can potentially improve the quality of brain parcellations. Although a handful of studies have reported negligible sensitivity to spatial smoothing (25, 42, 61), it is not yet clear how sensitive other existing parcellations are to the scale of spatial smoothing. In addition, preliminary results have shown that prewhitening fMRI time series can improve the quality of parcellation by removing spurious correlations between voxels caused by autocorrelation (32). Prewhitening has also been shown to be a promising preprocessing step for biomarker discovery (138, 179) and estimation of task-activated voxels (180). It is yet to be investigated whether prewhitening has a significant impact on the quality of brain parcellations.

CONCLUSIONS

There is currently no consensus over which brain parcellation is the most accurate in identifying functionally relevant regions of the brain. In the absence of ground truth, brain parcellations have been evaluated based on how similar they are to an ideal brain parcellation, how reproducible they are, and how generalizable they are. It has become evident no single parcellation has all the characteristics of an ideal parcellation. Assessing parcellation methods based on their effectiveness, reliability, or external sources of information however does not evaluate brain parcellations’ utility in different contexts. Therefore, we proposed brain parcellations be evaluated in the context of their intended application. Rather than identifying a single optimal brain parcellation, we have argued for constructing a catalog of different brain parcellation applications and identifying optimal brain parcellations for different application categories separately. Identification of the optimal parcellation for different contexts should be done mainly in a data-driven manner where without making prior assumptions about what parcellations are appropriate for a given context, several parcellations at different granularities are used and their performance in identifying the signal of interest are compared. The parcellation(s) that maximally identify the signal of interest are deemed appropriate for that context. Ideally, once this optimization process has been done in different contexts, patterns will emerge that reveal which parcellations are more appropriate for each context.

GRANTS

This work was supported by National Institutes of Health (NIH) Grant MH060662, NIH Grant DA038984, National Science Foundation (NSF) Grant CMMI: 1634445, NSF IIS1850204, 1S10OD017974-01, and National Institute of Biomedical Imaging and Bioengineering (NIBIB) P41 EB027061.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

P.M., T.I.N., K.O.L., and G.A. conceived and designed research; P.M. interpreted results of experiments; P.M., A.T.D., and Q.D. prepared figures; P.M. drafted manuscript; P.M., A.T.D., T.I.N., K.O.L., and G.A. edited and revised manuscript; P.M., A.T.D., T.I.N., K.O.L., and G.A. approved final version of manuscript.