Characterizing variation in the functional connectome: promise and pitfalls

Abstract

The functional MRI (fMRI) community has zealously embraced resting state or intrinsic functional connectivity approaches to mapping brain organization. Having demonstrated their utility for charting the large-scale functional architecture of the brain, the field is now leveraging task-independent methods for the investigation of phenotypic variation and the identification of biomarkers for clinical conditions. Enthusiasm aside, questions regarding the significance and validity of intrinsic brain phenomena remain. Here, we discuss these challenges and outline current developments that, in moving the field toward discovery science, permit a shift from cartography toward a mechanistic understanding of the neural bases of variation in cognition, emotion and behavior.

Characterizing phenotypic variation

Nearly two decades after the seminal description of the phenomenon [1], resting state or intrinsic functional connectivity (iFC; Box 1) research is booming. Having amply demonstrated the utility of iFC approaches for mapping the functional architecture of the brain [2,3], researchers are now beginning to tackle cognitive and clinical neuroscience questions concerning the neural bases of interindividual phenotypic variation (i.e., brain-behavior relationships). In the past, fMRI research was restricted to the examination of specific cognitive constructs and tasks adapted for the scanner environment. However, fMRI researchers can now examine the relationship between brain activity and any phenotypic variable quantified inside or outside the scanner (e.g., psychiatric diagnoses; cognitive, behavioral or physical states or traits; task performance), using a single imaging dataset. The possibilities are exhilarating – studies have already examined the neural correlates of variation along several spectra of behavior, including memory function [4], social competence [5], personality [6], and social network size and rank in macaque monkeys [7]. Enthusiasm must be tempered, however, by concerns regarding sample size, motion and other nuisance factors, as well as a lack of consensus regarding data processing strategies. Most importantly, the neurophysiological bases of intrinsic brain activity and iFC remain poorly understood. Here, we discuss these challenges and outline how propagation of the methods and ethos of discovery science can help to address them.

Box 1. Same phenomenon, many names.

Although long a condition of interest for researchers working with a variety of modalities including electroencephalography (EEG) and positron emission tomography (PET), the work of Biswal and colleagues [1] marked the birth of the field of study of rest using fMRI (see [89] for an excellent historical account). Although certainly an appropriate designation during the early years of the field, the term ‘resting state’ now seems somewhat of a misnomer. That is, in light of evidence for the ubiquitous and universal nature of the correlated fluctuations of interest, the term ‘resting state’ no longer accurately captures the phenomenon in question, motivating a search for alternatives.

Extant designations vary according to whether they capture the imaging method (e.g., R-fMRI), analytic approach (e.g., resting state functional connectivity) or the biological phenomenon itself, and it is difficult to identify a label that encompasses all aspects. Here, our preference is to refer to the biological phenomenon, without bias toward a specific imaging modality or analytic approach. Accordingly, while we recognize the historical significance of ‘resting state,’ we use the term ‘intrinsic activity,’ defined by Raichle [90] as “ongoing neural and metabolic activity which is not directly associated with subjects’ performance of a task”. We also employ the corollaries ‘intrinsic functional connectivity’ (iFC) and ‘intrinsic connectivity networks’ (ICNs) to refer to the quantification of coherent intrinsic activity and the functional networks in which it occurs, respectively. Finally, we use the term ‘functional connectome’ [24] to refer to the complete set of intrinsic functional connections in the brain.

It is our hope that, as the field moves toward a better understanding of the neuroanatomical and neurophysiological bases of intrinsic activity, a more specific nomenclature will emerge. Currently, however, we suggest that ‘intrinsic’ is preferable to ‘resting state’ for a number of reasons: (i) it captures something about the biological phenomenon itself, which is not specific to a particular modality such as fMRI; (ii) it depicts the ubiquity of the phenomenon – the persistence of intrinsic activity across sleep, sedation, task performance and coma, as well as its presence across multiple mammalian species; (iii) it does not limit the phenomenon described in terms of periodicity or frequency; (iv) it has already been adopted by several researchers (e.g., [57,90]); and finally, (v) as the field experiments with different states (e.g., scanning during sleep or while participants watch a video or listen to music), shedding the ‘resting state’ moniker will become more important.

The unconstrained nature of rest

A persistent criticism of resting state fMRI (R-fMRI) is the unconstrained, unknowable and variable nature of rest itself. In particular, concern is often expressed that interindividual or group differences in ‘resting’ cognition could be largely responsible for the results obtained. Unease regarding the influence of active cognition during rest primarily reflects a conflation of the intrinsic activity that underlies iFC and the relative increases in activity that occur in the default network during passive or resting state conditions. Specifically, whereas the relative increase in default network activity during passive conditions is indeed likely to reflect active cognition [8,9], intrinsic activity persists across, albeit moderated by, multiple states, including rest, task performance, sedation and sleep, and is also observed across species. As such, intrinsic activity represents a distinct phenomenon, likely with distinct neurophysiological bases, that does not support active cognitive processing [9].

The effect of participant current state cannot be disregarded entirely, however. Manipulations of participant resting cognitions [10] or mood [11] impact iFC significantly. Further, intrinsic activity is affected by whether participants are instructed to maintain their eyes open or closed [12,13], the prior performance of cognitive tasks [14] and factors such as substance withdrawal [15], drowsiness and sleep [16,17]. Perceptual processing may also have an impact [18]. Concern about such factors has largely been controverted by the moderate-to-high test-retest reliability demonstrated for indices of intrinsic activity [19] and iFC [12,20-23]. Even substantial variation associated with data collection site or scanner does not obscure iFC measures [24]. Consequently, if factors related to participant state vary randomly across a sample, their effects are likely to be negligible, but if systematic variation is suspected, for instance, between groups, concern is justified and interpretations should be tempered accordingly. One response to this challenge is comprehensive phenotypic characterization. This will allow investigators to control for state and trait differences among participants or to interrogate relationships associated with these differences themselves, by permitting the investigation of brain-behavior relationships for both categorical (e.g., group) and continuous measures of phenotypic variation in the context of group-level analyses [25,26]. Finally, an alternative approach involves avoiding ‘rest’ by exerting experimental control over participant state, for example, by scanning during natural sleep or passive conditions (listening to music or watching a movie) (e.g., [27]). Investigating the impact of such manipulations is an important next step.

Physiological noise and the global signal

The primary criticism leveled at Biswal et al. [1] – the extent to which R-fMRI phenomena can be explained by physiological processes such as vasomotion, rather than spontaneous neuronal activity – remains a concern. Signals associated with cardiac and respiratory processes account for 5-15% of the variance in intrinsic blood oxygenation level dependent (BOLD) activity [28-31]. Interindividual or group differences in factors affecting neurovascular coupling (e.g., age or disease processes) may therefore be particularly worrisome [32,33]. The study of very young [34] and elderly [35] populations should be accompanied by an awareness of these factors, as should studies of a variety of pathologic conditions including obesity [36], Alzheimer’s Disease [37] and stroke [38]. Ideally, physiological signals should be recorded and removed from R-fMRI data [28,39-41]. However, many researchers lack the necessary recording equipment or experience procedural difficulties (e.g., respiratory belt calibration). Although independent component analysis (ICA) approaches can remove physiological signals in the absence of explicit recordings [42,43], the predominant approach is nuisance signal regression. This entails the removal, via regression, of signals associated with motion, white matter and cerebrospinal fluid, as well as a global (mean) signal. Global signal regression (GSR) has been criticized for mathematically shifting the distribution of correlations so that approximately half are negative [44]. The neurophysiological validity of the resulting negative correlations (‘anticorrelations’) has not yet been established. On the other hand, omitting GSR from analyses reduces sensitivity and anatomical specificity, prompting its continued use, despite the associated caveats.

Alternatives to GSR have been proposed [45,46], which, although promising, do not resolve the question of what it means when two regions that were positively correlated or unrelated before correction become negatively correlated afterwards. More problematic is the temptation to over-interpret negative correlations – as ‘inhibitory’ interactions (e.g., [47]). Further, electrophysiological work suggests that the global signal is correlated with an oscillatory neuronal signal present throughout the brain [48]. Although this observation does not necessarily invalidate the use of GSR, it prompts caution in discussing ‘negative’ iFC and emphasizes the need for direct examination of the neurophysiological bases of intrinsic activity and the global signal in animal models [49].

Head motion: a recurring issue

The association between age and iFC constitutes one of the best studied phenotypic relationships to date. Recent evidence that previously reported developmental changes in iFC may, in part, reflect the effects of motion [50,51] is thus particularly troubling. The confounding effects of motion are not restricted to developmental studies but are a concern for all iFC studies, and indeed fMRI studies in general [50-52]. The solution remains unclear. Power et al. [50] propose that offending time points be removed prior to computing iFC. However, this is a destructive procedure that may violate analytic assumptions (e.g., temporal contiguity). Further, how and when to excise motion-corrupted frames is unclear [53], as are the limits of this approach: what proportion of time points can or should be removed? Van Dijk et al. [52] propose a less aggressive approach that involves including the mean frame-wise motion or number of micromovements as a nuisance covariate in group-level analyses. How to best quantify the success of such correction strategies and when to exclude participants outright remain open questions.

Structural variation

Volumetric and morphometric differences among participants, together with variation in the accuracy of template normalization, may also confound investigations of phenotypic differences in the functional connectome. Strategies to deal with these factors have been described, such as the inclusion of mean or voxel-wise covariates quantifying normalization accuracy [54] or morphometric measures [55], or the use of iFC maps themselves as a basis for inter-subject alignment [56]. Surface-based analyses [3,57,58] offer a particularly promising avenue. Yet, even in meticulously executed surface-based analyses, ambiguities remain, such as the commonly observed iFC between superior temporal areas and ventral somatomotor cortex, which may reflect true anatomic connectivity or blurring of signal across the Sylvian fissure [3]. In addition, interindividual variation in the locations of functional boundaries may not be resolved by alignment of gross anatomical features. High-resolution datasets and work in animal models may permit the disambiguation of these possibilities.

The case for discovery

Poldrack [59] has emphatically outlined the need for larger sample sizes, appropriate correction for multiple comparisons and robust statistical methods across the fMRI field as a whole. Simply put, inadequate sample sizes, methods and correction procedures induce a vicious cycle in which under-powered or methodologically weak studies are used in attempts to replicate the results of other weak studies, producing a large number of failures to replicate and a surfeit of false positives.

Figure 1 illustrates why these challenges are particularly salient for the examination of interindividual variation in the functional connectome. The plot shows the effect of sample size on a group-level correlation between age and iFC, revealing that sample sizes less than 100 produce wildly varying estimates of the ‘true’ effect (i.e., the effect obtained across all 1093 participants). Even though concerns can be mitigated by combining estimates of iFC across multiple scans [5] or by demonstrating reliability across scans or samples [6,55,60], this finding is sobering for studies of brain-behavior relationships using R-fMRI data, which, to date, have employed relatively small samples.

Figure 1.

Effect of sample size on the group-level correlation between age and posterior cingulate cortex (PCC) iFC (adapted from [24]). The correlation between age and PCC iFC was computed for each of a set of randomly sampled subgroups, ranging in size from 10 to 1,090. The plot shows the mean correlation ± 2 times the standard deviation (SD), computed across 10,000 iterations. The plot demonstrates that sample sizes less than 100 produce wildly varying estimates of the ‘true’ effect (the observed correlation between iFC and age, computed on the basis of 1093 participants and indicated by the solid horizontal line).

Going forward, these challenges may best be addressed by adopting the tools of discovery science and accruing large-scale, well-characterized datasets that permit the creation of test and replication samples (e.g., [3,57]). Several projects are already moving in that direction, including the 1000 Functional Connectomes Project (FCP) and International Neuroimaging Data-sharing Initiative (INDI; http://fcon_1000.projects.nitrc.org), the Human Connectome Project (HCP; http://www.humanconnectome.org), and the Superstruct project (http://sfari.org/funding/grants/abstracts/the-brain-genomics-superstruct-project). Data from FCP/INDI are already freely available to the community, while both the HCP and Superstruct projects will be making data available in the near future. The scientific multiplier effect of such efforts is already evident, with at least 24 papers published using FCP resources within two years of making these resources available. Propagation of the ethos of discovery science to the field as a whole is crucial to further progress in identifying the neural correlates of individual differences.

Analytic tools for discovery

Together with ICA approaches, seed-based correlation remains a popular method for deriving iFC because of its computational simplicity and amenability to group-level comparisons. Temporal- [61,62] and frequency domain-based [63-65] measures for characterizing intrinsic activity are also gaining popularity. More sophisticated methods are rapidly proliferating (see [66] for a review) as researchers adapt analyses and algorithms from other fields and computational infrastructures grow to match their demands. Some approaches, such as those aimed at detecting causal influences among regions, have been found wanting [67], although newer methods for the investigation of directional influences may be more effective [68]. A particularly promising line of research is the investigation of dynamic changes in iFC [69], a characteristic that may explain some of the phenotypic variation in iFC observed to date (e.g., ‘hyperconnectivity’ or ‘hypoconnectivity’ in one group relative to another may reflect more or less consistent iFC over time, respectively).

The next step is the development of methods capable of surveying the entire functional connectome for brain-behavior relationships in order to yield empirically grounded hypotheses that can be tested in replication datasets. Such approaches constitute a deviation from current practices, which require data reduction and/or a priori specification of regions or networks of interest. These requirements necessarily limit exploration and discard potentially meaningful information about interindividual variation. Acknowledging their genetic counterpart, new methods for exploration are termed Connectome-Wide Association Studies [70] and rely on multivariate, rather than univariate, statistical approaches (see Box 2).

Box 2. Multivariate prediction analysis.

When applied to the study of intrinsic activity, the goal of discovery science is to identify models that relate measures of that activity (such as iFC) to phenotypic variables. Prediction analysis provides a means for measuring how well these models generalize to independent data. This is complementary to inferential statistics, which measure the likelihood of such relationships arising by chance. In the prediction analysis framework, a model relating iFC to a phenotype is learned from a training dataset. This model is then applied to an independent test dataset to predict phenotypes. The resulting predictions are compared to the true phenotypes to estimate how well the model generalizes to the test dataset. Thus, prediction analysis provides a natural framework for evaluating biomarkers [21], performing real-time fMRI [91] and evaluating experimental tradeoffs [92].

Prediction analysis has been applied to functional neuroimaging data since the early 1990s [93] and more recently to IFC data [94]. Most, if not all, analysis methods can be applied in a predictive modeling framework but the majority of methods that have been applied to iFC are multivariate classification and regression methods (referred to as multivariate prediction analysis – MVPA). Multivariate methods are more sensitive to distributed patterns of iFC than their univariate counterparts. Additionally, they provide a means for evaluating the significance of an entire pattern using a single statistic, obviating the need to correct for multiple comparisons.

Although there are many circumstances in which high prediction accuracy is the ultimate goal of an analysis (e.g., predicting treatment outcome), in general, it is desirable that the model also be interpretable. Identifying the iFC measures (features) that are most important to the model is problematic and an open issue for MVPA research. Several feature selection algorithms have been proposed to address this issue, but there is no consensus on which is best [21]. We note that feature selection methods that rely on feature-by-feature statistical tests require correction for multiple comparisons.

MVPA classification has already been successfully used to identify potential iFC biomarkers of Alzheimer’s disease [95], major depression [96], schizophrenia [97], and autism [47], among others. MVPA classification and regression techniques have also been applied to identify biomarkers of age [98] and recent work has shown the utility of MVPA methods for deriving iFC models at the individual level [99]. An in-depth overview of the statistical pattern recognition methods underlying MVPA techniques can be found in [100].

Deep phenotyping

An early critique of R-fMRI [71] stated that rest was unlikely to be a productive way to understand network function. This assertion is incontrovertible – correlated intrinsic activity itself says nothing about the functions supported by the networks in which it occurs [9]. Such understanding can only be obtained by experimental manipulations in the context of task-based approaches or by relating interindividual variation in measures of intrinsic activity to variation in phenotypic measures. This highlights the importance of investment in the cognitive and behavioral constructs examined and the tools with which they are measured. As eloquently outlined by Bilder and colleagues, phenomics is now the rate-limiting step preventing the advance of discovery science in neuroimaging [72]. Obtaining community consensus on the phenotypic constructs to be explored (cognitive ontologies [73]) will ultimately transform cognitive and psychiatric neuroscience (Box 3).

Box 3. Clinical insights and applications.

Long frustrated by the challenges of designing task-probes amenable to clinical populations that differ in terms of intellectual, cognitive or behavioral functioning, researchers have readily welcomed the iFC approach (see Table 1 for a list of disorders and conditions studied to date). In addition to overcoming concerns regarding practice, floor and ceiling effects, iFC approaches have made the of data aggregation across imaging sites a reality. This is especially important for the study of clinical populations, many of which have a prevalence of 1% or lower in the general population, necessitating the aggregation of resources across sites to achieve appropriate sample sizes.

The transformative impact of the iFC approach for the fields of neurology and psychiatry extends beyond logistical considerations, fostering a shift in how we conceptualize neuropsychiatric illness. Whereas task-based studies encouraged a search for a ‘clinical homunculus’ that mapped distinct clinical phenomenology to altered function in circumscribed regions of the brain, the iFC approach emphasizes compromised functional and structural interactions as potential loci of dysfunction. Ultimately, the comprehensive assessment of both regional function and interregional connectivity are needed to provide the most complete characterization of the impact of pathological processes on the brain.

With respect to the future of clinical applications, the recent ADHD-200 Global Competition (http://fcon_1000.projects.nitrc.org/indi/adhd200/results.html) brought a key question to the forefront: do iFC approaches, and neuroimaging more broadly, have a role in the diagnosis of psychiatric illness? To answer this question, we look to models from the broader medical community. For example, when a patient visits their doctor with symptoms suggestive of a common cold, no blood test or diagnostics are ordered. In contrast, when the presenting symptoms are consistent with multiple possibly severe diagnoses, objective laboratory tests become of value. Applied to psychiatric illness, for a typical presentation of a condition such as Attention-Deficit/Hyperactivity Disorder (ADHD), the criteria in current diagnostic manuals are sufficient. However, when a clinical presentation cuts across diagnostic boundaries and clarification can directly impact treatment decision-making, imaging-based tools may have value in improving diagnostic accuracy. Similarly, imaging-based tools may help track response to treatment. However, as highlighted by the recent ADHD-200 Competition (http://fcon_1000.projects.nitrc.org/indi/adhd200), claims of clinical utility of purely imaging-based approaches in psychiatry are currently premature and potentially harmful. Further methodological innovation, combined with the generation of carefully characterized and well-coordinated datasets, is needed before imaging-based diagnostic tools can become a reality.

Lost in the cracks

Applying data-driven partitioning techniques (e.g., cluster analysis) to iFC data can parcellate the brain into distinct functional systems and units, revealing its functional ‘building blocks’ [3,74-76]. Such efforts typically focus on identifying functional units that are stable across individuals, rather than how they vary. Yet, interindividual variation in how functional areas are differentiated from one another may be of interest in itself. Cohen et al. [58] provided a key insight when they described ‘transition zones’: the boundaries between functional areas, indicated by sharp changes in iFC. Such transition zones are evident when mapping the confidence or stability of iFC-based parcellations (e.g., [3,57]) and iFC variability within a network across individuals (Figure 2). Interindividual variation in these transition zones does not appear to strictly follow structural variation, but rather variation in task-evoked activations [58,77]. As such, examination of links between variation in functional zones and behavior holds promise: initial studies suggest that the functional connections most strongly related to phenotypic variables [5,6] and to the magnitude of task-evoked activation [77] are those that exhibit the greatest variability across individuals and lay within these transition zones. The role of intraindividual dynamic variation in the strength of iFC [69] in the creation of these transition zones also merits investigation.

Figure 2.

Interindividual variation in functional boundaries or transition zones (adapted from [24]). The figure shows the spatial distribution of voxel-wise coefficients of variation (absolute value) for (a) fractional Amplitude of Low Frequency Fluctuations (fALFF: a frequency domain-based measure of intrinsic activity); (b) iFC associated with a seed placed in the intraparietal sulcus (IPS); (c) iFC associated with a seed placed in posterior cingulate cortex (PCC); and (d) iFC within an ICA-based network identified as the default network. For the purpose of visualization, coefficients of variation (CV) were rank-ordered, so that the relative degree of variation across participants at a given voxel is shown. Ranking coefficients of variation in this way clearly delineates regions of greatest interindividual variability, thus demarcating putative functional boundaries or transition zones.

Toward a mechanistic understanding

We have provided an overview of some of the main challenges facing intrinsic brain research as the field moves into an era of discovery. We hold that assimilation of the methods and ethos of discovery science will propel the field beyond simply mapping the brain’s functional organization toward understanding how interindividual variation in brain organization and function underlie normal and abnormal variation in cognition, emotion and behavior. Looking ahead, a mechanistic understanding of brain-behavior relationships will demand multimodal and translational approaches. Studies in animal models permit direct structural, pharmacological, molecular and genetic experimental manipulations that will provide causal explanations of intrinsic brain phenomena, as well as the disruptions associated with clinical disorders. The time scales of animal development also provide experimentally tractable timeframes within which to study questions pertinent to human development and developmental psychopathology. Already, non-invasive investigations in humans using EEG [78] and MEG [79,80] have demonstrated relationships between spatial and temporal indices of oscillatory electrophysiological activity and iFC measures. Together with invasive studies in non-human primates [48,81,82] and humans [83,84], these studies provide strong evidence that the signal fluctuations underlying iFC arise from the same neuronal substrate as activity occurring on faster time scales, including those relevant to information processing and behavior [84]. A compelling complementary line of research involves the investigation of the impact of intrinsic (ongoing) brain activity on perception and behavior [85,86]. Finally, computational modeling work employing physiologically realistic constraints [87,88] has provided compelling accounts of the emergence of intrinsic fluctuations on the timescales captured by fMRI from neuronal interactions occurring at faster timescales, as well as plausible explanations for some of the more puzzling characteristics of intrinsic activity (e.g., anti-phase relationships between networks). Continued exploitation of these important lines of research is sure to spark the next wave of breakthroughs in our understanding of brain and behavior.

Table 1.

Number of publications in which iFC or resting state approaches have been used to study a variety of disorders and conditions.

Disorder/Condition	# studies
Schizophrenia	63
Alzheimer’s Disease	62
Depression	46
Mild Cognitive Impairment (MCI)	43
Aging	42
Epilepsy	32
Substance Dependence	28
ADHD	17
Multiple Sclerosis	14
Autism	14
Parkinson’s Disease	12
Pain	11
Anxiety Disorders	9
Sleep	9
Miscellaneous Neurological Disorders	9
Stroke	8
Obsessive Compulsive Disorder (OCD)	8
Posttraumatic Stress Disorder (PTSD)	8
Amnesia	6
Brain Lesions	6
Dementia	4
Seizure	4
Trauma	4
Bipolar Disorder	3
Personality Disorders	2
Cerebral Palsy	2
Fetal Alcohol Syndrome	2
Migraine	2
Psychopathy	2
Learning Disabilities	1
Tourette Syndrome	1