Stepwise Connectivity of the Modal Cortex Reveals the Multimodal Organization of the Human Brain

Abstract

How human beings integrate information from external sources and internal cognition to produce a coherent experience is still not well understood. During the past decades, anatomical, neurophysiological and neuroimaging research in multimodal integration have stood out in the effort to understand the perceptual binding properties of the brain. Areas in the human lateral occipitotemporal, prefrontal, and posterior parietal cortices have been associated with sensory multimodal processing. Even though this, rather patchy, organization of brain regions gives us a glimpse of the perceptual convergence, the articulation of the flow of information from modality-related to the more parallel cognitive processing systems remains elusive. Using a method called stepwise functional connectivity analysis, the present study analyzes the functional connectome and transitions from primary sensory cortices to higher-order brain systems. We identify the large-scale multimodal integration network and essential connectivity axes for perceptual integration in the human brain.

Introduction

Humans have the ability to process physical and mental information to ultimately generate a sense of reality. The processes by which perceptual information is captured and integrated to create a holistic unitary experience of the subject's world are still under debate. The perceptual feature integration problem in the brain, initially originated in vision research as the “binding problem” (Treisman, 1996; Reynolds and Desimone, 1999; Wolfe and Cave, 1999), has been an intriguing issue for many decades. Brain anatomical and functional patterns suggest the existence of areas with high local modularity and hierarchical connections in sensory cortices (Maunsell and van Essen, 1983; Felleman and Van Essen, 1991; Distler et al., 1993; Ungerleider and Haxby, 1994; Sepulcre et al., 2010), as well as integrative association areas that receive widespread projections from distributed brain systems (Pandya and Kuypers, 1969; Jones and Powell, 1970; Mesulam, 1990; Eguíluz et al., 2005; Salvador et al., 2005; Mesulam, 2008; Buckner et al., 2009). In this sense, a specific set of regions, now known as cortical hubs, merge the highest number of functional large distant connections in the human brain, leading to interpretations of these regions as the top hierarchical areas for integration (Buckner et al., 2009; Sepulcre et al., 2010). Nevertheless, it is still not well understood how the brain manages to integrate these two archetypical extremes or, in other words, how the transitions from modular sensory regions to parallel-organized heteromodal and limbic processing systems take place.

In recent years, anatomical, neurophysiological, and neuroimaging research on multimodal integration has provided insights into the binding of three main perceptual modalities in the nervous system: vision, touch and audition. For instance, areas such as the posterior temporal lobe and the lateral occipitotemporal junction (LOTJ), as well as areas in the posterior parietal lobe, have been consistently described as critical for bimodal or trimodal integration processing (Beauchamp, 2005; Beauchamp et al., 2004; Calvert, 2001; Driver and Noesselt, 2008). A large region covering the entire superior temporal sulcus (STS) appears to be essential for trimodal integration in nonhuman primates (Driver and Noesselt, 2008). Other brain regions at the subcortical level, such as the superior colliculus, have also been described as multimodal processors [for a study in cat, see the study by Wallace et al. (1998)]. Moreover, rather than integrating multimodal information in isolated or disconnected regions, functional MRI activation studies suggest that perceptual multimodal binding is likely to be achieved via mutual interaction of multiple regions (Downar et al., 2000; Corbetta and Shulman, 2002).

In this study, we aim to identify the functional connectome of the modal brain (visual, auditory, and somatosensory cortices) by using a novel method we call stepwise functional connectivity (SFC) (see Fig. 1). We have specifically developed SFC as a network analysis technique to explore the convergence and interactions of sensory systems at the connectivity level. While most functional connectivity and resting-state MRI studies emphasize the separation and isolation of networks in the brain (for instance, independent component analysis and K-means approaches), here we focus on a complementary question that represents a new challenge for network neuroscience: how are brain systems bound together? By using SFC analysis, we sought to not only elucidate the main areas of multimodal integration but also to untangle the complex connectivity transitions that take place from primary to higher-order cognitive distributed systems of the brain.

Figure 1.

Stepwise functional connectivity analysis for identifying the brain connectome of modal cortex. A, The functional connectivity map of the posterior cingulate cortex is computed using a conventional seed-based approach, which is displayed on a cortical surface and as a network graph (star-net topology). Only one link-step connection is computed in this approach. B, On the other hand, SFC analysis takes full advantage of the whole connectivity association matrix of the individual brains to analyze the functional connectivity of seed regions using a wide range of link-step distances. C illustrates SFC analysis in detail. From any node (j; red nodes) in the brain, we compute the number of pathways (D_ji^l) that connect to a voxel in an a priori seed node (i; blue node) with a particular number of links l (where l is the specific link-step distance). Similar to the conventional seed-based approach, in the one-link-step distance case, only direct connections to the seed are considered (small arrows in C-a). In higher-order step distances, we explore connectivity patterns beyond direct connections, subsequently expanding to the rest of the brain (C-b, C-c). Note that for schematic purposes only, one seed voxel node (blue color) is displayed in C-a, C-b, and C-c.

Materials and Methods

Overview.

We use resting-state functional connectivity MRI (fcMRI) and SFC analyses to investigate the large-scale integration networks of the three main sensory (visual, somatosensory, and auditory) cortices in the human brain. Core analyses were based on a data set of 100 healthy young adults (data set 1; mean age, 21.3 years; 37% male). Another data set of 100 subjects (data set 2; mean age, 20.83 years; 39% male) was used for replication and complementary analyses.

As a general workflow, we performed the following steps. (1) We first explored the SFC patterns of primary cortices to describe the main convergence regions of multimodal integration (data set 1). (2) Then, we focused our analyses on characterizing the discovered multimodal regions to describe the different cores and functional connectivity axes within the multimodal network (data set 2). (3) Finally, we used an interconnector network analysis to explore specific functional connectivity profiles between pairs of sensory cortices (data set 2).

Participants.

All subjects participated in the MRI studies for payment. Subjects were recruited as part of the Brain Genomic Superstruct Project (Buckner et al., 2011; Yeo et al., 2011), a neuroimaging collaborative effort across multiple laboratories at Harvard University, the Massachusetts General Hospital, and the greater Boston area. Inclusion criteria were as follows: fMRI signal-to-noise ratio >100, no artifacts detected in the MR data, no self-reported neurological or psychiatric diseases, and psychoactive medications history. All participants had normal or corrected-to-normal vision and were right-handed native English speakers. Participants provided written informed consent in accordance with Helsinki Declaration and guidelines set by institutional review boards of Harvard University or Partners Healthcare.

MRI acquisition procedures.

Scanning was acquired on a 3 Tesla TimTrio system (Siemens) using the 12-channel phased-array head coil supplied by the vendor. High-resolution 3D T1-weighted magnetization multiecho images for structural anatomic reference (multiecho MPRAGE) (van der Kouwe et al., 2008) and a gradient-echo echo-planar pulse sequence (EPI) sensitive to blood oxygenation level-dependent (BOLD) contrast for functional imaging data were obtained. Multiecho MPRAGE parameters were as follows: TR, 2200 ms; TI, 1100 ms; TE, 1.54 ms for image 1 to 7.01 ms for image 4; flip angle, 7°; 1.2 × 1.2 × 1.2 mm; FOV, 230. EPI parameters were as follows: TR, 3000 ms; TE, 30 ms; flip angle, 85°; 3 × 3 × 3 mm voxels; FOV, 216; 47 slices. Although one or two runs were acquired per each subject, we used only the first run (each run lasted 6.2 min, 124 time points) to keep data as homogeneous as possible across subjects. Functional images covered the whole brain including the entire cerebellum, and all slices were aligned to the anterior commissure/posterior commissure plane using an automated alignment procedure (van der Kouwe et al., 2005). During the functional runs, the participants were instructed to remain still, stay awake, and keep their eyes open. Head motion was restricted using a pillow and foam, and earplugs were used to attenuate scanner noise.

fMRI and fcMRI preprocessing.

MRI preprocessing steps were optimized for fcMRI analysis (Fox et al., 2005; Vincent et al., 2006; Van Dijk et al., 2010) extending from the approach developed by Biswal et al. (1995). First, we performed conventional preprocessing of functional MRI such as removal of first four volumes to allow for T1-equilibration effects, compensation of systematic slice-dependent time shifts, motion correction, and normalization to the MNI atlas space (SPM2; Wellcome Department of Cognitive Neurology, London, UK) to yield a volumetric time series resampled at 2 mm cubic voxels. It is essential to remark that all network connectivity analyses of subjects' images were done preserving the index information of MNI coordinates. This approach allowed us to obtain degree of SFC maps that are comparable within the sample. Next, we performed more specific preprocessing for functional connectivity analysis. Temporal filtering was applied to the MRI data to remove constant offsets and linear trends over the data while retaining frequencies below 0.08 Hz. Moreover, several sources of spurious variance, along with their temporal derivatives, were removed through linear regression, including (1) six parameters obtained from correction for rigid-body head motion, (2) the intensity signal averaged over the entire brain, (3) the intensity signal averaged over the body of the lateral ventricles, and (4) the signal averaged over white matter in the centrum semiovale. This regression procedure removes variance, such as physiological artifacts, that are unlikely to represent signal of neuronal origin (Van Dijk et al., 2010). However, it is well known that removal of global signal also causes a shift in the distribution of correlation coefficients forcing negative correlations to increase, making the interpretation of the negative correlation ambiguous (Chang and Glover, 2009; Murphy et al., 2009; Van Dijk et al., 2010). Therefore, in this study only correlations exceeding a positive and false discovery rate (FDR)-corrected threshold (see below, Stepwise functional connectivity analysis) were used to avoid this concern. As a final step before the network analysis, we downsampled the fcMRI data to 8 mm isotropic voxels for computational efficiency.

It is important to remark that fcMRI measures intrinsic activity correlations between brain regions (Biswal et al., 1995; Fox and Raichle, 2007; Lu et al., 2011) and can reflect monosynaptic as well as polysynaptic signal relationships between regions. Direct but also indirect connectivity driven by common sources, such as physiological noise or overlapping spatial smooth, could generate significant correlations. The rationale as well as the caveats and limitations of fcMRI have been discussed previously in detail (Van Dijk et al., 2010).

Stepwise functional connectivity analysis.

While a great part of functional connectivity and resting-state MRI studies focus on the separation or isolation of networks in the brain, here we focus on a complementary and substantially different question that represents a new challenge for network neuroscience: how are brain systems bound together? We used SFC analysis to map the connectivity patterns of brain seed regions at different step (or “link-step”) distances. In a sense, the field of tract tracing techniques that use transneuronal tracers to investigate multisynaptic/multistep circuits of brain targets allowing the visualization of entire functional neuronal networks (Ugolini, 2010; Nhan and Callaway, 2012) conceptually inspired us to develop the SFC analysis in the framework of functional connectivity. As an input to the SFC analysis, we first computed the individual association matrices from each participant's whole-brain network by computing the Pearson R correlations of each voxel to every other voxel time course in the preprocessed BOLD images (see Fig. 1B). To perform this analysis, each individual BOLD image was previously converted to an N-by-M matrix, where N was the image voxels in MNI space, and M was the 124 acquisition time points. A mask of 4652 voxels covering the whole brain (cortex, subcortex, brain stem, and cerebellum) was used to extract the time courses. As a result, a 4652 × 4652 matrix of Pearson R, or product-moment, correlation coefficients was obtained for each individual. After this point, only positive correlations of the association matrix were taken into account. Next, we applied an FDR (Benjamini and Hochberg) correction at the 0.001 level to control for the rate of false positives in the final network adjacency matrices. This method introduces in each individual a customized and high r threshold that allows the elimination of network links with low temporal correlation, which are likely to be attributable to noise signal. Finally, we binarized the resulting FDR thresholded matrices to obtain undirected and unweighted graphs for each individual that will serve as input data for the SFC analysis. All SFC analyses were performed at the individual level for each participant.

SFC analysis aims to characterize regions that connect to specific seed brain areas at different levels of link-step distances (see Fig. 1). In our framework, a step refers to the number of links (edges) that belongs to a path connecting a node to the seed (or target) area. In this sense, link step and path length are analogous concepts. However, we prefer to use link step to avoid confusion with the graph theory terms of average path length and shortest path used when analyzing pairs of nodes.

In SFC analysis, the degree of stepwise connectivity of a voxel j for a given step distance l and a seed area i (Fig. 1C, Dji^l) is computed from the count of all paths that (1) connect voxel j and any voxel in seed area i, and (2) have an exact length of l.

It is easy to see that SFC is also related to transition probabilities in an information flow analysis between the seed area and voxels across the rest of the brain. Given the lack of directionality information provided by fcMRI data, in SFC we did not include any restrictions about recurrent pathways crossing the seed regions multiple times. On the other hand, an important aspect of the SFC approach is that that the link-step concept has no obvious physiological meaning as it depends on, among other things, the resolution of the images. Therefore, we explored a wide range of link-step distances, from 1 to 20, to characterize the progression of the derived maps. Although the amount of overlap between consecutive steps is expected to be high, we aimed to see meaningful relative changes between pairs of maps. As seen in Figure 2, the SFC patterns are topographically dissimilar between consecutive maps from steps one to six, and become stable for link-step distances above seven. The cortical map of this final stable state collapses into regions that were described previously as cortical hubs (Buckner et al., 2009; Sepulcre et al., 2010). Based on this analysis, in our results we show only maps up to seven steps.

Figure 2.

Spatial correlation across steps. Voxel-level spatial correlations between consecutive pairs of SFC maps were calculated for each sensory modality to check the topological stability of our results. The results showed stable correlation coefficients after a six to seven link-step distance (arrows). Therefore, the range from one to seven steps was considered sufficient for the characterization of the connectivity patterns.

SFC analysis uses a priori selected seed voxels of interest. In the first part of the study, we selected seed regions in the primary visual and auditory cortices based on probabilistic cytoarchitectonic maps. Briefly, we used the SPM Anatomy Toolbox (Amunts et al., 2000; Eickhoff et al., 2006a,b) to select the spatial coordinates of primary visual (BA17, V1) and primary auditory (BA22, A1) cortices for accurate determinations of histological MRI corresponding to our sample resolution. For the primary somatosensory cortical seed, we focused our analysis on the hand area due to its central role in the sense of human touch. MNI coordinates of the hand somatosensory seed region were selected from a previous task activation study (Buckner et al., 2011). For all three primary cortex seeds, we created a cube region of eight voxels, where the reference MNI coordinates, as shown in Table 1, correspond to the more lateral, anterior, and superior voxel. We used equal-sized cube regions of eight voxels rather than single-voxel seeds to achieve (1) a comparable degree of connectivity maps between modalities and (2) graded and not binary results in the one-step-distance condition. Importantly, our seed regions confined within the modular borders of primary sensory cortices (Sepulcre et al., 2010). Therefore, D_ji^l of a brain voxel represents the sum of the number of pathways that connect to any one of the eight voxels in the seed region. The same eight-voxel cube region of interest strategy was used below, in the sections Characterization of the multimodal integration network and Interconnectors between modality pairs. To create the seeds of interest for the characterization of the multimodal network in data set 2, we used the local maxima intensity peaks from the five-step distance map of Figure 6A. Both left and right hemispheres were included in all the SFC analyses, but only left or right seeds were considered wherever appropriate. Figures 3–5 show results for each left sensory modality seed. To get a combined SFC map of all modalities together, Figures 6A and 7 use the same SFC strategy but with all three seeds (visual, auditory, and somatosensory) simultaneously. In other words, for every voxel in the brain, we calculated the number of paths to all the three seed regions for a given step distance. We call this analysis the combined SFC analysis. As a final step before surface projection, we computed one-sample t tests across the entire sample of 100 subjects to obtain a statistical map for every seed and for every SFC map with a level of significance of p < 0.0001 (uncorrected).

Table 1.

Locations of seed regions utilized in the stepwise functional connectivity analysis

Name	Coordinates
L visual	−14, −78, 8
L auditory	−54, −14, 8
L somatosensory	−42, −29, 65
R visual	10, −78, 8
R auditory	58, −14, 8
R somatosensory	38, −29, 65
L parietal operculum	−58, −21, 25
L anterior insula+	−50, 3, −7
L superior parietal	−18, −45, 57
L lateral occipitotemporal	−50, −61, 1
L dorsal anterior cingulate+	−6, 2, 48
L dorsolateral prefrontal	−34, 43, 25

L, Left; R, right.

Figure 6.

Multimodal integration network. An SFC analysis combining the three sensory seeds simultaneously showed the complete map of the multimodal integration network of the human brain. A presents the three- and five-step maps of the combined SFC analysis. B presents the initial (binarized average of the one-step and two-step maps), intermediate (binarized average of the three- to five-step maps), and terminal (binarized average of the six- and seven-step maps) cores. It is noteworthy that the multimodal integration network (green cortex and nodes) acts as a strong network interface between the unimodal-related systems (red cortex and nodes) and the cortical hubs core (blue cortex and nodes). The sulcal map was provided by Caret software (Van Essen and Dierker, 2007). PCS, Posterior central sulcus; SMA, supplementary motor area.

Figure 3.

Visual cortex stepwise functional connectivity. Visual cortex SFC analysis revealed that visual cortex's direct connectivity follows three different pathways, two dorsal (a–c) and one ventromedial (a; inset, star). In addition to the conventional cortical surface and to avoid a ceiling visualization effect in the degree of connectivity of early visual regions, two flat projections centered on the occipital lobe (green square) are presented using the same (top inset) and a relaxed color-scale threshold (bottom inset). In subsequent steps, the visual cortex connectivity reached the frontal eye field (d), the multimodal network (e–h), and finally, the cortical hubs of the human brain. The visuotopic map was provided by Caret software (Van Essen and Dierker, 2007). Visuotopic areas: V1, V2, V3, V7, V8, and MT+; PCS, posterior central sulcus; SMA, supplementary motor area.

Figure 4.

Somatosensory cortex stepwise functional connectivity. Somatosensory cortex SFC analysis showed dense connections within the entire somatomotor cortex, strong direct connectivity with the secondary somatosensory area SII in OP4 (a; inset, star) and, to a lesser extent, to the LOTJ (b). To better visualize the degree of connectivity, conventional (bottom insets) and inflated projections (top insets) of the results using a relaxed color-scale threshold and centered on the insula region/postcentral gyrus (green square) are presented in the one-step and three-step maps. Primary somatosensory cortex had later connectivity to the multimodal network (c–f) and to the cortical hubs. PCS, Posterior central sulcus, SMA, supplementary motor area.

Figure 5.

Auditory cortex stepwise functional connectivity. Auditory cortex SFC analysis showed dense connections within the local auditory-related regions, strong direct connectivity with OP4 (a; inset, star) and, to a lesser extent, to the LOTJ (b). To better visualize the degree of connectivity, conventional (bottom inset) and inflated projections (top inset) of the results using a relaxed color-scale threshold and centered on the insula region (green square) are presented in the one-step and three-step maps. In the subsequent steps, auditory connectivity reached the multimodal network (c–f) and the cortical hubs. PCS, Posterior central sulcus; SMA, supplementary motor area.

Figure 7.

Multimodal integration network in the left and right hemispheres. We compared the left and right hemispheres to check for brain asymmetries in the multimodal integration network. Sensory cortices of the right hemisphere have connectivity to a wider multimodal region in the temporoparietal junction than the left hemisphere (a and b). On the other hand, with a higher number of steps, only primary sensory cortices of the left hemisphere have connectivity to VLPFC and part of the Broca's language area (b). STG, Superior temporal gyrus; VLPFC, ventrolateral prefrontal cortex.

Interconnectors between modality pairs.

Besides the a priori selected seed regions, no other prior constraints were introduced in the computation of SFC. However, it is possible that brain regions have meaningful functional connectivity routes that do not necessarily follow the main or massive connectivity patterns revealed by the SFC method. For instance, primary regions could have less numerous but more direct links with other primary regions that might be washed out with our stepwise approach. This is the main reason to use a complementary method that helps us to capture potentially interesting patterns of connectivity between pairs of cerebral areas. First, we detect the voxel nodes that have direct and indirect connectivity to both seed regions, a and b. A direct interconnector node is defined as a node that has connections to both seeds in one link-step distance. Therefore, a direct interconnector node is a node directly located between targets. An indirect interconnector node is defined as a node that has direct connections to one of the targets (one link-step distance) but indirect connections to the other target (two link-steps distance). Later, we compute the degree connectivity of a given interconnector node by summing the total number of paths to the target nodes. Other levels of connectivity such as indirect connections in three-, four-, or five-link-step distances were analyzed, but because indirect interconnectors that combined one- and two-link-step connectivity already collapsed in the trimodal regions (see Fig. 12), we restricted our results to the bimodal findings. In summary, this interconnector analysis strategy provides information about interface regions that bond two sensory modalities (see Fig. 10).

Figure 12.

Bimodal interconnectors between visual, somatosensory, and auditory cortices. A, B, Bimodal integration regions in the human brain were described between visual and auditory (a), visual and somatosensory (b, c) and auditory and somatosensory (d) modalities. PMTG, Posterior middle temporal gyrus; DMT+, dorsal MT+.

Figure 10.

Characterization of the multimodal integration network: dorsal anterior cingulate cortex and dorsolateral prefrontal cortex. A, The dorsal anterior cingulate+ multimodal region was densely connected to the parietal operculum and anterior insula+ in both direct and indirect manners (one- and three-step maps). B, The multimodal dorsolateral prefrontal cortex region is connected in one-step distance to the anterior part of the anterior insula+ region and to the posterior part of the parietal operculum (one-step map). Subsequently, dorsolateral prefrontal cortex showed connectivity to the rest of multimodal network regions (three-step map). SMA, Supplementary motor area.

Visualization.

As a final step, all resulting maps were projected onto the cerebral hemispheres of the Population-Average Landmark and Surface-based (PALS) surface (PALS-B12) provided with Caret software (Van Essen and Dierker, 2007) using the “enclosing voxel algorithm” and “multifiducial mapping” settings. To aid visualization, all surface SFC images were displayed using a normalized color scale from 0 to 1, where 0 is the intensity corresponding to the one-sample t test p value of 0.0001, and 1 is the maximum intensity of the image corresponding to the smallest one-sample t test p value. Using the stepwise results of Figure 6A, we created the overlap map of Figure 6B. The masks for early, intermediate, and late steps were made using the average of the SFC maps for one to two, three to four, and and four to seven step distances, respectively. We used Pajek software (De Nooy et al., 2005) and the Kamada and Kawai (1989) energy layout for the network graph displays in Figures 6B and 9B. The Kamada and Kawai (1989) algorithm is a force layout method based on a network energy minimization procedure that takes into account the difference between geometric and pairwise shortest-path distances of nodes in the graph. Only nodes from the partition of the three components shown in Figure 6B (initial-, intermediate-, and terminal-step cores) were included in the network graph. We used the same color partition for brain surface and network topological spaces in Figure 6B. In Figure 9, we performed an average-linkage hierarchical clustering analysis to study the partitions of the multimodal integration network. We included all nodes from the seed regions used in the below section Characterization of the multimodal integration network.

Figure 9.

Characterization of the multimodal integration network: superior parietal cortex and lateral occipitotemporal junction. A, The superior parietal region of the multimodal network displayed direct connections mostly to the parietal operculum, but also to the frontal eye field, lateral occipitotemporal junction, dorsal anterior cingulate+, and anterior insula+ (one-step map). In later stepwise conditions, a superior parietal connectivity pattern was concentrated in the parietal operculum and anterior insula+ core (three-step map). B, The connectivity of the lateral occipitotemporal junction of the multimodal network diffused locally in a wide area at the temporal–parietal–occipital lobe confluence and also had dense connectivity to superior parietal cortex and parietal operculum multimodal regions (one-step map). Later, the connectivity converged within the parietal operculum and anterior insula+ core (three-step map). PCS, Posterior central sulcus.

Results

Stepwise functional connectivity of the primary visual cortex

When exploring visual cortex with SFC, we find that the direct functional connections from the early visual seed in V1/BA17 are mostly limited to within the occipital lobe, especially to its medial and dorsal areas (Fig. 3, a). There is a gradient of a decreasing degree of connectivity from the seed region to the rest of the lobe that, interestingly, expands dorsally from V1 to V2, V3, and V7 (Fig. 3, b, details in flat projections) and also laterally from V1 to V2 and MT+ (Fig. 3, c, details in flat projections). A third branch of connectivity is following the ventromedial part of the occipital lobe, from V1 to V2v, V3v, V4, and V8, and posterior parahippocampus (Fig. 3, a, details in flat projections). After the initial steps of connectivity, all visual pathways converge to a set of distinct regions in the frontal eye field (FEF), superior parietal cortex (SPC; BA1/2/5), parietal operculum [or operculum parietale (OP); BA40], anterior insula (AI)/ventral premotor cortex (VPMC) in the frontal operculum (AI+), and dorsal anterior cingulated cortex/supplementary motor area (DACC+; BA6/24/32) cortices (Fig. 3, e–h). Finally, in the late steps, the connectivity of the visual cortex reaches distributed cortical regions, now known as the cortical hubs of the human functional brain (Buckner et al., 2009; Sepulcre et al., 2010).

Stepwise functional connectivity of the primary somatosensory cortex

In the early step, somatosensory cortex displays a regional–local functional connectivity all along the somatomotor cortex (Fig. 4, a) and, to a lesser extent, also to the LOTJ (Fig. 4, b). Figure 4 (a) shows that primary somatosensory cortex has a significant and direct connectivity to the secondary somatosensory area [operculum parietale 4 (OP4)/BA43] (Eickhoff et al., 2007) in the ventral and anterior part of the parietal operculum (Fig. 4, a, star in inflated projection). With increasing steps, somatosensory cortex connects to the same constellation of regions as the visual cortex, which includes the SPC, OP, AI+, and DACC+ (Fig. 4, c–f). Again similarly to the visual cortex, the high-order SFC maps of the somatosensory cortex show connectivity patterns reaching the cortical hubs network.

Stepwise functional connectivity of the primary auditory cortex

In the first step, the primary auditory cortex exhibits a dense and predominant regional–local functional connectivity, specifically along auditory related areas such as the perisylvian secondary auditory areas (Fig. 5, a). Although, to a lesser extent, auditory cortex also exhibits direct connections to somatomotor and the LOTJ (Fig. 5, b). Similarly to somatosensory cortex, initial steps in the SFC analysis reveal a significant connectivity to a secondary representation area (OP4/BA43) in the parietal operculum (Fig. 5, a, star in inflated projection), an area slightly anterior to the one described in the somatosensory SFC analysis. In later steps, the auditory cortex connectivity reaches first other perisylvian areas such as a ventral premotor cortex and operculum parietale 1 (OP1) (Fig. 5, d, e, inflated projection), and then the same set of regions described in the previous two sensory modalities, SPC, OP, AI+, and DACC+ (Fig. 5, c–f). In the final steps, the connectivity of the auditory cortex reaches the cortical hubs network.

Multimodal integration network

In Figure 6 we used a combined approach to highlight the topological convergence of the stepwise connectivity patterns in the three major sensory modalities that were explored (Fig. 6A). In addition to the regions mentioned previously (SPC, OP, AI+, and DACC+), we found more clearly that one area in dorsolateral prefrontal cortex (DLPFC; BA10/46) and another area in the LOTJ, in the confluence region of BA19/22/37/39 and touching the superior temporal sulcus, provide also strong nexuses of convergence from all three sensory modalities (Fig. 6A,B). In other words, concrete and consistent regions emerge as a common destiny for the sensory modalities and serve as transition bridges from perception to higher-order cognitive regions such as the cortical hubs (Fig. 6B, green areas and nodes).

Multimodal integration network in the left and right hemispheres

We found left/right hemisphere asymmetries in the multimodal integration network, particularly in the vicinity of the temporoparietal junction (TPJ). The OP multimodal integration region on the right side of the brain is more extensive than the one obtained in the left hemisphere (Fig. 7, a, b). In the right hemisphere, the OP region merges with the superior temporal gyrus, while in the left hemisphere the OP is isolated. In addition, the maps of the late steps show important left/right differences in the ventrolateral prefrontal cortex, especially in Broca's language area (Fig. 7, b).

The TPJ has been classically described as a region affected in the spatial neglect syndrome, especially in the right hemisphere (Friedrich et al., 1998; Ro et al., 1998; Vallar, 1998). Neglect syndrome is an attention disturbance in which subjects are unaware of spatial stimuli in one hemifield. The fact that the multimodal region in the right hemisphere is larger in that particular location may account for the higher rate of neglect patients with right TPJ lesions. The characterization of multimodal integration regions strongly supports a network theory of the neglect condition. Although frequently described after focal TPJ lesions, neglect syndrome seems to be a network disease (Mesulam, 1981; Downar et al., 2000; He et al., 2007). For instance, other regions such as DACC or prefrontal cortex, particular in the vicinity of BA44, have been implicated in neglect patients (Vallar, 1998). Therefore, it is not surprising that the regions that generate neglect are included in the multimodal network described in this study. Even newly characterized regions of the multimodal network such as SPC have been associated with neglect (Mesulam, 1999). The network framework brings new insights to bear on neglect syndrome. The disruption of connectivity axes between multimodal regions may explain problematic points of previous “TPJ-centrist” interpretations. For instance, an appropriate integration of perceptual, attentional and motor planning information seems critical for a rapid access of motor actions (Ikeda et al., 1999). It is well known that neglect patients have a motor deficit, particularly in initiating movements (Corbetta and Shulman, 2002), and an interruption of connectivity from OP, AI+, or SPC to DACC+, which include the presupplementary motor area, can explain this aspect of the syndrome.

Characterization of the multimodal integration network

After isolating the multimodal network in the first data set, we aimed to confirm that these regions make up a robust coherent network using the second data set. In this section, we used SFC analysis again, but this time to characterize the main functional connectivity axes of the previously obtained multimodal regions.

OP (Fig. 8A) and AI+ (B) stepwise connectivity maps show that both regions form a strong axis of connectivity between them. For higher-order steps, OP and AI+ also exhibit strong connectivity with the DACC+ and SPC regions. As expected for an SFC analysis at rest, the stepwise connectivity maps of OP and AI+ eventually reach the stable state in the cortical hubs regions, but in fewer steps than the primary sensory cortices. In summary, AI+ and OP stepwise connectivity maps show a dense core of connectivity between both regions, and in later steps reach other multimodal nodes but never significantly return to the primary cortices.

Figure 8.

Characterization of the multimodal integration network: parietal operculum and anterior insula. A, B, Parietal operculum and anterior insula+ regions showed strong direct connectivity between them (one-step maps). In both cases, the other region represented the highest degree of connectivity in their respective maps. In intermediate steps, they are connected to the rest of the multimodal network (three-step maps). SMA, Supplementary motor area.

When analyzing the SPC region, we find that this region has strong direct coupling with the OP and then a dense connectivity to the OP-AI+ axis (Fig. 9A). On the other hand, the LOTJ's direct connectivity spreads locally across a broad region of the temporal–parietal–occipital junction (Fig. 9B). The LOTJ also has direct coupling with the SPC and OP regions, forming a posterior brain connectivity triangle. As in SPC, LOTJ connectivity converges to the OP–AI+ axis with further steps.

The DACC+ (Fig. 10A) stepwise connectivity results show that this region is directly coupled with the OP–AI+ axis. In this sense, it seems that DACC+ is actually an important component of this connectivity core (compare Figs. 8A,B, 10). On the other hand, DLPFC stepwise maps display a distinctive pattern of connectivity compared to other multimodal regions (Fig. 10B). With one step, DLPFC has connections to OP and AI+. However, the connectivity profile is more posterior in the OP region and more anterior in the AI+ region than the pattern displayed by the other multimodal regions (Fig. 10B, inset, stars). Although these differences are subtle, the more posterior part of the OP region and the more anterior part of the AI+ have been related to another functional network, namely, the frontoparietal control network (Yeo et al., 2011). The intimacy by which both networks are interdigitated in the inferior parietal and insula regions suggests that some of these multimodal network nodes might be implicated in the interrelation of perceptual integration and cognitive control functions.

Finally, to explicitly characterize the modular relationship between the multimodal regions, we performed an average-linkage hierarchical clustering analysis of the multimodal integration network (Fig. 11A,B). Briefly, we first averaged the FDR-corrected whole-brain association matrices (positive correlation coefficients) from data set 2. Then, we extracted a 48 × 48 matrix containing the seed voxels of OP, AI+, SPC, LOTJ, DACC+, and DLPFC regions (eight voxels per region) and performed the average-linkage hierarchical clustering analysis. The hierarchical partition shows two main network modules (cutoff criterion, r > 0.2), one integrated by SPC and the LOTJ (Fig. 11A,B, a), and the other by the OP, AI+, DACC+, and DLPFC (Fig. 11A,B, b). We observe that the connectivity between the two modules, a and b, is mostly through the connections of the SPC–OP axis (Fig. 11B). On the other hand, there is also a meaningful partition within the module b (cutoff criterion, r > 0.4): submodule b₁ formed by OP, AI+, and part of DACC+, and submodule b₂ formed by DLPFC and part of DACC+ (Fig. 11B). Figure 11C illustrates the main connectivity axes described in this study, especially highlighting the SPC–OP and OP–AI+ axes.

Figure 11.

Modules and connectivity axes of the multimodal integration network. A, B, The average-linkage hierarchical clustering analysis of the multimodal integration network (A) confirmed two main set of regions: (1) module a, formed by superior parietal cortex (dark yellow) and lateral occipitotemporal junction (light yellow), and (2) module b, formed by the parietal operculum (dark blue), anterior insula+ (violet), dorsal anterior cingulate+ (blue), and dorsolateral prefrontal cortex (light blue; B). The dendrogram shows, at a cutoff criterion of r > 0.2, the main partition in a and b modules but also a meaningful subdivision within module b at cutoff criterion r > 0.4. Submodule b₁ is formed by parietal operculum (dark blue), anterior insula+ (violet), and part of dorsal anterior cingulate+ (blue), and submodule b₂ is formed by dorsolateral prefrontal cortex (light blue) and part of dorsal anterior cingulate+ (blue). C, These results pointed out the existence of major cores and functional connectivity axes within the multimodal integration network. For instance, the superior parietal cortex–parietal operculum connectivity axis was important for the communication between modules a and b. The parietal operculum, anterior insula+, and part of the dorsal anterior cingulate nodes formed a central core in the overall organization of the multimodal network.

Interconnectors between modality pairs

To further characterize patterns of functional connectivity between pairs of sensory modalities, we used another novel network analytical strategy that detects bimodal brain regions. The analysis of bimodal interconnectors is significantly different from the previous SFC analysis; it specifically targets the connections between two primary regions rather than the unconstrained transitions of connectivity revealed by SFC. Yet, similar to the SFC analysis, it uses a stepwise approach to detect direct and indirect connections for all the possible pairs of sensory modalities: visual–auditory (V–A), visual–somatosensory, (V–S) and auditory–somatosensory (A–S) (Fig. 12).

The V–A results demonstrate that the posterior middle temporal gyrus, an area anterior to the LOTJ multimodal region, is engaged in the merging of direct connections between the visual and auditory cortex (Fig. 12A, a). An analogous area has been extensively reported as visual–auditory bimodal integrator using other neuroimaging techniques (Driver and Noesselt, 2008). The V–S findings show a more distributed pattern, where SPC (Fig. 12A, b) and dorsal and lateral occipital (Fig. 12A, c) regions display strong visual and somatosensory bimodality. Additionally, a motor region known to be related in oculomotor processing shows interconnector properties between visual and somatosensory cortices (Fig. 12A, b, anterior part). Primary motor and somatosensory cortices are locally interlocked by mutual connections across the central sulcus (Pandya and Kuypers, 1969; Jones and Powell, 1970). Therefore, it is not surprising that oculomotor areas have a deep visual–somatosensory interconnectivity suggesting early integration of somatomotor and visual processing for oculomotor functions. Finally, OP connections, such as OP4 (BA43), are especially critical for the bimodal interconnectivity of auditory and somatosensory primary cortices (Fig. 12A, d). Four regions in the upper bank of the lateral sulcus can be distinguished in humans, OP1 to OP4, which correspond to S2, parietal ventral (PV), and ventral somatosensory areas in monkeys (Disbrow et al., 2000; Eickhoff et al., 2006a,b). Our findings in the bimodal interconnector analysis suggest that OP4 is not only a somatosensory secondary representation, but also a plausible early bimodal interconnector that may influence the strong modularity described previously between the somatomotor and auditory cortices (Yeo et al., 2011). Indeed, PV in monkeys is connected to both primary somatosensory as well as medial auditory belt areas (Disbrow et al., 2003).

Finally, we find that indirect interconnectors are less specific for detecting bimodal integration. Maps of indirect interconnectors show trimodal rather than bimodal regions, although with some unique features depending on the pair of modalities. For instance, we find extensive occipital, including MT+, and FEF engagement when visual modality is being analyzed. Figure 12B is a diagram summarizing key bimodal integrators in the human brain.

Discussion

Throughout the history of neuroscience, scientists have been studying where and how brain systems integrate perceptual modalities to be incorporated into the more complex texture of cognition (Mesulam, 1998). A recent perspective that emphasizes functional connections of the brain promises to further our understanding of the underlying network structure of the human brain through the study of the connectome. If articulated networks occur as a whole in the brain to finally enable human cognition, then subdivisions of sensory, multimodal, and cognitive processing should be studied in terms of interdigitated systems and network interactions rather than segregated parts (Goldman-Rakic, 1988). In this study, we used a novel approach called SFC analysis to identify sensory cortex interactions and transitional connections between systems of the human brain.

Multimodal integration, salience-attention processing, and other overlapping concepts

The multimodal network integrates the primary cortices and connects them to the parallel systems at the top of the brain functional hierarchy. In the visual system, we found that stepwise connectivity patterns move along the dorsal and ventral visual streams. Furthermore, we uncovered some differences compared to various interpretations of these streams (Kim et al., 1999; Corbetta et al., 2008). First, the dorsal visual connectivity transitions move through different visual-related regions along two branches, one from V1 to V7 and another from V1 to MT+. On the other hand, the ventral connectivity stream in our findings is a medial pathway that connects V1 to other ventromedial visual regions to finally reach V8 and the posterior parahippocampus. In all cases, the multimodal network, especially SPC and OP, are the common brain regions for the convergence of the three visual streams. Contrary to prior interpretations, we do not see major stepwise connectivity to anterior parts of the temporal lobe in the ventral stream or a dominant connectivity, for instance, between the SPC and FEF in the dorsal stream. In summary, visual streams mostly merge in nodes of the multimodal network and not in other plausible regions such as the intraparietal sulcus (IPS), FEF, or middle prefrontal gyrus, as postulated previously (Corbetta et al., 2008). However, it is possible that these regions and other connectivity axes may gain importance during task-related activity. In this sense, IPS–FEF or SPC–FEF connections may be functionally more engaged, compared to what our results suggest, during visual and goal-directed stimulus–response tasks.

When considering the results of visual, auditory, and somatosensory modalities together, our findings are less exposed to the classic dorsal versus ventral dichotomic interpretation of the perceptual or attention-related systems (Corbetta and Shulman, 2002). Our interpretation is that by using the multimodal network, the three major primary sensory systems of the brain take advantage of both ventral and dorsal connections to the same extent. Of particular interest is the integration of dorsal and ventral regions through the SPC–OP axis and the posterior triangle of connectivity between the SPC, OP, and LOTJ. In the past, connections between the SPC and OP have been extensively reported, for instance, in nonhuman primate anatomical studies. Monkey Area 7b (analogous to our large OP region) is selectively connected with BA5 (analogous to our large SPC region) and also with insular cortex and the supplementary somatosensory area (Cavada and Goldman-Rakic, 1989). Other findings have shown strong evidence of connectivity from BA5 to the upper bank of the lateral sulcus in the perisylvian region (Jones and Powell, 1970), or from OP areas to BA5 in the superior parietal lobe (Disbrow et al., 2003).

Not surprisingly, several of the multimodal network regions characterized in this study have been described previously in the attention and stimuli salience processing literature. For instance, anterior components of the multimodal integration network, especially AI+ and DACC+, have been described as part of the so-called salience network (Seeley et al., 2007; Bressler and Menon, 2010), and regions similar to OP and AI+ have been implicated in multimodal and ventral attentional processing (Corbetta and Shulman, 2002; Downar et al., 2002; Seeley et al., 2007; Burton et al., 2008; Eckert et al., 2009; Menon and Uddin, 2010). Although we cannot describe the network changes that presumably take place during task in this study, the tremendous overlap between our findings, especially in AI+ and DACC+, and prior studies using attentional paradigms (Dosenbach et al., 2006), suggest that concepts such as multimodal connectivity integration, salience, or goal-directed attention all refer to a great extent to a common brain mechanism. Due to our functional connectivity approach, we focus on the multimodal interpretation of our findings. Yet it is obvious that many different cognitive processes bring into play, to some degree, multisensory integrated representations and presumably activate similar cortical regions.

“From Sensation to Cognition”

The title of this section is taken from the paper by Mesulam (1998, p. 1013). SFC is able to provide a meaningful picture of the large-scale brain processing from sensory to cognitive-related cortices. After one decade of isolating and discovering brain functional connectivity networks, research focus has now shifted to understanding how networks interact and merge in the brain. In this study, we found that the cortex is structured in such a manner that uses the multimodal integration network to interface between external information and cognitive hubs. Cortical hubs, and particularly the default mode network (DMN), are associated with autobiographical, self, and social functions (Buckner et al., 2008). Due to the connectivity properties of the cortical hubs, they are expected to be at the top of the hierarchical structure of the brain at rest (Goldman-Rakic, 1988; Buckner et al., 2009; Bressler and Menon, 2010; Sepulcre et al., 2010). In this sense, our data support the traditional view of sensory-specific information feeding forward into higher multimodal, heteromodal, and transmodal convergence zones (Driver and Noesselt, 2008) and information flowing in the cortex in a hierarchical configuration (Hubel and Wiesel, 1962; Pandya and Kuypers, 1969; Jones and Powell, 1970; Ungerleider and Desimone, 1986; Felleman and Van Essen, 1991; Clavagnier et al., 2004).

In their seminal monkey anatomical tracing study, Jones and Powell (1970) used an approach conceptually analogous with SFC. Although the direct comparison between Jones and Powell's (1970) results and ours would be in several ways anachronistic, we found striking similarities. The authors aimed to analyze the convergence of sensory pathways through the analysis of “the sequence of association connections passing outwards from the primary sensory areas as though following the successive steps in a (supposed) sequence of cortical function…to identify regions of convergence within the cortex” (Jones and Powell, 1970, p. 794). Using that strategy, Jones and Powell (1970) found that somatosensory cortex (S1) send fibers to (1) secondary somatosensory regions (S2), then to (2) BA5, then to (3) a large portion of BA7, and finally to (4) frontal and temporal limbic and heteromodal regions—regions that are known today as the DMN. In other words, from a local connectivity to the somatomotor and S2 region, somatosensory connections progress to multimodal regions such as BA5 (analogous to our SPC region) and BA7 (analogous to our OP region), and from there the connectivity spreads to DMN in the monkey (analogous to the human DMN). Interestingly, the STS has been reported as an important region for sensory integration in monkeys (Driver and Noesselt, 2008), and, as shown by Jones and Powell (1970), pathways from the three major modalities converge along the entire STS. In humans we only found a small portion of the STS involved in multimodal processing as part of the LOTJ findings.

Finally, of special relevance are our findings in the DLPFC multimodal region. Although the cortical hubs represent the top of the brain network hierarchy, other networks play important cognitive roles during specific tasks such as the frontoparietal control network (Vincent et al., 2008; Spreng et al., 2010). The anatomical proximity between the regions of the multimodal network and the frontoparietal control network is especially noticeable in the DLPFC stepwise connectivity maps. It has already been postulated that anterior insula and dorsal cingulate cortex may function as a dynamic switcher between the mutually exclusive frontoparietal control and default mode networks (Sridharan et al., 2008; Bressler and Menon, 2010). In this sense, it may be possible that some regions of the multimodal network, especially the anterior submodule, including DLPFC, are involved in cognitive interactions or transitions between active and passive tasks. On the other hand, the sensory awareness seems to relay on specific yet flexible binding between sensory-related processing regions and frontal and parietal cognitive networks that are related to active control of information and working memory (Engel and Singer, 2001). In our work this is especially relevant for the DLPFC region. The DLPFC connectivity network directly merges the multimodal and frontoparietal control networks and may contribute to the fine spatial–temporal integration of sensory information into highly distributed networks.

Conclusions

The findings of this study extend our knowledge of how the human brain network architecture transitions from perceptual cortex to multimodal regions, and ultimately to areas that support more conscious and high-order cognitive functions. A particularly interesting area for future exploration concerns the application of SFC analysis to neuropsychiatric disorders (such as neglect, autism, sensory integration dysfunction disorder, or ADHD) that are suspected to be a result of dysfunctional integration of perceptual or attentional brain systems.