Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Sep 1.
Published in final edited form as: Neuroimage. 2013 Apr 21;78:396–401. doi: 10.1016/j.neuroimage.2013.04.047

Parietal connectivity mediates multisensory facilitation

David Brang 1,2, Zack Taich 2, Steven A Hillyard 3, Marcia Grabowecky 1, VS Ramachandran 2
PMCID: PMC3672392  NIHMSID: NIHMS471071  PMID: 23611862

Abstract

Our senses interact in daily life through multisensory integration, facilitating perceptual processes and behavioral responses. The neural mechanisms proposed to underlie this multisensory facilitation include anatomical connections directly linking early sensory areas, indirect connections to higher-order multisensory regions, as well as thalamic connections. Here we examine the relationship between white matter connectivity, as assessed with diffusion tensor imaging, and individual differences in multisensory facilitation and provide the first demonstration of a relationship between anatomical connectivity and multisensory processing in typically developed individuals. Using a whole-brain analysis and contrasting anatomical models of multisensory processing we found that increased connectivity between parietal regions and early sensory areas was associated with the facilitation of reaction times to multisensory (auditory-visual) stimuli. Furthermore, building on prior animal work suggesting the involvement of the superior colliculus in this process, using probabilistic tractography we determined that the strongest cortical projection area connected with the superior colliculus includes the region of connectivity implicated in our independent whole-brain analysis.

Keywords: diffusion tensor, cross-modal, bimodal, redundant target, redundant signals, superior colliculus

1. Introduction

Our senses provide us with independent and sometimes dissimilar information, yet we need to detect, localize, and respond to events in the world based on unitary and coherent percepts. This perceptual unity is achieved through multisensory integration, which combines and facilitates the processing of temporally and spatially congruent information from different sensory modalities (e.g., Stein, Meredith, Wolf, 1993). The redundant-target paradigm (e.g., Cappe et al., 2010; Hershenson, 1962; Miller, 1982; Brang et al., 2012) demonstrates a basic and well-replicated finding that participants are faster to respond to a multisensory stimulus (a sound and a light combined) relative to a unisensory target (the sound or light in isolation), due to the facilitatory effects of multisensory integration. This facilitation is generally attributed to integration at the level of individual neurons, such that the response to a multisensory stimulus is greater than the response to either unisensory stimulus in isolation1, and has been proposed as a driving force behind the evolution of multisensory processes in both animals and humans (for a review see Stein and Stanford, 2008).

Multisensory integration reliably occurs in brain areas that receive inputs from multiple primary sensory modalities, including the cortex, midbrain, and thalamus (Jones and Powell, 1970) with the extent of integration varying according to task demands and stimulus complexity. Cells in the superior colliculus (SC) in particular are involved in orienting to a multisensory stimulus and initiate directed eye and head movements through projections to brainstem motor nuclei (Gandhi and Katnani, 2011; Krebs et al., 2010). Temporal, parietal, and frontal association areas have demonstrated multisensory integration during a wide array of tasks both with functional imaging (Macaluso and Driver, 2005) and single unit recordings (e.g., Bruce, Desimone, Gross, 1981) and lesions to these regions are typically associated with deficits in multisensory processing (e.g., Teuber, 1966).

Several mechanisms have been proposed to underlie multisensory integration, differing in the anatomical pathways involved and the modulatory role of connectivity. Multisensory convergence models argue that information is relayed from the primary sensory modalities through afferent connections to multisensory neurons in temporal, parietal, and frontal cortical regions (Driver and Noesselt, 2008; Jones and Powell, 1970) leading to behavioral facilitation though projections to the primary motor cortex (Molholm et al., 2006) and the superior colliculus (Jiang et al., 2001). A second, direct anatomical model attributes multisensory integration to direct anatomical connections between the primary sensory modalities, including between unimodal visual and auditory cortical areas (Beer, Plank, Greenlee, 2011; Falchier et al., 2002; Lewis and Noppeney, 2010; Romei et al., 2009; Falchier et al., 2010). Questioning the necessity of anatomical connections between the senses in mediating multisensory integration, recent evidence suggests substantial multisensory processing occurs in areas typically considered unisensory (Ghazanfar and Schroeder, 2006), including primary and secondary auditory cortex (Kayser, Petkov, Logothetis, 2008). Indeed, laminar profiles of auditory cortex show both feedforward and feedback projections from visual cortex, indicating that visual information is relayed to auditory cortex in the initial bottom-up flow of processing (Schroeder and Foxe, 2002).

These classes of models make different predictions regarding the effect intraindividual variability in anatomical connectivity has on multisensory processing. As both increased connectivity and increased myelination are related to increased processing speed (Turken et al., 2008) both multisensory convergence and direct anatomical models predict that multisensory processing would be facilitated by high white matter coherence along anatomical pathways, but differ on where in the brain these critical connections lie. On the other hand, processes relying on multisensory responses in unisensory regions assign less importance to variations in long-range anatomical connectivity and predict relatively little effect of individual differences in connectivity on multisensory processing. Here we examine the role of anatomical connectivity, as assessed with diffusion tensor imaging (DTI), in human multisensory facilitation. Multisensory facilitation was quantified with a redundant-target paradigm in which subjects responded as quickly as possible to auditory, visual, and auditory-visual stimuli. Subjects typically respond faster to multisensory targets than the timing predicted by statistical summation of the unisensory targets (exceeding what is known as the race-model prediction; Miller, 1982); the degree to which subjects’ average multisensory response speed exceeds that of the race-model reflects their level of multisensory facilitation.

2. Materials and Methods

2.1 Subjects

Twenty-seven subjects (22.7±3.2 years, fifteen females, 24 right-handed) participated in the study. Subjects had no history of neurological disorders, and gave their informed consent to participate in the experiment.

2.2 Individual differences in multisensory facilitation

2.2.1 Redundant-target paradigm

Participants were seated in front of a PC screen (refresh rate 100 Hz) with their eyes 57 cm from the center of the screen. The stimuli were delivered using E-Prime software (Psychology Software Tools, Inc.). Participants pressed the space bar on a PC keyboard to start each trial. Trials began with a 1.5 second fixation cross followed by a variable random delay (500 – 1500 ms) preceding either a salient visual stimulus (a red “X”, 12 cd/m2 presented at fixation subtending 0.7 degrees of visual angle, presented for 100 ms against a white background, 65 cd/m2), a salient auditory stimulus (a 3500 hz 64 dB tone presented for 100 ms via headphones), or both targets simultaneously in the multisensory condition. Catch trials were also included to discourage anticipatory responding. Participants were instructed to respond via E-prime’s Serial Response box with the index finger of their dominant hand as soon as they perceived a stimulus. There were 4 blocks of 74 trials; each block began with 4 randomly selected trials that were treated as practice trials and excluded from analysis, followed by 20 trials in each condition (auditory, visual, and multisensory) plus 10 blank catch trials. As such, each participant generated 80 reaction times for each experimental condition.

2.2.2 Redundant-target paradigm analysis

As the multisensory condition presented participants with two targets compared to a single target presented in either the auditory or visual conditions, some facilitation of reaction times is attributable to two independent stimuli contributing to response generation and execution. To calculate the extent of multisensory benefit in addition to that which is predicted by the redundant nature of the multisensory condition, the independent race model was used as a comparison for multisensory benefits compared to the joint probability of responses from either sensory stimulus alone (Laurienti et al., 2006; Miller, 1982, 1986). The independent race model utilizes cumulative distribution functions (CDFs) to compare relative probabilities for each condition within 1 ms time windows and is constructed from the joint probabilities of the auditory alone and visual alone conditions [(Pr Auditory + Pr Visual) − (Pr Auditory * Pr Visual)]. If the cumulative distribution function for the multisensory condition exceeds that which would be predicted by the race-model then the model is violated. These violations of the race model are thought to reflect true multisensory integration at the neural level (Laurienti et al., 2006; Miller, 1982). Accordingly, each subjects’ data were divided into 1 ms reaction time bins, used to create individual CDFs for each condition (auditory, visual, and multisensory). Next, the race-model predictions at each time bin were computed for each subject based on the auditory and visual CDFs. Finally, average race-model predictions were calculated for each subject from the mean of the response times in the race-model CDFs for comparison against the raw multisensory response times. To confirm that subjects’ multisensory response times were faster than those of the race model we compared the conditions using a two-tailed independent t-test. Multisensory facilitation for each participant was defined as the difference between mean race-model prediction minus the observed mean response to multisensory stimuli.

2.3 Whole-brain analysis

2.3.1 MRI acquisition

MR imaging was acquired on a General Electric 1.5T Excite HDx MRI scanner. The T1-weighted anatomical dataset was acquired in the sagittal plane with a 3D MPRAGE sequence (TR = 10.73 ms, TE =2.8 ms, TI = 1000 ms, FOV = 25 cm, ip angle = 8°, whole brain, slice thickness = 1 mm, 176 slices). DTI was acquired along 51 directions with a b-value of 1000 s/mm2 (as well as one image with no diffusion weighting) with a voxel-size of 1.875 × 1.875 × 2.5 mm3 (TR = 13.2 s, TE = 80.4 ms, FOV = 24 cm, 47 oblique slices AC/PC aligned encompassing the whole brain, and 2.5 mm slice thickness).

2.3.2 DTI analysis

DTI measurements yield information about white matter coherence within a region. The most consistent measure derived from DTI is fractional anisotropy (FA), with higher values reflecting greater numbers of axons, reduced axonal diameter, reduced fiber crossing, and increased myelination (Johansen-Berg Heidi and Rushworth, 2009). FA was calculated using FMRIB Software Library (FSL; Smith et al., 2004) and we conducted a whole-brain voxel-wise statistical analysis of the FA data using TBSS (Tract-Based Spatial Statistics; Smith et al., 2007; part of FSL). First, FA images were created by fitting a tensor model to the raw diffusion data using FDT, and then brain-extracted using BET (Smith, 2002). All subjects’ FA data were then aligned into a common space (MNI152) using the nonlinear registration tool FNIRT (Andersson, Jenkinson, Smith, 2007) which uses a b-spline representation of the registration warp field (Rueckert et al., 1999). Next, the mean FA image was created and thinned to create a mean FA skeleton that represents the centers of all tracts common to the group. Each subject’s aligned FA data were then projected onto this skeleton and the resulting data fed into voxel-wise cross-subject statistics. Voxel-wise permutation-based testing and inference was performed using Randomise in FSL (10,000 permutations) to test for a relationship between FA and multisensory facilitation. Statistical maps were family-wise error corrected using p < .05 using FSL’s threshold-free cluster enhancement (TFCE) metric (Smith and Nichols, 2009), in which raw voxel-wise t-statistics are adjusted based on local spatial clusters, producing whole-brain corrected volumes. Raw data from significant regions are displayed in Figure 2 where coefficients are reported as descriptive values only as these values were selected from a whole-brain analysis and will be biased to include positively skewed noise within voxels that reach significance, resulting in inflation of the true coefficient size.

Figure 2.

Figure 2

Open in a new tab

(a) Areas of significant correlation between FA and multisensory facilitation (red), obtained with FSL whole-brain corrected TBSS analysis, in white matter underlying the right parietal lobe and primary motor cortex (M1). Results displayed on radiologically oriented MNI brain. (b) Correlation coefficients relating behavioral performance on the multisensory facilitation task with FA extracted from parietal ROI (yellow circles) and M1 ROI (green squares) and derived from the whole-brain analysis; coefficients are reported as descriptive values.

2.4 Multisensory models

2.4.1 Model development and testing

After the TBSS analysis we used FSL tools Bedpostx and Probtrackx (Behrens et al., 2003) to generate collections of tracts based on anatomical connectivity along four groups of pathways to test for correlations between average FA along these tracts with multisensory facilitation. Bedpostx uses Monte Carlo Markov chain sampling to build up diffusion parameters at each voxel. Probtrackx was used to identify tracts that passed through two various waypoints (5000 streamlines from each voxel), while excluding all tracts that passed through areas which were not white matter in order to avoid impossible tracts (using an inverted, binarized version of the white matter parcellation masks generated by FreeSurfer). Waypoints were generated from the automatic cortical parcellation in FreeSurfer (2009 Destrieux Atlas; Destrieux et al., 2010; Fischl et al., 2004).

Exploring connectivity directly between auditory and visual areas (Aud-Vis model), tracts were identified between the auditory cortex (transverse temporal gyrus) and visual cortex (V1 and V2, probabilistically defined based on cytoarchitechtonic whole-brain histology; Fischl et al., 2008). The other groups of pathways examined connectivity from the auditory and visual cortices to two distinct parietal regions and one temporal region, selected based on evidence from past research of their potential involvement in multisensory processing, and included the superior parietal lobule (SPL model), intraparietal sulcus (IPS model), and posterior superior temporal sulcus (pSTS model). The SPL and IPS were generated automatically by Freesurfer and the pSTS was segmented from a posterior segment of the superior temporal sulcus label in Freesurfer (Beauchamp et al., 2008). One subjects’ probabilistic tractography in the right hemisphere yielded no connections between auditory areas and the SPL, and so this subject’s right hemisphere SPL model was excluded from all analyses.

Each subjects’ probabilistically defined pathways were normalized according to the initial number of seed voxels in each waypoint producing normalized probability density functions for each tract. Pathways were thresholded to include only voxels containing greater than 5% of the total number of fibers to eliminate spurious connections (results were unchanged at thresholds of 3% and 8%) generating a mask image for each path in each subject from which average FA was extracted for subsequent analysis; masks were subsequently registered to MNI space and averaged across subjects for display purposes.

2.4.2 Model analysis

After generation of masks for the each of the Aud-Vis, SPL, IPS, and pSTS models, mean data from each hemisphere was extracted from the four models resulting in eight values per subject. Pearson correlations for the left and right hemispheres were computed for multisensory facilitation values (dependent variable) with mean FA values extracted from each of the four models (independent variables of Aud-Vis, SPL, IPS, pSTS model pathways). In order to examine the spatial specificity of each correlation observed between multisensory processing and FA localized within these tracts, we used a permutation-based analysis (1000 permutations) to test each model against the null-distribution that a result equal to or greater than the observed Pearson correlation would be obtained by correlating N randomly selected white matter voxels with multisensory facilitation value (where N is equal to the number of voxels present in each model in each subject). Randomly selected white matter voxels were drawn from the same hemisphere as each respective model and sampled independently for each subject on each permutation iteration.

2.5 Connectivity seeded in the superior colliculus

Two of the authors (DB, ZT) generated masks of the superior colliculi in all participants. FSL tools Bedpostx and Probtrackx (Behrens et al., 2003) were used to generate probabilistic tractography seeded in the superior collicuclus (SC) for all participants. Probtrackx was used to identify tracts (5000 streamlines from each voxel) originating in the SC of each hemisphere, limited to the respective hemisphere by parcellation masks generated for T1 images in Freesurfer. SC tracts were normalized according to the initial number of seed voxels in the SC, producing normalized probability density functions for each participant. SC tracts were subsequently registered to MNI space, and thresholded to include only voxels in containing greater than .05% of the total fibers to eliminate spurious connections, and summed across participants to generate a spatial histogram reflecting similar SC tracks across individuals. Results were empirically the same when examining the median normalized probability density functions across the group.

3. Results

3.1 Individual differences in multisensory facilitation

Subjects’ mean response times were 388 ms (s.d. = 92) for auditory targets, 384 ms (s.d. = 56) for visual targets, and 307 ms (s.d. = 53) for multisensory targets (Figure 1, main plot)2. Race-model predictions were calculated from the joint independent probability of unisensory target reaction times [(auditory + visual) − (auditory x visual)]; mean race-model predictions were 331 ms (s.d. = 57). Multisensory response times significantly exceeded predictions of the race-model, t(26) = 10.27, p < .001, and difference scores between subjects’ race-model predictions and multisensory reaction times yielded an average multisensory facilitation time of 24 ms, ranging from 1 to 46 ms across subjects (Figure 1, sub-plot).

Figure 1.

Figure 1

Open in a new tab

Main panel: Group averaged cumulative distribution functions for the three stimulus conditions (auditory, visual, multisensory) and computed race-model distribution. Sub panel: Histogram showing individual subjects’ race-model - multisensory difference scores. Overlaid black line reflects normal distribution.

3.2 Whole-brain analysis

The whole-brain, multiple comparisons corrected analysis found significant (corrected p < .05) positive correlations between multisensory facilitation and FA in two regions: white matter underlying the right parietal lobe (center of gravity (COG) 19.8 (x), −50.9 (y), 42.9 (z); pmin 0.042, cluster size 231 voxels; Figure 2) and the right primary motor cortex (M1; COG 29.4 (x), −20.3 (y), 39.2 (z); pmin 0.045, cluster size 209 voxels). No significant negative correlations were observed (all corrected p > .92).

3.3 Multisensory models

In addition to FA values reflecting white matter coherence, DTI also yields information on the orientation of white matter connectivity within a voxel. Specifically, probabilistic tractography estimates the most likely path of diffusion from a given region in the brain, non-specifically reflecting afferent or efferent connectivity. As the whole-brain analysis is limited to identifying differences localized within white matter, the cortical areas involved in the identified network cannot be readily identified purely based on spatial proximity. In order to elucidate the specific tract implicated in multisensory facilitation, we used probabilistic tractography to construct pathways directly between auditory and visual areas (Aud-Vis model), replicating prior work by Beer and colleagues (2011), and from auditory and visual areas to the SPL, IPS, and pSTS.

FA values for each subject were extracted from all four models in each hemisphere and correlated with multisensory facilitation values. Results revealed a significant relationship between anatomical connectivity and multisensory facilitation for left hemisphere IPS (r = .33, p < .05) and SPL (r = .39, p<.05) models, but not pSTS (r = .26, p = .15) or Aud-Vis models (r = −.02, p = .87). A similar pattern was identified in the right hemisphere, with a significant relationship identified for the IPS model (r = .37, p <.05), marginally significant relationship for the SPL model (r = .27, p = .10), but non-significant findings for pSTS (r = .09, p = .55) or the Aud-Vis model (r = .18, p = .24). Steiger’s z-test for comparing dependent correlations (Steiger, 1980) was used to identify which, if any, of these correlations were significantly different from one another. The only marginal effect to emerge was between left hemisphere SPL and Aud-Vis models (z = 1.70, p = .089).

3.4 Connectivity seeded in the superior colliculus

As described above, past animal work has highlighted multisensory facilitation in the SC. In order to examine whether the significant regions identified with our whole-brain analysis showed prominent connections with the SC, using probabilistic tractography we identified unconstrained anatomical connections between the SC and the cortex. Consistently across our 27 subjects, the main pathway interconnecting the SC and the cortex was the parietal lobe in both hemispheres (Figure 3). Critically, this region showed marked overlap with the region identified in our whole-brain analysis (Figure 2) in which FA correlated positively with multisensory facilitation. Furthermore, repeating the correlation between multisensory facilitation and FA contained within anatomical tracts connecting the superior colliculi with both hemispheres accounted for 22% of the variance [r = .47, p < .01] in our participants’ individual differences in multisensory facilitation.

Figure 3.

Figure 3

Open in a new tab

Group overlap in unconstrained probabilistic tractography seeded in the left (left three images) and right (right three images) superior colliculi. Overlapping pathways range from light blue (overlap from 14 subjects) to dark blue (overlap from all 27 subjects). The left two images also show overlap with the area of significant correlation between FA and multisensory facilitation (red).

4. Discussion

This study demonstrates a prominent relationship between multisensory processing and anatomical connectivity in healthy adults, suggesting that increased white matter coherence between the parietal lobe and modality specific cortical areas provides a more efficient network for multisensory processes. Specifically, using a whole-brain analysis and correcting for multiple comparisons, we identified a positive relationship between multisensory facilitation and FA in white matter underlying the right parietal lobe. Using probabilistic tractography to construct reliable anatomical pathways underlying direct Aud-Vis, and indirect IPS, SPL, and pSTS pathways, we demonstrated significant relationships for IPS and SPL connections in both hemispheres. Lastly, examining anatomical connections between the SC and cortex, we demonstrated the most prominent pathway lay between the parietal lobe and SC, overlapping with the region of white matter critically involved in multisensory facilitation. These results provide strong support for models of multisensory facilitation based on anatomical connectivity with parietal regions.

The present results suggest an important role of either or both human SPL and IPS regions in mediating multisensory facilitation. Previous work has highlighted the role of the IPS in multisensory facilitation (for a review see Stein and Stanford, 2008), and that the development of this pathway is critically necessary for SC neurons to demonstrate enhanced multisensory gain (Jiang et al., 2001). More limited research implicates the SPL’s involvement in this process, and the present findings are in alignment with human intracranial recordings measured from the surface of the brain during a multisensory facilitation task (Molholm et al., 2006). In that study, behavioral facilitation was associated with enhanced activity in the SPL between 120 and 160 ms, demonstrating the timing at which this process onsets, but also suggesting that the SPL is a critical region that benefits from multisensory information3. However, as recordings in these patients were limited to the surfaces of cortical gyri, no electrodes were placed along the sulcus in which the IPS lays. Accordingly, it is possible that the activity recorded at SPL electrode sites was voltage projected from IPS neurons.

The demonstration of the role of white matter underlying the parietal lobe demonstrated here further builds on this model to suggest that the parietal lobe integrates information from the primary sensory modalities, in turn facilitating behavioral response. Our results point to two further mechanisms for the final stage of the facilitation of manual response times. In addition to white matter underlying the SPL and IPS, our whole-brain analysis also identified a significant positive correlation in right primary motor cortex. This cluster may reflect connections originating in the parietal lobe and terminating in primary motor cortex, leading to behavioral output, consistent with suggestions put forth by Molholm and colleagues (Molholm et al., 2006). However, as subjects in the present study were predominately right handed (3 left, 24 right) and responded with their dominant hand during the task, we would have expected a correlation reflecting behavioral output to be present in the left hemisphere as opposed to the right. One possible alternative account is that this area of connectivity underlying motor cortex reflects an indirect pathway from the SPL or IPS to the SC (Figure 3) with final output to primary motor cortex. Anatomical studies support the feasibility of this network as projections from cortex to the SC support multisensory processes in the midbrain (Jiang et al., 2001; Wallace and Stein, 2000) and as the SC is interconnected with thalamic relays (for a review see Boehnke and Munoz, 2008). In particular, the SC provides input to the medial pulvinar (Abramson and Chalupa, 1988; Cappe et al., 2009a) which in turn projects to parietal, motor and prefrontal regions (Boehnke and Munoz, 2008). A final consideration is that the SC and parietal lobe reflect phylogenetically old and new pathways, respectively, subserving parallel multisensory facilitation processes, an issue that can be better examined with the integration of DTI and electrophysiological methods.

One limitation of the current method is that DTI measures the movement of water molecules and cannot assess the directionality of white matter connections. As such, we cannot exclude the possibility that the present results reflect increased coherence of feedback connections from the parietal lobe to the primary sensory modalities (Macaluso and Driver, 2005; Schroeder and Foxe, 2005) as opposed to feed-forward convergence prior to motor output. Furthermore, the resolution afforded by DTI necessitates a simplification of the anatomical pathways that comprise these complex connections (Cappe et al., 2009b) raising the possibility that the lack of observed correlation with the Aud-Vis model is due to lack of sensitivity. However, as we were successful in identifying an anatomical pathway between auditory and visual cortices using probabilistic tractography, replicating prior DTI studies in humans (Beer, Plank, Greenlee, 2011) and anatomical tracer studies in animals (Falchier et al., 2002), it is perhaps more plausible that this pathway is critically involved in other forms of multisensory processing. This would be consistent with results from EEG studies showing fast transfer of information between auditory and visual areas (for a review see Senkowski et al., 2008), and suggestions that low-level interactions may mediate more basic and automatic perceptual processes, including salient illusions such as the sound-induced illusory flash paradigm (Bhattacharya, Shams, Shimojo, 2002; Mishra et al., 2007; Shams, Kamitani, Shimojo, 2000) and the enhancement of visual cortex excitability to sounds (Romei et al., 2009).

Finally, while the present study examined the role of anatomical connectivity on multisensory facilitation, there is a wealth of evidence suggesting that synchronized oscillatory activity (e.g., Lakatos et al., 2007) can also explain aspects of the unified and dynamic nature of multisensory integration (Senkowski et al., 2006). These functional connections typically utilize pre-existing anatomical networks (Skudlarski et al., 2008, though see Greicius et al., 2009) and include mechanisms such as phase-resetting, monosynaptic stimulation, and decreased response latency (for a review see Schroeder and Lakatos, 2009). As the hard-wired constraints of anatomical connections may lack the flexibility required for the generation of novel percepts from multisensory information (Senkowski et al., 2008), a critical understanding of the relative influences of individual differences in anatomical and functional connectivity on multisensory processing is a critical next step.

4.1 Conclusions

As multisensory processing varies widely across the population and is impaired in numerous clinical conditions including autism (Collignon et al., 2012) and schizophrenia (Williams et al., 2010) in addition to healthy aging (Setti et al., 2011) there is a pressing need to understand the differences in the multisensory network that underlie intrasubject variability in multisensory processing. Critically, these data strongly suggest a modulatory role of anatomical connectivity underlying the parietal lobe in multisensory facilitation, implicating abnormal patterns of connectivity as a candidate mechanism for deficient multisensory processes. However, future work will be needed to examine whether these specific pathways between the SPL and IPS to primary sensory modalities are directly implicated in these conditions or are part of a broader pattern of abnormal connectivity. This work additionally provides the foundation to test whether anatomical connectivity plays a modulatory role in all forms of multisensory integration and whether the associated anatomical pathways are task dependent. Indeed, multisensory interactions include many sensory modalities at many levels, ranging from simple sensory stimuli to linguistic processes and the precise network involved will likely differ based on stimuli and the type of interaction.

Highlights.

  • Faster response times to multisensory stimuli mediated by anatomical connectivity

  • Whole-brain analysis identified connections in the right parietal lobe

  • Probabilistic tractography implicates connectivity between parietal lobe and colliculus

Acknowledgments

This study was supported by NSF Grant BCS-1029084, NIMH Grant 7P50MH086385, NINDS Grant 2T32NS047987, and Abe Pollin.

Footnotes

1

Multisensory neurons exhibit different ranges of summed activity, with less than a quarter of neurons in the SC showing superadditivity and the majority showing subadditive integration (Perrault et al., 2005).

2

Prior studies using similar behavioral designs have typically yielded mean response times in the range of 50 to 100 ms faster than those observed here (e.g., Molholm et al., 2002).

3

In the three patients studied by (Molholm et al., 2006), all showed multisensory effects in the superior parietal lobule even though one patient showed no behavioral gain.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Abramson BP, Chalupa LM. Multiple pathways from the superior colliculus to the extrageniculate visual thalamus of the cat. Journal of Comparative Neurology. 1988;271(3):397–418. doi: 10.1002/cne.902710308. [DOI] [PubMed] [Google Scholar]
  2. Andersson JLR, Jenkinson M, Smith S. In Practice. 2007. Jun, Non-linear registration aka spatial normalisation FMRIB technial report TR07JA2. [Google Scholar]
  3. Beauchamp MS, Yasar NE, Frye RE, Ro T. Touch, sound and vision in human superior temporal sulcus. Neuroimage. 2008;41(3):1011–1020. doi: 10.1016/j.neuroimage.2008.03.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Beer AL, Plank T, Greenlee MW. Diffusion tensor imaging shows white matter tracts between human auditory and visual cortex. Experimental Brain Research. 2011;213(2–3):299–308. doi: 10.1007/s00221-011-2715-y. [DOI] [PubMed] [Google Scholar]
  5. Behrens TEJ, Woolrich MW, Jenkinson M, Johansen-Berg H, Nunes RG, Clare S, Matthews PM, Brady JM, Smith SM. Characterization and propagation of uncertainty in diffusion-weighted MR imaging. Magnetic Resonance in Medicine. 2003;50(5):1077–1088. doi: 10.1002/mrm.10609. [DOI] [PubMed] [Google Scholar]
  6. Bhattacharya J, Shams L, Shimojo S. Sound-induced illusory flash perception: Role of gamma band responses. Neuroreport. 2002;13(14):1727. doi: 10.1097/00001756-200210070-00007. [DOI] [PubMed] [Google Scholar]
  7. Boehnke SE, Munoz DP. On the importance of the transient visual response in the superior colliculus. Current Opinion in Neurobiology. 2008;18(6):544–551. doi: 10.1016/j.conb.2008.11.004. [DOI] [PubMed] [Google Scholar]
  8. Brang D, Williams LE, Ramachandran VS. Grapheme-color synesthetes show enhanced crossmodal processing between auditory and visual modalities. Cortex. 2012;48(5):630–637. doi: 10.1016/j.cortex.2011.06.008. [DOI] [PubMed] [Google Scholar]
  9. Bruce C, Desimone R, Gross CG. Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. Journal of Neurophysiology. 1981;46(2):369–384. doi: 10.1152/jn.1981.46.2.369. [DOI] [PubMed] [Google Scholar]
  10. Cappe C, Morel A, Barone P, Rouiller EM. The thalamocortical projection systems in primate: an anatomical support for multisensory and sensorimotor interplay. Cerebral Cortex. 2009a;19(9):2025–2037. doi: 10.1093/cercor/bhn228. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Cappe C, Rouiller EM, Barone P. Multisensory anatomical pathways. Hearing Research. 2009b;258(1–2):28–36. doi: 10.1016/j.heares.2009.04.017. [DOI] [PubMed] [Google Scholar]
  12. Cappe C, Thut G, Romei V, Murray MM. Auditory – Visual multisensory interactions in humans: Timing, topography, directionality, and sources. The Journal of Neuroscience. 2010;30(38):12572–12580. doi: 10.1523/JNEUROSCI.1099-10.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Collignon O, Charbonneau G, Peters F, Nassim M, Lassonde M, Lepore F, Mottron L, Bertone A. Reduced multisensory facilitation in persons with autism. Cortex. 2012 doi: 10.1016/j.cortex.2012.06.001. [DOI] [PubMed] [Google Scholar]
  14. Destrieux C, Fischl B, Dale A, Halgren E. Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature. NeuroImage. 2010;53(1):1–15. doi: 10.1016/j.neuroimage.2010.06.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Driver J, Noesselt T. Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron. 2008;57(1):11–23. doi: 10.1016/j.neuron.2007.12.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Falchier A, Clavagnier S, Barone P, Kennedy H. Anatomical evidence of multimodal integration in primate striate cortex. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience. 2002;22(13):5749–5759. doi: 10.1523/JNEUROSCI.22-13-05749.2002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Falchier A, Schroeder CE, Hackett TA, Lakatos P, Nascimento-Silva S, Ulbert I, Karmos G, Smiley JF. Projection from visual areas V2 and prostriata to caudal auditory cortex in the monkey. Cerebral Cortex. 2010;20(7):1529–1538. doi: 10.1093/cercor/bhp213. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Fischl B, Rajendran N, Busa E, Augustinack J, Hinds O, Yeo BTT, Mohlberg H, Amunts K, Zilles K. Cortical folding patterns and predicting cytoarchitecture. Cerebral Cortex. 2008;18(8):1973. doi: 10.1093/cercor/bhm225. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Fischl B, van der Kouwe A, Destrieux C, Halgren E, Ségonne F, Salat DH, Busa E, Seidman LJ, Goldstein J, Kennedy D, et al. Automatically parcellating the human cerebral cortex. Cerebral Cortex. 2004;14(1):11. doi: 10.1093/cercor/bhg087. [DOI] [PubMed] [Google Scholar]
  20. Gandhi NJ, Katnani HA. Motor functions of the superior colliculus. Annual Review of Neuroscience. 2011;34(1):205–231. doi: 10.1146/annurev-neuro-061010-113728. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Ghazanfar AA, Schroeder CE. Is neocortex essentially multisensory? Trends in Cognitive Sciences. 2006;10(6):278–285. doi: 10.1016/j.tics.2006.04.008. [DOI] [PubMed] [Google Scholar]
  22. Greicius MD, Supekar K, Menon V, Dougherty RF. Resting-state functional connectivity reflects structural connectivity in the default mode network. Cerebral Cortex. 2009;19(1):72–78. doi: 10.1093/cercor/bhn059. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Hershenson M. Reaction time as a measure of intersensory facilitation. Journal of Experimental Psychology. 1962;63(3):289–293. doi: 10.1037/h0039516. [DOI] [PubMed] [Google Scholar]
  24. Jiang W, Wallace MT, Jiang H, Vaughan JW, Stein BE. Two cortical areas mediate multisensory integration in superior colliculus neurons. Journal of Neurophysiology. 2001;85(2):506–522. doi: 10.1152/jn.2001.85.2.506. [DOI] [PubMed] [Google Scholar]
  25. Heidi Johansen-Berg, Rushworth MFS. Using diffusion imaging to study human connectional anatomy. Annual Review of Neuroscience. 2009;32:75–94. doi: 10.1146/annurev.neuro.051508.135735. [DOI] [PubMed] [Google Scholar]
  26. Jones EG, Powell TPS. Connexions of the somatic sensory cortex of the rhesus monkey III. - thalamic connexions. Brain. 1970;93(1):37–56. doi: 10.1093/brain/93.1.37. [DOI] [PubMed] [Google Scholar]
  27. Kayser C, Petkov CI, Logothetis NK. Visual modulation of neurons in auditory cortex. Cerebral Cortex. 2008;18(7):1560–1574. doi: 10.1093/cercor/bhm187. [DOI] [PubMed] [Google Scholar]
  28. Krebs RM, Woldorff MG, Tempelmann C, Bodammer N, Noesselt T, Boehler CN, Scheich H, Hopf JM, Duzel E, Heinze HJ. High-field FMRI reveals brain activation patterns underlying saccade execution in the human superior colliculus. PloS One. 2010;5(1):e8691. doi: 10.1371/journal.pone.0008691. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Lakatos P, Chen C, O’Connell MN, Mills A, Schroeder CE. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron. 2007;53(2):279–292. doi: 10.1016/j.neuron.2006.12.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Laurienti PJ, Burdette JH, Maldjian JA, Wallace MT. Enhanced multisensory integration in older adults. Neurobiology of Aging. 2006;27(8):1155–1163. doi: 10.1016/j.neurobiolaging.2005.05.024. [DOI] [PubMed] [Google Scholar]
  31. Lewis R, Noppeney U. Audiovisual synchrony improves motion discrimination via enhanced connectivity between early visual and auditory areas. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience. 2010;30(37):12329–12339. doi: 10.1523/JNEUROSCI.5745-09.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Macaluso E, Driver J. Multisensory spatial interactions: A window onto functional integration in the human brain. Trends in Neurosciences. 2005;28(5):264–271. doi: 10.1016/j.tins.2005.03.008. [DOI] [PubMed] [Google Scholar]
  33. Miller J. Timecourse of coactivation in bimodal divided attention. Perception & Psychophysics. 1986;40(5):331–343. doi: 10.3758/bf03203025. [DOI] [PubMed] [Google Scholar]
  34. Miller J. Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology. 1982;14(2):247–279. doi: 10.1016/0010-0285(82)90010-x. [DOI] [PubMed] [Google Scholar]
  35. Mishra J, Martinez A, Sejnowski TJ, Hillyard SA. Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. Journal of Neuroscience. 2007;27(15):4120. doi: 10.1523/JNEUROSCI.4912-06.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ. Multisensory auditory – visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cognitive Brain Research. 2002;14(1):115–128. doi: 10.1016/s0926-6410(02)00066-6. [DOI] [PubMed] [Google Scholar]
  37. Molholm S, Sehatpour P, Mehta AD, Shpaner M, Manuel Gomez-Ramirez, Ortigue S, Dyke JP, Schwartz TH, Foxe JJ. Audio-visual multisensory integration in superior parietal lobule revealed by human intracranial recordings. Journal of Neurophysiology. 2006;96(2):721–729. doi: 10.1152/jn.00285.2006. [DOI] [PubMed] [Google Scholar]
  38. Perrault TJ, Vaughan JW, Stein BE, Wallace MT. Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. Journal of Neurophysiology. 2005;93(5):2575–2586. doi: 10.1152/jn.00926.2004. [DOI] [PubMed] [Google Scholar]
  39. Romei V, Murray MM, Cappe C, Thut G. Preperceptual and stimulus-selective enhancement of low-level human visual cortex excitability by sounds. Current Biology. 2009;19(21):1799–1805. doi: 10.1016/j.cub.2009.09.027. [DOI] [PubMed] [Google Scholar]
  40. Rueckert D, Sonoda LI, Hayes C, Hill DLG, Leach MO, Hawkes DJ. Nonrigid registration using free-form deformations: application to breast MR images. Medical Imaging, IEEE Transactions on. 1999;18(8):712–721. doi: 10.1109/42.796284. [DOI] [PubMed] [Google Scholar]
  41. Schroeder CE, Lakatos P. Low-frequency neuronal oscillations as instruments of sensory selection. Trends in Neurosciences. 2009;32(1):9–18. doi: 10.1016/j.tins.2008.09.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Schroeder CE, Foxe J. Multisensory contributions to low-level,‘unisensory’processing. Current Opinion in Neurobiology. 2005;15(4):454–458. doi: 10.1016/j.conb.2005.06.008. [DOI] [PubMed] [Google Scholar]
  43. Schroeder CE, Foxe JJ. The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Research. Cognitive Brain Research. 2002;14(1):187–198. doi: 10.1016/s0926-6410(02)00073-3. [DOI] [PubMed] [Google Scholar]
  44. Senkowski D, Schneider TR, Foxe JJ, Engel AK. Crossmodal binding through neural coherence: Implications for multisensory processing. Trends in Neurosciences. 2008;31(8):401–409. doi: 10.1016/j.tins.2008.05.002. [DOI] [PubMed] [Google Scholar]
  45. Senkowski D, Molholm S, Manuel Gomez-Ramirez, Foxe JJ. Oscillatory beta activity predicts response speed during a multisensory audiovisual reaction time task: A high-density electrical mapping study. Cerebral Cortex {(New} York, {NY:} 1991) 2006;16(11):1556–1565. doi: 10.1093/cercor/bhj091. [DOI] [PubMed] [Google Scholar]
  46. Setti A, Burke KE, Kenny RA, Newell FN. Is inefficient multisensory processing associated with falls in older people? Experimental Brain Research. 2011;209(3):375–384. doi: 10.1007/s00221-011-2560-z. [DOI] [PubMed] [Google Scholar]
  47. Shams L, Kamitani Y, Shimojo S. Illusions: What you see is what you hear. Nature. 2000;408:788. doi: 10.1038/35048669. [DOI] [PubMed] [Google Scholar]
  48. Skudlarski P, Jagannathan K, Calhoun VD, Hampson M, Skudlarska BA, Pearlson G. Measuring brain connectivity: Diffusion tensor imaging validates resting state temporal correlations. NeuroImage. 2008;43(3):554–561. doi: 10.1016/j.neuroimage.2008.07.063. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Smith S, Nichols T. Threshold-free cluster enhancement: Addressing problems of smoothing, threshold dependence and localisation in cluster inference. NeuroImage. 2009;44(1):83–98. doi: 10.1016/j.neuroimage.2008.03.061. [DOI] [PubMed] [Google Scholar]
  50. Smith SM. Fast robust automated brain extraction. Human Brain Mapping. 2002;17(3):143–155. doi: 10.1002/hbm.10062. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Smith SM, Heidi Johansen-Berg, Jenkinson M, Rueckert D, Nichols TE, Miller KL, Robson MD, Jones DK, Klein JC, Bartsch AJ, et al. Acquisition and voxelwise analysis of multi-subject diffusion data with tract-based spatial statistics. Nature Protocols. 2007;2(3):499–503. doi: 10.1038/nprot.2007.45. [DOI] [PubMed] [Google Scholar]
  52. Smith SM, Jenkinson M, Woolrich MW, Beckmann CF, Behrens TEJ, Heidi Johansen-Berg, Bannister PR, De Luca M, Drobnjak I, Flitney DE, et al. Advances in functional and structural MR image analysis and implementation as FSL. NeuroImage. 2004;23(Suppl 1):S208–219. doi: 10.1016/j.neuroimage.2004.07.051. [DOI] [PubMed] [Google Scholar]
  53. Steiger JH. Tests for comparing elements of a correlation matrix. Psychological Bulletin. 1980;87:245–251. [Google Scholar]
  54. Stein BE, Meredith MA, Wolf S. The merging of the senses. MIT Press; Cambridge, MA: 1993. [Google Scholar]
  55. Stein BE, Stanford TR. Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews Neuroscience. 2008;9(4):255–266. doi: 10.1038/nrn2331. [DOI] [PubMed] [Google Scholar]
  56. Teuber HL. Alterations of perception after brain injury. In: Eccles JC, editor. Brain and Conscious Experience. Springer; Berlin-Heidelberg-New York: 1966. pp. 182–216. [Google Scholar]
  57. Turken AU, Susan Whitfield-Gabrieli, Bammer R, Baldo JV, Dronkers NF, Gabrieli JDE. Cognitive processing speed and the structure of white matter pathways: Convergent evidence from normal variation and lesion studies. NeuroImage. 2008;42(2):1032–1044. doi: 10.1016/j.neuroimage.2008.03.057. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Wallace MT, Stein BE. Onset of cross-modal synthesis in the neonatal superior colliculus is gated by the development of cortical influences. Journal of Neurophysiology. 2000;83(6):3578–3582. doi: 10.1152/jn.2000.83.6.3578. [DOI] [PubMed] [Google Scholar]
  59. Williams LE, Light GA, Braff DL, Ramachandran VS. Reduced multisensory integration in patients with schizophrenia on a target detection task. Neuropsychologia. 2010;48(10):3128–3136. doi: 10.1016/j.neuropsychologia.2010.06.028. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES