The Behavioral Relevance of Multisensory Neural Response Interactions

Abstract

Sensory information can interact to impact perception and behavior. Foods are appreciated according to their appearance, smell, taste and texture. Athletes and dancers combine visual, auditory, and somatosensory information to coordinate their movements. Under laboratory settings, detection and discrimination are likewise facilitated by multisensory signals. Research over the past several decades has shown that the requisite anatomy exists to support interactions between sensory systems in regions canonically designated as exclusively unisensory in their function and, more recently, that neural response interactions occur within these same regions, including even primary cortices and thalamic nuclei, at early post-stimulus latencies. Here, we review evidence concerning direct links between early, low-level neural response interactions and behavioral measures of multisensory integration.

Introduction

There are myriad everyday situations where information from the different senses provides either redundant or complementary information to facilitate perception and behavior. One example of such multisensory interactions is speech perception in noisy environments, where facial information improves comprehension (e.g., Sumby and Pollack, 1954). Other examples include the detection and localization of stimuli either in naturalistic or laboratory settings, where performance is often facilitated by multisensory stimuli (reviewed in Stein and Meredith, 1993; see also Murray et al., 2005; Zampini et al., 2007; Tajadura-Jimenez et al., 2009 for examples using auditory-somatosensory stimuli in humans). The facilitation of reaction times to multisensory stimuli is one instantiation of a redundant signals effect (Raab, 1962). On the one hand, this effect could be explained by truly independent processing of each sensory modality, such that the faster of the two mediates response execution (typically, a button-press or eye movement) on any given trial. When there are two sources of information, performance is facilitated because the probability of either of the two sources leading to a fast response is higher than either source alone – a purely statistical phenomenon referred to as probability summation. Under this framework, no neural response interactions are required. However, notable examples exist for neural response interactions, even when probability summation fully accounts for behavioral gains (e.g., Murray et al., 2001; Sperdin et al., 2009). On the other hand, the facilitation can exceed expectations based on probability summation (Miller, 1982), in which case neural response interactions, at some stage prior to response initiation, need to be invoked. One corollary of this facilitation is the importance of identifying those neural response interactions that are (causally) linked to behavioral improvements.

A Paradigm Shift for Models of Multisensory Processing

Research over the past decade or so has led to a significant paradigm shift in the manner in which neuroscientists conceive of the neural underpinnings of multisensory interactions on the one hand, and more generally the organization of the various sensory systems on the other hand. The traditional view held that the sensory systems were largely segregated at low-levels and early latencies of processing (e.g., Jones and Powell, 1970), with interactions and integration only occurring within higher-order brain regions, and at relatively late stages of processing. By consequence, any multisensory effects observed within low-level brain regions were presumed to be the product of feedback modulations from such higher-order structures. Instead, low-level and early multisensory effects have now been documented using anatomic, physiological, and brain imaging methods (reviewed in Wallace et al., 2004; Schroeder and Foxe, 2005; Ghazanfar and Schroeder, 2006; Kayser and Logothetis, 2007; Senkowski et al., 2008; Stein and Stanford, 2008). This new framework has consequently spurred interest in determining the precise circumstances when multisensory interactions and their behavioral consequences will and will not occur.

The Case of Auditory-Somatosensory Interactions

The case of auditory-somatosensory neural response interactions, wherein the response to the multisensory stimulus does not equal the summed responses from the constituent unisensory conditions, is illustrative of this abovementioned paradigm shift. Humans and non-human primates exhibit non-linear neural response interactions within the initial post-stimulus processing stages (Foxe et al., 2000; Schroeder et al., 2001; Murray et al., 2005; Lakatos et al., 2007). These interactions manifest at early latencies within secondary (also termed belt) regions of auditory cortex adjacent to primary (also termed core) auditory cortices (Schroeder et al., 2001, 2003; Fu et al., 2003; Kayser et al., 2005; Murray et al., 2005; Gonzalez Andino et al., 2005a; see also Cappe and Barone, 2005; Hackett et al., 2007a,b; Smiley et al., 2007; Cappe et al., 2009; for corresponding anatomic data). These early and low-level effects are seen despite paradigmatic variations in terms of passive stimulus presentation versus performance of a simple stimulus detection task (in the case of studies in humans) or even the use of anesthetics (in the case of studies in non-human primates).

The robustness of auditory-somatosensory interactions is also supported, albeit indirectly, by the cumulative psychophysical findings in humans. Facilitative effects on reaction time speed have now been observed not only when the stimuli are presented to the same location in space, but also when the stimuli are spatially misaligned. This is the case for left-right (Murray et al., 2005), front-back (Zampini et al., 2007), as well as near-far (Tajadura-Jimenez et al., 2009), spatial disparities (see also Gillmeister and Eimer, 2007; Yau et al., 2009). Such findings have been used to generate hypotheses concerning the spatial representation of auditory and somatosensory information within regions, and at latencies when the initial response interactions are observed. The rationale is predicated on the so-called “spatial rule” of multisensory interactions, which stipulates that the receptive field organization of a neuron (or neural population) is a determining feature of multisensory interactions and their quality (Stein and Meredith, 1993). Based on this principle and the above findings, it has been hypothesized that the initial auditory-somatosensory neural response interactions are occurring within brain regions whose neuronal population consists of large (potentially 360°) auditory spatial representations and unilateral somatosensory (i.e., hand) representations. Some support for this hypothesis is found in electrophysiological studies in humans (Murray et al., 2005) and monkeys (Fu et al., 2003) that varied the spatial position of the stimuli. For example, Murray et al. (2005) performed source estimations of neural response interactions and showed effects within the left caudal auditory cortices when the somatosensory stimulus was to the right hand, irrespective of whether the sound was within the left or right hemispace (and vice versa).

These kinds of results suggest that spatial information is not a determining factor in whether facilitative effects at a population level will be observed (though see Lakatos et al., 2007 for data concerning oscillatory activity within primary auditory cortex in response to contralateral and ipsilateral somatosensory input). We would hasten to note, however, that effects would not be expected to circumvent constraints enforced by a given neuron's receptive field properties or spatial tuning (e.g., Murray and Spierer, 2009, for a discussion of such issues). To address this more directly, we recently introduced a new paradigm combining a stimulus detection task, like that described above, with intermittent probes about the spatial location of stimuli on the preceding trial. In this way, we could assess whether the task-relevance of spatial information is sufficient for limiting facilitative effects to spatially aligned conditions (Sperdin et al., 2010). This was not the case. Rather, performance on stimulus detection was facilitated to an equal extent – both when stimuli were spatially aligned and misaligned. Still, other findings indicate that the particular body surface stimulated (Tajadura-Jimenez et al., 2009; see also Fu et al., 2003), and acoustic features of the sounds (Yau et al., 2009; see also commentary by Foxe, 2009) may play determinant roles in the pattern of behavioral (and neurophysiological) effects one observes.

More generally, and in part because of the large consistency in the above effects, auditory-somatosensory interactions represent a situation in which one might reasonably conclude that early effects within low-level cortices are relatively automatic, and unaffected by cognitive factors (Kayser et al., 2005). By extension, one hypothesis is that these interactions are not causally linked to behavioral outcome, as they appear to be robust – irrespective of whether or not anesthetics are used, and irrespective of variations in task demands. Closer inspection of some of the details of the extant studies, however, reveal several challenges with using the majority of the above studies to generate hypotheses regarding links between early and low-level neural response interactions and behavior. In the case of studies in monkeys, none included the performance of a task. However, electrophysiological recordings in Schroeder et al. (2001) and Lakatos et al. (2007), and hemodynamic imaging (Kayser et al., 2005), have been performed in awake and fixating animals. It will not be surprising if the coming years begin introducing behavioral tasks into their recording setups in animals (e.g., Komura et al., 2005; Hirokawa et al., 2008 for such types of studies in rats). In the case of studies of auditory-somatosensory interactions in humans, only the study by Murray et al. (2005) included a behavioral task – in particular a simple detection paradigm. Moreover, this study introduced some advances in electrical neuroimaging analyses of scalp-recorded electroencephalographic data and event-related potentials (Michel et al., 2004, 2009; Murray et al., 2008, 2009) to the domain of multisensory research. These analyses, along with application of source estimations, have the promise of facilitating the comparison of findings from humans with those from animal models (e.g., Gonzalez Andino et al., 2005a), as well as those from EEG (or MEG) with those from fMRI (e.g., compare localization in Foxe et al., 2002 with that in Murray et al., 2005). Of particular benefit is that these kinds of analyses circumvent two major statistical pitfalls of traditional voltage waveform analyses by using reference- independent measurements and by taking advantage of the added information from multi-channel recordings. In addition, and of high relevance for the multisensory researcher, these kinds of analyses permit the differentiation of two major families of neurophysiological mechanisms of multisensory interactions depicted in Figure 1. On the one hand, electrical neuroimaging analyses can differentiate whether an effect at a given latency follows from modulations in response strength vs response topography. The former would be consistent with a gain modulation, whereas the latter would forcibly follow from changes in the underlying configuration of intracranial sources according to Helmholz's principles (Lehmann, 1987). That is, these analyses can statistically determine if and when there are generator configurations uniquely active under multisensory conditions. On the other hand, electrical neuroimaging analyses – in large part because they are reference- independent – also allow for the differentiation of supra-additive and sub-additive interactions. This is particularly useful because the directionality of effects observed at individual voltage waveforms will vary with the choice of the reference (as will the presence and latency of statistical effects).

Figure 1.

Two potential varieties of multisensory interactions assessable by applying a linear model to the analysis of event-related potentials. The linear model involves comparing the summed responses from unisensory conditions with the response to the multisensory stimulus. In this figure, the level of activity (arbitrary units) within fictive brain regions is illustrated within the blue discs. In panel (A) modulations in response strength are illustrated, wherein the same set of brain regions is observed in response to the summed unisensory and multisensory conditions, albeit with greater magnitude in the latter case. This is illustrative of a supra-additive gain modulation. The colored topographic maps illustrate what one might observe in the ERP data. In panel (B) modulations in the configuration of brain regions active under multisensory stimulus conditions are illustrated, such that brain regions otherwise inactive under unisensory conditions are observed. In terms of event-related potential analyses, this latter mechanism would manifest as a modulation in the topography of the electric field at the scalp, which is illustrated in the voltage maps below. It should be noted that these two mechanisms, i.e., gain and generator modulations, can co-occur.

Early, Low-Level Auditory-Somatosensory Interactions are Linked to Fast Stimulus Detection

In Sperdin et al. (2009) we showed that early non-linear neural response interactions within low-level auditory cortices influence subsequent reaction time speed. To reach this conclusion, we sorted ERPs to auditory, somatosensory, and combined auditory-somatosensory multisensory stimuli according to a median split of reaction times during a detection task with these stimuli (see Figure 2). In this way we could separate both behavioral and electrophysiological responses according to whether the reaction time on a given trial was relatively fast or slow. At a behavioral level, only trials leading to fast reaction times produced facilitation in excess of predictions based on probability summation (and, therefore, necessitating invocation of neural response interactions). By contrast, the facilitation in the case of multisensory trials leading to slow reaction times was fully explained by probability summation.

Figure 2.

A schematic of the median split analysis approach applied in Sperdin et al. (2009). For each subject and stimulus condition, trials were sorted according to RT speed. Those with RTs faster than the median were considered “fast” and those slower than the median were considered “slow”. Event-related potentials were likewise separately averaged according to RT speed, and compared using the linear model schematized in Figure 1. Data were analyzed in using a multi-factorial within subjects design.

At a neurophysiological level, we observed non-linear neural response interactions over two time periods within the initial 200-ms post- stimulus onset (Figure 3). Over the 40–84 ms post-stimulus period, supra-additive modulations in response strength were observed when reaction times were ultimately fast (and by extension when an explanation of the reaction time facilitation based on probability summation did not suffice). No non-linear interactions were observed over this time period when reaction times were ultimately slow (and by extension when probability summation fully accounted for the reaction time facilitation). These early non-linear interactions, in the case of trials leading to fast reaction times, were moreover localized to posterior regions of the superior temporal cortex that have been repeatedly documented as an auditory-somatosensory convergence and integration zone (Foxe et al., 2000, 2002; Schroeder et al., 2001, 2003; Fu et al., 2003; Cappe and Barone, 2005; Gonzalez Andino et al., 2005a; Kayser et al., 2005; Murray et al., 2005; Hackett et al., 2007a,b; Smiley et al., 2007; Cappe et al., 2009). Over the 86–128 ms post-stimulus period, supra-additive modulations in response strength were observed independent of the ultimate speed of reaction times (and of whether probability summation accounted for the reaction time facilitation). That is, the presence or absence of violation of probability summation at a behavioral level was not a determining factor in whether non-linear neural response interactions were observed. Rather, early non-linear interactions were limited to trials leading to fast reaction times and also to violation of probability summation. Whether both of these psychophysical features (i.e., reaction time speed and violation of probability summation) are the outcome of early non-linear interactions awaits further investigation.

Figure 3.

Evidence for the impact of early non-linear and supra-additive neural response interactions on RT speed. The top panels illustrate global field power waveforms in response to multisensory stimulus pairs, and the summed unisensory responses for trials producing fast and slow RTs (left and right panels, respectively). While non-linear neural response interactions began at 40 ms post-stimulus for trials producing fast RTs, such was only the case from 86 ms onwards for trials producing slow RTs. The middle portion illustrates the difference in source estimations over the 40–84 ms post-stimulus period between responses to multisensory stimulus pairs and summed unisensory responses for trials producing fast and slow RTs (red and green framed images, respectively). The sagittal slice is shown at x = −53 mm using the Talairach and Tournoux (1988) coordinate system. The bottom panel illustrates the mean scalar value of differential activity within a cluster of 25 solution points within the left superior temporal cortex. There were significantly greater non-linear multisensory neural response interactions in the case of trials producing fast RTs.

Another important finding regarding the underlying mechanism of auditory-somatosensory interactions is that there was no evidence in our analyses for modulations in response topography and, therefore, no evidence that auditory-somatosensory multisensory interactions recruit distinct configurations of brain regions. Rather, our results are consistent with a mechanism based on changes in the gain of responses within brain regions already active under unisensory conditions (see also Gonzalez Andino et al., 2005a; Murray et al., 2005). The specific pattern leading to early supra-additive non-linear interactions in the case of fast trials is also noteworthy (c.f. Figures 2 and 3 in Sperdin et al., 2009). Responses to the multisensory condition did not significantly differ as a function of reaction time speed. Rather, responses to unisensory stimuli were significantly weaker when reaction times were ultimately fast. At a mechanistic level, this pattern would suggest that changes in unisensory processing may be at the root of whether performance is fast or slow.

It is likewise of note that it was the earlier period of non-linear interactions that modulated with later reaction time speed (i.e., that at 40–84 ms), as opposed to the later period that did not exhibit such a modulation (i.e., that at 86–128 ms). This pattern suggests that interactions that are behaviorally relevant during the time course of post-stimulus brain responses may be dissociable from those that are not. The electrophysiological effect at 40–84 ms appears to be linked to the relative (i.e., faster vs slower for a given participant) rather than absolute reaction time. In other words, electrophysiological effects were consistently observed over the 40–84 ms period despite the fact that what was labeled as fast reaction times varied across participants (i.e., individual means for “fast” multisensory trials ranged from 218–521 ms; mean ± sem = 309 ± 34 ms). Further research applying single-trial analysis methods (e.g., De Lucia et al., 2007, 2010; Murray et al., 2009) will be required to shed further light on this aspect of our results.

Toward Identifying Mechanisms for Behaviorally-Relevant Multisensory Interactions

Advances in animal recordings

While Sperdin et al. (2009) showed that early non-linear neural response interactions measured in ERPs from humans differ as a function of reaction time speed, they do not in and of themselves identify the underlying mechanism. In many regards, it can be argued that the converse situation (i.e., exquisite mechanistic information in the absence of links to performance or perception) exists in the overwhelming majority of studies of multisensory processing in animals, because often no task is required or recordings are conducted in anesthetized preparations. Such being said, some data do exist. For example, one of the earliest (to our knowledge) multisensory neurophysiological experiments in awake and behaving monkeys suggested that sensory-related responses within pre-central neurons was not related (at least in some obvious way) to reaction time (Lamarre et al., 1983). By contrast, the latency of movement-related activity corresponded to the animal's reaction times. However, a more detailed analysis of responses from a subset of six neurons recorded by Lamarre et al. (1983) indicated that the facilitation of reaction times following multisensory (auditory-visual) stimulation was not due to the speeding up of activity within motor cortices (Miller et al., 2001). Any facilitation of neural responses – they contended – was likely occurring at early processing stages that were not recorded in their study. In agreement with this prediction, Wang et al. (2008) indeed observed facilitation of neural response latencies within macaque primary visual cortex in response to multisensory vs visual stimuli. This effect was task-dependent, such that facilitation was only observed during an active discrimination task requiring saccadic eye movements and was absent when the monkey was passively viewing. Similarly, recordings from the superior colliculus of the anesthetized cat show there to be an initial response enhancement expressed as shortened response latencies and stronger response magnitude under multisensory conditions (Rowland and Stein, 2007, 2008; Rowland et al., 2007). However, any direct link to behavior is obfuscated by the use of an anesthetized preparation. More germane, this sampling of studies highlight the added information provided by the analysis of dynamic information within firing rates, and the potential ability to link neural activity (whether firing rates, post- synaptic potentials, or other varieties) to performance/perception when experiments are conducted in awake and behaving preparations.

Transcranial magnetic stimulation (TMS) as a tool for identifying behaviorally-relevant multisensory interactions

Most recently, our own group (in collaboration with that of Gregor Thut and Vincenzo Romei) has focused on using single-pulse transcranial magnetic stimulation (TMS) in combination with psychophysics to identify causal links between neurophysiological and behavioral metrics of multisensory interactions. In one study, we showed that single-pulse TMS applied to the occipital pole affects simple reaction times to visual and auditory stimuli in equally large but opposite ways (Romei et al., 2007). That is, TMS significantly slowed reaction times to visual stimuli and significantly facilitated reaction times to auditory stimuli (there were no reliable effects on reaction times to multisensory stimuli). Moreover, these effects were temporally delimited, occurring over the 60–90ms period, and the beneficial interaction effect of combined unisensory auditory and TMS-induced visual cortex stimulation matched and was correlated with the RT-facilitation after external multisensory AV stimulation without TMS. This pattern suggests that multisensory interactions occur between the stimulus-evoked auditory and TMS-induced visual cortex activities. Further evidence for such was provided by a follow-up experiment showing that auditory input enhances excitability within the visual cortex itself (using phosphene-induction via TMS as a measure) over a similarly early time period (75–120 ms) (see also Ramos-Estebanez et al., 2007).

We recently extended this latter finding by showing that auditory-driven changes in visual excitability depend on the quality of the sound, such that structured sounds (as opposed to white noise versions) signaling approach/looming resulted in the highest excitability enhancement, and, furthermore, occurred at pre-perceptual levels (i.e., at latencies and stimulus durations too short for reliable psychophysical discrimination of the sound types) (Romei et al., 2009). The collective findings provide indications of the behavioral relevance of early auditory inputs into low-level visual cortices. Identifying the precise neurophysiological mechanism will undoubtedly benefit from simultaneous EEG-TMS acquisitions currently underway in our laboratories (for recent methodological reviews see, e.g., Ilmoniemi and Kičicč, 2010; Miniussi and Thut, 2010; Thut and Pascual-Leone, 2010). This kind of approach would likewise prove informative in fully unraveling the spatio-temporal dynamics of the behavioral relevance of specific auditory-somatosensory interactions.

Attention and arousal

Variations in multisensory interactions that in turn impact reaction time might simply follow from fluctuations in participants’ level of attention, such that fast trials were the result of high levels of attention and vice versa. Both spatial attention (Talsma and Woldorff, 2005) and selective attention (Talsma et al., 2007) can indeed modulate auditory-visual multisensory integration. In these studies, attention resulted in larger and/or supra-additive effects within the initial 200-ms post- stimulus presentation. In Sperdin et al. (2009), however, subjects were instructed to attend to both sensory modalities (i.e., audition and touch) and received no instructions regarding the spatial position of the stimuli. Rather, they performed a simple detection task irrespective of the spatial position of the stimuli (see also Sperdin et al., 2010, for an examination of when spatial attention was engaged). Therefore, it is unlikely that our participants were modulating their spatial or selective attention in a systematic manner – though we cannot unequivocally rule such out. Nonetheless, that our behavioral results show a redundant signals effect would not be predicted if the participants had selectively attended (systematically) to one or the other sensory modality. In terms of spatial attention, all eight stimulus conditions (i.e., four unisensory and four multisensory; see Murray et al., 2005 for details) were equally probable within a block of trials, and the fact that all spatial combinations resulted in multisensory facilitation of RTs (detailed in Murray et al., 2005) would suggest that participants indeed attended to both left and right hemispaces simultaneously. Finally, examination of the distribution of trials producing fast and slow RTs showed there to be an even distribution throughout the duration of the experiment. This would argue against a systematic effect of attention, arousal or fatigue.

Neural oscillations

Another possible mechanism is based on studies implicating a role of oscillatory activity in multisensory interactions. Lakatos et al. (2007) demonstrated that somatosensory inputs into supragranular layers of primary auditory cortices can serve to reset the phase of ongoing oscillatory activity that in turn modulates the responsiveness to auditory stimuli across the cortical layers. The phase of the reset oscillations was linked to whether the auditory (and by extension multisensory) response was enhanced or suppressed (see also Lakatos et al., 2008, for evidence of the role of delta phase in reaction times to visual stimuli, and more recently Lakatos et al., 2009 for evidence for the role of attention in these reset phenomena). Response amplification occurred under multisensory conditions when somatosensory inputs into primary auditory cortex led to an optimal phase of ongoing theta (∼7 Hz) and gamma (∼35 Hz) band activity. It will be important to determine if and how these kinds of effects might manifest elsewhere (e.g., in regions of the superior temporal cortex where auditory-somatosensory interactions were observed in Sperdin et al., 2009) and might impact task performance. Nonetheless, that responses to multisensory were amplified could provide not only a mechanism for enhanced attention and perception (cf. Lakatos et al., 2007, 2008, 2009; Schroeder et al., 2008; Schroeder and Lakatos, 2009), but also faster performance with such stimuli (i.e., because higher amplitude and steeper sloping responses will meet thresholds earlier; cf. Martuzzi et al., 2007, for an example of a study examining response dynamics in humans during an auditory-visual detection task). In the case of the present study, trials producing fast reaction times may be those where ongoing oscillations (in primary auditory cortex or elsewhere) are reset such that their phase is optimal for response enhancement. Along these lines, the impact of pre-stimulus oscillatory activity on stimulus-related responses has been demonstrated in humans, such that pre-stimulus alpha activity can predict the accuracy of perception (Romei et al., 2008; see also Silvanto and Pascual-Leone, 2008; Busch et al., 2009; Mathewson et al., 2009; Britz and Michel, 2010). Such being said, no pre-stimulus effects were observed in our analyses, even when pre-stimulus baseline correction was not performed. However, we – of course – cannot entirely exclude the possibility of pre-stimulus effects that are not phase-locked to stimulus onset. It may be possible to resolve the contributions of specific frequencies within specific brain regions to multisensory interactions and behavior by applying single-trial time-frequency analyses subsequent to source estimations (e.g., Gonzalez Andino et al., 2005a,b; see also Van Zaen et al., 2010 for methods for adaptive frequency tracking).

A role for post-stimulus oscillatory activity has also been advocated as a mechanism. Oscillatory activity within the beta frequency range (13–30 Hz) has been proposed to contribute to reaction time speed. After applying a Morlet wavelet transform to EEG data acquired during the completion of an auditory-visual simple detection task, Senkowski et al. (2006) observed a negative correlation between evoked beta power over the 50–170ms time period and reaction time speed (when measured collectively across unisensory and multisensory conditions). Greater beta power was thus linked to faster reaction times, independent of stimulus condition, though separate analyses in this study also indicate that beta power was significantly stronger for multisensory conditions. Consequently, it is challenging to resolve whether any correlation was driven by the variability across stimulus conditions (i.e., data from the same participant from the different stimulus conditions was treated as independent measurements in the correlation analysis). Others have obtained the opposite result. Pogosyan et al. (2009) observed that voluntary reaction times were slowed after the entrainment of motor cortical activity at 20 Hz using transcranial alternating-current stimulation. Clearly, additional research is required to resolve these discrepant findings and to draw closer links between alterations in oscillatory activity and multisensory interactions specifically, as opposed to general influences on reaction time speed irrespective of stimulation conditions.

While we did not specifically examine oscillatory activity in our work (though this is the focus of ongoing analyses), it may be the case that modulations in synchronous neural activity are in turn driving the differences we observed in terms of response strength. That is, stronger responses may be the consequence of more synchronous activity, whereas weaker responses may be the consequence of less synchronous activity. One possibility, then, is that enhanced synchrony leads to faster processing (e.g., by meeting a threshold level more quickly) either in the brain region(s) exhibiting enhanced synchrony or in a downstream target region. This variety of mechanism would also be consistent with our observation of weaker responses to multisensory vs unisensory conditions on trials when reaction times were fast. If a mechanism based on fluctuations in neural synchrony is indeed at the root of whether reaction times are ultimately fast, then an immediately ensuing question is to identify the cause/mediator for such fluctuations. Another possibility, which is not exclusive to the above, is that distinct anatomic pathways are involved in trials producing faster vs slower reaction times. There is now evidence, at least in non-human primates, for both cortico-cortical (Cappe and Barone, 2005; Hackett et al., 2007a; Smiley et al., 2007) and cortico-thalamo-cortical (Hackett et al., 2007b; Cappe et al., 2009) pathways; whereby, auditory and somatosensory information can converge and interact. Whether there is any variation in the routing of responses that would in turn result in faster or slower reaction times is unknown in the case of auditory-somatosensory interactions, but there is evidence that suggests that such is occurring during a simple visual detection task (Saron et al., 2003; Martuzzi et al., 2006). While we cannot unequivocally rule out this latter possibility as a contributing mechanism, our analyses (both of the surface-recorded ERPs and source estimations thereof) would suggest that our effects derive from modulations in the strength of activity of a common network of brain regions (though the sensitivity of ERPs to activity in sub-cortical structures is limited).

Outlook

There is a growing body of evidence highlighting the behavioral and perceptual relevance of early and low-level multisensory phenomena. The early stages of such research are bound to focus more on demonstrating such links, rather than on identifying their precise neurophysiological mechanisms. However, it is also the case that such findings are likely a harbinger of future advances in the analysis of multisensory datasets either because of improvements in signal analysis, either in humans (e.g., Gonzalez Andino et al., 2005a; Murray et al., 2005, 2008; Sperdin et al., 2009) or in animals (e.g., Rowland and Stein, 2008), or because of the improved feasibility of conducting experiments in awake and behaving animals (e.g., Wang et al., 2008; Cappe et al., 2010). Such notwithstanding, it will be essential – in our view – to go beyond the demonstration of correlates of behavioral outcome. Instead, causal evidence will be needed to draw firmer conclusions about underlying mechanisms. One promising direction is the combination of different brain imaging methods in humans to render greater mechanistic information. The urgency for this line of research also comes from accumulating evidence showing that multisensory interactions play a central role during both development (e.g., Lickliter and Bahrick, 2000; Neil et al., 2006; Wallace et al., 2006; Polley et al., 2008) and aging (e.g., Laurienti et al., 2006; Hugenschmidt et al., 2009), as well as in neurodegenerative and neuropsychiatric illnesses (e.g., Ross et al., 2007; Blau et al., 2009; see also comment by Wallace, 2009). Consequently, there is growing interest in applying multisensory phenomena as a potential diagnostic and rehabilitation tool. In conclusion, the field of multisensory research now seems well-poised to conjoin its neurophysiological and psychophysical findings.

Conflict of Interest Statement

This research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Key Concept

Multisensory interactions

Operationally, we use this term to refer to any instance where information from one sensory modality affects the processing and/or response to that from another sensory modality.

Redundant signals effect

The improvement in stimulus detection and discrimination performance following presentation of multiple stimuli.

Probability summation

The increased likelihood of improved performance by presenting multiple, independent stimuli – each competing to mediate motor responses. By analogy, the likelihood of rolling at least one ‘6’ on a die increases the more often it is rolled. Facilitation exceeding probability summation is a hallmark of neural response interactions (though the converse need not be true).

Neural response interactions

To assess multisensory integration, responses to multisensory stimulus pairs are typically contrasted with the summed responses to the constituent unisensory stimuli. Differences are indicative of synergistic responses under multisensory conditions.

Spatial rule

This principle of multisensory interactions, originally described in the seminal works of Stein and Meredith (1993), stipulates that facilitative effects depend on external stimuli falling within the excitatory zone of a neuron's receptive field for each sensory modality. This zone need not be synonymous with the external position of the stimuli across the senses.

Source estimations

This refers to solutions to the bio-electromagnetic inverse problem, which is the reconstruction of intracranial sources based on surface recordings. A fuller treatment can be found in Michel et al. (2004) or Grave de Peralta Menendez et al. (2004).

Electrical neuroimaging analyses

This is a set of analyses of EEG data involving reference-independent measures (as opposed to individual ERP waveforms) of the entire electric field recorded at the scalp, and the estimation of intracranial sources (e.g., Michel et al., 2004; Murray et al., 2008).

Event-related potential

This is the time-locked average of EEG epochs to an external or internal event. The term was first introduced by Vaughan Jr. (1969).

Response strength vs response topography

Strength refers to the spatial standard deviation across the electric field at the scalp, quantified as the root mean square of the voltage measurements across the electrode montage. Topography refers to the shape of the electric field with changes quantified as the root mean square of the difference between two strength-normalized measurements across electrodes.

Supra-additive and sub-additive interactions

These terms refer to when the response to multisensory stimuli are greater or less than, respectively, the summed responses to the constituent unisensory conditions. In the case of electrical neuroimaging analyses of event-related potentials, we restrict the use of these terms to instances of modulations in response strength rather than response topography.

Biography

Micah M. Murray is Associate Director of the Electroencephalography Brain Mapping Core of the Center for Biomedical Imaging and a Senior Research Scientist within the Neuropsychology and Neurorehabilitation Service and Radiology Service at the Centre Hospitalier Universitaire Vaudois and University of Lausanne, Switzerland. His research focuses on brain dynamics of sensory and cognitive processes in humans, combining tools from psychophysics and neuropsychology, electroencephalography, functional magnetic resonance imaging, and transcranial magnetic stimulation.