Skip to main content
NCBI home page
The Journal of Neuroscience logoLink to The Journal of Neuroscience
. 2012 Oct 31;32(44):15338–15344. doi: 10.1523/JNEUROSCI.1796-12.2012

Multisensory Representation of Frequency across Audition and Touch: High Density Electrical Mapping Reveals Early Sensory-Perceptual Coupling

John S Butler 1,✉, John J Foxe 1, Ian C Fiebelkorn 1, Manuel R Mercier 1, Sophie Molholm 1,✉
PMCID: PMC3664421  NIHMSID: NIHMS418859  PMID: 23115172

Abstract

The frequency of environmental vibrations is sampled by two of the major sensory systems, audition and touch, notwithstanding that these signals are transduced through very different physical media and entirely separate sensory epithelia. Psychophysical studies have shown that manipulating frequency in audition or touch can have a significant cross-sensory impact on perceived frequency in the other sensory system, pointing to intimate links between these senses during computation of frequency. In this regard, the frequency of a vibratory event can be thought of as a multisensory perceptual construct. In turn, electrophysiological studies point to temporally early multisensory interactions that occur in hierarchically early sensory regions where convergent inputs from the auditory and somatosensory systems are to be found. A key question pertains to the level of processing at which the multisensory integration of featural information, such as frequency, occurs. Do the sensory systems calculate frequency independently before this information is combined, or is this feature calculated in an integrated fashion during preattentive sensory processing? The well characterized mismatch negativity, an electrophysiological response that indexes preattentive detection of a change within the context of a regular pattern of stimulation, served as our dependent measure. High-density electrophysiological recordings were made in humans while they were presented with separate blocks of somatosensory, auditory, and audio-somatosensory “standards” and “deviants,” where the deviant differed in frequency. Multisensory effects were identified beginning at ∼200 ms, with the multisensory mismatch negativity (MMN) significantly different from the sum of the unisensory MMNs. This provides compelling evidence for preattentive coupling between the somatosensory and auditory channels in the cortical representation of frequency.

Introduction

In a now classic study, Jousmäki and Hari (1998) demonstrated the powerful influence of sounds on somatosensory experience in the so-called “parchment-skin illusion.” To elicit this illusion, a participant rubs their hands together while the sounds of their hands are fed back to them through head phones, but the sound is distorted to emphasize the high or the low frequencies on playback. When asked to describe what their hands feel like, participants tend to describe their hands as feeling smooth and dry like parchment paper for the high-frequency condition. For the low-frequency manipulation, they report that their hands feel rough and moist. This and other behavioral studies (Guest et al., 2002; Wilson et al., 2009, 2010a,b; Yau et al., 2009) make clear that there is substantial convergence between the auditory and somatosensory systems in the computation of frequency, especially when the stimuli are well matched and thus likely to reflect the same event (Foxe, 2009).

These findings raise the question of when in the information processing stream auditory and somatosensory inputs interact to influence the perception of frequency. It has become increasingly apparent that processing in hierarchically early sensory regions can be modulated by inputs from nonprimary sensory modalities, and that this occurs within the sensory-perceptual information processing timeframe. Indeed, there is corroborative evidence for early interactions between auditory and somatosensory channels ranging from postmortem anatomical tracer studies, to invasive recordings in regions of sensory cortex, and to scalp electrical recordings and neuroimaging in humans (see Foxe and Schroeder, 2005). An intriguing possibility, then, is that multisensory perception of frequency results from coupling between the sensory channels early in the information processing hierarchy during sensory-perceptual processing. An alternate account could be that the unisensory systems operate largely independently to extract feature-level information, before multisensory convergence and integration in higher-order association areas (see Calvert et al., 1998; Driver and Noesselt, 2008).

Here we used the MMN (for mismatch negativity) to test whether auditory and somatosensory frequency representations interact within sensory cortical regions early in time and preattentively to influence frequency representations. The MMN is elicited by changes in a pattern of stimulation and occurs whether or not the participant is attending the information (Näätänen, 1992; Näätänen et al., 2007), but only in response to stimulus changes that can be perceived when the stimuli are actively attended (Amenedo and Escera, 2000) and thus is tightly linked to perceptual experience. The neural locus of MMN generation in sensory cortex is considered to reflect where the deviating feature is processed (Molholm et al., 2005; Butler et al., 2011) and thus to provide insight into the representation of feature-specific sensory information. We reasoned that if auditory and somatosensory frequency information interacts to influence perception during sensory-perceptual stages of processing, the MMN in response to combined auditory and somatosensory deviants would display multisensory integrative properties (Stein and Meredith, 1993). Alternatively, if calculation of frequency is sensory specific and there is no intersensory coupling at the sensory-perceptual level, the MMN to combined somatosensory and auditory frequency deviants should reflect a linear summation of the MMNs evoked by these deviants when presented in isolation (e.g., Molholm et al., 2002).

Materials and Methods

Participants.

Twelve participants (seven male) ranging in age from 19–32 years (mean 23.6) with normal hearing completed the experiment for a modest fee of $12 per hour. All participants were right handed and reported normal hearing and no known neurological deficits. All participants gave written informed consent, and all procedures were approved by the ethical review board of the City College New York and the Albert Einstein College of Medicine. Ethical guidelines were in accordance with the Declaration of Helsinki.

Stimuli and paradigm.

Participants viewed a silent movie with subtitles during stimulus delivery and were instructed to ignore the sounds and vibrations. The standard stimuli (p = 0.9) and the deviant stimuli (p = 0.1) were 200 Hz tones/vibrations of 100 ms duration and 400 Hz tones/vibrations of 100 ms. All stimuli were sinusoids convolved with a trapezoid such that there was a 5 ms rise at the onset and a 5 ms ramp at the offset. Tones were presented to the right ear via headphones (Sennheiser HD600). Somatosensory stimulation was presented via a low-cost linear amplifier (Piezo Systems) to the index finger of the right hand. The stimulator was wrapped in gauze, the hand placed on a sound dampening surface, and pre-experimental testing was performed to ensure that sounds from the tactile vibrator were fully masked. Stimuli were presented with an interstimulus interval of 1000 ms.

All experiments were carried out in a darkened acoustically and electrically shielded room. The auditory and somatosensory stimuli could be presented alone or simultaneously as a pair. There were three blocked conditions in which: (1) only somatosensory standards and deviants were presented (the sMMN condition); (2) only auditory standard and deviant stimuli were presented (the aMMN condition); and (3) somatosensory-auditory standard pairs and deviant pairs were presented (saMMN). The order of presentation of the blocks was counterbalanced across participants.

EEG recording and analysis.

Electrical brain activity was recorded using a BioSemi system 168 channel EEG system. The data were recorded at 512 Hz and low-pass filtered at 30 Hz (12 dB/octave). The data were analyzed offline using the Matlab programming language (MathWorks). Epochs of 600 ms with 100 ms pre-stimulus baseline were extracted from the continuous data. An automatic artifact rejection criterion of ±85 μV was applied across all electrodes in the array. Trials with more than six artifact channels were rejected. In trials with less than six such channels, we interpolated any remaining bad channels using a nearest neighbor spline (Perrin et al., 1987; Perrin et al., 1989). The data were re-referenced to the average of the mastoid channels. The average accepted trials per condition were ∼90 deviants and ∼800 standards.

Mismatch responses.

The MMN response was visualized by subtracting the standard response from the deviant response. To test for the presence of the MMN, mean amplitude measurements were obtained in a 20 ms window centered at the group mean peak amplitude in the difference wave between 100 and 300ms. From the literature, the midline frontal site (Fz) and the central left site (C3) were identified as regions of interest for both the somatosensory and auditory MMN conditions (Näätänen, 1992; Butler et al., 2011). A 2 (condition: standard vs deviant) × 2 (electrode site: Fz vs C3) repeated-measures ANOVA was used to test for the presence of the MMN. Given the dense recording montage for the planned comparisons and figures, each site of interest (i.e., Fz and C3) is represented by an average of the five nearest electrodes. This serves to increase the signal-to-noise ratio.

To provide a more encompassing description of the spatiotemporal properties of the MMNs, point-wise paired t tests between the standard and deviant responses were computed for all electrodes at each time point and are presented in statistical cluster plots (Molholm et al., 2002; De Sanctis et al., 2009). In this approach, to control for Type 1 errors a period of statistical significance is only considered if an alpha criterion of 0.05 or less is obtained for at least 11 consecutive sample points (representing a period of 21 ms) (Guthrie and Buchwald, 1991).

Multisensory responses.

To test for the presence of multisensory interactions in the processing of frequency, the multisensory MMN (saMMN) was compared to the sum of the unisensory MMNs (referred to hereafter as the sumMMN). The assumption was that if the auditory and somatosensory systems interact to influence frequency representations, the saMMN should differ from the sumMMN; that is, that it would show nonlinearity. Note that a linear outcome is a possibility, with previous investigations showing that the MMN system can act in a linear manner under conditions of two deviating features (with no significant differences between double-deviant MMNs and the sum of the respective single-deviant MMNs; Paavilainen et al., 2001; Wolff and Schröger, 2001). Mean amplitude measurements were obtained from the same 20 ms windows used to test for the presence of the MMN in the unisensory conditions and subjected to a 2 (multisensory vs sum) × 2 (electrodes site; Fz vs C3) repeated-measures ANOVA. To provide a comprehensive depiction of the spatiotemporal properties of the multisensory interactions, the saMMN and sumMMN were also compared with statistical cluster plots (described above, Mismatch responses).

Comparison of the topographies of the MMNs.

The topographical nonparametric statistical analysis (TANOVA), as implemented in the Cartool software (Lehmann and Skrandies, 1980), was used to test for differences in the neural generators underlying the multisensory saMMN as compared to those underlying the unisensory MMNs. This allowed us to test whether unique cortical areas were involved in generation of the multisensory MMN by comparing the saMMN to the sumMMN. The TANOVA procedure uses global dissimilarity and nonparametric randomized testing to statistically compare the topographies over time (Lehmann and Skrandies, 1980). Global dissimilarity is an index of configuration differences between two scalp distributions, independent of their strength as the data are normalized using the global field power. For each participant and time point, the global dissimilarity indexes a single value that varies between 0 and 2 (0, homogeneity; 2, inversion of topography). To create an empiric probability distribution against which the global dissimilarity can be tested for statistical significance, the Monte Carlo MANOVA is applied, as described previously (Manly, 1991). In the present application of the TANOVA, to control for type I errors, a period of statistical significance was only plotted and considered significant if an alpha criterion of 0.05 or less was obtained for at least 11 consecutive sample points (∼21 ms) (Guthrie and Buchwald, 1991; Foxe and Simpson, 2002; Butler et al., 2011).

Results

Mismatch response

Clear MMNs were observed for the aMMN and saMMN conditions. Inspection of the grand mean data revealed that the deviant response was more negative going than the standard response, with a maximum difference that was focused over frontal scalp at 170 ms (see Figs. 1 and 2). In comparison, the somatosensory MMN was smaller in amplitude and in duration, with a maximal standard/deviant amplitude difference that peaked at 220 ms over frontocentral scalp and a distribution similar to that of the aMMN and saMMNs in this later timeframe (see Figs. 1, 2A). Figure 2 shows the scalp distributions of the grand mean somatosensory, auditory, and somatosensory-auditory MMNs, as well as the distribution of the summed unisensory MMNs, at three time points. The first corresponds to the peak amplitude of the aMMN and saMMN, and the second to the peak amplitude of the sMMN. From this figure can be seen a slightly left-lateralized frontal negative distribution for all time periods for the aMMN, whereas the saMMN is bilateral at 170 ms, and then more left-lateralized at 220 ms. The sMMN shows a left-lateralized negativity at 220 ms. The sumMMN is shown for comparison to the saMMN. Differences are apparent at both time points, with the sumMMN greater in amplitude than the saMMN.

Figure 1.

Figure 1.

Open in a new tab

A, Grand Mean ERPS: event-related potentials (ERPS) for the somatosensory, auditory, and somatosensory-auditory conditions at the frontal central site for deviant (red), standard (green), and subtraction waveforms (black). B, Statistical Cluster Analysis: running t tests comparing the standard versus the deviant ERPs for the somatosensory, auditory, and somatosensory-auditory conditions. Electrodes are divided into five sections each representing 32 electrodes, the relative positions of which are color coded on the corresponding head. Significance is depicted for effects meeting an alpha (0.05) criterion and lasting for at least 11 consecutive sample points (∼20 ms).

Figure 2.

Figure 2.

Open in a new tab

Topographical plots of the grand average waves for the sMMN, aMMN, saMMN, and sumMMN [(s+a)MMN] (columns) at two time periods (rows).

Statistical tests confirmed the presence of MMNs for all three conditions. For the aMMN, there was a main effect of condition in the 160–180 ms time window (F(1,11) = 11.395, p = 0.006), which did not interact with the electrode site (F(1,11) = 0.334, p = 0.58). For the sMMN there was a main effect of condition in the 210–230 ms time window (F(1,11) = 4.907, p = 0.049) and no condition by electrode site interaction (F(1,11) = 0.406, p = 0.537). The saMMN was also found to be significant, with a main effect of condition in the 160–180 ms time window (F(1,11) = 6.689, p = 0.025) and no condition by electrode site interaction (F(1,11) = 0.006, p = 0.941). Note that while the timing of the somatosensory MMN was later than has been previously reported in the literature (Butler et al., 2011; Kekoni et al., 1997), this can readily be attributed to one of two factors: (1) the use of a different deviating feature (frequency vs duration), since the deviating feature impacts the neuronal generators of the MMN (Molholm et al., 2005) and consequently the scalp distribution and/or latency of the response (Giard et al., 1995); and (2) differences in perceived magnitude of deviance across the studies. That is, MMN latency increases for smaller deviances (Horváth et al., 2008), and in the present study the subjective magnitude of somatosensory deviance was small, which would be expected to lead to a relatively later MMN onset latency.

The statistical cluster plots show clear MMNs for all three conditions (see Fig. 1A), with the timing and topography of the MMN differing somewhat across the three. From these it can be seen that the aMMN onset at ∼100 ms, was focused over left frontal and central scalp (Fig. 1), and lasted for about 120 ms. The saMMN onset, slightly later at ∼120 ms, lasted for a shorter time over frontocentral scalp regions, offsetting at ∼190 ms, and had a more distributed topography that included right frontal scalp regions as well. The sMMN onset later still, at 200 ms with a more focal distribution over left central scalp, was of relatively short duration, offsetting only 50 ms later at ∼250 ms.

Multisensory effects

There was clear nonlinearity in the multisensory response, with the saMMN smaller in amplitude than the sumMMN (Fig. 3). Statistical comparison in the 210–230 ms window established the reliability of this difference, with a main effect of condition (F(1,11) = 5.350, p = 0.041), whereas condition and electrode site did not interact (F(1,11) = 0.78, p = 0.682). There was no main effect or condition by electrode site interaction for the earlier tested timeframe of 170–190 ms.

Figure 3.

Figure 3.

Open in a new tab

Multisensory effects: somatosensory-auditory (SomaAudio; black) and summed (Sum; blue) ERPS and their difference (Subtraction; gray) are shown for two electrode sites (Fz and C3).

To further characterize the spatiotemporal characteristics of this multisensory effect, statistical cluster plots comparing the saMMN and sumMMN were generated. This revealed a difference that was focused over left central scalp in the 200–240 ms time period (Fig. 4). TANOVA analysis did not reveal differences in the scalp distribution of the saMMN and the sumMMN

Figure 4.

Figure 4.

Open in a new tab

A, Statistical Cluster Analysis: running t tests comparing the saMMN versus the sumMMN. Significance is depicted for effects meeting an alpha (0.05) criterion and lasting for at least 10 consecutive sample points (∼20 ms). B, TANOVA Analysis: significant topographical between the saMMN versus the sumMMN are depicted for effects meeting a 1-alpha (0.05) criterion and lasting for at least 11 consecutive sample points (∼20 ms).

Discussion

The present results establish that interactions between auditory and somatosensory inputs can lead to the multisensory modulation of sensory-perceptual representations of frequency. While previous work has repeatedly shown that auditory and somatosensory inputs converge and are integrated early on in sensory cortices (Foxe et al., 2000, 2002; Schroeder et al., 2001; Meredith and Allman, 2009; Meredith et al., 2009; Keniston et al., 2010; Brandwein et al., 2011), to our knowledge this has not been associated with changes in the neural representation of a particular stimulus feature. The data show clear evidence of multisensory integrative properties in the MMN in that combined bisensory stimulation resulted in an MMN that was not the simple linear sum of the MMNs to the constituent unisensory frequency deviations. Statistical comparison of the topographies of the multisensory and composite unisensory MMNs (saMMN vs sumMMN) did not reveal significant differences in their scalp distributions. This suggests that integrative processing involved coupling of existing auditory and somatosensory circuits rather than recruitment of unique multisensory structures. The current data also support that integration of frequency information does not require directed attention. That is, there were clear multisensory effects on the representation of frequency, even though participants were not engaged in a frequency discrimination task or indeed any sort of task related to the auditory and somatosensory stimuli.

Recent psychophysical work has characterized the interplay of auditory and somatosensory frequency information (Yau et al., 2009; Wilson et al., 2010b). For example, Wilson and colleagues (2010) presented audio-somatosensory stimuli at threshold levels while participants engaged in a detection task. The distance between the frequencies of the auditory and somatosensory elements of the stimuli was manipulated, and the influence of frequency convergence versus divergence on detection levels was assayed. Audio-somatosensory detection levels showed the greatest facilitation compared to unisensory performance when the frequencies of the component stimuli were closest. Using a quite different approach, Yau et al. (2009) also showed that auditory and somatosensory frequency information interact to influence perception. In Yau et al. (2009), participants engaged in a two-alternative, forced-choice task in which they judged which of two sequentially presented somatosensory stimuli was higher in frequency. On most trials, a task-irrelevant auditory distractor was presented simultaneously with the comparator somatosensory stimulus. The sound could be a simple or a complex tone and it could be close or distant in frequency from the somatosensory stimulus. When the frequency of the auditory stimulus was close to that of the somatosensory stimulus, judgments of somatosensory frequency were drawn toward the frequency of the auditory stimulus, systematically affecting participants' performance. Here we provide initial evidence that such behavioral effects may be mediated by interactions between auditory and somatosensory inputs at the sensory-perceptual level of information processing.

It has become increasingly clear that processing in cortical sensory regions can be modulated by inputs from nonsensory modalities, and that this occurs within the sensory-perceptual information processing timeframe. While this idea was considered highly implausible when first proposed, it is now well accepted. Animal studies have revealed that there are monosynaptic connections between auditory and somatosensory and auditory and visual sensory cortices (Meredith et al., 2006; Keniston et al., 2010), and that nonprimary sensory inputs can modulate processing of the primary sensory input (Fu et al., 2003; Meredith and Allman, 2009; Meredith et al., 2009). Human neurophysiological studies have also shown early processing links between the auditory, somatosensory, and visual sensory systems (Foxe et al., 2000, 2002; Molholm et al., 2002; Schroeder and Foxe, 2002; Foxe and Schroeder, 2005; Murray et al., 2005; Schroeder and Foxe, 2005; Brandwein et al., 2011, 2012), with evidence from scalp electrical recordings that multisensory integration occurs early in sensory processing regions (as early as ∼50 ms poststimulus onset). The current findings reveal multisensory influences on the representation of frequency. Speculating on possible underlying mechanisms, the temporal alignment of the multisensory effect and the sMMN is suggestive of the multisensory modulation of somatosensory frequency representations. The subadditive nature of the multisensory effect further suggests that the somatosensory MMN was minimized when paired with the concurrent auditory stimulation. In support of this line of reasoning, the topographies of the aMMN and saMMN were very similar in this time-frame (Fig. 2), an observation supported by a post hoc analysis of topographical differences (i.e., a TANOVA did not reveal significant differences). Alternatively, one might consider that the auditory stimuli partially masked early sensory processing of the somatosensory stimuli and thus led to their reduced contribution to the saMMN. However, such an account would predict a diminution of the early sensory response, and there was no evidence for this (i.e., the multisensory and sum responses did not differ in this earlier timeframe, neither for the standard nor the deviant). Going forward, to better understand the nature and consequences of audio-somatosensory interactions, it will be essential to determine the extent to which the magnitude and direction of neural interactions are influenced by the similarity versus difference of frequency across sensory channels, and to characterize the perceptual consequences of these neural effects.

The mismatch negativity and multisensory processing

Only a few studies to date have used MMN to investigate multisensory influences on sensory-perceptual processing. These have largely focused on audio-visual speech processing, showing that visual speech affects sensory level speech representations in auditory cortex (Möttönen et al., 2002; Saint-Amour et al., 2007; Winkler et al., 2009), or on letter-sound combinations, showing that the presence of sound-congruent or sound-incongruent letters influenced the MMN (Froyen et al., 2008; Andres et al., 2011). These data suggest that both seen speech and seen letters automatically influence speech representations in the auditory cortex. A study by Pantev et al. (2009) further demonstrates the lasting effects of nonauditory influences on auditory representations with the use of the MMN. In this study, nonmusicians received either piano lessons, thus engaging in a multisensory learning experience (sensorimotor and auditory learning), or listened to the music produced by the first group and detected errors, a purely auditory learning experience. For both groups, the MMN was recorded to musical tokens that were similar to those they were trained on, before and after training. The results showed a clear amplification of the auditory MMN in the multisensory trained group in comparison to the auditory-alone trained group, strongly suggesting a lasting influence of the multisensory experience on auditory representations. With the present study, we extend the use of the MMN to interrogate multisensory processing at the feature-level of representation, showing not only that these multisensory inputs interact, but that they do so to influence the representation of frequency.

Additional considerations

In interpreting the current data, several factors should be considered. One is that increases in the frequency of an auditory stimulus are accompanied by systematic increases in perceived intensity (Stevens, 1934). Examination of the psychometric curves of Stevens (1934) suggests that the auditory frequency differences that we used could lead to small changes in the perceived loudness on the order of ∼2 dB. Nevertheless, the bulk of the auditory MMN should be driven by the much larger suprathreshold frequency deviation, and therefore an interpretation of interaction between the frequency representations is justified. Clearly, future work will need to control perceived intensity to fully segregate the role of frequency from intensity in audio-somatosensory interactions.

Another consideration is refractory effects. In a standard oddball paradigm, repeated stimulation at the same frequency leads to refractoriness of the responding cortical neurons in sensory cortex, while switching to another frequency stimulus invokes a “release” from refractoriness. Such refractory effects can confuse interpretation of the frequency MMN (Jacobsen and Schröger, 2001; Jacobsen et al., 2003), and one should consider whether low-level refractory effects and audio-somatosensory interactions therein might partially account for the multisensory effects observed in the present study. Refractory effects are seen in the early cortical evoked responses within the first 150 ms of stimulus processing, whereas here nonlinearity was only observed beginning at 200 ms following considerable auditory and somatosensory processing (see Fig. 1). Thus, the timing of the nonlinear multisensory effects are not in line with multisensory modulation of sensory-level refractory effects.

Finally, there is an open question regarding the spatial misalignment of the multisensory stimuli. Here, auditory stimulation was to the right ear (over headphones), whereas somatosensory stimulation was to the right index finger. Despite this obvious misalignment of the stimuli, robust effects were observed. This is consistent with previous work that has shown cortical multisensory integration and multisensory behavioral effects to be surprisingly impervious to spatial coregistration of the multisensory inputs (Fiebelkorn et al. 2011; Murray et al., 2005). Still, further work is needed to explore the highly reasonable idea that manipulations of spatial registration would impact how auditory and somatosensory inputs are integrated.

Conclusion

Psychophysical studies clearly demonstrate that somatosensory and auditory frequency information interacts to influence frequency judgments. Here, we took advantage of a well characterized electrophysiological component, the MMN, to test the hypothesis that auditory and somatosensory inputs interact to influence sensory-perceptual level processing of frequency information. Analyses of the data revealed nonlinearity in the audio-somatosensory MMN when compared to the sum of its unisensory counterparts. This finding is consistent with preattentive coupling between the somatosensory and auditory frequency processing streams during sensory-perceptual processing.

Footnotes

This study was supported by a grant from the National Institute of Mental Health (NIMH) to S.M. and J.J.F. (MH 85322). We thank Dr. P. DeSanctis and Edel Flynn for helpful discussions. The Cartool software (https://sites.google.com/site/fbmlab/cartool/cartool-registration) was programmed by Denis Brunet of the Functional Brain Mapping Laboratory, Geneva, Switzerland and is supported by the Center for Biomedical Imaging of Geneva and Lausanne.

References

  1. Amenedo E, Escera C. The accuracy of sound duration representation in the human brain determines the accuracy of behavioural perception. Eur J Neurosci. 2000;12:2570–2574. doi: 10.1046/j.1460-9568.2000.00114.x. [DOI] [PubMed] [Google Scholar]
  2. Andres AJ, Oram Cardy JE, Joanisse MF. Congruency of auditory sounds and visual letters modulates mismatch negativity and P300 event-related potentials. Int J Psychophysiol. 2011;79:137–146. doi: 10.1016/j.ijpsycho.2010.09.012. [DOI] [PubMed] [Google Scholar]
  3. Brandwein AB, Foxe JJ, Russo NN, Altschuler TS, Gomes H, Molholm S. The development of audiovisual multisensory integration across childhood and early adolescence: a high-density electrical mapping study. Cereb Cortex. 2011;21:1042–1055. doi: 10.1093/cercor/bhq170. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Brandwein AB, Foxe JJ, Butler JS, Russo NN, Altschuler TS, Gomes H, Molholm S. The development of multisensory integration in high-functioning autism: high-density electrical mapping and psychophysical measures reveal impairments in the processing of audiovisual inputs. Cereb Cortex. 2012 doi: 10.1093/cercor/bhs109. Advance online publication. Retrieved May 24, 2012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Butler JS, Molholm S, Fiebelkorn IC, Mercier MR, Schwartz TH, Foxe JJ. Common or redundant neural circuits for duration processing across audition and touch. J Neurosci. 2011;31:3400–3406. doi: 10.1523/JNEUROSCI.3296-10.2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Calvert GA, Brammer MJ, Iversen SD. Crossmodal identification. Trends Cogn Sci. 1998;2:247–253. doi: 10.1016/S1364-6613(98)01189-9. [DOI] [PubMed] [Google Scholar]
  7. De Sanctis P, Molholm S, Shpaner M, Ritter W, Foxe JJ. Right hemispheric contributions to fine auditory temporal discriminations: High-density electrical mapping of the Duration mismatch negativity (MMN) Front Integr Neurosci. 2009;3:5. doi: 10.3389/neuro.07.005.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Driver J, Noesselt T. Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron. 2008;57:11–23. doi: 10.1016/j.neuron.2007.12.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Fiebelkorn IC, Foxe JJ, Butler JS, Molholm S. Auditory facilitation of visual-target detection persists regardless of retinal eccentricity and despite wide audiovisual misalignments. Exp Brain Res. 2011;213:167–174. doi: 10.1007/s00221-011-2670-7. [DOI] [PubMed] [Google Scholar]
  10. Foxe JJ. Multisensory integration: frequency tuning of audio-tactile integration. Curr Biol. 2009;19:R373–R375. doi: 10.1016/j.cub.2009.03.029. [DOI] [PubMed] [Google Scholar]
  11. Foxe JJ, Schroeder CE. The case for feedforward multisensory convergence during early cortical processing. Neuroreport. 2005;16:419–423. doi: 10.1097/00001756-200504040-00001. [DOI] [PubMed] [Google Scholar]
  12. Foxe JJ, Simpson GV. Flow of activation from V1 to frontal cortex in humans. A framework for defining “early” visual processing. Exp Brain Res. 2002;142:139–150. doi: 10.1007/s00221-001-0906-7. [DOI] [PubMed] [Google Scholar]
  13. Foxe JJ, Morocz IA, Murray MM, Higgins BA, Javitt DC, Schroeder CE. Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Brain Res Cogn Brain Res. 2000;10:77–83. doi: 10.1016/s0926-6410(00)00024-0. [DOI] [PubMed] [Google Scholar]
  14. Foxe JJ, Wylie GR, Martinez A, Schroeder CE, Javitt DC, Guilfoyle D, Ritter W, Murray MM. Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. J Neurophysiol. 2002;88:540–543. doi: 10.1152/jn.2002.88.1.540. [DOI] [PubMed] [Google Scholar]
  15. Froyen D, Van Atteveldt N, Bonte M, Blomert L. Cross-modal enhancement of the MMN to speech-sounds indicates early and automatic integration of letters and speech-sounds. Neurosci Lett. 2008;430:23–28. doi: 10.1016/j.neulet.2007.10.014. [DOI] [PubMed] [Google Scholar]
  16. Fu KM, Johnston TA, Shah AS, Arnold L, Smiley J, Hackett TA, Garraghty PE, Schroeder CE. Auditory cortical neurons respond to somatosensory stimulation. J Neurosci. 2003;23:7510–7515. doi: 10.1523/JNEUROSCI.23-20-07510.2003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Giard MH, Lavikainen J, Reinikainen K, Perrin F, Bertrand O, Pernier J, Näätänen R. Separate representation of stimulus frequency, intensity, and duration in auditory sensory memory: an event-related potential and dipole-model analysis. J Cogn Neurosci. 1995;7:133–143. doi: 10.1162/jocn.1995.7.2.133. [DOI] [PubMed] [Google Scholar]
  18. Guest S, Catmur C, Lloyd D, Spence C. Audiotactile interactions in roughness perception. Exp Brain Res. 2002;146:161–171. doi: 10.1007/s00221-002-1164-z. [DOI] [PubMed] [Google Scholar]
  19. Guthrie D, Buchwald JS. Significance testing of difference potentials. Psychophysiology. 1991;28:240–244. doi: 10.1111/j.1469-8986.1991.tb00417.x. [DOI] [PubMed] [Google Scholar]
  20. Horváth J, Czigler I, Jacobsen T, Maess B, Schröger E, Winkler I. MMN or no MMN: no magnitude of deviance effect on the MMN amplitude. Psychophysiology. 2008;45:60–69. doi: 10.1111/j.1469-8986.2007.00599.x. [DOI] [PubMed] [Google Scholar]
  21. Jacobsen T, Schröger E. Is there pre-attentive memory-based comparison of pitch? Psychophysiology. 2001;38:723–727. [PubMed] [Google Scholar]
  22. Jacobsen T, Schröger E, Horenkamp T, Winkler I. Mismatch negativity to pitch change: varied stimulus proportions in controlling effects of neural refractoriness on human auditory event-related brain potentials. Neurosci Lett. 2003;344:79–82. doi: 10.1016/s0304-3940(03)00408-7. [DOI] [PubMed] [Google Scholar]
  23. Jousmäki V, Hari R. Parchment-skin illusion: sound-biased touch. Curr Biol. 1998;8:R190. doi: 10.1016/s0960-9822(98)70120-4. [DOI] [PubMed] [Google Scholar]
  24. Kekoni J, Hämäläinen H, Saarinen M, Gröhn J, Reinikainen K, Lehtokoski A, Näätänen R. Rate effect and mismatch responses in the somatosensory system: ERP-recordings in humans. Biol Psychol. 1997;46:125–142. doi: 10.1016/s0301-0511(97)05249-6. [DOI] [PubMed] [Google Scholar]
  25. Keniston LP, Henderson SC, Meredith MA. Neuroanatomical identification of crossmodal auditory inputs to interneurons in somatosensory cortex. Exp Brain Res. 2010;202:725–731. doi: 10.1007/s00221-010-2163-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Lehmann D, Skrandies W. Reference-free identification of components of checkerboard-evoked multichannel potential fields. Electroencephalogr Clin Neurophysiol. 1980;48:609–621. doi: 10.1016/0013-4694(80)90419-8. [DOI] [PubMed] [Google Scholar]
  27. Manly BFJ. Randomization and Monte Carlo methods in biology. Ed 1. New York: Chapman and Hall; 1991. [Google Scholar]
  28. Meredith MA, Allman BL. Subthreshold multisensory processing in cat auditory cortex. Neuroreport. 2009;20:126–131. doi: 10.1097/WNR.0b013e32831d7bb6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Meredith MA, Keniston LR, Dehner LR, Clemo HR. Crossmodal projections from somatosensory area SIV to the auditory field of the anterior ectosylvian sulcus (FAES) in Cat: further evidence for subthreshold forms of multisensory processing. Exp Brain Res. 2006;172:472–484. doi: 10.1007/s00221-006-0356-3. [DOI] [PubMed] [Google Scholar]
  30. Meredith MA, Allman BL, Keniston LP, Clemo HR. Auditory influences on non-auditory cortices. Hear Res. 2009;258:64–71. doi: 10.1016/j.heares.2009.03.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ. Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res. 2002;14:115–128. doi: 10.1016/s0926-6410(02)00066-6. [DOI] [PubMed] [Google Scholar]
  32. Molholm S, Martinez A, Ritter W, Javitt DC, Foxe JJ. The neural circuitry of pre-attentive auditory change-detection: an fMRI study of pitch and duration mismatch negativity generators. Cereb Cortex. 2005;15:545–551. doi: 10.1093/cercor/bhh155. [DOI] [PubMed] [Google Scholar]
  33. Möttönen R, Krause CM, Tiippana K, Sams M. Processing of changes in visual speech in the human auditory cortex. Brain Res Cogn Brain Res. 2002;13:417–425. doi: 10.1016/s0926-6410(02)00053-8. [DOI] [PubMed] [Google Scholar]
  34. Murray MM, Molholm S, Michel CM, Heslenfeld DJ, Ritter W, Javitt DC, Schroeder CE, Foxe JJ. Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb Cortex. 2005;15:963–974. doi: 10.1093/cercor/bhh197. [DOI] [PubMed] [Google Scholar]
  35. Näätänen R. Attention and brain function. Hillsdale, NJ: Erlbaum; 1992. [Google Scholar]
  36. Näätänen R, Paavilainen P, Rinne T, Alho K. The mismatch negativity (MMN) in basic research of central auditory processing: a review. Clin Neurophysiol. 2007;118:2544–2590. doi: 10.1016/j.clinph.2007.04.026. [DOI] [PubMed] [Google Scholar]
  37. Paavilainen P, Valppu S, Näätänen R. The additivity of the auditory feature analysis in the human brain as indexed by the mismatch negativity: 1+1 approximate to 2 but 1+1+1 < 3. Neurosci Lett. 2001;301:179–182. doi: 10.1016/s0304-3940(01)01635-4. [DOI] [PubMed] [Google Scholar]
  38. Pantev C, Lappe C, Herholz SC, Trainor L. Auditory-somatosensory integration and cortical plasticity in musical training. Ann NY Acad Sci. 2009;1169:143–150. doi: 10.1111/j.1749-6632.2009.04588.x. [DOI] [PubMed] [Google Scholar]
  39. Perrin F, Pernier J, Bertrand O, Giard MH, Echallier JF. Mapping of scalp potentials by surface spline interpolation. Electroencephalogr Clin Neurophysiol. 1987;66:75–81. doi: 10.1016/0013-4694(87)90141-6. [DOI] [PubMed] [Google Scholar]
  40. Perrin F, Pernier J, Bertrand O, Echallier JF. Spherical splines for scalp potential and current-density mapping. Electroencephalogr Clin Neurophysiol. 1989;72:184–187. doi: 10.1016/0013-4694(89)90180-6. [DOI] [PubMed] [Google Scholar]
  41. Saint-Amour D, De Sanctis P, Molholm S, Ritter W, Foxe JJ. Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia. 2007;45:587–597. doi: 10.1016/j.neuropsychologia.2006.03.036. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Schroeder CE, Foxe J. Multisensory contributions to low-level, ‘unisensory’ processing. Curr Opin Neurobiol. 2005;15:454–458. doi: 10.1016/j.conb.2005.06.008. [DOI] [PubMed] [Google Scholar]
  43. Schroeder CE, Foxe JJ. The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Res Cogn Brain Res. 2002;14:187–198. doi: 10.1016/s0926-6410(02)00073-3. [DOI] [PubMed] [Google Scholar]
  44. Schroeder CE, Lindsley RW, Specht C, Marcovici A, Smiley JF, Javitt DC. Somatosensory input to auditory association cortex in the macaque monkey. J Neurophysiol. 2001;85:1322–1327. doi: 10.1152/jn.2001.85.3.1322. [DOI] [PubMed] [Google Scholar]
  45. Stein B, Meredith MA. The merging of the senses. Cambridge, MA: MIT; 1993. [Google Scholar]
  46. Stevens SS. The volume and intensity of tones. Am J Psychol. 1934;46:397–408. [Google Scholar]
  47. Wilson EC, Reed CM, Braida LD. Integration of auditory and vibrotactile stimuli: effects of phase and stimulus-onset asynchrony. J Acoust Soc Am. 2009;126:1960–1974. doi: 10.1121/1.3204305. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Wilson EC, Braida LD, Reed CM. Perceptual interactions in the loudness of combined auditory and vibrotactile stimuli. J Acoust Soc Am. 2010a;127:3038–3043. doi: 10.1121/1.3377116. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Wilson EC, Reed CM, Braida LD. Integration of auditory and vibrotactile stimuli: effects of frequency. J Acoust Soc Am. 2010b;127:3044–3059. doi: 10.1121/1.3365318. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Winkler I, Horváth J, Weisz J, Trejo LJ. Deviance detection in congruent audiovisual speech: evidence for implicit integrated audiovisual memory representations. Biol Psychol. 2009;82:281–292. doi: 10.1016/j.biopsycho.2009.08.011. [DOI] [PubMed] [Google Scholar]
  51. Wolff C, Schröger E. Human pre-attentive auditory change-detection with single, double, and triple deviations as revealed by mismatch negativity additivity. Neurosci Lett. 2001;311:37–40. doi: 10.1016/s0304-3940(01)02135-8. [DOI] [PubMed] [Google Scholar]
  52. Yau JM, Olenczak JB, Dammann JF, Bensmaia SJ. Temporal frequency channels are linked across audition and touch. Curr Biol. 2009;19:561–566. doi: 10.1016/j.cub.2009.02.013. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from The Journal of Neuroscience are provided here courtesy of Society for Neuroscience

RESOURCES