Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2009 Jul 1.
Published in final edited form as: Neuroimage. 2008 Mar 20;41(3):1011–1020. doi: 10.1016/j.neuroimage.2008.03.015

Touch, Sound and Vision in Human Superior Temporal Sulcus

Michael S Beauchamp 1, Nafi E Yasar 1,2, Richard E Frye 3, Tony Ro 4
PMCID: PMC2409200  NIHMSID: NIHMS48764  PMID: 18440831

Abstract

Human superior temporal sulcus (STS) is thought to be a key brain area for multisensory integration. Many neuroimaging studies have reported integration of auditory and visual information in STS but less is known about the role of STS in integrating other sensory modalities. In macaque STS, the superior temporal polysensory area (STP) responds to somatosensory, auditory and visual stimulation. To determine if human STS contains a similar area, we measured brain responses to vibrotactile somatosensory, auditory and visual stimuli using blood-oxygen level dependent functional magnetic resonance imaging (BOLD fMRI). An area in human posterior STS, STSms (multisensory), responded to stimulation in all three modalities. STSms responded during both active and passive presentation of unisensory vibrotactile stimuli and showed larger responses for more intense vs. less intense tactile stimuli, hand vs. foot, and contralateral vs. ipsilateral tactile stimulation. STSms showed responses of similar magnitude for unisensory tactile and auditory stimulation, with an enhanced response to simultaneous auditory-tactile stimulation. We conclude that STSms is important for integrating information from the somatosensory as well as the auditory and visual modalities, and could be the human homolog of macaque STP.

Keywords: somatosensory, multisensory integration, motion, visual, auditory, tactile

Introduction

In everyday life, perceptual events often occur in multiple sensory modalities: we may feel our cell phone vibrate, hear it ring, or see the display flash, all indicating an incoming call. Where and how such multisensory processing occurs has intrigued philosophers, psychologists, and neuroscientists since at least the time of Aristotle (Aristotle, 350 B.C.E.). In the macaque monkey, an important multisensory region lies along the fundus of the posterior superior temporal sulcus (STS). This region was originally labeled the superior temporal polysensory (STP) area because single units in this area respond to visual, auditory and somatosensory stimulation (Bruce et al., 1981). Physiological and anatomical studies have delineated the cortical and subcortical connections and functional properties of macaque STP, also sometimes referred to as TPO (Padberg et al., 2003). Identifying the human homolog of macaque STP will allow us to generate additional hypotheses about the functional and anatomical properties of human STS (Beauchamp, 2005a).

In the banks of human posterior STS, neuroimaging studies have reported multisensory responses to auditory and visual stimulation (Beauchamp et al., 2004b; Calvert, 2001; Noesselt et al., 2007; Van Atteveldt et al., 2004; Wright et al., 2003). This region has been termed STSms, the STS multisensory region (Beauchamp et al., 2004a). Guided by the macaque literature, we wanted to determine if STSms was also important for processing somatosensory information. Previous human fMRI studies examining responses to somatosensory, auditory and visual stimulation have found regions responsive to all three modalities in parietal and frontal cortex, but not in the STS (Bremmer et al., 2001; Downar et al., 2000). Some studies of somatosensory processing have reported activity in STS (Burton et al., 2006; Disbrow et al., 2001; Golaszewski et al., 2002) but it is unclear if somatosensory, auditory and visual responses occur in human STSms as they do in macaque STP.

The primary goal of our experiments was to test the hypothesis that human STSms responds to somatosensory, auditory and visual stimulation. A secondary goal, contingent on the presence of somatosensory responses in STSms, was to test the hypothesis that multisensory integration between touch and sound occurs in STSms. Because a benchmark of multisensory integration is a differential response to multisensory compared with unisensory stimulation (Beauchamp, 2005b), we compared the response to multisensory and unisensory somatosensory and auditory stimulation. A final goal of the experiments was to characterize somatosensory and visual responses in STSms to a broad range of stimuli to allow an assessment of whether human STSms has similar response properties as macaque STP, above and beyond simply responding to touch, sound and vision.

Methods

We used a single-subject approach, identifying STSms on cortical surface models created for each individual subject. To allow us to devote the bulk of the experimental time to studying somatosensory responses in STSms, we used functional localizers (Saxe et al., 2006) to map visual responses in STSms in experiment 1 and visual and auditory responses in STSms in experiment 2. Table 1 lists a summary of the experimental conditions across experiments.

Table 1. Distribution of experimental conditions across subjects and experiments.

Each task refers to a separate experimental condition undertaken in a separate MR scan series (“run”). Every subject performed every task, but the number of scan series devoted to each task varied from subject to subject. The number in the scan series column shows the range across subjects. The design column shows the type of stimulus presentation paradigm (BD: block design; RER: rapid event-related). The number in the conditions column shows the number of different conditions in each task, including fixation baseline.

Task Scan series Design Conditions
Experiment 1: 12 Subjects
Visual localizer 1–3 BD 2
Vibrotactile somatosensory and auditory 3–6 RER 6
Experiment 2: 8 Subjects
Visual motion localizer 1 BD 3
Visual object localizer 1 BD 3
Auditory Localizer 1 BD 2
Vibrotactile somatosensory 6 RER 6
Open in a new tab

Subjects were recruited and informed consent was obtained in accordance with the University of Texas Committee for the Protection of Human Subjects. Eight subjects participated in experiment one (2M, 6F, mean age 26 yrs) and twelve subjects participated in experiment two (8M, 4F, mean age 27 yrs). Subjects’ data was anonymized with two letter experiment codes not corresponding to the subjects’ initials.

General MRI Methods

Participants were scanned using a 3 tesla whole-body MR scanner (Phillips Medical Systems, Bothell, WA). Anatomical images were collected using a magnetization-prepared 180 degrees radio-frequency pulses and rapid gradient-echo (MP-RAGE) sequence optimized for gray-white matter contrast with 1 mm thick sagittal slices and an in-plane resolution of 0.938 × 0.938 mm. Functional images were collected using a gradient-recalled-echo echo-planar-imaging sequence sensitive to the BOLD signal. Thirty-three axial slices were collected with an echo time (TE) of 30 ms and a flip angle of 90 degrees. Slice thickness was 3 mm and in-plane resolution was 2.75 mm × 2.75.

Experiment 1

Experimental Paradigm

As shown in Fig. 1, a three by two design was employed, with three categories of sensory stimulation (tactile-only, auditory-only, simultaneous tactile-auditory) and two intensities of stimulation (strong and weak). The trial duration was 2.75 sec, corresponding to an MRI repetition time (TR) of 2.75 seconds. Within each TR, acquisition was clustered so that imaging (with its accompanying sound and vibration) was completed in the first 2 seconds of the TR, followed by 0.75 seconds of silence. During the middle 500 ms of this silent interval, the stimulus was presented. A rapid event-related design was used. Each 5-minute scan series contained 110 trials (corresponding to 110 TRs) with 15 trials of each type and 20 trials of fixation baseline with no auditory or tactile stimulation.

Figure 1. Structure of stimulation trials in experiment 1.

Figure 1

Open in a new tab

A. Sequence of image acquisition and stimulation in experiment 1. Clustered MRI acquisition protocol. During the first 2 sec of each 2.75 sec trial, 33 images were acquired. Each image acquisition (indicated with thick vertical bars) produced sound and vibration from the scanner. Experimental stimuli were presented in the final 0.75 sec of the trial, when no MR image acquisition occurred.

B. Sensory stimulus during strong tactile trials. Green line indicates tactile stimulation, red line indicates auditory stimulation. Thick green portion indicates duration of a 200 Hz sinusoidal vibrotactile stimulus applied to the left hand. Gradual rise and fall of the thick green portion reflects the 4 Hz sinusoidal envelope applied to the driving waveform.

C. Sensory stimulus during strong auditory trials. Thick red line indicates duration of a 200 Hz sinusoid played in the left ear.

D. Sensory stimulus during strong tactile + auditory trials. Thick lines indicate simultaneous tactile + auditory stimulus presentation.

E. Unisensory tactile stimulus presented at low intensity.

F. Unisensory auditory stimulus presented at low intensity.

G. Multisensory tactile + auditory stimulus presented at low intensity.

Vibrotactile somatosensory stimuli were delivered using a piezoelectric bending element (Piezo Systems, Inc., Cambridge, MA) attached to the left hand using non-slip silicon elastic bandages. The qualitative percept of stimulation was akin to holding a ringing cell phone set to “vibrate” mode, without any accompanying auditory percept (the vibration of the benders was inaudible because of its low sound pressure level and the MR-compatible sound attenuating headphones worn by the subjects). Auditory stimuli were delivered to only the left channel (left ear) of the headphones to produce rough spatial correspondence with the left hand tactile stimulation.

The same waveform was used for vibrotactile stimulation (delivered via the piezoelectric benders) and auditory stimulation (delivered via headphones). A driving voltage was generated by a 24-bit PC sound card and amplified by a multichannel amplifier (Sony USA, New York, NY). As shown in Figure 1, the waveform consisted of a 200 Hz sinusoidal oscillation in a 500 ms envelope. To prevent onset and offset artifacts, the first and last 100 ms of the 500 ms envelope consisted of the first and second quarter-cycle of a 5 Hz sine wave, allowing the oscillation amplitude to gradually increase and decrease.

During experimental trials, subjects discriminated between the three trial types (tactile-only, auditory-only, or auditory-tactile) by pressing one of three buttons on a fiber optic response stick (Current Designs, Philadelphia, PA). No feedback was provided. Subjects were instructed to fixate central crosshairs, back-projected from an LCD projector (Sony Electronics, San Diego, CA) onto a Lucite screen (Da-Lite Inc., Warsaw, IN) and viewed through a mirror attached to the MR head coil. An MR-compatible eye-tracking system (Applied Science Laboratories, Bedford, MA) was used to monitor fixation and behavioral state.

Two intensities of stimulation were used: strong and weak. The intensities were adjusted for each subject in the MR scanner just prior to fMRI data collection, using the same driving waveform as used in the fMRI experiment. A strong tactile stimulus was delivered at a fixed intensity (10 dB attenuation equivalent to 30 V driving voltage and approximately 153 um displacement for four subjects; 15 dB attenuation, 17 V, 72 um for three subjects; 17 dB, 13 V, 72 um for five subjects). To set the level of the strong auditory stimulus, an auditory stimulus was presented at the same time as the strong tactile stimulus. Subjects used the MR-compatible response buttons to adjust the intensity of the auditory stimulus until it matched the perceived magnitude of the strong tactile stimulus (mean attenuation 16 dB ± 2dB SEM, mean sound pressure level 72 dB ± 2 dB). To set the level of the weak tactile stimulus, subjects decreased the intensity of the strong tactile stimulus until it was very weak but could still be detected on every presentation (50 ± 1 dB attenuation, 0.3 V ± 0.04 V, 1.6 ± 0.2 um displacement). This threshold was consistent with previous psychophysical studies using 200 Hz vibrotactile stimulation (Brisben et al., 1999). To set the level of the weak auditory stimulus, subjects adjusted the intensity of the auditory stimulus to match the intensity of a simultaneously presented weak tactile stimulus (42 dB ± 2 dB attenuation, 49 ± 2 dB SPL).

Visual Localizer

To identify visually responsive brain regions, a block-design visual localizer was conducted, in which subjects performed no task but alternately viewed 30-second excerpts from a movie (Winged Migration, Sony Pictures Classics) and fixation baseline.

Experiment 2

Experimental Paradigm

The vibrotactile somatosensory stimulus in Experiment 2 was delivered by five piezoelectric benders attached to the left and right hand and foot of the subject and the right hip. Trial duration and TR were both 2 seconds (clustered acquisition was not used) and there were five trial types, each containing stimulation of a single bender. The driving voltage consisted of a 200 Hz sine wave modulated by a 4 Hz square-wave envelope. There was no task during hand or foot stimulation. Hip stimulation trials (catch trials) required subjects to make an eye movement to a visual target (the word “TARGET”) in the upper right corner of the display screen, which was otherwise blank except for white fixation crosshairs (the target and fixation crosshairs were always present, so there were no visual transients associated with changes in the display). fMRI data from the catch trials were analyzed separately, so that oculomotor activations in catch trials would not confound the somatosensory activations measured in hand and foot trials; only the responses in hand and foot trials are reported here. In the rapid event-related design, each 5-minute scan series contained 150 trials (corresponding to 150 TRs) with 25 of each of the four types of hand and foot trials, 10 catch trials and 40 fixation baseline trials. Subjects performed 4 – 6 runs. A report on somatosensory responses in area MST using the data collected for experiment 2 has been previously published (Beauchamp et al., 2007).

Visual and Auditory Localizers

In separate scan series, subjects performed different auditory and visual localizers (see Table 1 for a summary). In the first localizer, subjects viewed low-contrast random moving dots presented in the left or right hemifields alternating with stationary dots. In the second localizer, subjects viewed real photographs of objects and scrambled photographs, alternating with fixation baseline. In the third localizer, subjects heard brief (1–2 second) recordings of a variety of non-linguistic stimuli, including recordings of animal calls, recordings of man-made objects (both manual and powered), scrambled versions of these recordings, and pure tones (Beauchamp et al., 2004b). Subjects performed a simple detection task during each localizer to ensure attention to the stimulus.

Experiment 1 and 2

fMRI Experimental Design and Data Analysis

fMRI data was analyzed using AFNI (Cox, 1996). Individual cortical surface models were created with FreeSurfer (Fischl et al., 1999) and visualized in SUMA (Argall et al., 2006). Localizer experiments were performed with a block design and analyzed with the general linear model by convolving the timing of each type of stimulation block with a gamma-variate function. Tactile experiments were conducted using a rapid event-related design, and analyzed with finite impulse response deconvolution. This allows estimation of the hemodynamic response to each trial type as if it had been presented in isolation in a slow event-related design.

To identify areas responding to auditory, visual and somatosensory stimulation, a modified conjunction analysis was used (Nichols et al., 2005). In each subject, the t-statistic of the contrast between stimulation vs. rest was independently calculated for each sensory modality in every voxel. This contrast revealed voxels that showed either a positive or negative BOLD response to sensory stimulation. Because a task-independent network of brain areas is deactivated (negative BOLD response) during any kind of sensory stimulation (Raichle et al., 2001) we selected only voxels showing a positive BOLD response to each sensory modality. This criterion was instantiated with the thresholding operation (Visual-t-statistic > x) AND (Auditory-t-statistic > x) AND (Tactile-t-statistic > x) where x is the unisensory threshold (Beauchamp, 2005b). All voxels passing this test were classified as “multisensory”, mapped to the cortical surface and classified as inside or outside the STS using an automated surface parcellation algorithm (Fischl et al., 2004). The time series from all multisensory STS voxels were converted to percent signal change and averaged to create an average time series for each subject. These time series were then averaged across subjects to create a grand mean.

A conjunction analysis was also used to create the mixed-effects group map. Individual subject brains were converted to standard space (Brett et al., 2002), and the percent signal change for each condition was entered into a voxel-wise ANOVA with subject as the random factor and condition as the fixed factor. A conjunction analysis was performed on the output of the ANOVA to find voxels showing a significant effect to each modality in isolation. All statistical inferences are based on between-subjects variance using a mixed-effects model, with stimulus type as the fixed factor and subject as the random factor.

Most statistical tests were performed only on the average time series created from all active voxels in each subject’s STS, mitigating the need to perform corrections for multiple comparisons. To create activation maps, a significance level of p < 0.05 (single voxel, uncorrected for multiple comparisons) was used for the single modality activation maps and p < 0.01 for the conjunction analysis. The actual probability of the observed STSms activations being due to chance fluctuations in the MR signal is considerably lower, approximately p < Pn, where P is the single-voxel p-value and n is the number of voxels in the STSms (Xiong et al., 1995). For individual subjects, mean n = 17; for the group map, n = 55.

Results

Experiment 1

Subjects received vibrotactile somatosensory stimulation on their left hand and auditory stimulation in their left ear while making behavioral responses with their right hand. To determine brain areas responsive to sensory stimulation, we focused our analysis on the right hemisphere, collapsing across different intensities of stimulation. As shown in Fig. 2A, tactile-only trials activated a broad network of frontal, parietal and temporal, including the post-central gyrus (the location of primary somatosensory cortex, S1), the parietal operculum (the location of secondary somatosensory cortex, S2), intraparietal sulcus, and the STS. Auditory-only trials activated a similar network of areas (including the STS) and the temporal plane, the location of core and belt areas of auditory cortex (Fig. 2B). The visual localizer activated occipital, temporal and parietal cortex, including the STS (Fig. 2C). To determine regions that responded to all three modalities, we performed a voxel-by-voxel conjunction analysis. Voxels concentrated in the parietal lobe and the STS were active in all three conditions (Fig. 2D). The mixed-effects group map showed a similar pattern, with a region of posterior STS responding to all three modalities (Fig. 2E). The center-of-mass of the STS activation in the group map was (52, 44, 15).

Figure 2. Responses to tactile, auditory and visual stimulation in experiment 1.

Figure 2

Open in a new tab

A. Lateral view of a single subject’s partially inflated right hemisphere. Colored regions responded significantly to tactile stimulation. Active regions in posterior STS are colored yellow, other active regions are colored purple. The fundus of the STS is shown as a white dashed line.

B. Single subject activation to auditory stimulation.

C. Single subject activation to visual stimulation.

D. Single subject conjunction map showing voxels responding to all three modalities. Circled yellow cluster shows the STS multisensory area, STSms.

E. Mixed-effects group map (n = 12). Voxels showing a significant response to all three modalities. Yellow cluster shows the STSms, with center of mass (52, 44, 15).

F. Single-subject MR time series from STSms. The dark black line shows the deconvolved event related response in a 16.5 second window following stimulation onset for three kinds of trials, collapsed across intensity of stimulation: Tac, tactile stimulation; Aud, auditory stimulation; Tac+Aud, tactile and auditory stimulation. The dashed line shows the mean unisensory response. The colored bars show the 500 ms stimulus duration.

G. Group average MR time series from STSms (n = 12). The dark black line shows the mean deconvolved event related response, the gray line shows ± 1 SEM.

After identifying STSms, we measured the degree of multisensory integration in STSms between tactile and auditory modalities. The evoked response in STSms to unisensory and multisensory trials was computed in each subject and averaged across subjects (Figs. 2F, G). The response resembled a classical hemodynamic response with a sharp increase followed by a slow return to baseline. Due to the relatively long TR (2.75 sec), the largest magnitude of response was observed in the second TR, 5.5 seconds following stimulus onset; this peak magnitude was used as a measure of the amplitude of response in different trials. Because the STSms was defined without reference to the multisensory response, unbiased statistical comparisons could be performed between multisensory and unisensory responses (Simmons et al., 2007).

The response was similar in unisensory tactile and auditory trials (0.30% vs. 0.31% ± 0.02% SEM for both). In multisensory tactile-auditory trials, the response was 23% larger than the maximum unisensory response and 24% larger than the average unisensory response (0.38% ± 0.02% SEM vs. 0.31% ± 0.02% SEM, paired t-test with 11 degrees of freedom, p = 0.0001). The response in the STSms to each of the six trials types was also entered into a three-factor mixed-effect ANOVA with stimulus modality (tactile, auditory, tactile-auditory) and intensity (weak, strong) as fixed factors and subject as a random factor. The most significant effect was modality (F(2,22) = 10.3, p = 0.0007) driven by the increased response to multisensory stimulation. There was also a significant effect of intensity (F(1,11) = 16.1, p = 0.002), reflecting a larger response to strong compared with weak stimuli (0.37% ± 0.02% vs. 0.29% ± 0.02%). The interaction between modality and intensity was not significant (F(2,22) = 0.1, p = 0.9) showing that the degree of multisensory enhancement did not differ between weak and strong multisensory trials.

Behavioral Data

In order to ensure attention to the sensory stimulus, subjects performed a simple three-alternative forced choice on stimulus modality. Performance was high for strong trials (88%, 93%, 96% for tactile, auditory and tactile-auditory strong trials, respectively) and weak trials (85%, 93%, 75%). Subjects frequently confused weak tactile-auditory trials with weak tactile-only and auditory-only trials. Consistent with the accuracy data, reaction time was also longer for weak trials (953, 926, 921 ms for tactile, auditory and tactile-auditory strong trials; 953, 950, 1062 ms for weak trials). To determine the significance of these effects, an ANOVA was performed. There was a significant effect of intensity (F(1,11) = 9.7, p = 0.01) and an interaction between modality and intensity (F(2,22) = 7.2, p = 0.004) driven by the poorer performance in the weak tactile-auditory condition.

Experiment 2

In experiment 1, subjects performed a discrimination task, manually pressing a button in response to each sensory stimulus. It could be argued that the observed STS activations were the result of cognitive processes involved in task performance, rather than simple sensory responses. To address this possibility, in experiment 2 subjects received somatosensory vibrotactile stimulation on their hands and feet that did not require a behavioral response (Beauchamp et al., 2007).

Because tactile stimuli were delivered bilaterally, we expected responses to be evoked in both left and right hemispheres. Consistent with this, we observed activation in the left and right postcentral gyrus, parietal operculum and STS (Fig. 3A). Localizers were used to map auditory and visually-responsive brain regions. Auditory responses were observed in the temporal plane, inferior frontal cortex, and the STS (Fig. 3B) while visual responses were found primarily in the occipital lobe and the STS. A conjunction analysis revealed a focus of trisensory activation in posterior STS in the single-subject (Fig. 3D) and group average activation maps (Fig. 3E). The center-of-mass of the average STS activation was (56, 41, 14) in the right hemisphere and (−44, 35, 13) in the left hemisphere.

Figure 3. Brain areas responding to auditory, visual and tactile stimulation in experiment 2.

Figure 3

Open in a new tab

A. Lateral view of a single subject’s partially inflated right hemisphere. Colored regions responded significantly to tactile stimulation. Active regions in posterior STS are colored yellow, other active regions are colored purple. The fundus of the STS is shown as a white dashed line.

B. Single subject activation to auditory stimulation.

C. Single subject activation to visual stimulation.

D. Single subject conjunction map showing voxels responding to all three modalities. Circled yellow cluster shows the STS multisensory area, STSms.

E. Mixed-effects group map (n = 8). Voxels showing a significant response to all three modalities. Yellow cluster shows the STSms, with center of mass (−44, 35, 13) in left hemisphere and (56, 41, 14) in right hemisphere.

The event-related design used for the tactile experiment allowed us to extract the average hemodynamic responses to single stimulation trials (Fig. 4A). The strongest response was to contralateral hand stimulation (0.25%), which was significantly greater than the response to ipsilateral hand stimulation (0.18%, paired t-test with 7 degrees of freedom, p = 0.02) contralateral foot stimulation (0.21%, p = 0.02) and ipsilateral foot stimulation (0.19%, p = 0.02). In order to determine the functional properties of the STSms, we also calculated the average evoked response during the different stimulus conditions presented in the visual and auditory block-design localizers. STSms showed a strong response to low-contrast moving points, with a greater response to contralateral than ipsilateral motion (Fig. 4B; 0.45% vs. 0.29%, p = 0.004). STSms also responded to static images (Fig. 4C), although significantly weaker than the response to moving points (0.13%, p = 0.03). There was no significant difference in the response to real photographs compared with the response to scrambled photographs (0.13% for both). Auditory stimulation produced a strong response that was equivalent in magnitude (0.41%, p = 0.4) to the strongest visual stimulus (contralateral moving points) but was significantly greater than the response to the other visual stimuli (p = 0.0004) although these comparisons must be interpreted cautiously because auditory and visual stimuli were presented in different scan series.

Figure 4. Timecourse of average evoked BOLD response (n = 8 subjects) in the STS multisensory area.

Figure 4

Open in a new tab

A. Response to contralateral hand (CH), ipsilateral hand (IH), contralateral foot (CF), and ipsilateral foot (IF) stimulation. Colored bars illustrate 2 second stimulus duration. Black lines show mean, gray lines show ± 1 SEM.

B. Response to low-contrast moving points in the contralateral (CVF) and ipsilateral (IVF) visual field. Response to auditory stimuli (AUD) is shown for comparison. Colored bars illustrate 20 second stimulus duration (followed by fixation baseline).

C. Response to real (REAL) and scrambled (SCR) photographs.

In macaque monkeys, area STP is located anterior and superior to areas MST and MT. To determine the relationship between human MT, MST and STSms, previously described techniques (Beauchamp et al., 2007; Huk et al., 2002) were used to create maps of all three areas in two hemispheres (Fig. 5). MT was located in the posterior bank of the ascending limb of the posterior inferior temporal sulcus. MST was located anterior in the ascending limb of the sulcus, extending onto middle temporal gyrus. STSms was located on the posterior bank and fundus of the STS, just anterior to MST. Across subjects, a consistent anatomical landmark for STSms was the inflection point in the posterior superior temporal sulcus where it angles upwards towards the parietal lobe. The anatomical positioning of MT, MST and STSms in human cortex was similar to that of MT, MST and STP in macaque cortex (Fig. 5D).

Figure 5. Relationship between the STS multisensory area (STSms) and areas MT and MST.

Figure 5

Open in a new tab

A. Lateral view of a single subject’s partially inflated left hemisphere. Colored regions responded significantly to all three modalities. Active regions in posterior STS are colored yellow, other active regions are colored purple. The fundus of the STS is shown as a white dashed line. Red box indicates the region enlarged in B.

B. Composite map showing multisensory activation and localizer defined MT and MST. White outline shows STSms, blue outline shows MST, green outline shows MT.

C. Composite map in an additional hemisphere from a different subject.

D. Relationship between macaque area STP and macaque areas MT and MST. The top panel shows a lateral view of a macaque brain (Dickson et al., 2001). The fundus of the STS is shown as a white dashed line. The bottom panel shows an inflated view of the brain, with labeled areas from (Lewis and Van Essen, 2000b): MT, MST (MSTdp+MSTm) and STP (TPOi+TPOc).

Discussion

Guided by the literature on macaque STP, we hypothesized that human STS should contain an area that responds to somatosensory, auditory and visual stimulation. Data from twenty subjects in two separate imaging experiments supported this hypothesis.

Tactile Responses in STSms

Previous studies have reported somatosensory responses in human STS (Burton et al., 2006; Disbrow et al., 2001; Golaszewski et al., 2002). The present results are the first to show that these responses are co-localized with auditory and visual responses. The results of experiment 1 might have reflected a general cognitive process important for the behavioral task rather than a modality-specific sensory response in STS. However, passive presentation of somatosensory stimuli in experiment 2 evoked a similar magnitude of response as experiment 1 suggesting that the behavioral task is not required for somatosensory STS responses. The magnitude of STSms response was modulated by the intensity of the tactile stimulation and by the body site of stimulation, further supporting the conclusion that these reflect sensory processing rather than task performance.

Multisensory Integration in STSms

Previous studies have shown that posterior STS responds more to multisensory auditory-visual stimuli than to unisensory auditory or visual stimuli (Beauchamp et al., 2004b; Calvert, 2001; Hein et al., 2007; Noesselt et al., 2007; Raij et al., 2000; Van Atteveldt et al., 2004). Consistent with these results, we observed a larger response for multisensory auditory-tactile stimuli than unisensory auditory or tactile stimulation. The degree of enhancement for auditory-tactile multisensory stimulation compared to the maximum unisensory response in the present study was 23%, similar to the 17% enhancement for auditory-visual multisensory stimuli in STSms observed in a previous study (Beauchamp et al., 2004b). These results add to a body of evidence showing multisensory interactions between touch and sound in auditory cortex, sometimes extending into the STS (Foxe et al., 2002; Kayser et al., 2005; Murray et al., 2005; Schroeder et al., 2001).

In the present study, “super-additive” multisensory responses were not observed. That is, the response to auditory-tactile stimuli was greater than the response to auditory or tactile stimuli in isolation, but was not greater than the summed response to auditory and tactile unisensory stimuli (Stein and Meredith, 1993). Previous fMRI studies of auditory-visual integration in STS (Beauchamp et al., 2004a; Beauchamp et al., 2004b; Hein et al., 2007; Van Atteveldt et al., 2004; van Atteveldt et al., 2007) and auditory-tactile integration in auditory cortex (Kayser et al., 2005) have also not observed super-additive changes in the BOLD signal, perhaps because only a few single neurons show super-additivity (Laurienti et al., 2005; Perrault et al., 2005). Supporting this idea, in single-unit recording studies, only a small fraction of STP neurons respond to both auditory and tactile stimulation (Bruce et al., 1981; Hikosaka et al., 1988); the same is true in multisensory regions of cat cortex (Clemo et al., 2007). Conversely, many single neurons may show no response to a sensory stimulus in isolation, but the same stimulus may modulate responses when presented with other sensory modalities (Allman and Meredith, 2007). In macaque auditory cortex, auditory-tactile integration increases as the auditory stimulus decreases in intensity (Lakatos et al., 2007) consistent with the so-called law of inverse effectiveness (Stein and Meredith, 1993). In the present experiment, differences in auditory-tactile integration were not observed for weak and strong tactile stimuli, possible because all of the auditory stimuli used were well above threshold.

Double label studies show that projections into STP from parietal and temporal lobe (carrying visual and auditory information, respectively) project to non-overlapping, but often adjacent, patches of cortex (Hackett et al., 2007; Seltzer et al., 1996; Smiley et al., 2007). Functional responses in macaque STP are also unevenly distributed (Dahl et al., 2007). Consistent with these findings, in a high resolution fMRI study, human STSms was observed to contain a patchy distribution of auditory, visual and multisensory auditory-visual responses (Beauchamp et al., 2004a). It is not clear whether macaque STP or human STSms contains an additional, dedicated set of patches that respond preferentially to somatosensory stimulation, or whether somatosensory stimuli arrive in STSms within the previously described auditory, visual and multisensory patches.

Homology between macaque STP and the human STS multisensory area

We hypothesized that if human STSms is the homolog of macaque STP, it should share the same anatomical relationship with nearby identified areas, especially the adjacent area MST. Detailed functional mapping showed that human STSms was located just anterior to areas MST and MT, the same anatomical relationship that exists between MT, MST and STP in macaque cortex (Fig. 5D)(Lewis and Van Essen, 2000a).

If STSms is homologous to macaque STP, it should also have similar functional properties, above and beyond simply responding to the same three sensory modalities. We used previous electrophysiological and fMRI studies of macaque STP as a gauge to compare the functional properties of macaque STP with the functional activation of the human STS as measured in this study; simultaneous electrophysiological and fMRI studies have shown good correlation between multiunit activity, local field potentials and the BOLD response (Logothetis et al., 2001). Retinotopy in macaque STP, as measured with fMRI, is relatively crude (Nelissen et al., 2006). Receptive fields of single units in STP are large; most are limited to the contralateral visual field but about a third also respond to the ipsilateral visual field (Hikosaka et al., 1988). This would predict a significant ensemble BOLD fMRI response for ipsilateral stimulation, and a larger response for contralateral stimulation. This is exactly the BOLD signal we recorded from STSms: ipsilateral responses were significantly greater than zero, but significantly weaker than the response to contralateral visual stimulation. Macaque STP shows a significant fMRI response to moving compared with static stimuli (Nelissen et al., 2006) and visually-responsive macaque STP neurons are best activated by moving stimuli (Bruce et al., 1981; Hikosaka et al., 1988). Consistent with this finding, we observed significantly greater responses to moving compared with stationary stimuli in STSms, with only a weak response to static images. Macaque STP shows only a weak BOLD preference for shapes compared with scrambled shapes (Nelissen et al., 2006) and single STP neurons show little or no selectivity for shape (Bruce et al., 1981; Hikosaka et al., 1988). This matches our finding of no significant difference between real and scrambled static images in STSms. However, some neurons in TPO are face-selective (Baylis et al., 1987; Bruce et al., 1981) and human fMRI studies have described face selectivity in the posterior STS (Kanwisher et al., 1997).

In addition to similar visual processing profiles, the response selectivity of STSms to auditory and tactile stimuli was similar to that of macaque STP. Auditory-responsive STP neurons show broad-spectrum responses, with similar activity to very different sounds, such as pure tones, voices, white noise, and hand clapping (Bruce et al., 1981; Hikosaka et al., 1988). Consistent with this result, we saw robust activity in STSms to our auditory stimuli, which were pure tones in experiment 1 and a variety of animal, human and mechanical sounds in experiment 2. In tactile STP neurons, strong responses are evoked by cutaneous stimuli (Bruce et al., 1981; Hikosaka et al., 1988). The spatial preference of these neurons varies widely, from neurons that represent the entire body surface, to neurons that represent the contralateral body surface, to neurons that represent only the contralateral hand and arm. Estimating the ensemble response of these neurons, we would predict the largest responses to contralateral hand stimulation (which would activate all neurons) with the smallest responses to ipsilateral stimulation (which would activate only whole-body neurons). Consistent with this analysis, we observed the greatest BOLD activation in STSms for contralateral hand stimulation, and significantly weaker BOLD activation for ipsilateral hand and contralateral foot stimulation.

The role of multisensory responses in STSms

Visual biological motion is an especially potent activator of posterior STS (Beauchamp et al., 2002; Grossman and Blake, 2002). The STS is also important for processing speech, one of the main auditory cues used by humans to communicate (Price, 2000), with a special role for the integration of auditory and visual language cues (Callan et al., 2004; Calvert et al., 2000; Macaluso et al., 2004; Miller and D’Esposito, 2005; Saito et al., 2005; Schroeder et al., 2008; Sekiyama et al., 2003; van Atteveldt et al., 2007). STSms prefers real auditory stimuli to scrambled auditory stimuli (Beauchamp et al., 2004b) consistent with its role in the representation of sensory stimuli with meaning for the individual.

Some of the most important and meaningful types of sensory stimuli are social cues. The STS is thought to be an important node in the brain network for social cognition (Adolphs, 2003; Allison et al., 2000). Both human and non-human primates use visual, auditory and somatosensory cues to convey social information (Hauser and Konishi, 1999). Therefore, we speculate that multisensory integration of tactile responses in STSms might exist in the service of understanding the actions and intents of others. A firm pat on the back might be interpreted differently in the context of either a friendly greeting or a sharp reprimand. Integrating across modalities would allow the STSms to aid the individual in interpreting the ambiguous cues that abound in social interactions.

Acknowledgments

Benjamin Malmberg, David Strickland and Vips Patel assisted with subject recruitment and data collection. This research was supported in part by NSF grants 0642801 and 0642532 to MSB and TR and NIH NS046565 to RF. NIH S10 RR19186 provided partial funding for the purchase of the 3T scanner.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Adolphs R. Cognitive neuroscience of human social behaviour. Nat Rev Neurosci. 2003;4:165–178. doi: 10.1038/nrn1056. [DOI] [PubMed] [Google Scholar]
  2. Allison T, Puce A, McCarthy G. Social perception from visual cues: role of the STS region. Trends Cogn Sci. 2000;4:267–278. doi: 10.1016/s1364-6613(00)01501-1. [DOI] [PubMed] [Google Scholar]
  3. Allman BL, Meredith MA. Multisensory processing in “unimodal” neurons: cross-modal subthreshold auditory effects in cat extrastriate visual cortex. Journal of Neurophysiology. 2007;98:545–549. doi: 10.1152/jn.00173.2007. [DOI] [PubMed] [Google Scholar]
  4. Argall BD, Saad ZS, Beauchamp MS. Simplified intersubject averaging on the cortical surface using SUMA. Hum Brain Mapp. 2006;27:14–27. doi: 10.1002/hbm.20158. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Aristotle, 350 B.C.E. On Sense and the Sensible. The Internet Classics Archive, translated by J. I. Beare.
  6. Baylis GC, Rolls ET, Leonard CM. Functional subdivisions of the temporal lobe neocortex. Journal of Neuroscience. 1987;7:330–342. doi: 10.1523/JNEUROSCI.07-02-00330.1987. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Beauchamp MS. See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex. Current Opinion in Neurobiology. 2005a;15:145–153. doi: 10.1016/j.conb.2005.03.011. [DOI] [PubMed] [Google Scholar]
  8. Beauchamp MS. Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics. 2005b;3:93–114. doi: 10.1385/NI:3:2:093. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A. Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nature Neuroscience. 2004a;7:1190–1192. doi: 10.1038/nn1333. [DOI] [PubMed] [Google Scholar]
  10. Beauchamp MS, Lee KE, Argall BD, Martin A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron. 2004b;41:809–823. doi: 10.1016/s0896-6273(04)00070-4. [DOI] [PubMed] [Google Scholar]
  11. Beauchamp MS, Lee KE, Haxby JV, Martin A. Parallel visual motion processing streams for manipulable objects and human movements. Neuron. 2002;34:149–159. doi: 10.1016/s0896-6273(02)00642-6. [DOI] [PubMed] [Google Scholar]
  12. Beauchamp MS, Yasar NE, Kishan N, Ro T. Human MST but not MT responds to tactile stimulation. Journal of Neuroscience. 2007;27:8261–8267. doi: 10.1523/JNEUROSCI.0754-07.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Bremmer F, Schlack A, Shah NJ, Zafiris O, Kubischik M, Hoffmann K, Zilles K, Fink GR. Polymodal motion processing in posterior parietal and premotor cortex: a human fMRI study strongly implies equivalencies between humans and monkeys. Neuron. 2001;29:287–296. doi: 10.1016/s0896-6273(01)00198-2. [DOI] [PubMed] [Google Scholar]
  14. Brett M, Johnsrude IS, Owen AM. The problem of functional localization in the human brain. Nat Rev Neurosci. 2002;3:243–249. doi: 10.1038/nrn756. [DOI] [PubMed] [Google Scholar]
  15. Brisben AJ, Hsiao SS, Johnson KO. Detection of vibration transmitted through an object grasped in the hand. Journal of Neurophysiology. 1999;81:1548–1558. doi: 10.1152/jn.1999.81.4.1548. [DOI] [PubMed] [Google Scholar]
  16. Bruce C, Desimone R, Gross CG. Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. Journal of Neurophysiology. 1981;46:369–384. doi: 10.1152/jn.1981.46.2.369. [DOI] [PubMed] [Google Scholar]
  17. Burton H, McLaren DG, Sinclair RJ. Reading embossed capital letters: an fMRI study in blind and sighted individuals. Hum Brain Mapp. 2006;27:325–339. doi: 10.1002/hbm.20188. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Callan DE, Jones JA, Munhall K, Kroos C, Callan AM, Vatikiotis-Bateson E. Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. J Cogn Neurosci. 2004;16:805–816. doi: 10.1162/089892904970771. [DOI] [PubMed] [Google Scholar]
  19. Calvert GA. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cerebral Cortex. 2001;11:1110–1123. doi: 10.1093/cercor/11.12.1110. [DOI] [PubMed] [Google Scholar]
  20. Calvert GA, Campbell R, Brammer MJ. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol. 2000;10:649–657. doi: 10.1016/s0960-9822(00)00513-3. [DOI] [PubMed] [Google Scholar]
  21. Clemo HR, Allman BL, Donlan MA, Meredith MA. Sensory and multisensory representations within the cat rostral suprasylvian cortex. Journal of Comparative Neurology. 2007;503:110–127. doi: 10.1002/cne.21378. [DOI] [PubMed] [Google Scholar]
  22. Cox RW. AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages. Computers and Biomedical Research. 1996;29:162–173. doi: 10.1006/cbmr.1996.0014. [DOI] [PubMed] [Google Scholar]
  23. Dahl CD, Logothetis N, Kayser C. Functional organization of responses in the polysensory temporal region (STP) in the Macaque monkey. Society for Neuroscience; San Diego, CA: 2007. p. 714.713. [Google Scholar]
  24. Dickson J, Drury H, Van Essen DC. ‘The surface management system’ (SuMS) database: a surface-based database to aid cortical surface reconstruction, visualization and analysis. Philos Trans R Soc Lond B Biol Sci. 2001;356:1277–1292. doi: 10.1098/rstb.2001.0913. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Disbrow E, Roberts T, Poeppel D, Krubitzer L. Evidence for interhemispheric processing of inputs from the hands in human S2 and PV. Journal of Neurophysiology. 2001;85:2236–2244. doi: 10.1152/jn.2001.85.5.2236. [DOI] [PubMed] [Google Scholar]
  26. Downar J, Crawley AP, Mikulis DJ, Davis KD. A multimodal cortical network for the detection of changes in the sensory environment. Nat Neurosci. 2000;3:277–283. doi: 10.1038/72991. [DOI] [PubMed] [Google Scholar]
  27. Fischl B, Sereno MI, Dale AM. Cortical surface-based analysis. II: Inflation, flattening, and a surface-based coordinate system. Neuroimage. 1999;9:195–207. doi: 10.1006/nimg.1998.0396. [DOI] [PubMed] [Google Scholar]
  28. Fischl B, van der Kouwe A, Destrieux C, Halgren E, Segonne F, Salat DH, Busa E, Seidman LJ, Goldstein J, Kennedy D, Caviness V, Makris N, Rosen B, Dale AM. Automatically parcellating the human cerebral cortex. Cerebral Cortex. 2004;14:11–22. doi: 10.1093/cercor/bhg087. [DOI] [PubMed] [Google Scholar]
  29. Foxe JJ, Wylie GR, Martinez A, Schroeder CE, Javitt DC, Guilfoyle D, Ritter W, Murray MM. Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. Journal of Neurophysiology. 2002;88:540–543. doi: 10.1152/jn.2002.88.1.540. [DOI] [PubMed] [Google Scholar]
  30. Golaszewski SM, Siedentopf CM, Baldauf E, Koppelstaetter F, Eisner W, Unterrainer J, Guendisch GM, Mottaghy FM, Felber SR. Functional magnetic resonance imaging of the human sensorimotor cortex using a novel vibrotactile stimulator. Neuroimage. 2002;17:421–430. doi: 10.1006/nimg.2002.1195. [DOI] [PubMed] [Google Scholar]
  31. Grossman ED, Blake R. Brain Areas Active during Visual Perception of Biological Motion. Neuron. 2002;35:1167–1175. doi: 10.1016/s0896-6273(02)00897-8. [DOI] [PubMed] [Google Scholar]
  32. Hackett TA, De La Mothe LA, Ulbert I, Karmos G, Smiley J, Schroeder CE. Multisensory convergence in auditory cortex, II. Thalamocortical connections of the caudal superior temporal plane. Journal of Comparative Neurology. 2007;502:924–952. doi: 10.1002/cne.21326. [DOI] [PubMed] [Google Scholar]
  33. Hauser MD, Konishi M. The design of animal communication. MIT Press; Cambridge, Ma: 1999. [Google Scholar]
  34. Hein G, Doehrmann O, Muller NG, Kaiser J, Muckli L, Naumer MJ. Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. Journal of Neuroscience. 2007;27:7881–7887. doi: 10.1523/JNEUROSCI.1740-07.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Hikosaka K, Iwai E, Saito H, Tanaka K. Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey. Journal of Neurophysiology. 1988;60:1615–1637. doi: 10.1152/jn.1988.60.5.1615. [DOI] [PubMed] [Google Scholar]
  36. Huk AC, Dougherty RF, Heeger DJ. Retinotopy and Functional Subdivision of Human Areas MT and MST. Journal of Neuroscience. 2002;22:7195–7205. doi: 10.1523/JNEUROSCI.22-16-07195.2002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Kanwisher N, McDermott J, Chun MM. The fusiform face area: a module in human extrastriate cortex specialized for face perception. Journal of Neuroscience. 1997;17:4302–4311. doi: 10.1523/JNEUROSCI.17-11-04302.1997. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Kayser C, Petkov CI, Augath M, Logothetis NK. Integration of touch and sound in auditory cortex. Neuron. 2005;48:373–384. doi: 10.1016/j.neuron.2005.09.018. [DOI] [PubMed] [Google Scholar]
  39. Lakatos P, Chen CM, O’Connell MN, Mills A, Schroeder CE. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron. 2007;53:279–292. doi: 10.1016/j.neuron.2006.12.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Laurienti PJ, Perrault TJ, Jr, Stanford TR, Wallace MT, Stein BE. On The Use of Superadditivity as a Metric for Characterizing Multisensory Integration in Functional Neuroimaging Studies. Experimental Brain Research. 2005 doi: 10.1007/s00221-005-2370-2. [DOI] [PubMed] [Google Scholar]
  41. Lewis JW, Van Essen DC. Corticocortical connections of visual, sensorimotor, and multimodal processing areas in the parietal lobe of the macaque monkey. Journal of Comparative Neurology. 2000a;428:112–137. doi: 10.1002/1096-9861(20001204)428:1<112::aid-cne8>3.0.co;2-9. [DOI] [PubMed] [Google Scholar]
  42. Lewis JW, Van Essen DC. Mapping of architectonic subdivisions in the macaque monkey, with emphasis on parieto-occipital cortex. Journal of Comparative Neurology. 2000b;428:79–111. doi: 10.1002/1096-9861(20001204)428:1<79::aid-cne7>3.0.co;2-q. [DOI] [PubMed] [Google Scholar]
  43. Logothetis NK, Pauls J, Augath M, Trinath T, Oeltermann A. Neurophysiological investigation of the basis of the fMRI signal. Nature. 2001;412:150–157. doi: 10.1038/35084005. [DOI] [PubMed] [Google Scholar]
  44. Macaluso E, George N, Dolan R, Spence C, Driver J. Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage. 2004;21:725–732. doi: 10.1016/j.neuroimage.2003.09.049. [DOI] [PubMed] [Google Scholar]
  45. Miller LM, D’Esposito M. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. Journal of Neuroscience. 2005;25:5884–5893. doi: 10.1523/JNEUROSCI.0896-05.2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Murray MM, Molholm S, Michel CM, Heslenfeld DJ, Ritter W, Javitt DC, Schroeder CE, Foxe JJ. Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cerebral Cortex. 2005;15:963–974. doi: 10.1093/cercor/bhh197. [DOI] [PubMed] [Google Scholar]
  47. Nelissen K, Vanduffel W, Orban GA. Charting the lower superior temporal region, a new motion-sensitive region in monkey superior temporal sulcus. Journal of Neuroscience. 2006;26:5929–5947. doi: 10.1523/JNEUROSCI.0824-06.2006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Nichols T, Brett M, Andersson J, Wager T, Poline JB. Valid Conjunction Inference with the Minimum Statistic. Neuroimage. 2005;25:653–660. doi: 10.1016/j.neuroimage.2004.12.005. [DOI] [PubMed] [Google Scholar]
  49. Noesselt T, Rieger JW, Schoenfeld MA, Kanowski M, Hinrichs H, Heinze HJ, Driver J. Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices. Journal of Neuroscience. 2007;27:11431–11441. doi: 10.1523/JNEUROSCI.2252-07.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Padberg J, Seltzer B, Cusick CG. Architectonics and cortical connections of the upper bank of the superior temporal sulcus in the rhesus monkey: an analysis in the tangential plane. Journal of Comparative Neurology. 2003;467:418–434. doi: 10.1002/cne.10932. [DOI] [PubMed] [Google Scholar]
  51. Perrault TJ, Jr, Vaughan JW, Stein BE, Wallace MT. Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. Journal of Neurophysiology. 2005;93:2575–2586. doi: 10.1152/jn.00926.2004. [DOI] [PubMed] [Google Scholar]
  52. Price CJ. The anatomy of language: contributions from functional neuroimaging. Journal of Anatomy. 2000;197(Pt 3):335–359. doi: 10.1046/j.1469-7580.2000.19730335.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Raichle ME, MacLeod AM, Snyder AZ, Powers WJ, Gusnard DA, Shulman GL. A default mode of brain function. Proc Natl Acad Sci U S A. 2001;98:676–682. doi: 10.1073/pnas.98.2.676. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Raij T, Uutela K, Hari R. Audiovisual integration of letters in the human brain. Neuron. 2000;28:617–625. doi: 10.1016/s0896-6273(00)00138-0. [DOI] [PubMed] [Google Scholar]
  55. Saito DN, Yoshimura K, Kochiyama T, Okada T, Honda M, Sadato N. Cross-modal binding and activated attentional networks during audio-visual speech integration: a functional MRI study. Cerebral Cortex. 2005;15:1750–1760. doi: 10.1093/cercor/bhi052. [DOI] [PubMed] [Google Scholar]
  56. Saxe R, Brett M, Kanwisher N. Divide and conquer: a defense of functional localizers. Neuroimage. 2006;30:1088–1096. doi: 10.1016/j.neuroimage.2005.12.062. discussion 1097–1089. [DOI] [PubMed] [Google Scholar]
  57. Schroeder CE, Lakatos P, Kajikawa Y, Partan S, Puce A. Neuronal oscillations and visual amplification of speech. Trends Cogn Sci. 2008;12:106–113. doi: 10.1016/j.tics.2008.01.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Schroeder CE, Lindsley RW, Specht C, Marcovici A, Smiley JF, Javitt DC. Somatosensory input to auditory association cortex in the macaque monkey. Journal of Neurophysiology. 2001;85:1322–1327. doi: 10.1152/jn.2001.85.3.1322. [DOI] [PubMed] [Google Scholar]
  59. Sekiyama K, Kanno I, Miura S, Sugita Y. Auditory-visual speech perception examined by fMRI and PET. Neurosci Res. 2003;47:277–287. doi: 10.1016/s0168-0102(03)00214-1. [DOI] [PubMed] [Google Scholar]
  60. Seltzer B, Cola MG, Gutierrez C, Massee M, Weldon C, Cusick CG. Overlapping and nonoverlapping cortical projections to cortex of the superior temporal sulcus in the rhesus monkey: double anterograde tracer studies. Journal of Comparative Neurology. 1996;370:173–190. doi: 10.1002/(SICI)1096-9861(19960624)370:2<173::AID-CNE4>3.0.CO;2-#. [DOI] [PubMed] [Google Scholar]
  61. Simmons WK, Bellgowan PS, Martin A. Measuring selectivity in fMRI data. Nat Neurosci. 2007;10:4–5. doi: 10.1038/nn0107-4. [DOI] [PubMed] [Google Scholar]
  62. Smiley JF, Hackett TA, Ulbert I, Karmas G, Lakatos P, Javitt DC, Schroeder CE. Multisensory convergence in auditory cortex, I. Cortical connections of the caudal superior temporal plane in macaque monkeys. Journal of Comparative Neurology. 2007;502:894–923. doi: 10.1002/cne.21325. [DOI] [PubMed] [Google Scholar]
  63. Stein BE, Meredith MA. The Merging of the Senses. MIT Press; 1993. [Google Scholar]
  64. Van Atteveldt N, Formisano E, Goebel R, Blomert L. Integration of letters and speech sounds in the human brain. Neuron. 2004;43:271–282. doi: 10.1016/j.neuron.2004.06.025. [DOI] [PubMed] [Google Scholar]
  65. van Atteveldt NM, Formisano E, Blomert L, Goebel R. The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cerebral Cortex. 2007;17:962–974. doi: 10.1093/cercor/bhl007. [DOI] [PubMed] [Google Scholar]
  66. Wright TM, Pelphrey KA, Allison T, McKeown MJ, McCarthy G. Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cerebral Cortex. 2003;13:1034–1043. doi: 10.1093/cercor/13.10.1034. [DOI] [PubMed] [Google Scholar]
  67. Xiong JH, Gao JH, Lancaster JL, Fox PT. Clustered Pixels Analysis For Functional MRI Activation Studies Of the Human Brain. Human Brain Mapping. 1995;3:287–301. [Google Scholar]

RESOURCES