Neural Correlates of Auditory–Visual Stimulus Onset Asynchrony Detection

Abstract

Intersensory temporal synchrony is an ubiquitous sensory attribute that has proven to be critical for binding multisensory inputs, sometimes erroneously leading to dramatic perceptual illusions. However, little is known about how the brain detects temporal synchrony between multimodal sensory inputs. We used positron emission tomography to demonstrate that detecting auditory–visual stimulus onset asynchrony activates a large-scale neural network of insular, posterior parietal, prefrontal, and cerebellar areas with the highest and task-specific activity localized to the right insula. Interregional covariance analysis further showed significant task-related functional interactions between the insula, the posterior thalamus, and superior colliculus. Based on these results and the available electrophysiological and anatomical connectivity data in animals, we propose that the insula, via its known short-latency connections with the tectal system, mediates temporally defined auditory–visual interaction at an early stage of cortical processing permitting phenomena such as the ventriloquist and the McGurk illusions.

A fundamental brain function is to integrate information available to multiple sensory modalities producing a unified representation of the external world. Although multimodal sensory inputs from a single object or event normally coincide both in space and time, intersensory integration mechanisms seem to rely more critically on their temporal than spatial congruence (e.g., the ventriloquist effect) (Bertelson and Radeau, 1981). Indeed, the ability to detect and use temporal synchrony in associating multimodal sensory stimuli (e.g., sounds and visual events) has been demonstrated in human infants as young as 2 months (Lewkowicz, 1996,2000), and is believed to be operational at birth providing an innate capacity on which intermodal perception and associative learning are based (Spelke, 1987; Bahrick, 1992). Yet, the neural correlates for this basic process remain unknown. Using positron emission tomography (PET), we studied normal human subjects while performing a task requiring detection of auditory–visual stimulus onset asynchrony as well as a matched control condition. The PET experiment was designed for both paired image subtraction and correlational analysis methods. Brain regions specifically involved in temporal synchrony–asynchrony detection process were postulated where regional cerebral blood flow (rCBF) responses during task performance are significantly higher than during the control condition and appropriately modulated as a function of increasing task demand.

MATERIALS AND METHODS

Subjects. Twelve right-handed healthy volunteers (seven men, five women, ages 27–56 years) participated in behavioral and PET experiments after giving written informed consent.

Behavioral tasks. Subjects' ability to detect intermodal temporal mismatch between simple stationary auditory and visual stimuli was assessed in two separate auditory–visual (AV) and visual–auditory (VA) conditions. In both conditions, the number of trials with asynchronous stimuli was equal to that of trials with synchronous stimuli. In AV, asynchronous trials were presented with the sound preceding the visual stimulus, whereas in VA, the visual stimulus preceded the sound. Subjects pressed one of two buttons (with the right index or middle finger) to indicate whether tones and visual stimuli were perceived as being synchronous or asynchronous. Each subject performed 10 blocks in random order. Each block consisted of 50 trials: 25 trials with synchronous stimuli (zero AV delay) randomly intermixed with 25 trials with asynchronous stimuli (AV delay of 50, 75, 100, 150, or 200 msec or VA delay of 100, 150, 200, 250, or 300 msec between onsets of stimuli). Choice of these delays was based on the findings of previous behavioral studies (Dixon and Spitz, 1980; Lewkowicz, 1996). The visual stimulus consisted of a circle 4 cm in diameter, green or yellow in color (25 trials with each color, randomly intermixed), presented against a black background at the center of a computer monitor ∼50 cm from the subjects' eyes (2° visual angle). Auditory stimuli were 2000 Hz tones presented binaurally at 90 dB via well fitting headphones. Both visual and auditory stimuli were 100 msec in duration and were presented randomly every 2 or 3 sec to avoid anticipatory responses. Reaction times (measured from onset of second stimulus to onset of button press) >1500 or <150 msec were excluded from analysis.

Subjects were then studied with PET while performing AV and VA conditions. Three levels of difficulty per condition (AV1, 2, 3 and VA1, 2, 3) were established by varying the amount of intermodal delay while keeping other task components constant. Behavioral data from the initial six subjects were used to construct AV and VA delays applied during the PET experiment. Subjects performed AV1, AV2, and AV3 with delays of 242, 142, and 56 msec and VA1, VA2, and VA3 with delays of 300, 208, and 114 msec, respectively, predicted to correspond (within each condition) to 97, 80, and 70% correct responses, respectively. To control for brain regions activated by the sensorimotor response and by attention to visual and auditory stimuli, subjects also performed a control condition during which stimuli were identical to those of AV and VA but were always synchronous; subjects pressed one of the two buttons to indicate the color of visual stimuli. To maintain attention to sound during the control condition, subjects were instructed to respond only if the sound was present. Sound was omitted in approximately five trials of the control condition only during training and during 30 sec before radioisotope injection. Thus, during active scanning of the control condition, the number of auditory and visual stimuli was identical to that in AV and VA conditions. Control condition, AV 1, 2, 3, and VA 1, 2, and 3 were performed in random order.

PET methods. Scans were obtained in three-dimensional mode using a GE Advance PET tomograph (Waukesha, WI) with an axial field of view of 15.3 cm, covering the whole brain. Task performance began 30 sec before bolus infusion of 10 mCi of [H₂¹⁵O] (half life, 2.1 min) via a left cubital vein catheter. Scanning was started when a rising brain radioactivity count was first detected (∼20–30 sec after radioisotope injection) and continued for 60 sec thereafter. Arterial blood was not sampled, and the radioactive counts were used as a measure of rCBF. Interscan interval was 10 min. A transmission scan was obtained (with headphones in place) before each session and used to correct for radioactivity attenuation. Head movement was minimized by using a thermoplastic mask molded to each subject's head and attached to the scanner bed. Attenuation-corrected scans were reconstructed into 35 transaxial planes, 4.25 mm apart, with an in-plane center resolution of 6.5 mm full width at half maximum (FWHM) in each direction. SPM99b software (http://www.fil.ion.ucl.ac.uk/spm) was used for realignment, normalization to a standard stereotactic space (Montreal Neurological Institute brain template), and smoothing with an isotropic Gaussian filter of 12 mm to accommodate individual variability in gyral anatomy.

After correcting for variations in global blood flow (normalized to 50 ml/100 ml/min) using ANCOVA, differences between experimental conditions (12 subjects, three replications per condition) were statistically tested for each voxel (search volume was fromz = −50 to z = 80) using (SPM96) (Friston et al., 1995). The resulting whole brain statistical parametric maps (SPMs) based on the t statistic (transformed to normalized z scores) had a final spatial resolution ofx =10.4, y = 11.8, and z = 13.4 mm (FWHM). A statistical significance threshold of peak activity (Z > 3.09; p < 0.05 corrected for multiple comparisons) was used. For interregional covariance analysis, synchrony–asynchrony detection conditions (AV1, 2, 3 and VA1, 2, 3) were treated as one task, excluding control condition scans, thus measuring within-task, across-subject covariance with preselected reference regions (Horwitz, 1991).

RESULTS

Behavioral tasks

As expected, the longer the delay between visual and auditory stimuli the more readily their temporal synchrony–asynchrony was detected (Fig. 1). Similar thresholds were obtained in previous behavioral studies using various audiovisual bimodal presentations (e.g., speech, colliding, or moving–sounding objects) (Dixon and Spitz, 1980; Lewkowicz, 1996), indicating that, regardless of stimulus characteristics, humans are less sensitive to temporal mismatch between heteromodal than between same-modality auditory or visual stimuli (for review, see Nichelli, 1993). Subjects were faster and more accurate in AV than VA conditions (Fig. 1), a finding consistent with previous studies showing that humans characteristically tolerate greater temporal discrepancies between visual and auditory events when sight precedes sound than when the sound occurs first (Dixon and Spitz, 1980; Lewkowicz, 1996).

Fig. 1.

Subjects' performance while detecting auditory–visual synchrony–asynchrony at different intermodal delays. Means (± SEM) of percentage of correct responses (a) and response times (b) of 12 subjects. AV (open circles), Sound leading; VA (solid circles), light leading. Percentage of correct responses significantly increased (F(4,11) = 45.4;p < 0.0001 and 61.7; p < 0.0001) while response time decreased (F(4,11) = 9.9; p < 0.0034 and 18.7; p < 0.0001), as a function of intermodal delay for AV and VA, respectively. At delays of 100, 150, and 200 msec, subjects were faster and more accurate in AV than VA conditions (t test,p < 0.006).

During PET, percentage of correct responses and reaction times were: 96.8 ± 0.7 and 467.1 ± 26.0 for control condition; 91.8 ± 2.3 and 423.3 ± 28.9 for AV1; 84.0 ± 3.2 and 470.0 ± 32.5 for AV2; 69.8 ± 2.9 and 530.5 ± 38.0 for AV3; 87.7 ± 2.8 and 427.8 ± 33.8 for VA1; 81.0 ± 2.8 and 491.3 ± 41.6 for VA2; 67.9 ± 3.9 and 545.0 ± 38.1 for VA3; respectively.

PET results

Using voxel-based image subtraction, AV and VA were compared with the control condition (C) yielding two contrasts: (AV − C) and (VA − C), where AV and VA are the average of rCBF responses to AV1, 2, and 3 and VA1, 2, and 3, respectively (Table1). With either contrast, no significant activation was detected in the sensorimotor, occipital, or temporal regions (p > 0.05 uncorrected), indicating that responses to sensorimotor, visual, and auditory stimulation were adequately controlled for. AV − C contrast showed significant activation of the right anterior insular cortex, right ventrolateral prefrontal cortex, right inferior parietal lobule, and left cerebellar hemisphere (Table 1). Highly similar activation clusters were obtained by the contrast (VA − C) (Table 1). Both contrasts also activated a homologous area in the left insular region; however, this activation was below statistical significance. Figure2 shows areas significantly activated in common to both (AV − C) and (VA − C) without significant interaction between the contrasts (p > 0.05), i.e., independent of whether the sound or visual stimulus occurred first.

Table 1.

Brain regions activated during auditory–visual synchrony–asynchrony detection (AV and VA) relative to control condition (C)

Brain region	x, y, z(mm)	Z score
AV-C
Right insula	36, 24, −4	5.72
Right inferior frontal gyrus	48, 36, 20	5.58
Right inferior parietal lobule	44, −58, 54	4.78
Left cerebellum	−52, −64, −36	5.60
VA-C
Right insula	38, 22, −6	5.98
Right inferior frontal gyrus	44, 32, 18	4.89
Left inferior frontal gyrus	−36, 50, 2	4.78
Right inferior parietal lobule	48, −52, 44	5.07
Left cerebellum	−28, −58, −46	4.96

Z scores correspond to voxels of peak activity.x, y, z, Stereotaxic coordinates referring to the atlas of Talairach and Tournoux.

Fig. 2.

Brain regions activated in common to both auditory–visual and visual–auditory synchrony–asynchrony detection conditions. Statistical parametric maps were superimposed on axial views of a normalized representative subject's brain.a, Right inferior parietal lobule (46, −54, 48;z = 5.15; p < 0.006);b, right ventrolateral prefrontal cortex (48, 34, 18;z = 5.60; p < 0.001);c, right anterior insular cortex (36, 24, −4;z = 6.57; p < 0.0001); andd, left cerebellar hemisphere (−28, −58, −48;z = 5.43; p < 0.002).p values corrected for multiple comparisons.

We then applied regression analysis to identify voxels with rCBF responses that positively correlate with increasing task demand, i.e., decreasing intermodal delay. These were found only in the right insular region with the voxel of maximal correlation lying precisely within the activation cluster shown independently by subtraction analysis (Fig.3). Because only three levels of task difficulty were examined for linear rCBF correlation, this finding does not exclude task-related response modulation in other brain regions. However, the results of this analysis preclude any interpretation that insular response shown in subtraction analysis is caused by “de-activation” induced by some component of the control condition. The location of peak voxel in both analyses was within the stereotaxic coordinates of the insula as identified in previous PET and functional magnetic resonance imaging studies (Calvert et al., 1997; Buchel et al., 1998).

Fig. 3.

Statistical parametric map (thresholded atz > 4.63; p < 0.05 corrected for multiple comparisons) showing voxels with significant incremental rCBF response to increasing task demand superimposed on sagittal (a) and axial (b) views of a normalized representative subject's brain. Voxel with highest covariance: x = 38, y = 24, z = −4; z = 6.42; p < 0.0001.

Finally, we further examined nonsubtracted rCBF data for interregional covariance as an estimate of functional connectivity between brain regions activated during synchrony–asynchrony detection conditions (Horwitz, 1991). Normalized rCBF values (72 scans) at peak insular, prefrontal, posterior parietal, and cerebellar activation clusters (Fig. 2) were used as covariates of interest. Positive rCBF correlations with the right insula were found in the posterior midbrain (in the region of the superior colliculus), right posterior thalamus, right precuneus, right prefrontal cortex, and left insula (Fig.4). Areas with significant rCBF correlation with the right prefrontal, posterior parietal, and left cerebellar areas are listed in Table 2. We specifically examined the occipital and superior temporal regions for rCBF correlations with the right insula, right prefrontal, posterior parietal, and left cerebellar areas, but no significant correlations were found (Fig. 4, Table 2).

Fig. 4.

Brain regions with significant functional interactions with the right insula during synchrony–asynchrony detection. Correlations are displayed as statistical maps (z > 3.09) superimposed on sagittal views of a normalized representative subject's brain. Coordinates (x, y, z) of voxels of maximal positive rCBF correlation with the reference voxel (36, 24, −4): a, left insula (−36, 12, 0); b, posterior midbrain (in the region of the superior colliculus; −2, −28, −12); c, right precuneus (12, −80, 48); d, right posterior thalamus (18, −22, 8); and e, right prefrontal cortex (32, 48, 20).

Table 2.

Brain areas with positive rCBF correlation during synchrony–asynchrony detection task

Brain region (Brodmann's area)	x, y, z(mm)	r
Right insula (x 36,y 24, z −4)
Left insula	−36, 12, 0	0.33*
Posterior midbrain	−2, −28,  −12	0.49
Right precuneus	12, −80, 48	0.35*
Right posterior thalamus	18, −22, 8	0.42*
Right prefrontal cortex	32, 48, 20	0.51
Right inferior frontal gyrus (48, 34, 18)
Right orbital gyrus (BA11)	22, 20, −30	0.29*
Right medial prefrontal cortex (BA8/32)	2, 24, 48	0.29*
Left insula	−36, 24, −2	0.41*
Inferior parietal lobule (BA 40)	52, −52, 48	0.69
Left cerebellum	44, −74, −34	0.62
Right inferior parietal lobule (46, −54, 48)
Right inferior frontal gyrus	36, 58, 2	0.62
Middle frontal gyrus	50, 14, 36	0.68
Left cerebellum (−28, −58, −48)
Left inferior frontal gyrus	28, 62, −4	0.45
Right orbital gyrus	40, 68, −30	0.44
Left insula	−30, 16, −8	0.34*
Left sensori-motor cortex	−28, −40, 38	0.242-160

Reference voxels are peak activity voxels in right insular, right prefrontal, right parietal, and left cerebellar regions shown by subtraction analysis (Fig. 2).

x, y, z stereotaxic coordinates of voxels with maximal correlation; r, Pearson's correlation coefficient.

F2-160: p < 0.01,

^*p < 0.005, other, <0.0001 (p values (Fisher'sr to z).

DISCUSSION

The results demonstrate that visual–auditory temporal synchrony–asynchrony detection predominantly activates a large-scale neural network of multisensory cortical areas in the insular, posterior parietal, and prefrontal regions. Both cognitive subtraction and task-demand correlational analyses concur that the right insula is more actively involved in this process than the classical association cortices in the prefrontal and posterior parietal regions. This function has not been previously assigned to the human insula and would have not been readily predicted given the traditionally proposed insular functions (Penfield and Faulk, 1955; Mesulam and Mufson, 1984;Augustine, 1996). These have been derived mainly from brain lesion and epilepsy studies that have implicated the insula in autonomic, visceral, somatosensory, vestibular, smell, taste, and language processing (Penfield and Faulk, 1955; Mesulam and Mufson, 1984;Augustine, 1996). Similarly, functional neuroimaging studies showed activation of the insula under a wide variety of behavioral conditions and mostly emphasized its role in visceral and limbic functions. These included pain perception, anticipatory anxiety, reflex conditioning, and associative learning (Buchel et al., 1998; Fischer et al., 1998;Ploghaus et al., 1999). Although many of these studies applied behavioral tasks requiring association of visual and auditory information and in that respect, generally agree with our findings, none of previous functional imaging studies specifically investigated intersensory temporal synchrony as the basis for insula activity (Calvert et al., 1997; Buchel et al., 1998).

Interregional covariance analysis was complementary to conventional cognitive subtraction showing task-related functional interactions between the insular, the dorsolateral prefrontal, and posterior parietal areas. In addition, the posterior thalamus and superior colliculus also showed significant functional interaction with the right insula; the region with the highest and task-specific activity, indicating the involvement of the tectal system in auditory–visual temporal synchrony–asynchrony detection (Fig. 4). Electrophysiological and anatomical connectivity studies in primates and other mammals indicate that the insular cortex may receive auditory and/or visual inputs via multiple parallel pathways from the auditory cortex, the temporal, prefrontal, and posterior parietal multisensory regions as well as via subcortical projections from the superior colliculus through the posterior group of thalamic nuclei (magnocelluar medial geniculate and suprageniculate nuclei) (Graybiel, 1973; Guldin and Markowitsch, 1984; Hicks et al., 1988) (Mesulam and Mufson, 1984). Compatible with the direct tecto-thalamo-insular pathways are the short insular neuronal latencies reported for visual and auditory stimulation in electrophysiological studies (Sudakov et al., 1971; Benvento and Loe, 1975; Hicks et al., 1988). Similar subcortical pathways from the superior colliculus via the posterior thalamus are known to project to the amygdala and are believed to mediate limbic processes such as reflex conditioning to subconsciously perceived visual and auditory inputs (Morris et al., 1999). Based on our results and the available anatomical and electrophysiological data from animal studies, we can specifically propose that intersensory temporal processing is similarly mediated via subcortical tecto-thalamo-insular pathways. These would allow detection of temporal correspondence between visual and auditory inputs and their interaction at an early level of cortical processing, perhaps in parallel with the primary sensory and parasensory association cortices. In support of this hypothesis, are the results of single unit studies showing that both insular and superior collicular neuronal responses to combinations of visual and auditory stimuli are characteristically modulated by the amount of intermodal temporal delay (Loe and Benevento, 1969; Benevento et al., 1977; Meredith et al., 1987).

At the behavioral level, early interaction of visual and auditory inputs has been proposed by several studies to explain two perceptual illusions (the McGurk and ventriloquist effects) that result from binding of conflicting multimodal information based on their temporal synchrony (McGurk and MacDonald, 1976; Driver, 1996). The McGurk illusion occurs when a sound of a syllable (e.g., “ba”) is temporally synchronized with lip movements silently uttering a different syllable (e.g., “ga”) producing the perception of another syllable (typically “da”). According to the early (or prelabeling) integration models of audiovisual speech perception, the McGurk effect is evidence that voice and lip-movement cues are combined before the unimodal acoustic and visual information are assigned to a phoneme or word category (McGurk and MacDonald, 1976; Schwartz et al., 1998). A similar mechanism has also been proposed to explain the ventriloquist effect in humans [and a similar phenomenon in animals, e.g., cats respond to temporally synchronous but spatially disparate auditory and visual stimuli by moving to a position halfway between the two sources (Stein and Meredith, 1993)] in which audiovisual integration is believed to occur early in the course of cortical processing, before the process of spatial selective attention is complete (Driver, 1996).