The roles of physical and physiological simultaneity in audiovisual multisensory facilitation

Abstract

A series of experiments measured the audiovisual stimulus onset asynchrony (SOA_AV), yielding facilitative multisensory integration. We evaluated (1) the range of SOA_AV over which facilitation occurred when unisensory stimuli were weak; (2) whether the range of SOA_AV producing facilitation supported the hypothesis that physiological simultaneity of unisensory activity governs multisensory facilitation; and (3) whether AV multisensory facilitation depended on relative stimulus intensity. We compared response-time distributions to unisensory auditory (A) and visual (V) stimuli with those to AV stimuli over a wide range (300 and 20 ms increments) of SOA_AV, across four conditions of varying stimulus intensity. In condition 1, the intensity of unisensory stimuli was adjusted such that d′ ≈ 2. In condition 2, V stimulus intensity was increased (d′ > 4), while A stimulus intensity was as in condition 1. In condition 3, A stimulus intensity was increased (d′ > 4) while V stimulus intensity was as in condition 1. In condition 4, both A and V stimulus intensities were increased to clearly suprathreshold levels (d′ > 4). Across all conditions of stimulus intensity, significant multisensory facilitation occurred exclusively for simultaneously presented A and V stimuli. In addition, facilitation increased as stimulus intensity increased, in disagreement with inverse effectiveness. These results indicate that the requirements for facilitative multisensory integration include both physical and physiological simultaneity.

1. Introduction

1.1. Multisensory integration

Meredith (2002) identified two classes of multisensory convergence: areal and neuronal. Areal convergence occurs when unisensory neurons from different modalities merely coexist within a brain region but do not interact. Neuronal convergence occurs when unisensory neurons from two or more modalities make synaptic contact onto recipient “multisensory” neurons. Multisensory integration occurs when the response of multisensory neurons to convergent unisensory input differs qualitatively and quantitatively from that elicited by the individual unisensory inputs alone (Calvert, 2001).

1.2. The redundant signals effect

A well-known behavioral manifestation of facilitative multisensory integration is the decrease in response time (RT) to pairings of unisensory stimuli presented over multiple sensory channels, where RT to the multisensory combination is faster than to either unisensory signal alone. This enhancement in the speed of processing has been termed the “redundant signals effect,” or RSE (Miller, 1982). The RSE is not an exclusively multisensory phenomenon because it also occurs when the redundant signals occur within a single sensory modality (Iacoboni & Zaidel, 2003; Miller, 1986; Miniussi, Girelli, & Marzi, 1998; Molholm et al., 2002; Mordkoff & Yantis, 1991; Murray, Foxe, Higgins, Javitt, & Schroeder, 2001; Supek et al., 1999), but this paradigm has been widely employed in multisensory research.

Miller (1982) compared two models that could potentially account for the RSE: separate activation or “race” models and neural coactivation models. Race models assume that each unisensory (redundant) signal is processed independently, such that on each trial the channel that processes the stimulus most quickly, and thus initiates the overt response, “wins” the race. According to race models, redundancy gains can result solely from statistical probability summation and no neural interaction between the activated sensory channels is required (although such interactions are not ruled out). In contrast, neural coactivation models posit that signals from each unisensory channel interact in order to initiate the response. Coactivation across the two channels accumulates until a response criterion is reached, which can occur before the same criterion is reached by activation within either individual channel.

Miller (1982) derived a mathematical inequality describing the race model that specifies an upper limit on the cumulative probability (CP) of obtaining speeded RT to redundant stimuli. Miller's inequality asserts that the CP of obtaining the fastest responses to redundant signals must be less than or equal to the CP of obtaining the fastest responses to individual stimuli. Specifically, the race model states that for pairs of stimuli, i.e., auditory (A) and visual (V), at a given AV stimulus onset asynchrony (SOA_AV), at a given response latency (t), that

$${\displaystyle \mathrm{CP}(\mathrm{RT}<t|\mathrm{AV})\le \mathrm{CP}(\mathrm{RT}<t-{\mathrm{SOA}}_{\mathrm{AV}}|\mathrm{A})+\mathrm{CP}(\mathrm{RT}<t|\mathrm{V}),}$$ (1)

where A is delayed relative to V, and

$${\displaystyle \mathrm{CP}(\mathrm{RT}<t|\mathrm{AV})\le \mathrm{CP}(\mathrm{RT}<t|\mathrm{A})+\mathrm{CP}(\mathrm{RT}<t-{\mathrm{SOA}}_{\mathrm{AV}}|\mathrm{V}),}$$ (2)

where V is delayed relative to A. CP(RT < t|AV) is the CP of obtaining an RT faster than time (t) in response to the presentation of the A and V stimuli. This CP must be less than or equal to the sum of the CPs of obtaining RTs faster than time in response to the individual unisensory stimuli, CP(RT < t − SOA_AV | A) + CP(RT < T | V) or CP(RT < t | A) + CP(RT < t − SOA_AV | V). Violations of Miller's inequality signify that probability summation cannot account for decreased RTs in response to redundant signals and implies that neural coactivation has occurred. The evidence that neural coactivation is a causal mechanism for the AV RSE comes from numerous behavioral and electrophysiological studies (Giard & Peronnet, 1999; Miller, 1982, 1986; Molholm et al., 2002, 2006; see, however, Otto & Mamassian, 2012, for a challenge to this standard interpretation).

1.3. The “rules” of multisensory integration: Spatial, temporal, and intensive

Unisensory stimuli that are closely aligned in space (the “spatial rule”) and time (the “temporal rule”) are more likely to produce multisensory response facilitation than are stimuli that are temporally and/or spatially disparate, where the latter may even result in response suppression (Holmes & Spence, 2005; Meredith & Allman, 2009; Meredith & Stein, 1986; Meredith, Nemitz, & Stein, 1987). Response facilitation may be especially robust for weak (i.e. near threshold) unisensory stimuli (the “inverse effectiveness rule”). Inverse effectiveness is most conspicuous in the responses of multisensory neurons in the superior colliculus (Meredith & Stein, 1986).

1.3.1. The inverse effectiveness rule

Results from both behavioral and electrophysiological studies have been interpreted to support the inverse effectiveness rule (Callan, Callan, Kroos, & Vatikiotis-Bateson, 2001; Diederich & Colonius, 2004; Frassinetti, Bolognini, & Làdavas, 2002; Lakatos, Chen, O'Connell, Mills, & Schroeder, 2007; Serino, Farnè, Rinaldesi, Haggard, & Làdavas, 2007; Senkowski, Saint-Amour, Höfle, & Foxe, 2011). The generality of the inverse effectiveness rule has recently been questioned on the grounds that previous studies sampled an insufficient or incorrect range of stimulus intensities to critically test this hypothesis (Holmes, 2007, 2009; Leavitt, Javitt, & Foxe, 2007; Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007). Studies employing a range of stimulus intensities, whose results have been described as “generally consistent” with the inverse effectiveness rule (Lakatos et al., 2007), report maximal multisensory facilitation at intermediate (not the lowest) levels of stimulus intensity. Other studies (Senkowski et al., 2011), while testing a wide range of stimulus intensities, may still not have sampled at the lowest end of the perceptible intensity continuum.

1.3.2. The temporal rule: Physiological simultaneity

Stimuli are intrinsically processed at different rates by different sensory modalities. For example, human auditory evoked cortical potentials onset as early as 10 ms (Celesia & Puletti, 1971) whereas the C1 component of the visual evoked potential, which reflects activity in early retinotopically mapped visual areas (V1/V2), onsets much later at 45–60 ms (Clark & Hillyard, 1996; Foxe & Simpson, 2002; Foxe et al., 2008; Jeffreys & Axford, 1972; Murray et al., 2001). In addition to processing speed differences between sensory channels due to intrinsic factors such as unequal pathway length, dissimilar axonal conduction velocity and/or variations in synaptic complexity, there is the variable influence of stimulus intensity. Simple RT is inversely related to stimulus intensity, a phenomenon known as Piéron's law (Jas´kowski, 1985; Mansfield, 1973; Piéron, 1952; Prestrude, 1971; Roufs, 1963), and intensity-dependent variations in the latency of neural activity occur in the primate visual system (Barlow, Snodderly, & Swadlow, 1978; Maunsell et al., 1999) and in the auditory system of both cat (Eggermont, 1998; Phillips, 1998) and human (Stufflebeam, Poeppel, Rowley, & Roberts, 1998).

There is general agreement that multisensory facilitation requires that the neural activity evoked by unisensory stimuli converge synchronously onto multisensory coincidence detectors—that is, that the unisensory inputs must exhibit physiological simultaneity (Hershenson, 1962; Miller, 1986; Raab, 1962). A direct example is that multisensory neurons in the cat's superior colliculus exhibit optimal multisensory integration when the activity elicited by unisensory stimulation occurs at roughly the same post-stimulus latency (Stein & Meredith, 1993). Likewise, human behavioral data indirectly suggest that multisensory facilitation of simple RT occurs at stimulus onset asynchronies (SOAs) corresponding to the difference in simple RT to the unisensory stimuli, where this RT difference is presumed to reflect the difference in intrinsic processing speed (Diederich & Colonius, 2004; Hershenson, 1962; Miller, 1986). If optimal multisensory facilitation requires the synchronous convergence of unisensory signals onto a multisensory coincidence detector, and if the post-stimulus latency of evoked activity depends on stimulus intensity, then changes in the relative intensities of the component unisensory visual and auditory stimuli should cause systematic changes in the SOA_AV at which optimal multisensory facilitation (neural coactivation) occurs.

1.4. The present experiment

We measure how the optimal SOA_AV for multisensory integration (as indexed by violations of the race model) depends on both absolute and relative unisensory stimulus intensities. An evaluation of the dependence of optimal SOA_AV on absolute stimulus intensity addresses the validity and generality of the inverse effectiveness rule, while its dependence on relative stimulus intensity tests the predictions of the strong version of the physiological simultaneity hypothesis.

2. Condition 1

Condition 1 measured the range of SOA_AV over which neural coactivation occurs for two relatively weak unisensory stimuli.

2.1. Method

2.1.1. Participants

Participants (n = 4; two male; mean age = 31 years) possessed normal (or corrected to normal) vision and normal hearing. All experiments were conducted in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki) for experiments involving humans. Prior to their participation in the study, all participants provided written informed consent. All procedures were approved by the institutional review board of North Dakota State University.

2.1.2. Stimuli and apparatus

Visual stimuli were circular Gabor patches which, when viewed from 114 cm, possessed a spatial frequency of 1 cycles/degree and a Gaussian envelope with a standard deviation of 1°. Gabor patches were centered at 2.25° eccentricity from fixation in the upper left visual quadrant (Figure 1). Stimuli were presented on a CRT (mean luminance = 60 cd/m²; monitor refresh rate = 100 Hz). Gabor contrast for individual participants ranged between 1% and 4% across all conditions (see individual conditions for details). For condition 1, Gabor contrast ranged from 1.0% to 1.7%. Visual stimulus duration was 100 ms.

Figure 1.

Visual stimuli were circular Gabor patches (1%–4% contrast) which possessed a spatial frequency of 1 cycle/degree and a Gaussian envelope with a standard deviation of 1°. Gabor patches were centered at 2.25° eccentricity from fixation (cross) in the upper left visual quadrant. Scale bar = 1°.

The auditory stimulus was a 1-KHz pure tone of variable loudness (range = 31.1–49.0 dB_A) presented via a speaker approximately co-located with the visual stimuli. Auditory stimulus duration was 100 ms.

2.1.3. Procedure

2.1.3.1. Pretest.

Prior to the experiment, all participants completed a pretest designed to equate the detectability of the unisensory stimuli. The pretest paradigm was a single-interval go/no-go signal detection task. Participants responded via button press as quickly and accurately as possible to the detection of any A or V stimulus. Participants performed 15 blocks of trials for a total of 30 trials per stimulus condition. Each block consisted of a total of 50 trials: 24 unisensory A stimuli (2 × 12 levels of dB attenuation), 24 unisensory V stimuli (2 × 12 contrasts), and 2 catch (no-signal) trials.

Sensitivity (d′) was calculated according to the equation d′ = Z_H − Z_FA, where Z_H denotes the Z-transformed hit rate (hits/signal trials) and Z_FA denotes the Z-transformed false-alarm rate (false alarms/no-signal trials). Nonlinear least-squares regression to a logistic function interpolated stimulus intensities yielding criterion performance (d′ ≈ 2).

2.1.3.2. Experiment.

A single-interval go/no-go signal detection task was employed. Trials commenced with the appearance of a fixation cross. After a variable interval (1,000–2,000 ms), the first stimulus (S1) was presented. The second stimulus (S2) was presented following a variable SOA. Participants responded via button press as quickly and accurately as possible to the detection of any stimulus. RT was recorded to the nearest millisecond. Trials terminated after subject response or after 1,500 ms. Figure 2 illustrates the sequence of events in multisensory trials.

Figure 2.

Multisensory trials commenced with the appearance of a fixation cross. After a variable interval (1,000–2,000 ms) the first stimulus (S1) was presented. The second stimulus (S2) was presented following a variable SOA. Participants responded via button press as quickly and accurately as possible to the detection of any stimulus. Response time (RT) was recorded to the nearest millisecond. Trials terminated after subject response or after 1,500 ms.

Participants completed a total of 34 blocks of 75 trials each, for a grand total of 2,550 trials. Each block included 9 V trials, 9 A trials, 9 no-stimulus catch trials, and 3 AV trials at each of 16 SOAs ranging (in 20-ms increments) from − 100 ms (A→V) to 200 ms (V→A). Each subject's RT distribution was trimmed to exclude outliers (e.g., 100 ms > RT > 1,000 ms). The trimmed RT distributions possessed nearly equal numbers of trials contributed by each subject and subsequent analyses in all experimental conditions were conducted on RT distributions pooled across participants.

The intensities of the unisensory stimuli were adjusted (as necessary) after every 18 trials to ensure a mean sensitivity of d′ ≈ 2, which corresponds to an 87% correct response rate in a two-alternative forced-choice task, where 75% correct is typically taken as threshold performance. Thus, stimuli in condition 1 were sufficiently strong that participants could reliably detect them, while sufficiently weak that performance in this condition could be meaningfully contrasted to that observed in subsequent experimental conditions where stimuli were highly suprathreshold (d′ > 4; 99.9% correct response rate).

2.1.4. Data analysis

2.1.4.1. Multisensory response facilitation/inhibition.

To test whether RTs to multisensory stimuli were significantly faster (or slower) than RTs to unisensory stimuli, mean RT in all 16 SOA_AV conditions was compared with the fastest (and slowest) mean unisensory RTs (A or V) using independent samples t-tests.

2.1.4.2. Multisensory sensitivity enhancement.

Mean sensitivity (d′) at each SOA_AV was calculated using a bootstrapping procedure (Foster & Bischof, 1991). Response distributions for each SOA_AV condition (hits) were combined with an equal size sample taken at random from the no-signal condition (false alarms). The distribution of hits and false alarms was exhaustively sampled (with replacement, 1,000 iterations) to generate sampling distributions of d′ from which means and standard errors were obtained.

2.1.4.3. Neural coactivation (Miller's inequality analysis).

RT data were trimmed (100 ms > RT > 1,000 ms) and cumulative distribution functions (CDFs) were created for each stimulus condition (A, V, AV). Sampling distributions were bootstrapped by exhaustively resampling each CDF (with replacement, 1,000 iterations). A sampling distribution of Miller's inequality values was generated by subtracting the multisensory CDF predicted by probability summation, AV_P = (A + V), from the CDF observed for each multisensory condition at each iteration such that MI_f = AV − AV_P, where MI_f indicates facilitative MI. This process yielded a mean value (and standard error) of Miller's inequality for each SOA_AV. Figures 8(c) and (d), condition 4, provide a graphic illustration of CDF comparisons between unisensory and simultaneous multisensory conditions.

Figure 8.

Test of the race model for condition 4. Panel (a) shows spectrum-coded mean values of Miller's inequality as a function of RT and SOA_AV. Panel (b) shows a magnified view of the region exhibiting significant (p < 0.05) violations of the race model, which occurred at SOA_AV values of −20 and 0 ms. Panel (c) plots the mean value (thick line) and 95% confidence intervals (thin lines) for Miller's inequality as a function of RT for simultaneous SOA_AV. Panel (c) shows the CP distributions for the two unisensory conditions (A: purple; V: blue) and their sum (A + V: black), which is the CP distribution predicted by the race model. The CP distribution observed for AV stimulation at an SOA_AV of 0 ms (AV_o) is shown in red. Panel (d) plots the mean value (thick line) and 95% confidence intervals (thin lines) for Miller's inequality as a function of RT for simultaneous SOA_AV.

2.2. Results and discussion

2.2.1. Multisensory response facilitation/inhibition

Mean RTs for condition 1 are plotted as a function of SOA_AV in Figure 3 (a). While possessing equally detectable stimulus intensities (d′ ≈ 2), mean RT to the unisensory A stimulus was nevertheless faster (424.6 ms) than to the unisensory V stimulus (431.9 ms). This difference in RT did not reach significance (t₁₈₃₁ = − 1.206, p = 0.228) and likely reflects the shorter latency of auditory versus visual cortical responses (Celesia & Puletti, 1971; Jeffreys & Axford, 1972). To assess whether multisensory stimulation produced response facilitation, mean RT in each AV condition was compared with that in the fastest unisensory condition (A). Green bars identify those AV SOAs where results of independent samples t-tests indicated that significant multisensory response enhancement occurred. Significant response facilitation occurred at SOA_AV values of − 60, − 40, − 20, 0, 20, and 40 ms (t₁₃₀₃ = − 2.891, p = 0.004; t₁₃₀₃ = − 2.944, p = 0.003; t₁₂₉₅ = − 3.560, p < 0.001; t₁₂₈₈ = − 4.620, p < 0.001; t₁₃₀₃ = − 2.933, p = 0.003; and t₁₂₉₀ = − 1.957, p = 0.051; respectively). To assess whether multisensory stimulation might produce response inhibition, mean RT in AV conditions was compared with that in the slowest unisensory condition (V). Red bars indicate that significant response inhibition occurred at SOA_AV of 140, 160, 180, and 200 ms (t₁₂₉₃ = 2.712, p = 0.007; t₁₂₉₈ = 2.626, p = 0.009; t₁₂₉₅ = 2.141, p = 0.032; and t₁₂₉₈ = 4.100, p < 0.001; respectively).

Figure 3.

Mean RTs in the four experimental conditions plotted as a function of SOA_AV. Mean RT in each AV condition was compared with that in the fastest unisensory condition (A or V). Green bars identify those AV SOAs where results of independent samples t-tests indicated that significant multisensory response enhancement occurred. Mean RT in each AV condition was also compared with that in the slowest unisensory condition (A or V). Red bars identify those AV SOAs for which significant multisensory response inhibition occurred.

2.2.2. Multisensory sensitivity enhancement

If sensitivities to the unisensory A and V stimuli combine probabilistically then the sensitivity in multisensory (AV) conditions (d′_AVp) should equal the quadratic sum of the unisensory sensitivities (Campbell & Green, 1965; Legge, 1984):

$${\displaystyle d_{\mathrm{AVp}}^{\prime }=\sqrt{{(d_{\mathrm{Ao}}^{\prime })}^2+{(d_{\mathrm{Vo}}^{\prime })}^2}}$$ (3)

.

Figure 4 plots the mean bootstrapped values of observed sensitivity (d′_AVo, ±2 and ±3 SEM) in all 16 multisensory conditions as a function of SOA_AV. The average observed unisensory sensitivities were d′_Ao = 1.911, and d′_Vo = 1.910, and the multisensory sensitivity predicted by probability summation (d′_AVp = 2.699) is plotted as a horizontal line. Although there is a significant multisensory facilitation of mean RT, sensitivity to multisensory stimuli does not differ significantly (p > 0.01) from that predicted by probability summation at any SOA_AV.

Figure 4.

Mean bootstrapped values of observed sensitivity (d′_AVo, ±2 and ±3 SEM) in all 16 multisensory conditions as a function of SOA_AV. The average observed unisensory sensitivities were d′_Ao = 1.911 and d′_Vo = 1.910, and the multisensory sensitivity predicted by probability summation (d′_AVp = 2.699) is plotted as a horizontal line. Although there is a significant multisensory facilitation of mean RT, sensitivity to multisensory stimuli does not differ significantly (p > 0.01) from that predicted by probability summation at any SOA_AV.

2.2.3. Neural coactivation (Miller's inequality analysis)

Figure 5(a) spectrum codes the value of Miller's inequality as a function of RT and SOA_AV. Figure 5 (b) is a magnified view of the region exhibiting significant (p < 0.05) violations of the race model (positive values). Noteworthy is that positive values of the inequality occur exclusively at an SOA_AV value of 0 ms (physical simultaneity), across a range of RT (200–325 ms) with a peak value (0.0360) occurring at 292 ms (Figure 5c).

Figure 5.

Test of the race model for condition 1. Panel (a) shows spectrum-coded mean values of Miller's inequality as a function of RT and SOA_AV. Panel (b) shows a magnified view of the region exhibiting a significant (p < 0.05) violation of the race model that occurred exclusively at an SOA_AV of 0 ms. Panel (c) plots the mean value (thick line) and 95% confidence intervals (thin lines) for Miller's inequality as a function of RT for simultaneous SOA_AV.

3. Condition 2

Condition 2 was designed to reveal how changing the intensity of the V stimulus might influence its integration with the relatively weak A stimulus. Specifically, we tested whether increasing the contrast of the V stimulus caused a change in the optimal SOA_AV for multisensory integration relative to condition 1. If physiological simultaneity determines multisensory facilitation, then increasing the intensity of the V stimulus (which increases the speed of visual processing and reduces mean RT to V stimuli) should shift the optimal SOA_AV to more negative values such that, relative to condition 1, the A stimulus will now need to be presented earlier in time with respect to the V stimulus. Conversely, the optimal SOA_AV will remain unchanged if physical simultaneity is critical for multisensory integration.

3.1. Method

The participants in condition 2 were the same as in condition 1. The experimental methods and procedures were as in condition 1 except that V stimulus contrast was increased to levels producing a criterion sensitivity of d′ > 4 for each participant (contrast range = 3.4%–4.0%). To ensure this criterion sensitivity, d′ was calculated for unisensory A and V stimuli after every 18 trials throughout the experiment and V stimulus contrast and A stimulus intensity were adjusted as necessary.

3.2. Results and discussion

3.2.1. Multisensory response facilitation/inhibition

Mean RTs in condition 2 are plotted as a function of SOA_AV in Figure 3 (b). As expected, mean RT to the unisensory V stimulus was now significantly faster (315.8 ms) than to the unisensory A stimulus (450.0 ms) (t₁₉₈₄ = − 25.158, p < 0.001). This highly significant reversal in the relative speed of responses to A and V stimuli reflects the approximately threefold increase in V stimulus contrast relative to condition 1, and is an illustration of Piéron's law. To assess whether multisensory stimulation produced response facilitation, mean RT in AV conditions was compared with that in the fastest unisensory condition (V). The green bars indicate that significant response facilitation occurred at SOA_AV values of 0 and 20 ms (t₁₅₉₇ = − 2.272, p = 0.023 and t₁₅₉₇ = − 2.160, p = 0.031, respectively). There was no multisensory response inhibition at any SOA_AV.

3.2.2. Neural coactivation (Miller's inequality analysis)

Figure 6(a) spectrum codes the values of Miller's inequality as a function of RT and SOA_AV for condition 2. A magnified view of the region exhibiting significant (p < 0.05) violations of the race model is shown in Figure 6 (b). As in condition 1, significant violations of Miller's inequality occurred for physically simultaneous A and V stimuli, as well as at an SOA_AV of 20 ms. At an SOA_AV of 0 ms (Figure 6c), violations occurred for physically simultaneous stimuli across a range of RT (217–290 ms), with a peak at 244 ms. The fact that the greatest violations of Miller's inequality occur when unisensory stimuli are physically simultaneous is clearly incompatible with the strong version of the physiological simultaneity hypothesis, which posits that neural coactivation should occur at an SOA that closely corresponds to the difference in mean RT to the unisensory stimuli. The results of condition 2 also clearly show that neural coactivation can occur when a more rapidly processed stimulus (V) precedes a more slowly processed stimulus (A). The results of condition 2 are also inconsistent with the inverse effectiveness rule, because increasing the intensity of the V stimulus actually increased the magnitude of the violation of Miller's inequality (0.0360 vs. 0.0530 in conditions 1 and 2, respectively).

Figure 6.

Test of the race model for condition 2. Panel (a) shows spectrum-coded mean values of Miller's inequality as a function of RT and SOA_AV. Panel (b) shows a magnified view of the region exhibiting significant (p < 0.05) violations of the race model that occurred at SOA_AV values of 0 and 20 ms. Panel (c) plots the mean value (thick line) and 95% confidence intervals (thin lines) for Miller's inequality as a function of RT for simultaneous SOA_AV.

4. Condition 3

Condition 3 is complimentary to condition 2 and was designed to reveal how changing the intensity of the A stimulus might influence its integration with a relatively weak V stimulus. Specifically, we tested whether increasing the intensity (loudness) of the A stimulus caused a change in the optimal SOA_AV for multisensory integration relative to condition 1. Again, if physiological simultaneity is necessary for multisensory facilitation, then increasing the intensity of the A stimulus (which will increase the speed of auditory processing and reduce mean RT to A stimuli) should shift the optimal SOA_AV for neural coactivation to more positive values such that, relative to condition 1, the V stimulus would need to be presented earlier with respect to the A stimulus. Conversely, the optimal SOA_AV will not change if physical simultaneity determines multisensory integration.

4.1. Method

The participants in condition 3 were the same as in conditions 1 and 2. The experimental methods and procedures were as in condition 1, except that A stimulus intensity was increased to levels producing a criterion sensitivity d′ > 4 (intensity range = 44.5–49.0 dB_A). In order to ensure criterion sensitivity, d′ was calculated for unisensory A and V stimuli after every 18 trials throughout the experiment and V stimulus contrast and A stimulus intensity were adjusted as necessary.

4.2. Results and discussion

4.2.1. Multisensory response facilitation/inhibition

Mean RTs in condition 3 are plotted as a function of SOA_AV in Figure 3 (c). As expected, mean RT to the unisensory A stimulus (301.5 ms) was significantly faster than that to the unisensory V stimulus (441.7 ms) (t₁₉₇₈ = − 32.247, p < 0.001). The significantly faster mean response to the intense A stimulus in condition 3, relative to conditions 1 and 2, is another illustration of Piéron's law. To assess whether multisensory stimulation produced response facilitation, mean RT in AV conditions was compared with that in the fastest unisensory condition (A). Green bars indicate that significant response facilitation occurred at SOA_AV values of − 20 and 0 ms (t₁₆₁₉ = − 2.340, p = 0.019; t₁₆₁₆ = − 4.145, p < 0.001, respectively). There was no multisensory response inhibition at any SOA_AV.

4.2.2. Neural coactivation (Miller's inequality analysis)

Figure 7(a) spectrum codes the values of Miller's inequality as a function of RT and SOA_AV for condition 3. A magnified view of the region exhibiting significant violations of the race model is shown in Figure 7 (b). At an SOA_AV of 0 ms (Figure 7c), positive values of Miller's inequality occur across a range of RT (192–330 ms), with a peak at 288 ms. As in conditions 1 and 2, significant violations of Miller's inequality occurred for physically simultaneous A and V stimuli, as well as at an SOA_AV of − 20 and − 40 ms. Once again, the fact that violations of Miller's inequality have the greatest magnitude when unisensory stimuli are physically simultaneous is clearly incompatible with the strong version of the physiological simultaneity hypothesis, and the results of condition 3 confirm that neural coactivation can occur when a more rapidly processed stimulus (A) precedes the more slowly processed stimulus (V). The results of condition 3 are also inconsistent with the inverse effectiveness rule, because increasing the intensity of the A stimulus significantly increased the magnitude of the violation of Miller's inequality (0.0360 vs. 0.1113 in conditions 1 and 3, respectively).

Figure 7.

Test of the race model for condition 3. Panel (a) shows spectrum-coded mean values of Miller's inequality as a function of RT and SOA_AV. Panel (b) shows a magnified view of the region exhibiting significant (p < 0.05) violations of the race model, which occurred at SOA_AV values of − 40, − 20, and 0 ms. Panel (c) plots the mean value (thick line) and 95% confidence intervals (thin lines) for Miller's inequality as a function of RT for simultaneous SOA_AV.

5. Condition 4

Finally, condition 4 tested whether increasing the intensities of both A and V stimuli to clearly suprathreshold levels (d′ > 4) would result in a pattern of optimal SOA_AV for multisensory integration that resembled that of condition 1.

5.1. Method

The participants in condition 4 were the same as in condition 1. The experimental paradigm replicated condition 1 with the following exception. Both V and A stimuli were increased in intensity to levels producing criterion sensitivities of d′ > 4.

5.2. Results and discussion

5.2.1. Multisensory response facilitation/inhibition

Mean RTs in condition 4 are plotted as a function of SOA_AV in Figure 3 (d). As in condition 1, mean RT to the unisensory A stimulus was significantly faster (286.3 ms) than that to the unisensory V stimulus (316.7 ms) (t₂₄₀₂ = −8.74, p < 0.01). The significantly faster mean responses to the highly suprathreshold A and V stimuli in condition 4, relative to condition 1 (t₂₁₂₁ = −28.338, p < 0.01 and t₂₁₁₂ = −26.030, p < 0.01, respectively), again illustrate Piéron's law. To assess whether multisensory stimulation produced response facilitation, mean RT in AV conditions was compared with that in the fastest unisensory condition (A). The green bars indicate that significant response facilitation occurred at SOA_AV values of −80, −20, and 0 ms (t₁₆₁₃ = −2.03, p = 0.043; t₁₆₁₃ = −2.67, p = 0.008; and t₁₆₁₀ = −5.72, p < 0.001, respectively). There was no multisensory response inhibition at any SOA_AV.

5.2.2. Neural coactivation (Miller's inequality analysis)

Figure 8(a) spectrum codes the values of Miller's inequality as a function of RT and SOA_AV for condition 4. A magnified view of the region exhibiting significant violations of the race model is shown in Figure 8 (b). As in condition 3, positive values of Miller's inequality occurred for SOA_AV − 20 and 0 ms, a result that is incompatible with the strong version of the physiological simultaneity hypothesis. At an SOA_AV of 0 ms, positive values of Miller's inequality occur across a range of RT (171–257 ms), with a peak at 232 ms. The results of condition 4 are directly contradictory to the inverse effectiveness rule, because increasing the intensity of both unisensory stimuli significantly increased the magnitude of the violation of Miller's inequality (0.0360 vs. 0.1678 in conditions 1 and 4, respectively).

Figures 8(c) and (d) illustrate how the RT data were analyzed in order to compute values of Miller's inequality and establish confidence intervals. CP distributions as a function of RT were constructed for the A (purple) and V (blue) unisensory stimulus conditions, and for their multisensory combination, AV_o, at all values of SOA_AV (SOA_AV = 0 ms is shown in red). The sum of the CP distributions for the unisensory conditions (A + V) is the CP predicted by the race model (black). The value of Miller's inequality is computed at each value of RT by subtraction: CP(AV_o) − CP(A + V). When the CP of observed RT exceeds that predicted by the race model, the value of Miller's inequality is positive. These violations are represented by the segment of the AV_o CP distribution that lies above the (A + V) CP distribution in Figure 8 (c). Miller's inequality is plotted (thick line) as a function of RT (from 150 to 350 ms) in Figure 8 (d). Confidence intervals were derived by a bootstrapping procedure (Foster & Bischof, 1991) where the AV_o and A + V CP distributions were sampled exhaustively with replacement to generate a sampling distribution of Miller's inequality (N = 1,000). Thin lines in Figures 5–7(c) and 8(d) plot 95% confidence intervals for the value of Miller's inequality.

6. General discussion

Taken together, the results of the four experimental conditions reveal that audiovisual multisensory facilitation, as indexed by significant violations of the race model, occurs only over a narrow range of stimulus onset asynchronies which invariably includes physical simultaneity. Manipulations of stimulus intensity that changed the speed of unisensory processing, as revealed by significant alterations of mean RT, had no influence on the range of SOA_AV over which multisensory interaction occurred. Moreover, when the range of SOA_AV over which significant violations of the race model occurred did extend beyond physical simultaneity (condition 2: 20 ms; condition 3: −20 and −40 ms; condition 4: −20 ms), the extension was such that the more rapidly processed stimulus needed to precede the more slowly processed stimulus, a result that is, in fact, opposite to the strong version of the physiological simultaneity hypothesis (Hershenson, 1962), which predicts that the more slowly processed stimulus should need a “head start” in order to arrive at some central site simultaneously with a more rapidly processed stimulus. It should be noted that violations of Miller's inequality at all non-zero SOAs were smaller than those at simultaneity. Finally, there was a surprising lack of evidence for multisensory facilitation with respect to sensitivity (d′).

6.1. Inverse effectiveness

Although the rule of inverse effectiveness has sometimes been upheld as a universally observed characteristic of multisensory integration (Lakatos et al., 2007; Meredith & Stein, 1986), a number of studies have reported findings that are inconsistent with this rule (Lakatos et al., 2007; Ross et al., 2007). Our results likewise do not support the inverse effectiveness rule because the magnitude of the violations of Miller's inequality we observed generally increased with increasing stimulus intensity (at SOA_AV = 0 ms: 0.0360, 0.0530, 0.1113, and 0.1678 in conditions 1–4, respectively). Our results are, however, consistent with a reinterpretation (Holmes, 2007) of the results of Alvarado, Vaughan, Stanford, and Stein (2007), who showed that whereas multisensory enhancement in neurons in the cat's superior colliculus obeyed the rule of inverse effectiveness when analyzed in terms of relative spike rate increase, they demonstrated the behavior we observe, viz., increasing effectiveness with increasing stimulus intensity, when analyzed in terms of absolute spike rates. On the contrary, the range of SOA_AV over which statistically significant decreases in simple mean RT occur is wider for the weakest stimuli (condition 1) than for the other conditions (Figure 3), but increases in the time window of integration is not what is typically meant by inverse effectiveness.

6.2. Intensity-adjusted latency coding

Although physiological activity resulting from two unisensory signals must simultaneously converge on a multisensory “coincidence detector” for facilitative MI to occur, we find that such facilitation occurs nearly exclusively for physically simultaneous multisensory occurrences, independent of factors that differentially affect unisensory processing time, such as stimulus intensity. This makes sense from an ecological perspective, for if multisensory facilitation has aided survival, the advantage it confers must be the enhanced processing of genuine physical events (e.g., the sights and sounds of predators or prey which, because they have a common cause, are physically simultaneous), not merely to physiological simultaneities some (potentially large) fraction of which are accidental. Although the evolutionary advantage of this result is undeniable, the mechanism whereby it is accomplished is unclear.

Consider the case in which facilitative MI occurs at a coincidence detector where the response latencies for a given pair of simultaneously presented unisensory stimuli are exactly equal. Varying the relative intensities (and hence the processing speed) of the physically simultaneous unisensory stimuli will cause their physiological responses to convergence at this detector asynchronously, and yet we find that physically simultaneous stimuli integrate despite variations in intensity. Conversely, the physically asynchronous occurrence of two unisensory stimuli can, depending on their relative intensity, result in spuriously coincident physiological responses that nevertheless fail to result in facilitation. This suggests that the time of arrival at multisensory sites must be corrected for latency differences caused by adventitious variations in stimulus intensity. In other words, the system must “take account” of differences in physiological response latency that are unrelated to true unisensory SOA in order to reject false correspondences. It is well known that neural networks can undergo Hebbian learning, wherein the strength of synaptic connections is modified based on experience. The biophysical bases of temporal coincidence detection could involve mechanisms of spike-time dependent synaptic plasticity (Song, Miller, & Abbott, 2000) and/or synaptic scaling (Turrigiano, Leslie, Desai, Rutherford, & Nelson, 1998). Our novel idea that synapses might additionally be “tuned” to particular stimulus intensities is purely speculative. In short, however, physiological response simultaneity is a necessary but not sufficient condition for facilitative MI.

One possible mechanism for achieving this outcome is shown in Figure 9. Assume a neural network of AV coincidence detectors (labeled 1–5 in Figure 9) which integrate unisensory inputs (i.e., fire) exclusively when A and V inputs arrive synchronously, thus reflecting the necessary condition of physiological simultaneity. The coincidence detectors receive input from both A and V sensory receptors via delay lines and are thus place coded (Jeffress, 1948). A given afferent A or V signal ultimately supplies input to the entire network of coincidence detectors at a continuum of latencies. In Jeffress' (1948) theory of binaural sound localization, differences in the time of arrival of sound at the two ears (the interaural time difference) result in physiological convergence that varies in location within the network of coincidence detectors such that the spatial location of the sound source is read out by the relative spatial location of the activated coincidence detector. This explanation can be adapted to explain our results.

Figure 9.

An illustration of how learned associations (Bayesian priors) between physical simultaneity and relative stimulus intensity might be represented in a place-coded delay line network of coincidence detectors. Panels (a–c) refer to situations where physically simultaneous A and V events vary in relative intensity. In panel (a), the A and V inputs are of nominally equivalent relative intensity (denoted by the similar size of the delay lines, lettering, and the circular input and synaptic symbols), and the afferent signals (coded red) converge synchronously on and activate coincidence detector 3 (red). In panels (b) and (c), simultaneous A and V events vary in intensity. The more intense stimulus (thick line) propagates through the network more rapidly than the less intense stimulus (thin line), and their signals thus converge on coincidence detectors 1 or 5. Panel (d) illustrates two inputs of nominal equivalent intensity where the visual stimulus is delayed in time of onset relative to the auditory stimulus (represented here as added pathway length).

We begin by assuming that organisms have ubiquitous access to the physiological activity generated by genuinely simultaneous multisensory events in their environment, that is, that they have access to ground truth. Thus, over the course of their development organisms will accrue robust Bayesian priors with respect to multisensory convergence. The class of multisensory stimulation for which exact latency/intensity information is known results from self-generated events. For example, tapping an object with the hand (or a tool), or throwing a projectile whose impact with a nearby surface produces both visual and auditory consequences, etc., produces afferent visual, auditory, and haptic/kinesthetic signals of known common origin. Because these signals routinely vary in intensity, the latencies at which these signals converge at multisensory coincidence detectors can, over repeated stimulation, result in the accumulation of intensity-adjusted probability distributions that form the basis for computing the posterior probability that any set of incoming multisensory signals have a common origin.

One way in which the Bayesian priors could be instantiated in a delay-line network is illustrated in Figure 9.

Figures 9(a–c) refer to situations where physically simultaneous A and V events vary in relative intensity. In Figure 9 (a), the A and V inputs are of nominally equivalent relative intensity (denoted by the similar size of the delay lines, lettering, and the circular input and synaptic symbols), and the afferent signals (coded red) converge synchronously on and activate coincidence detector 3 (red). No other coincidence detector receives simultaneous input. Figures 9(b) and (c) describe situations where simultaneous A and V events vary in intensity. The more intense stimuli (thick lines) propagate through the network more rapidly than the less intense stimuli (thin lines), and their signals thus converge on coincidence detector 1 or 5. Because we are assuming that in each case the physical origin of the afferent signals is known to have been common, physical stimulus synchrony can be disambiguated and correctly read out by the network only if the coincidence detectors are trained (learn) to fire when the constituent inputs possess the appropriate relative intensities. Thus, in Figure 9 (d), two inputs of nominal equivalent intensity are illustrated where the visual stimulus is delayed in time of onset relative to the auditory stimulus (represented here as added pathway length). Being of nominal equivalent intensity, the two signals propagate through the network at similar speeds and converge on coincidence detector 4. However, because this coincidence detector has “learned” that physically simultaneous A and V events only converge at its location when they possess unequal intensities (intensity tuning is indicated by the sizes of the circular synaptic contacts), it rejects this physiological simultaneity as spurious and does not integrate. Animations demonstrating the four conditions of Figure 9 are available as a supplementary flash file.