Interactions between space and effectiveness in human multisensory performance

Abstract

Several stimulus factors are important in multisensory integration, including the spatial and temporal relationships of the paired stimuli as well as their effectiveness. Changes in these factors have been shown to dramatically change the nature and magnitude of multisensory interactions. Typically, these factors are considered in isolation, although there is a growing appreciation for the fact that they are likely to be strongly interrelated. Here, we examined interactions between two of these factors – spatial location and effectiveness - in dictating performance in the localization of an audiovisual target. A psychophysical experiment was conducted in which participants reported the perceived location of visual flashes and auditory noise bursts presented alone and in combination. Stimuli were presented at four spatial locations relative to fixation (0°, 30°, 60°, 90°) and at two intensity levels (high, low). Multisensory combinations were always spatially coincident and of the matching intensity (high-high or low-low). In responding to visual stimuli alone, localization accuracy decreased and response times (RTs) increased as stimuli were presented at more eccentric locations. In responding to auditory stimuli, performance was poorest at the 30 and 60° locations. For both visual and auditory stimuli, accuracy was greater and RTs were faster for more intense stimuli. For responses to visual-auditory stimulus combinations, performance enhancements were found at locations in which the unisensory performance was lowest, results concordant with the concept of inverse effectiveness. RTs for these multisensory presentations frequently violated race-model predictions, implying integration of these inputs, and a significant location-by-intensity interaction was observed. Performance gains under multisensory conditions were larger as stimuli were positioned at more peripheral locations, and this increase was most pronounced for the low-intensity conditions. These results provide strong support that the effects of stimulus location and effectiveness on multisensory integration are interdependent, with both contributing to the overall effectiveness of the stimuli in driving the resultant multisensory response.

1. Introduction

Our brains are continually receiving sensory information from the environment. Each sensory system is tasked with receiving and processing this information and each accomplishes this task in different ways. Often, information from one modality is accompanied by corresponding information in another, particularly when this information is derived from the same event. To process such stimuli more efficiently, our brains integrate this information, often in ways that result in substantial changes in behavior and perception (G. A. Calvert, Spence, C. , Stein, B.E., 2004; Murray & Wallace, 2012; B.E. Stein & Meredith, 1993). Several of the more familiar and compelling examples of these multisensory-mediated changes in behavior include improvements in target detection (Frassinetti, Bolognini, & Ladavas, 2002; Lovelace, Stein, & Wallace, 2003), improvements in target localization and orientation (Ohshiro, Angelaki, & DeAngelis, 2011; B. E. Stein, Huneycutt, & Meredith, 1988), and speeding of reaction or response times (RTs; Amlot, Walker, Driver, & Spence, 2003; G. A. Calvert & Thesen, 2004; Corneil, Van Wanrooij, Munoz, & Van Opstal, 2002; Diederich, Colonius, Bockhorst, & Tabeling, 2003; Forster, Cavina-Pratesi, Aglioti, & Berlucchi, 2002; Frens, Van Opstal, & Van der Willigen, 1995; Harrington & Peck, 1998; Hershenson, 1962; Hughes, Reuter-Lorenz, Nozawa, & Fendrich, 1994; Molholm, et al., 2002).

In an effort to “decide” what should be integrated (and what should not be integrated), the brain capitalizes on the statistical regularities of cues from the different senses that provide important information as to the probability that they are related (Alais & Burr, 2004; Altieri, Stevenson, Wallace, & Wenger, 2015; Baier, Kleinschmidt, & Muller, 2006; den Ouden, Friston, Daw, McIntosh, & Stephan, 2009; Massaro, 1984; McIntosh & Gonzalez-Lima, 1998; Polley, et al., 2008; Shinn-Cunningham, 2008; Wallace & Stein, 2007). Thus, multisensory integration (and its resultant behavioral and perceptual benefits) is in part determined by physical factors associated with the stimuli to be paired. Several stimulus factors have been identified as being integral to this process. Preeminent among these are the spatial and temporal relationships of the stimuli and their relative effectiveness. As a general rule, the more spatially and temporally proximate two signals are, the more likely they are to influence one another’s processing. Furthermore, weakly effective sensory signals typically result in the largest multisensory gains when they are paired, a phenomenon known as inverse effectiveness.

These stimulus-dependent factors and their influence on multisensory processing have proven to be remarkably robust across a wide array of experimental measures. These include: the activity of individual neurons in animal models [space (Meredith & Stein, 1986a), time (Meredith, Nemitz, & Stein, 1987), effectiveness (Meredith & Stein, 1986b)], neural responses in humans as measured by fMRI and PET [space (Macaluso, George, Dolan, Spence, & Driver, 2004), time (Macaluso, et al., 2004; L. M. Miller & D'Esposito, 2005; Stevenson, Altieri, Kim, Pisoni, & James, 2010; Stevenson, VanDerKlok, Pisoni, & James, 2011), effectiveness (James & Stevenson, 2012; James, Stevenson, & Kim, 2009, 2012; Nath & Beauchamp, 2011; Stevenson & James, 2009; Stevenson, Kim, & James, 2009; Werner & Noppeney, 2009)] and EEG [space (Zhou, Zhang, Tan, & Han, 2004), time (Schall, Quigley, Onat, & Konig, 2009; Senkowski, Talsma, Grigutsch, Herrmann, & Woldorff, 2007; Talsma, Senkowski, & Woldorff, 2009), effectiveness (Stevenson, Bushmakin, et al., 2012)], as well as human behavior and perception [space (Bolognini, Frassinetti, Serino, & Ladavas, 2005; Frassinetti, et al., 2002) but see (Murray, et al., 2005), time (Conrey & Pisoni, 2006; Dixon & Spitz, 1980; Hillock, Powers, & Wallace, 2011; Keetels & Vroomen, 2005; Stevenson & Wallace, 2013; Stevenson, Zemtsov, & Wallace, 2012a; van Atteveldt, Formisano, Blomert, & Goebel, 2007; van Wassenhove, Grant, & Poeppel, 2007; Wallace, Roberson, et al., 2004), effectiveness (Stevenson, Zemtsov, & Wallace, 2012b; Sumby & Pollack, 1954) but see (Chandrasekaran, Lemus, Trubanova, Gondan, & Ghazanfar, 2011; Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007)]. It should also be noted here that, aside from these bottom-up factors, other higher-level factors such as task, semantic congruence, and context are likely to also be very important in dictating the final response (Foxe, 2008; Otto, Dassy, & Mamassian, 2013; Stevenson, Wallace, & Altieri, 2014; Ten Oever, Sack, Wheat, Bien, & Van Atteveldt, 2013).

Although these factors have largely been studied in an independent manner in this prior work (e.g., exclusive manipulation of the spatial relationship of the paired stimuli), there is an intuitive interdependency between them that has not been thoroughly explored. For example, manipulating the absolute spatial location of multisensory stimuli impacts the relative effectiveness of these stimuli because of, for example, changes in the sensory acuity of the peripheral organs. Indeed, recent neurophysiological (Carriere, Royal, & Wallace, 2008; Krueger, Royal, Fister, & Wallace, 2009; Royal, Carriere, & Wallace, 2009) and psychophysical (Cappe, Thelen, Romei, Thut, & Murray, 2012; Macaluso, et al., 2004; Stevenson, Fister, Barnett, Nidiffer, & Wallace, 2012) studies have begun to shed light on the nature of these interdependencies.

These studies serve as motivation for the current study, which seeks to examine the interdependency of spatial location and stimulus effectiveness in dictating one aspect of human performance – target localization. The work is predicated on the evidence that manipulations of the location of visual or auditory stimuli results in changes in the accuracy related to detecting the location or changes in the location of the stimuli (Bock, 1993; Carlile, Leong, & Hyams, 1997; Mills, 1958, 1960; Yost, 1974). Therefore, our hypothesis was that changing the location of a stimulus should result in changes in the effectiveness of that stimulus. In turn, the magnitude of behavioral gains from multisensory presentations should reflect this change of effectiveness across space in a manner mirroring inverse effectiveness, providing insights into how space and effectiveness interact to dictate multisensory responses. To explore this hypothesis, we tested individuals’ ability to localize visual, auditory, and paired audiovisual targets as a function of both stimulus location and stimulus intensity. By examining localization accuracy and RTs, we then characterized the multisensory gains seen in responses to these different stimulus combinations (Stevenson, Ghose, et al., 2014). Testing the principles of multisensory integration together and investigating interactions between them would lend support to the notion that the principles are strongly interrelated, and provide novel mechanistic insights into the nature of such interactions.

2. Methods

2.1. Participants

Participants included fifty-one Vanderbilt undergraduate students (21 male, mean age =18.9, STD = 1, age range =18-21 and were compensated with class credit. All recruitment and experimental procedures were approved by the Vanderbilt University Institutional Review Board. Exclusionary criteria, applied prior to in-depth data analysis, included a failure to detect foveal stimuli (at 0°) above 80% rate (N = 5), or the failure to report foveal, synchronous stimuli as synchronous at a 50% rate (N = 5). Finally, one subject was excluded for repeatedly pressing a single button on the response box for the entirety of the experiment. This study is part of a larger study investigating the interaction of spatial, temporal, and effectiveness factors on multisensory processing (Krueger Fister, Stevenson, Nidiffer, Barnett, & Wallace, in revision; Stevenson, Fister, et al., 2012).

2.2. Stimuli

Visual and auditory stimuli were presented using E-Prime version 2.0.8.79 (Psychology Software Tools, Inc; PST). Visual stimuli were presented on two Samsung Sync Master 2233RZ monitors at 100 HZ arranged so that each monitor crossed the circumference of circle centered on the participants nasium at a distance of 46cm at 0°, 30°, 60°, and 90° azimuth, with all presentations in the right visual field (Figure 1a, b). All visual stimuli were white circles measuring 7mm in diameter, or approximately 1° of visual angle. Visual stimulus durations were 10 ms, with timing confirmed using a Hameg 507 oscilloscope with a photovoltaic cell. Visual stimuli were presented at two luminance levels, 7.1 cd/m² (low) and 215 cd/m² (high) with a black background of 0.28 cd/m², measured with a Minolta Chroma Meter CS-100. Visual stimuli were presented at each spatial location (4) and each salience level (2), for a total of eight visual-only conditions.

Figure 1. Stimulus Apparatus and Trial Structure.

(A, B) Auditory and visual stimuli were presented on a two-monitor array such that stimuli locations at 0°, 30°, 60°, and 90° were equidistant from the nasium. (C) Trial Structure for the task. After fixating a cross for 500 – 1000 ms, subjects were presented with auditory, visual or multisensory stimulus. The subjects then had 2000 ms to respond with the location of the stimulus or indicate that there was no stimulus.

Auditory stimuli were presented via four separate speakers mounted on the top of the two monitors at 0°, 30°, 60°, and 90° azimuths angled toward participant, matching the visual presentations. Speakers were mounted 2cm, or 2.5° above their respective visual presentation. Auditory stimuli consisted of a frozen white-noise burst generated at 44100 Hz with the Matlab rand function with a 5 ms rise/fall cosine gate. Auditory stimulus duration was held constant at 10 ms, with timing confirmed using a Hameg 507 oscilloscope. Auditory stimuli were presented at two intensity levels, 46 dB SPL (low) and 64 dB SPL (high), with a background noise at 41 dB SPL, measured with a Larson Davis sound level meter, Model 814. Auditory stimuli were presented at each spatial location (4) and each salience level (2), for a total of eight auditory-only conditions.

Audiovisual (AV) conditions consisted of pairs of the auditory and visual stimuli described above. Presentations were always spatially coincident, and salience levels were always matched (high-high and low-low). AV conditions were presented at each spatial location (4) and each salience level (2) for a total of eight AV conditions. Additionally, a “blank” no stimulus condition was also included in which no auditory or visual stimulus was presented while all aspects of the trial remained consistent. In total 25 unique conditions were presented, eight visual only, eight auditory only, eight AV, and one blank. In addition to these trials that are relevant to this report, additional presentations including temporal synchrony variations of these stimuli were also included. These modulations are reported elsewhere (Stevenson, Fister, et al., 2012) and are incorporated in these analyses only in the exclusionary criteria listed above.

2.3. Procedure

Participants were seated inside an unlit WhisperRoom™ (SE 2000 Series) with their forehead placed against a Headspot (University of Houston Optometry) forehead rest locked in place, with a chinrest and chair height adjusted to the forehead rest. Participants were asked to fixate towards a cross at all times, and were monitored by close circuit infrared cameras throughout the experiment to ensure fixation. Participants were instructed to make all responses as quickly and as accurately as possible. Each trial began with a fixation screen for 1 s with a fixation cross in visual center at 0° elevation 0° azimuth (Figure 1c). Fixation was followed by blank screen with a randomly jittered time between 500 and 1000 ms, with stimulus presentation immediately after. Following stimulus presentation, a response screen appeared including the prompt, “Where was it?” below a fixation cross, with the response options 1-4 displayed on screen at each spatial location coincident with stimulus presentation, and a fifth option (5) for no stimulus detected. Participants completed the spatial location task by responding via a five-button PST serial response box where 1 = 0°, 2 = 30°, 3 = 60°, 4 = 90°, and 5 = no stimulus. Hence, the chance of a correct response from a random guess was 20%. Following the participant’s response, the fixation cross appeared and the subsequent trial began.

Participants completed four sessions, each lasting approximately 20 minutes. Each session included five stimulus presentations with each of the 25 conditions in a randomized order for a total of 125 trials per session. Across the four sessions, participants completed 20 trials per condition, for a total of 600 trials per participant. Participants were given breaks in between sessions as needed. Including breaks, total experiment time was approximately 90 minutes.

2.4. Analysis

An α-level of p = 0.05 was used for all analyses. Response accuracies were measured for the spatial localization task over each spatial location and stimulus intensity. Accuracies were measured as percent of total trials per condition in which the subject judged the location of the stimulus correctly. A three-way ANOVA (location × intensity × sensory modality) was used as an omnibus test with two-way ANOVAs (location × intensity) used as follow-up tests. Multisensory enhancement was found by comparing multisensory performance against the greater performance of the unisensory components. Enhancements were compared across spatial location and stimulus intensity using a three-way, repeated-measures ANOVA. Within each significant main effect, significance of gain of each condition was determined by a protected, one-sample t-test.

Only trials in which subjects correctly localized stimuli and only responses made within 2000 ms were considered in RT analysis. RT cumulative distribution functions (CDFs), specifying at each time point the proportion of total responses that were made, were computed for each condition (i.e., auditory, visual and multisensory; high and low intensity; and 0, 30, 60, and 90°) and subject. In order to account for stimulus redundancy in multisensory stimuli when assessing multisensory gain, multisensory CDFs were compared to an independent race-model computed from a distribution comprised of the minima of randomly selected auditory and visual RTs without replacement (Raab, 1962). In this model, for all reaction times t > 0:

$$\widehat{p}(RT<t\mid S_{AV})=p(RT<t\mid S_A)+p(RT<t\mid S_V)-p(RT<t\mid S_A)\times p(RT<t\mid S_V)$$

This race model prediction $\left[\widehat{p}\left(R{T}_{AV}<t\right)\right]$ represents a statistical threshold (viz. probability summation) such that probabilities along the multisensory distribution that are greater than the race model prediction are said to violate the race model and thus are indicative of integration of unisensory components. The race model relies on an assumption of context invariance. That is, it implies that the processing of one signal is independent of the processing of another (Ashby & Townsend, 1986; Luce, 1986; Townsend & Wenger, 2004). Significant race-model violations were determined using the Kolmogorov-Smirnoff (KS) Test for each CDF for each condition on an individual–subject basis.

To assess the extent of race-model violations [V_AV(t)] across space and intensity, the predicted CDF was subtracted from the multisensory CDF for each subject and condition in which

$$V_{AV}\left(t\right)=p(RT<t\mid S_{AV})-\widehat{p}(RT<t\mid S_{AV})$$

Group race-model inequality distributions were computed by averaging V_AV(t) across all subjects at each RT, t. These resulting distributions are known as the race-model inequalities (J. Miller, 1982). From these distributions, race-model violation distributions were computed by averaging only positive portions of individual subjects’ race-model inequalities (i.e., V_AV > 0). These distributions were compared across stimulus location and intensity in two ways. First, following Colonius and Diederich (2006), area under the portions of the race-model violation distributions were computed for each subject and condition and then analyzed using repeated-measures ANOVA. Comparisons between intensity levels or stimulus location were carried out via two-sample t-tests. Second, for individual subjects, race-model inequalities were compared across intensity level (high - low) and spatial location (0° − 30°, etc.). This difference-of-difference measure is a powerful means to highlight interaction effects, and has been used previously in a number of domains, including fMRI (James, Kim, & Stevenson, 2009; James & Stevenson, 2012; Stevenson, et al., 2009), ERPs (Stevenson, Bushmakin, et al., 2012) and in examining RT distributions (Stevenson, Fister, et al., 2012). In addition, such difference-of-difference measures were used to initially reveal simple interaction effects with RT measures (Sternberg, 1969a, 1969b, 1975, 1998). For the resulting contrasts, probability differences were placed in 100 ms bins and bins were compared using two-sample t-tests (for example, high - low at 0° was compared to high - low at 30°).

3. Results

3.1. Changes in localization accuracy as a function of stimulus location and intensity

In measures of accuracy, where subjects were asked to identify stimulus location, an omnibus three-way ANOVA (factors of modality [i.e., visual alone, auditory alone, visual-auditory], location and intensity) showed significant main effects, significant two-way interactions, and a significant three-way interaction. These statistics are summarized in Table 1. Follow-up, two-way ANOVAs (factors of stimulus location and intensity) revealed significant main effects of spatial location for the auditory (F_(3,117) = 14.306, p = 5.2 × 10⁻⁸, partial η² = 0.27), visual (F_(3,117) = 286.961, p = 1.7 × 10⁻⁵⁴, partial η² = 0.88), and multisensory (F_(3,117) = 20.041, p = 1.5 × 10⁻¹⁰, partial η² = 0.34) conditions (Figure 2a, b). For these spatial manipulations, subjects were less accurate at judging the location of visual stimuli at peripheral locations compared to stimuli that were centrally located (0° vs. 90°; t = 26.36, p = 1.84 × 10⁻²⁶, d = 8.44). A similar pattern of decreasing accuracy with increasing eccentricity held for auditory stimuli presented from center to 60° (t = 5.50, p = 2.6 × 10⁻⁶, d = 1.76), albeit at a slower rate than in the visual condition. However, auditory localization accuracy increased for the most peripheral (90°) location (t = 3.37, p = 0.002, d = 1.08).

Table 1.

Results from three-way ANOVA

Effect	dfn	dfd	F	P
A – Sensory Modality	2	78	417.771	2.11×10−42
B – Intensity Level	1	39	104.786	1.31 ×10−12
C – Spatial Location	3	117	124.385	3.06 ×10−36
A × B	2	78	87.767	1.08 ×10−20
A × C	6	234	120.268	1.29 ×10−68
B × C	3	117	11.155	1.71 ×10−06
A × B × C	6	234	30.42	6.92 ×10−27

Figure 2. Accuracy Performance Across Space and Intensity.

(A, B) Localization performance declined for auditory and visual stimuli for high (A) and low (B) intensity conditions. (C) To assess multisensory enhancement, the greatest unisensory performance was subtracted from the multisensory performance. Enhancements across space follow the pattern of inverse effectiveness.

Main effects of intensity (Figure 2a, b) were highly significant in the visual and multisensory conditions (F_(1,39) = 195.076, p = 1.1 × 10⁻¹⁶, partial η² = 0.83 and F_(1,39) = 14.553, p = 4.7 × 10⁻⁴, partial η² = 0.27, respectively). In contrast, the auditory intensity effect failed to reach significance (F_(1,39) = 2.162, p = 0.15). Thus, for both the visual and audiovisual conditions, high-intensity stimuli were more accurately localized than low-intensity stimuli. In addition, significant interactions between location and intensity were found in both the auditory and visual modalities (F_(3,117) = 9.47, p = 1.2 × 10⁻⁵, partial η² = 0.20 and F_(3,117) = 38.31, p = 1.9 × 10⁻¹⁷, partial η² = 0.50, respectively), but not in the multisensory condition (F_(3,117) = 2.08, p = 0.11).

In summarizing these results, localization accuracy was strongly influenced by the location of auditory, visual, and multisensory stimuli. Accuracy was found to decrease for the localization of visual and multisensory targets as intensity was decreased. The effects of changing intensity and location were found to interact for unisensory auditory and visual conditions, but not under multisensory conditions.

Along with examining absolute performance as a function of stimulus location and intensity, we also assessed the amount of multisensory gain, indexed as a comparison between multisensory accuracy and the better of the two unisensory accuracies. This measure yields an effect of spatial location (F_(3,117) = 3.663, P = 0.014, partial η² = 0.09) which is driven by increased multisensory gain at 30° in both high (t = 3.033, p = 0.004, d =0.97) and low (t = 3.499, p = 0.001, d = 1.12) intensities and in the low intensity condition at 60° (t = 2.448 p = 0.02, d = 0.79). In contrast, no significant interaction between location and intensity was observed (F_(3,117) = 1.76, p = 0.16).

3.2. Changes in mean response times as a function of stimulus location and intensity

Like for accuracy in stimulus localization, RTs showed a main effect for spatial location in auditory (F_(3,117) = 22.178, p = 1.9 × 10⁻¹¹, partial η² = 0.36), visual (F_(3,117) = 102.758, p , p = 4.9 × 10⁻³³, partial η² = 0.72), and multisensory (F_(3,117) = 75.847, p = 1.3 × 10⁻²⁷, partial η² = 0.66) conditions (Figure 3, bottom). For the visual condition, subjects were slower to respond to stimuli located in the periphery compared to central stimuli (0° vs. 90°; t = 10.80, p = 2.8 × 10⁻¹³, d = 3.46). The auditory spatial effect had a similar pattern as that seen for accuracy. RTs first increased as stimuli were located at more peripheral locations, but with a rate of increase slower than that seen for vision. This effect reached its maximum at 60° (0° vs. 60°; t = 7.19, p = 1.2 × 10⁻⁸, d = 2.30), after which RTs were faster at 90° (60° vs. 90°; t = 4.47, p= 6.6 × 10⁻⁵, d = 1.43). A main effect of intensity on reaction times was only significant for visual stimuli (F_(1,39) = 5.463, p = 2.9 × 10⁻⁶, partial η² = 0.12; Figure 3). We found significant interactions in RTs between spatial location and intensity in the visual and multisensory conditions (F_(3,117) = 5.754, P = 0.001, partial η² = 0.13, F_(3,117) = 3.476, p = 0.018, partial η² = 0.08, respectively), but not in the auditory condition (Figure 3).

Figure 3. Mean Response Times Across Space and Intensity.

(A, B) Mean RTs generally declined with more peripheral presentations for high (A) and low (B) intensity conditions. Likewise, mean RTs were slower for low- than high-intensity stimulus presentations.

3.3. Changes in multisensory response times as a function of stimulus location and intensity: race-model analyses

From performance for the two unisensory conditions, we calculated a race-model prediction to which we compared the multisensory cumulative distribution function (CDF, Figure 4a-b). When the multisensory CDF falls significantly to the left of the modeled CDF, it is considered a violation of the race model, and thus multisensory facilitation (J. Miller, 1982). There were significant race-model violations present in all subjects but one, with just over half of subjects (N = 21) having violations in 5 or more of the multisensory conditions.

Figure 4. Response Time CDFs and Race-Model Inequalities.

(A – D) Step-by-step analysis of response time CDFs for high and low intensities at 0° location. (A and B) Group average multisensory (purple lines) and race-model (black line) CDF for high (A) and low (B) intensity stimuli at the central location. (C) Group average race-model inequality for high and low intensities at 0° location, computed by subtracting race-model CDFs from multisensory CDFs for each subject. (D)Group average race-model violations for high and low intensities at 0° location. Race model violations were calculated by averaging only positive portions of each subject’s race-model inequality. (E – H) CDF analyses shown in (C) and (D) for all stimulus conditions and average areas under these curves. (E) Race-model inequalities for all spatial locations and intensity levels averaged across subjects. (F) Average area under the curve for race-model inequalities for all conditions across subjects. (G) Race-model violation curves for all spatial locations and intensity levels averaged across subjects. Curves decrease in amplitude across space for high intensity stimuli while low intensity curves increase in amplitude across space. (H) Average area under the curve for race model violations, as in (F), showed a significant interaction driven by differences primarily at 60°. Central locations showed no difference for intensity.

A comprehensive comparison of multisensory performance with race-model predictions revealed that for all conditions (8) and all individuals (40), there were 200 significant race-model violations out of 320 possible (p = 4.5 × 10⁻⁶). Across conditions, the number of subjects with significant violations was between 23 and 29 (out of the 40 possible). In general, more race-model violations were found at peripheral locations and lower intensities (Table 2).

Table 2.

Race Model Violations per Condition

Intensity	Location	Violations	p
High	0	25	0.036585
	30	23	0.080702
	60	27	0.010944
	90	23	0.080702
Low	0	23	0.080702
	30	23	0.080702
	60	29	0.002103
	90	27	0.010944

In an additional test for multisensory facilitation, we also calculated the race-model inequality by subtracting the race-model CDF from the multisensory CDF (Figure 4c,e). In this measure, all positive values indicate race-model violations, clear evidence for a lack of independence between visual and auditory processes. Race-model violation distributions are shown for all subjects in Figure 4d,g. As a general rule, race-model violations occurred in the earliest epochs of the response distributions. To quantitatively assess these multisensory gains in RT, we calculated the area under the curve for individual race-model violations (Figure 4h). The main effect of location was significant in these analyses (F_(3,117) = 2.905, p = 0.038, partial η² = 0.07), with race-model violations increasing as stimuli moved more peripherally. The main effect of intensity was also significant (F_(3,117) = 5.265, p = 0.027, partial η² = 0.12), with violations being greater for low- vs. high-intensity stimuli. Area-under-the-curve calculations also yielded a significant interaction between location and intensity (F_(3,117) = 2.788, p = 0.044, partial η² = 0.07). Thus, as stimuli were presented at increasingly peripheral locations, there was greater multisensory facilitation in the low-intensity condition than in the high-intensity condition. Pair-wise comparisons revealed a significant difference between the two intensity conditions at the 60° location (t = 2.609, p = 0.012, d = 0.84). In contrast, area under the race-model inequality curves were not statistically different between the high- and low-intensity conditions at 0°, 30°, and 90° (t = 1.104, p = 0.28; t = 0.25, p = 0.8; and t = 1.402, p = 0.17, respectively).

Finally, analyses were also carried out to identify points along the CDFs that showed significant enhancement in an interaction of location and intensity. First, race-model violations for low-intensity stimuli were contrasted with their high-intensity counterparts (Figure 5a). Positive values represent greater violation in the high intensity condition. These distributions tended to become more negative for peripheral stimuli (Table 3). To test this, we then made comparisons of these CDF contrasts between the central location and each of the peripheral locations, which revealed an interesting pattern of interactions (Figure 5b, c, and d). Here, positive values represent greater race-model violations for low-intensity and peripheral stimuli. There were no significant bins when comparing between 0° and 30°. In contrast, for the 0° − 60° and 0° − 90° contrasts there were a number of significant bins, as denoted by the asterisks (p < 0.05, Table 3). These analyses thus reveal that for increasingly peripheral locations, there was a greater multisensory facilitation of RT for low-intensity stimuli when compared to high-intensity stimuli, indicating an interaction of location and intensity in multisensory facilitation of RTs.

Figure 5. Response Time CDF Interactions.

(A) CDF contrasts for main effect of intensity were calculated by subtracting each subject’s low intensity race model violation curve from their high intensity curve and averaging across subjects. Positive values represent greater enhancement in high intensity conditions. There is a general trend for the CDF contrasts to be more negative in the periphery. (B,C,D) CDFs from (A) were subtracted to produce interaction CDFs. CDFs were divided into 100ms bins for statistical analysis. Shaded regions represent the 95% confidence intervals. (B) The 0° − 30° interaction yielded no significant bins. (C,D) Significant bins were present in the 0° − 60° and 0° − 90° interactions.

Table 3.

Bin-by-bin CDF comparisons

Contrast	Bin (ms)	t	p
0° high - low	1500	0.932	0.0858
60° high - low	1000	0.943	0.0717
	1100	0.984	0.0205
	1200	0.99	0.0128
	1300	0.99	0.0164
	1400	0.937	0.0802
	1700	0.955	0.0566
90° high - low	500	0.935	0.0821
	600	0.93	0.0879
	700	0.974	0.0328
	800	0.981	0.0242
	900	0.923	0.0979
	1800	0.953	0.0595
	1900	0.935	0.0826
0° - 60° x	800	0.958	0.0529
high - low	900	0.988	0.0156
interaction	1000	0.985	0.0188
	1100	0.992	0.0103
	1200	0.992	0.0104
	1300	0.992	0.0106
	1700	0.972	0.0351
0° - 90° x	600	0.962	0.0477
high - low	700	0.970	0.0375
interaction	800	0.985	0.019
	900	0.98	0.0251

4. Discussion

Here, we present a novel finding showing an interactive effect between the location and intensity of paired audiovisual stimuli in dictating multisensory gains in human performance. Consistent with prior work (Bock, 1993; Carlile, et al., 1997; Green & Swets, 1966; Hecht, Shlaer, & Pirenne, 1942), when visual and auditory stimuli are presented more peripherally or at lower intensity, localization performance suffers. In contrast, the pairing of these stimuli as they become less effective results in larger multisensory gains. Indeed, the greatest effect is seen for low intensity stimuli presented at peripheral locations. A similar pattern occurs for response times. Here subjects became slower to respond to peripheral and lower intensity stimuli with the greatest speeding of RTs when these two manipulations were combined. Collectively, these results are consistent with the principle of inverse effectiveness, and suggest that in addition to being linked to stimulus intensity, effectiveness can also be modulated by changing the location of the stimuli, with comparable effects on the multisensory interactive product.

Although accuracy in localizing visual stimuli decreased monotonically across space, auditory stimuli produced a response minimum at 60°. This pattern of performance is likely explainable by changes in interaural level as a function of azimuth, and where the poorest auditory localization accuracy has been found to be at intermediate locations (i.e., not at 90°) (Moore, 2004). Unlike interaural time differences that increase monotonically from 0° to 90° (Feddersen, Sandel, Taeas, & Jeffress, 1957), maximum interaural level differences occur between 45° and 70°, depending on frequency (Shaw, 1974). Despite the lack of an interaction effect for multisensory gains in localization accuracy, the results provide support for the notion that multisensory gains across space follow the principle of inverse effectiveness. Another possible account for the apparent increase in accuracy at 90° is an edge effect (Durlach & Braida, 1969; Weber, Green, & Luce, 1977). The result of this would be an artificial increase of the auditory accuracy at 90° due to subjects using this extreme as an “anchor.” If this is the case, it is possible that our results are underestimating the multisensory effects on localization accuracy and therefore masking a potential interaction in the accuracy domain. Further work should be conducted to better account for the possibility of such an effect.

Another consideration is related to the differences in how the stimuli are perceived. Most of the incorrect judgments of visual stimuli were misses whereas auditory errors were largely mislocalizations (Supplementary Figure 1). This provides some interesting insights into ways stimulus effectiveness can be manipulated. Due to the spatial acuity of the visual system, it is likely that multisensory enhancement during localization isn’t observed during conditions where a visual stimulus is readily detected (Hairston, Laurienti, Mishra, Burdette, & Wallace, 2003). However, by reducing the intensity of the visual signal (and by shifting its location into the periphery) it becomes less detectible (i.e., less “effective”) and thus less informative of the location of a multisensory stimulus. This is in contrast to the decrease in effectiveness resulting in the mislocalization of the auditory stimulus. It is possible that a sub-threshold visual signal can still be integrated (Aller, Giani, Conrad, Watanabe, & Noppeney, 2015) during a localization task to cause multisensory enhancement without capturing the behavior. This poses an interesting question concerning how various types of information (e.g., presence or location) can be integrated in different tasks (e.g., detection or localization).

In addition to our analysis of response accuracy, we also characterized changes in response times as a function of varying stimulus location and intensity. These results, assessed relative to a race-model predictive framework (which accounts for non-interactive statistical facilitation), revealed three principal findings. First, there were more race-model violations for lower-intensity stimuli, as expected from the principle of inverse effectiveness. Second, as was seen with response accuracies, stimuli located at more peripheral locations also resulted in more race-model violations, providing a convergent measure relating stimulus location to stimulus effectiveness (and the principle of inverse effectiveness). Third, consistent with these patterns, the greatest multisensory-mediated gains in performance were seen for low-intensity stimuli positioned at peripheral locations.

The similarity in the effects of changing stimulus location and intensity on multisensory performance, coupled with the clear interaction between these factors, strongly suggest a unifying mechanism that is dictating the final multisensory product. This novel finding supports the hypothesis that a number of stimulus-driven modulations of multisensory enhancement may be a manifestation of one larger principle possibly guided by the relative effectiveness of the stimuli. This does not contradict previous research, but instead highlights the limited scope of past studies in which the principles of integration were studied in isolation. Inverse effectiveness seems to be a central mechanism for dictating multisensory enhancements across stimulus-level manipulations such as, in the case of the current study, location and intensity. While inverse effectiveness is most often associated with manipulations of stimulus intensity, it is important to note here that effectiveness refers to the general ability of a stimulus to drive a response (B. E. Stein, Stanford, Ramachandran, Perrault, & Rowland, 2009), a concept not only restricted to stimulus intensity. Indeed, relationships following the pattern of inverse effectiveness have been discovered through alternate manipulations such as task difficulty (S. Kim, Stevenson, & James, 2012), cue reliability (Ohshiro, et al., 2011), and motion coherence level (R. Kim, Peters, & Shams, 2012).

The main effects of the current work clearly illustrate the interaction between space and intensity in dictating multisensory localization. However, the results do not say whether the effects of space and intensity are equal. Likely, modulations in space and intensity carry certain weights per unit of modulation (e.g., degrees and decibels) and are not likely to scale linearly. Future work will be aimed at determining the contribution to effectiveness that these stimulus parameters carry and the equilibration of the modulation units. These data could be very instructive in informing models of multisensory integration such as the Time-Window of Integration model (TWIN) (Colonius & Diederich, 2004, 2011; Colonius, Diederich, & Steenken, 2009; Diederich & Colonius, 2009), where the termination of unisensory processes within a given window of time is likely affected by manipulations of location in addition to intensity, thus affecting the final interactive product.

Ongoing research related to this study suggests that the interdependence of the principles of multisensory integration is not restricted to space and intensity. Interactions have been observed, for example, between temporal and spatial properties of audiovisual stimuli (Macaluso, et al., 2004; Slutsky & Recanzone, 2001; Spence, Baddeley, Zampini, James, & Shore, 2003; Stevenson, Fister, et al., 2012) and between timing and effectiveness (Krueger Fister, et al., in revision). Recent evidence from animal physiology suggests that, even on a neural level, stimulus factors such as location and intensity are not independent (Carriere, et al., 2008; Krueger, et al., 2009; Royal, et al., 2009). It is perhaps not surprising that the impacts that space, timing and effectiveness have on multisensory integration are all interlinked.

That space, time and effectiveness are all interlinked makes sense given that these stimulus parameters are not dissociable in real environments. Changing the location of a multisensory stimulus, which often includes a change in distance from the observer, often leads to changes in effective intensity and its temporal structure. Multisensory integration is established early in life (Bremner, Lewkowicz, & Spence, 2012; Lewkowicz & Ghazanfar, 2009; Massaro, 1984; Massaro, Thompson, Barron, & Laren, 1986; Neil, Chee-Ruiter, Scheier, Lewkowicz, & Shimojo, 2006; Wallace, Carriere, Perrault, Vaughan, & Stein, 2006; Wallace & Stein, 1997) and is highly susceptible to the statistics of our perceptual environment (Baier, et al., 2006; den Ouden, et al., 2009; McIntosh & Gonzalez-Lima, 1998) throughout development (Carriere, et al., 2007; Polley, et al., 2008; Wallace, Perrault, Hairston, & Stein, 2004; Wallace & Stein, 2007) and even in adulthood (A. R. Powers, 3rd, Hevey, & Wallace, 2012; A. R. Powers, Hillock, & Wallace, 2009; Schlesinger, Stevenson, Shotwell, & Wallace, 2014; Stevenson, Wilson, Powers, & Wallace, 2013). Hence, in the real world, multisensory stimuli are specified by their spatial and temporal correlations; correlations that interact with (and that are partly derived from) the relative effectiveness of the stimuli in determining the degree of multisensory facilitation that will be generated.

Highlights.

• Changing location or intensity reduces stimulus effectiveness.

• Reduced stimulus effectiveness is associated with poorer performance.

• Decreasing stimulus effectiveness causes an increase in multisensory gains.

• Space and intensity interact, causing more gains for low level, peripheral stimuli.