Auditory Enhancement of Visual Phosphene Perception: The Effect of Temporal and Spatial Factors and of Stimulus Intensity

Abstract

Multisensory integration of information from different sensory modalities is an essential component of perception. Neurophysiological studies have revealed that audio-visual interactions occur early in time and even within sensory cortical areas believed to be modality-specific. Here we investigated the effect of auditory stimuli on visual perception of phosphenes induced by transcranial magnetic stimulation (TMS) delivered to the occipital visual cortex. TMS applied at subthreshold intensity led to the perception of phosphenes when coupled with an auditory stimulus presented within close spatiotemporal congruency at the expected retinotopic location of the phosphene percept. The effect was maximal when the auditory stimulus preceded the occipital TMS pulse by 40 ms. Follow-up experiments confirmed a high degree of temporal and spatial specificity of this facilitatory effect. Furthermore, audiovisual facilitation was only present at subthreshold TMS intensity for the phosphenes, suggesting that suboptimal levels of excitability within unisensory cortices may be better suited for enhanced cross-modal interactions. Overall, our findings reveal early auditory–visual interactions due to the enhancement of visual cortical excitability by auditory stimuli. These interactions may reflect an underlying anatomical connectivity between unisensory cortices.

Introduction

Information regarding external events typically reach our brain through independent sensory systems that integrate these multisensory inputs into a unified percept.

The frontal, parietal and temporal lobes of the primate brain contain neurons that respond to more than one sensory modality and consequently have been identified as sites of multimodal integration [12]. According to a hierarchical model, sensory information converges in these higher level areas through feedforward pathways arising from unimodal areas [17]. Recently, the notion that multisensory interactions occur only at the level of these high-order areas has been challenged by anatomical, neuroimaging and electrophysiological data suggesting that crossmodal structures also exert relevant feedback modulation within early, modality-specific areas [22, 34].

It is noteworthy that a variety of constraints regarding multisensory interactions in low-level unisensory cortices have been identified including spatial, temporal, as well as semantic and associative [3, 8, 22, 34, 36]. The characterization of such functional features is crucial in order to understand the causal interplay between different senses that affect brain areas and responses. Indeed, different types of constraints on multisensory interactions might likely arise at different points in time during sensory processing (in accord with different temporal windows for extracting relevant stimulus features) and may therefore reflect distinct functional mechanisms subserved by specific networks [12, 19, 20].

Based on these considerations, the present study aimed at exploring the effects of spatial (i.e. stimulus congruency) and temporal (i.e. time window) factors as well as stimulus intensity on crossmodal interactions in early visual areas. We have taken advantage of the fact that the perception of visual phosphenes can be induced by Trancranial Magnetic Stimulation (TMS). TMS delivered over the occipital cortex induces transient visual sensations (i.e. phosphenes) that occur at a precise spatial locations in visual space. The TMS intensity threshold that is needed to generate phosphene is believed to provide a measure of the excitability of the visual cortex [2, 11, 18]. By using this approach we aimed to investigate crossmodal interactions directly by studying the effect of auditory stimuli on the perception of TMS-induced visual phosphenes [2, 28, 30, 31].

Materials and Methods

Participants

Eight participants took part in each of the four experiments (Experiment 1: 8 females, mean age 22; Experiment 2: 7 females, mean age 26; Experiment 3: 6 females, mean age 25; Experiment 4: 8 females, mean age 21).

All participants were right-handed (except one in experiment 4) according to the Oldfield handedness questionnaire [27] and reported normal hearing and normal or corrected-to-normal vision. None of the participants had any contraindication to TMS. All were naïve to both the experimental procedure and the purpose of the study and gave written informed consent prior to participating. The experiment was carried out in accordance with the ethical standards of the Declaration of Helsinki (BMJ 1991; 302: 1194) and was approved by the ethical committee of the University of Milano-Bicocca. All the accepted recommendations for the use and safety of TMS were applied [32].

Phosphene-TMS screening session

Given the subjective nature of phosphene perception, an initial screening session was carried out in order to ascertain if subjects could reliably report phosphenes following single TMS pulses delivered to the occipital cortex. Only subjects who reported robust, reliable and stable phosphenes were enrolled in the study [11]. Approximately 60% of the screened subjects fulfilled these criteria, allowing for four groups of 8 participants described above.

Participants sat in an armchair wearing an elastic swimming cap. The cap was used to mark the site of stimulation and ensure reliable and reproducible coil placement throughout the experiment. Participants also wore a specially designed blindfold to prevent the possibility of any ambient light perception. Participants adapted to darkness for 15 minutes to stabilize the level of visual cortex excitability prior to performing the TMS experiment [11]. Single-pulse TMS was delivered using a Magstim Super Rapid Transcranial Magnetic Stimulator (Magstim, Whitland, UK) connected with a figure-of-eight coil (70 mm diameter). The optimal site of occipital stimulation for inducing phosphenes in the peripheral visual field was initially determined for each participant using a mapping procedure (for details, see [11]). In experiments 1,2 and 4, phosphenes were perceived in every participant at an eccentricity of at least 30° within in the periphery of the contralateral visual hemifield. In experiment 3, phosphenes were perceived at an eccentricity of less then 10° (see below). The optimal scalp position was then marked on the elastic swimming cap. This site was located in the left hemispheres for 38% of the participants in Experiment 1, for 63% of the participants in Experiment 2, for 25% of the participants in Experiment 3, for 63% of the participants in Experiment 4.

The spatial location of the perceived phosphenes was marked by the experimenter on a board placed in front of the participant. This was done in order to position the speaker delivering the auditory stimuli at exactly the location of the reported phosphene in Experiments 1-3-4 (see Figure 1).

Figure 1.

Overview of the experimental settings and results. Each panel plots the mean change of perceived phophenes (%) in the Crossmodal Trials from the Unimodal occipital TMS condition (i.e. Baseline, absence of the auditory stimulus) as a function of ISIs (TMS pulse delay with respect to Sound; negative values indicate that the sound preceded the TMS pulse) and Spatial Congruency (Same Side, i.e. black bars, vs. Opposite Side, i.e. grey bars). The crossmodal enhancement effect was apparent only for subthreshold TMS pulses combined with a spatially congruent auditory stimulation delivered at 40 ms before the TMS pulse (Experiment 1). The asterisk indicates a significant difference with respect to all the other conditions (p<0.05). Error bars represent the Standard Error (SE).

Finally, after identifying the optimal scalp position for the induction of reliable and robust phosphenes, the individual’s phosphene threshold (PT, defined as the stimulator output intensity inducing phosphenes in half of the trials) was determined [11]. The PT corresponded to a mean TMS intensity of 64% (± 12% SD) of the maximal stimulator output across all participants.

Experiment 1

Subjects remained blindfolded throughout the experiment and TMS was applied over the occipital pole at constant subthreshold TMS intensity (80% of individually defined PTs). Auditory stimuli consisted of a 20-ms burst of white noise (intensity 60 dB), delivered from two external piezoelectric loudspeakers (0.4W, 8Ω). The loudspeakers were placed on a plastic semicircular perimeter device (height 40 cm, length 200 cm) that was fixed on to the surface of a table at a distance of 40 cm from the participant. One loudspeaker was placed exactly at the same spatial location as the perceived phosphene in both the vertical and the horizontal meridians (Same Side: auditory stimulus ipsilateral to phosphenes), while the other was placed at the corresponding position in the opposite hemifield (Opposite Side: auditory stimulus contralateral to phosphenes).

During the experimental session, TMS pulses were delivered alone (Unimodal condition), or paired with the auditory stimulus presented in the right or left hemifield (Crossmodal condition). In the crossmodal conditions, the auditory stimulus could precede (−80, −60, −40 or −20 ms), follow (+20 or +40 ms) or be synchronous to the TMS pulse. These interstimulus intervals (ISIs) were chosen in light of previous work investigating the temporal profile of crossmodal interactions in nonhuman primates [35]. Catch trials, i.e. left or right unimodal auditory stimuli without TMS, were also randomly presented. The participant’s task was to press the space bar of the keyboard to indicate the perception of a phosphene.

Stimuli were delivered with an inter-trial interval that was randomly varied between 5–7 sec to minimize the potential of any carry-over effect of TMS on visual cortical excitability. In total, there were 306 trials, equally distributed in three blocks (each lasting approximately 9 minutes): 210 Crossmodal Stimuli (i.e. 15 trials for each of the 14 possible crossmodal combinations); 66 Unimodal Stimuli (corresponding to the 20% of trials) and 30 Catch Trials (10%). Sequence and timing of the stimuli and responses recording were all under computer control (E-Prime Software, Psychology Software Tools, Inc).

Experiment 2

A second experiment was conducted to further explore the effect of spatial correspondence of auditory stimuli on phosphene perception and as a function of perceived phosphene location. Here, the same procedure of the Experiment 1 was adopted with the exception that now the auditory stimuli could appear in either the same or opposite visual hemifield (Same vs. Opposite Sides), but without an exact spatial correspondence to the perceived phosphene location. The auditory stimulus was delivered from two loudspeakers located horizontally at ear level and at an eccentricity of 40° to the left and right of midline. This setting was used for every participant regardless of the perceived location of the phosphene (located on average at 30 deg in the contralateral visual field).

Experiment 3

This experiment further investigated the spatial constraints of any crossmodal interaction by inducing phosphenes in the central visual field, rather than in the visual periphery. Again, the same procedure of experiment 1 was used but now phosphenes were induced at an eccentricity of less than 10°. Given the central location of phosphenes, only one auditory stimulus was presented from a single loudspeaker placed exactly at the perceived phosphene location (i.e. Same Side).

Experiment 4

Here we investigated the effect of stimulus intensity. Following the procedure of Experiment 1, we now set the intensity of the TMS pulse at a suprathreshold level (i.e. 120% of individual PT).

Statistical Analysis

The effect of Auditory Stimulation on Phosphenes Perception was determined by analyzing the difference in the percentage of phosphene detections between Crossmodal (i.e. occipital TMS plus Auditory Stimulus) and Unimodal (i.e. TMS alone) trials (trials, Δ Accuracy = Perceived Phosphenes in Crossmodal Trials – Perceived Phosphenes in Unimodal Trials). To normalize the data distribution, the percentage of phosphene detection was then converted to the arcsin of the square root of the raw values [40].

In Experiments 1, 2 and 4, the mean change in Phosphenes Detection in Crossmodal Trials from the Unimodal Trials condition was subjected to a 2-way repeated measures ANOVA, including factors of Spatial Congruency and ISI. In Experiment 3 the ANOVA included only the Factor ISI. Pairwise comparisons were run using the Newman-Keuls test. Finally, the effect size of the ANOVA was measured by calculating the partial Eta Squared (p η²) to quantify the degree of association between an effect and the dependent variable, i.e. the proportion of the total variance that is attributable to the main factor or interaction [7].

Results

In every experiment, all subjects reliably reported phosphenes within the contralateral visual field at a constant location and with a stable PT as assessed by t-tests carried out on the mean PT for the first half versus the second half of trials (Experiment 1: First half = 36% vs. Second half = 40%, p = 0.36; Experiment 2: 41% vs. 37%, p = 0.32; Experiment 3: 39% vs. 41%, p = 0.46; Experiment 4: 68% vs. 64%, p = 0.34). The F, p, and p η² values of Main effects and interactions are summarized in Table 1.

Table 1.

Summary of experimental results. The statistical values (F statistic, p-value and p η² value from ANOVAs are shown for each experiment.

	Factors (df)	F	p-level	p η2
Experiment 1	Spatial Congruency (1,7)	3.08	0.1	0.3
	ISI (6,42)	2.87	0.02†	0.3
	Spatial Congruency * ISI (6,42)	2.54	0.04†	0.3
Experiment 2	Spatial Congruency (1,7)	0.055	0.82	0.01
	ISI (6,42)	0.58	0.74	0.08
	Spatial Congruency * ISI (6,42)	0.42	0.86	0.06
Experiment 3	ISI (6,42)	0.41	0.87	0.06
Experiment 4	Spatial Congruency (1,7)	0.1	0.75	0.02
	ISI (6,42)	0.3	0.93	0.04
	Spatial Congruency * ISI (6,42)	1.47	0.21	0.17

^†indicates a significant effect.

In Experiment 1 (peripheral phosphenes induced by subthreshold TMS), the main effect of ISI showed an improvement of phosphene perception when the auditory stimulus was delivered 40 ms before (68%) as compared to it 40 ms after the TMS pulse (55%, p < 0.05) (see Table 1).

Additionally, the significant interaction Spatial Congruency x ISI showed that phosphene perception was significantly enhanced by a spatially congruent auditory stimulus delivered 40 ms before the TMS pulse (78%, p< 0.05 for all comparisons) (Figure 1A).

In Experiment 2, in which the location of auditory stimuli was lateralized to the same hemifield but without exactly matching the phosphene location, the effect of sound was not statistically significant (Table 1). Thus, visual cortical excitability was only modulated by auditory stimulus presented in strict spatial alignment with the phosphene location (Figure 1B).

In Experiment 3, there was no evidence of crossmodal interactions for phosphenes induced within the central visual field (Table 1). This suggests that the central field representation of visual cortex was less susceptible to auditory crossmodal influences (Figure 1C).

Finally, the absence of a significant effect using suprathreshold TMS in Experiment 4 suggests that the level of visual cortical excitability within the optimal time window for the arrival of the auditory input represents a critical factor to allow for crossmodal interactions (Table 1 and Figure 1D).

Discussion

The present study shows that auditory stimulation can facilitate the perception of sub-threshold TMS-induced phosphenes. Crucially, this crossmodal facilitatory effect held a high degree of temporal and spatial specificity and was present only for low-intensity TMS pulses (i.e. below the subject’s PT for phosphenes).

We observed a critical temporal window in which crossmodal interactions induced this perceptual facilitatory effect. Specifically, auditory modulation of phosphene perception was maximal when the auditory stimulus preceded the occipital TMS pulse by 40 ms. This timing is in agreement with event-related potential recordings in humans of relatively early auditory influences on visual processing (<50 ms from stimulus onset) [13, 26].

Moreover, we found that crossmodal facilitation only occurred when specific spatial criteria were met. First, a strict spatial correspondence between the location of the auditory stimulus and that of the perceived phosphene was required. Specifically, it was not enough to present an ipsilateral auditory stimulus, but a rather rigorous audio-visual spatial alignment was necessary. Second, the effect of auditory signals occurred only for phosphenes perceived within the visual periphery (> 30 deg), but not central visual field.

Finally, the auditory enhancement effect was observed only when occipital TMS was delivered at subthreshold intensity. In contrast, suprathreshold TMS did not benefit from additional auditory stimulation, suggesting that multisensory integration depends on the relative physiological salience of the visual information.

A likely interpretation of these results is that an auditory stimulus characterized by precise spatial-temporal properties would be effective in enhancing the excitability of the visual cortex, thus enhancing perception. The spatial-temporal constraints of this facilitatory effect are in general agreement with previously reported visuo-auditory and visuo-tactile interactions described in unimodal visual cortices and heteromodal areas [6, 8, 14, 16, 22–24, 39].

The effect of stimulus intensity is reminiscent of neurophysiological evidence of an “inverse effectiveness rule” for multisensory integration. Recordings made from neurons within the cat superior colliculus have shown that the salience of the unimodal signals represents a major determinant of the advantage resulting from their integration [25]. Our results support the view that multisensory integration can lead to an enhancement in the salience of initial subthreshold events, which in turn increases the likelihood of detecting and/or responding to a presented event [37, 38]. However, a different account may suggest that the absence of crossmodal effects with salient unimodal stimuli may simply reflect a ceiling effect when using stronger unisensory inputs, or constraints from the dynamic range of neural firing [15]. However, phosphenes were detected at just 15% above threshold in the unimodal condition, suggesting that a ceiling effect is unlikely. Noteworthy for supratheshold TMS stimuli, there was even a tendency towards a suppression of phosphene perception in the crossmodal trials (from 65% in the unimodal condition to 60% in the −40 ms crossmodal condition), as opposed to the enhancement for subthreshold TMS (unimodal = 40%, crossmodal = 78%), with the mean change in the two TMS conditions being significantly different (p<0.001). Therefore, it appears that in our paradigm stimulus intensity in sensory-specific cortices might predict the extent of crossmodal interactions, with higher multisensory gain for low-level signals [3, 38]. Multisensory integration may be “optimal” in a formal sense weighting each modality’s contribution by its reliability and variability for the stimulus feature in question [1, 9].

The spatial constraints as well as the early latency of the audiovisual interactions reported here provide important clues to the underlying neurophysiological mechanisms. Previous studies proposed that multisensory integration occurs through several mechanisms, involving subcortical structures [5, 10] as well as direct and indirect cortico-cortical [4, 10, 29] and feedback connections [8, 12, 21, 34]. An auditory input activates the primary auditory cortex within 15 ms of stimulus presentation and is then transmitted to unisensory visual areas through one of two possible pathways [13, 17, 27, 36]. The first pathway consists of a direct, feedfoward projection from primary or associative auditory cortices to early visual areas [10, 29]. In this view, early crossmodal interactions could be generated in primary or secondary visual regions. This could potentially explain the predominance of crossmodal effects within the peripheral, but not central visual field found in the present study, given that anatomical connections between low-level visual and auditory areas appear to target mainly the peripheral visual field representations, whereas central visual areas have considerably weaker auditory inputs [10]. Alternatively, an indirect pathway may be implicated in which feed-forward auditory inputs reach areas of multisensory convergence (e.g. the superior temporal polysensory region in the STS) and are then transmitted via feedback connections to earlier unisensory visual areas [2, 3, 13, 17, 24, 25, 27, 28, 36].

Crossmodal influences on unimodal visual cortex (as described here) may also implicate higher-order mechanisms. Top-down modulatory influences from supramodal attentional areas, such as frontal and parietal cortices, may be important for enhancing crossmodal interactions conforming to fine-graded spatial constraints particularly for orienting attention [8, 22]. Presumably, once the combination of sensory stimuli is deemed to be behaviorally relevant (for example, by spatial congruency), supramodal attentional mechanisms may enhance crossmodal interactions through feedback modulatory connections [22].

Alternatively, our findings may also reflect retinotopic additive effects of the crossmodal stimulation on the spontaneous activity within occipital cortex. It has previously been shown that regions in human frontal cortex such as frontal-eye-filed (FEF), may exert modulatory top-down influences on retinotopic visual cortex, inducing an increase of activity in the cortical representation of the peripheral visual field, but a decrease it in the areas representing the central field [33]. This later observation is in broad agreement with the findings reported here.

Overall, the present study extends previous findings relative to the crossmodal modulation of phosphene perception [2, 28, 30, 31] highlighting some critical key features for these sensory interactions. It should be mentioned that similar studies have reported a slightly different temporal profile suggesting that auditory input can enhance visual cortical excitability over a longer time window after auditory stimulus onset [30, 31]. However, there are several methodological differences that might account for this discrepancy. First, contrary to other studies, we did not test for interactions occurring at longer latencies, as reported for example by Romei and coworkers (2007, 2009). Second, these same authors used auditory pure tones as auditory cues [31], that were always perceived centrally: Therefore the crossmodal spatial congruency effects were not assessed. Furthermore, when white-noise auditory stimuli were used [30], the chronometry of potential crossmodal interactions was not assessed. Given these methodological differences, our results are not necessarily in contradiction with previous evidence. Rather, all together they suggest that different crossmodal mechanisms could be recruited depending on the auditory stimulus selectivity, the perceptual gain as well as temporal and spatial features.

In conclusion, our findings are in agreement with current notions that crossmodal interactions facilitate the detection of single unimodal events particularly if their intensity is weak. These crossmodal effects appear to be related to spatial and temporal factors and on the overall intensity of physiological salience of the stimulus components. These constraints may provide a relevant adaptive advantage by enhancing the orienting and response to external spatial events, particularly under conditions of impoverished sensory signals.