The multifaceted interplay between attention and multisensory integration

Abstract

Multisensory integration has often been characterized as an automatic process. Recent findings suggest that multisensory integration can occur across various stages of stimulus processing that are linked to, and can be modulated by, attention. Stimulus-driven, bottom-up mechanisms induced by cross-modal interactions can automatically capture attention towards multisensory events, particularly when competition to focus elsewhere is relatively low. Conversely, top-down attention can facilitate the integration of multisensory inputs and lead to a spread of attention across sensory modalities. These findings point to a more intimate and multifaceted interplay between attention and multisensory integration than was previously thought. We review developments in our understanding of the interactions between attention and multisensory processing, and propose a framework that unifies previous, apparently discordant findings.

Bidirectional Influences of Multisensory Integration and Attention

Our brains are continually inundated with stimulus input arriving through our various sensory pathways. The processes involved in synthesizing and organizing this multisensory deluge of inputs are fundamental to effective perception and cognitive functioning. Although the combination of information across the senses has been investigated since psychology became an experimental discipline [1], the last decade has seen a sharp increase of interest in this question. Within this context, the issue of how attentional shifts in one modality can affect orienting in other modalities has inspired a great deal of this recent research [2], but, perhaps surprisingly, the role that attention plays during multisensory integration itself has been largely overlooked.

Multisensory integration generally refers to the set of processes by which information arriving from different sensory modalities (e.g., vision, audition, touch) interacts and influences processing in the other modality, including how these sensory inputs are combined together to yield a unified perceptual experience of multisensory events (Box 1). Seminal studies investigating the response patterns of single neurons [3-5] in anesthetized animals identified a number of stimulus-driven factors, most notably the temporal and spatial concordance of cross-sensory inputs, as major determinants for multisensory integration [3, 6] (see refs [3, 7, 8] for a detailed discussion). This work has inspired theoretical and empirical research on multisensory integration across a variety of species (including humans), brain structures (including cortical and subcortical networks) [9-14], and methodological perspectives [2, 15-19].

Box 1: Multisensory Integration.

In non-human single-cell recording studies, multisensory integration has been commonly characterized in terms of the firing pattern of neurons that are responsive to concurrent input from more than one sensory modality [23, 80]. Multisensory responses in these particular neurons (i.e., when input in both modalities is present) are typically strongest when the cross-modal inputs are spatially aligned, presented in approximate temporal synchrony, and evoke relatively weak responses to unisensory presentations. Interestingly, response depression has also been described in this context, following spatially misaligned inputs. As such, multisensory integration reflects relative modulations in efficacy with which sensory inputs are processed and can compete for processing resources [23]. Neurons exhibiting these properties have been most extensively studied in the superior colliculus, a midbrain structure that contains multisensory convergence zones [4, 5] and is involved in the reflexive orienting of attention, both covert and overt, towards salient stimuli.

In keeping with animal physiology, human behavioral, electrophysiological, and neuroimaging studies have shown that spatiotemporally concordant inputs to multiple modalities often result in non-linear (super- or sub-additive) brain potentials and faster response latencies [8, 26]. In addition, several other aspects of multisensory integration have been identified in human research. Some of these aspects were originally hinted at by the modality appropriateness hypothesis [29] and the unity assumption [30, 31]. The modality appropriateness hypothesis posits that, in the case of bimodal stimulation, the sensory system with the higher acuity with respect to the relevant task plays a dominant role in the outcome of multisensory integration [32-36]. For example, the visual system usually dominates audiovisual spatial processes because it has a higher spatial acuity than the auditory system, whereas the auditory tends to impart more cross-modal influence in terms of temporal analysis. Recent approaches based on Bayesian integration have allowed formalising this idea under a computational framework [15, 16]. The unity assumption, on the other hand, relates to the degree to which an observer infers (not necessarily consciously) that two sensory inputs refer to s single unitary distal object or event [29, 31]. Both non-cognitive factors (temporal and spatial coincidence) and cognitive factors (prior knowledge and expectations) are thought to contribute to this process [30, 31, 66, 81].

Other behavioral studies in the human have focused on multisensory illusions that result from incongruency across the sensory modalities (i.e., cross-modal conflict; [82]) The ventriloquist phenomenon refers to the well-known illusory displacement of a sound toward the position of a spatially disparate visual stimulus [33, 83, 84], a dramatic effect that has been applied in a wide range of settings, within and beyond a purely scientific enquiry [85]. Another example is the McGurk effect [53], an illusion that occurs when speech sounds do not match the sight of simultaneously seen lip movements of the speaker, leading to a perception of a phoneme that is different than both the auditory and visual inputs. These illusions underscore the strong tendency to bind auditory and visual information that under normal (congruent) circumstances, helps reducing stimulus ambiguity.

The question within the scope of this review is how these stimulus-driven multisensory interactions interface with attentional mechanisms during the processes that lead to perception. Attention is a relatively broad cognitive concept that includes a set of mechanisms that determine how particular sensory input, perceptual objects, trains of thought, or courses of action are selected for further processing from an array of concurrent possible stimuli, objects, thoughts and actions (Box 2) [20]. Selection can occur in a top-down fashion, based on an item's relevance to the goals and intentions of the observer, or in a bottom-up fashion, whereby particularly salient stimuli can drive shifts of attention without voluntary control [21]. Interestingly, temporally and spatially aligned sensory inputs in different modalities have a higher likelihood to be favored for further processing, and thus to capture an individual's attention, than do stimuli that are not aligned [22-25]. This suggests that attention tends to orient more easily towards sensory input that possesses multisensory properties. Indeed, recent studies have shown that multisensory bottom-up processes can lead to a capture of attention [24]. From these findings, one might infer that the output of multisensory integration precedes attentional selection, and thus that it operates in a pre-attentional and largely automatic fashion.

Box 2: Attention.

Attention is an essential cognitive function that allows humans and other animals to continually and dynamically select particularly relevant stimuli from all the available information present in the external or internal environment, so that greater neural resources can be devoted to their processing. One relatively straightforward framework encapsulating many aspects of attention is that of biased competition [86]. According to this framework, attention is considered to be the process whereby the competing neural representations of stimulus inputs are arbitrated, either due to the greater intrinsic salience of certain stimuli or because they match better to the internal goals of the individual. Thus, attention may be oriented in a top-down fashion [87, 88], in which a selective biasing of the processing of events and objects that match the observer's goals is induced. In contrast, bottom-up, or stimulus-driven, attentional control refers to a mostly automatic mechanism in which salient events in the environment tend to summon processing resources, relatively independently of higher level goals or expectations [21, 89].

A fronto-parietal network of brain areas has been shown to be involved in allocating and controlling the direction of top-down attention by sending control signals that modulate the sensitivity of neurons in sensory brain regions [90-94]. Stimulus-driven attention employs various parts of the same network, appearing to operate in concert with subcortical networks that include the superior colliculus [92, 95] and regions in the right temporal-parietal junction [92].

It is still not well understood how top-down attention is controlled or allocated across the sensory modalities. Considerable overlap can be found in brain areas responsible for the top-down orienting of visual [91] and auditory [96] attention. In addition, attending to a specific location in one modality also affects the processing of stimuli presented at that location when presented in another sensory modality, suggesting that spatial attention may tend to be directed in a supramodal, or at least modality-coordinated fashion [56, 59, 62, 97, 98]. These multisensory links indicate that there is flexible attentional deployment across sensory modalities [99], making it unlikely that there are completely independent control mechanisms for each modality. It has been proposed that top-down spatial selective attention operates by increasing the sensitivity of neurons responsive to the attended stimulus feature [100, 101]. This mechanism might explain why attentional orienting can be operated in parallel across modalities, whereas attentional resolving - that is, the processing of relevant information within each modality - may be carried out more or less independently within each modality [102-104].

Complementing these findings, however, are behavioral, electrophysiological [8, 26, 27] and neuroimaging studies [28] in humans that have identified several higher level factors, such as voluntarily oriented spatial attention and semantic congruency, that can robustly influence how integration of information across the senses occurs (see Box 1) [29-31]. Some of these factors were originally hinted at by the modality appropriateness hypothesis [29]. This hypothesis posits that in the case of bimodal stimulation, the sensory system with the higher acuity with respect to the most critical aspects of a task plays a predominant role on how multisensory inputs are integrated [32-36] – e.g., visual for spatial characteristics, auditory for temporal aspects (see Box 1). Additional factors are related to the unity assumption [30, 31], which corresponds to the degree to which observers infer (consciously or not) that two sensory inputs originate from a single common cause [29, 31] (see Box 1). These, and other findings showing robust influences of cognitive factors on multisensory integration suggest a more flexible process than previously thought, whereby attention can affect the effectiveness of multisensory integration in a top-down fashion [2, 37-45].

How can these apparently opposing ideas regarding the interplay between attention and multisensory integration (i.e., bottom-up, highly automatic, multisensory integration processes that can perhaps capture attention vs. top-down attentional mechanisms that can modulate multisensory processing) have evolved in the literature? One possible reason is that studies investigating the role of top-down influences of attention on multisensory integration and studies investigating the bottom-up influence of multisensory integration on the orienting of attention have been conducted more or less independently from each other. This disconnection may thus have led to opposing, seemingly mutually exclusive conclusions. Considering recent findings from the fields of cognitive psychology and cognitive neuroscience, we propose a framework aimed at unifying these seemingly discordant findings (Figure 1).

Figure 1. A framework for the interactions between multisensory integration and attention.

(a) Sequence of Processing Steps. Inputs from the sense organs are thought to interact cross-modally at multiple phases of the processing pathways, including at very early stages [8]. Stimuli can be integrated automatically if a number of conditions are satisfied: (1) if initial saliency of one of the stimuli is at or above a critical threshold, preprocessing stages will attempt to spatio-temporally realign this stimulus with one of lesser salience [(2); spatio-temporal realignment]. The stimulus stream will then be monitored for congruence in stimulus patterns of the matched streams [(3); congruency detection]. If the realignment and/or congruency matching processes succeed, the neural responsiveness of brain areas in charge of processing the input streams will be increased [(4); recurrent stimulus driven sensitivity adjustments] to sustain the integration process. If stimuli cannot be realigned or when incongruency is detected, the sensory gain would tend to be decreased. Note that we consider these potential gain adjustments to be mainly stimulus driven, and therefore a reflection of the bottom-up driven shift part of the interaction between multisensory integration and attention. If none of the stimuli is of sufficient saliency, top-down attention may be necessary to set up an initial selection of to be-integrated-stimuli [(5);top-down sensory gain adjustments]. The resulting boost in sensory sensitivity due to a top-down gain manipulation might then be sufficient to initiate the processes of spatial temporal alignment and congruency matching that would otherwise not have occurred. In addition, top-down attention can modulate processing at essentially all of these stages of multisensory processing and integration. b) Three examples of interactive influences between multisensory integration and attention: (1) bottom-up multisensory integration, which can then drive a shift of attention, (2) the need for top-down attention for multisensory integration in the presence of many competing stimulus representations; (3) the spreading of attention across space and modality (visual to auditory).

A Framework for Interactions between Attention and Multisensory Integration

As noted above, and as reviewed in the sections to follow, the evidence in favor of the existence of bidirectional influences between attention and multisensory integration is considerable. Under certain circumstances the co-occurrence of stimuli in different modalities can lead preattentively and automatically to multisensory integration in a bottom-up fashion, which then makes the resulting event more likely to capture attention and thus, capitalize available processing resources [24]. In other situations, however, top-down directed attention can influence integration processes for selected stimulus combinations in the environment [40, 46]. Based on the pattern of findings in the literature, we propose a framework in which a key factor that determines the nature and directionality of these interactions is the stimulus complexity of the environment, and particularly the ongoing competition between the stimulus components in it.

More specifically, we propose (Figure 1a) that multisensory integration can, and will tend to occur more or less pre-attentively in a scene when the amount of competition between stimuli is low. So, for example, under circumstances in which the stimulation in a task-irrelevant sensory modality is sparse (Figure 1b, case 1), the stimuli in that modality will tend to be intrinsically more salient just because they occur infrequently. Because of their increased salience, these stimuli will tend to enhance the perceptual processing of corresponding (i.e., spatially and/or temporally aligned) sensory events in a concurrently active task-relevant modality. This stimulus-driven (i.e., bottom-up) cross-modal effect will tend to capture attention and processing resources, thereby facilitating the ability for attentional selection and/or attentional “resolving” (i.e. discrimination; see Box 2) of stimuli in the task-relevant modality. This multisensory-based enhancement can occur even in a context where attentional selection in the task-relevant sensory modality is challenging, such as in a difficult visual search task [24]. In contrast, when multiple stimuli within each modality are competing for processing resources, and thus the saliency for individual stimuli within the potentially facilitating modality is low, top-down selective attention is likely to be necessary in order for multisensory integration processes between the appropriately associated stimulus events to take place [41, 42, 46] (Figure 1b, case 2).

We focus below on three main forms under which these interactions between multisensory integration and attention can manifest themselves: (1) The Stimulus-driven influences of multisensory integration on attention; (2) The Influence of top-down directed attention on multisensory integration; and (3) The Spreading of attention across modalities, which contains aspects of both top-down and bottom-up mechanisms.

Stimulus-driven Influences of Multisensory Integration on Attention

A clear demonstration of the possible involvement of stimulus-driven multisensory integration on attentional selection was recently provided using a difficult visual search task [24]. In this study visual targets were presented amongst an array of similar distractor stimuli (see Figure 2). Search times for the visual target increased with an increasing number of display items (distractors), as has typically been found in demanding unisensory visual search tasks [47, 48]. In these displays, targets and distractors changed color at random moments and irrelevantly with respect to the task. However, when a brief, spatially uninformative sound was presented concurrently with a target color change, search times became much faster and relatively independent of the number of distractor items, providing evidence that the auditory co-occurrence made the visual target pop out from the background elements.

Figure 2. Multisensory integration mechanisms affecting visual attention in a bottom-up driven fashion.

(a) Cluttered displays containing a variable number of short line segments were presented. Display elements continuously changed color from green to red (or vice versa) at random moments and locations. A short tone pip could be presented simultaneously with the color change of the target element. Participants were required to detect and report the orientation of the target element, consisting of a horizontal or vertical line amongst ±45° tilted line distractors. (b) In the absence of a sound, search times increased linearly with the number of distractor items in the display (white squares). In contrast, when the sound was present, search times became much shorter and independent of set size, indicating that the target stimulus popped out of the background (black squares). (c). Search times as a function of the relative stimulus onset asynchrony (SOA) between color change of the target, and onset of the sound. Negative SOAs indicate that the tone preceded the visual target event, and positive SOAs indicate that the target event preceded the tone. (Data from condition with set size fixed at 48 elements). Adapted, with permission, from ref [24].

At first glance these findings might seem to contradict those of another visual-search study, in which auditory stimuli failed to facilitate the detection of a uniquely corresponding (temporally matched) visual event amongst visual distractors [49]. Here, however, not only the visual stimuli but also the accessory acoustic events changed at relatively high frequencies, and thus were all consequently of low saliency. These two findings suggest that a co-occurring sound renders it likely that a visual target pops out only when the sound is relatively rare and therefore a salient event by itself [50]. This conclusion is also consistent with data from another study, which showed that a sound can affect the processing of a synchronized visual input when it is a single salient event, but not when it is part of an ongoing stream [51].

The framework we are proposing here accounts for such bottom-up multisensory integration mechanisms because a relatively salient stimulus presented in one modality can elicit a neural response that is strong enough to be automatically linked to a weaker neural response to a stimulus in another modality (Figure 1b, case 1). We hypothesize that multisensory processes triggered by temporally aligned auditory input can affect the strength of the representation of the co-occurring visual stimulus, making it pop out of a background of competing stimuli. In contrast, when multiple stimuli are competing for processing resources, their neural representations may require top-down attention for the relevant cross-modally corresponding stimuli to be readily integrated and their processing effectively facilitated.

Influence of Top-Down Directed Attention on Multisensory Processing

As mentioned above, recent studies have provided evidence for the influence of top-down attention on multisensory integration processes. In a recent EEG study, for example, it was shown that spatial attention can strongly influence multiple stages of multisensory processing, beginning as early as 80 ms post-stimulus [40]. Complementarily, a recent fMRI study [34] showed increased activity in multiple brain areas, including the superior temporal sulcus, striate visual cortex, extrastriate visual cortex, and the superior colliculus, when attention was directed to visual lip movements that matched an auditory spoken sentence, compared to when attention was directed to simultaneously presented but non-matching lip movements (Figure 3). Thus, these results again provide evidence for the ability of top-down attention to influence multisensory integration processes. They are also consistent with EEG data from another study showing that allocating visual attention towards irrelevant lip movements in the visual scene interferes with the recognition of audiovisual speech signals from an attended speaker [52]. Together, these results demonstrate that top-down attention can modulate multisensory processing leading to either facilitation (in the case of rhymically congruent inputs) or interference (in the case of rhythmically incongruent inputs).

Figure 3. Effects of top-down spatial attention on audiovisual speech perception.

(a) Layout of the experiment. Sounds were presented from one central location, while two visually and laterally presented streams of lip-moments were played. One of these streams was congruent with the auditory speech signals while the other stream was incongruent. (b) Attention was selectively oriented to either the left of right visual streams, either of which could, in turn, be congruent or incongruent with the auditory speech stimuli. (c) Attending to the congruent stimuli resulted in increases in activation in several brain areas that are typically associated with multisensory integration. These areas included the superior temporal sulcus, and superior colliculus (central panel), as well as large parts of the retinotopically organized visual areas V1 and V2. Adapted, with permission, from ref [39].

Further evidence for the impact of top-down attention on multisensory processing derives from studies using audiovisual speech illusion paradigms [42-44]. For instance, diverting attention to a secondary task reduces susceptibility to the McGurk effect [53], whereby an auditory phoneme dubbed onto incongruent visual lip movements leads to an illusory auditory percept (see Box 1). Interestingly, this reduced susceptibility to the McGurk effect due to the allocation of processing resources to another task was observed regardless of the sensory modality (visual, auditory, or tactile) in which the secondary task was performed [42, 43]. This result suggests that a cognitive manipulation that diverts limited processing resources away from the stimulus input from one sensory modality from a perceptual event can also result in reduced integration of it with the input from other sensory modalities [54]. Thus, when competition between different stimulus sequences is relatively low, we propose that audiovisual speech stimuli may integrate relatively automatically. However, when a secondary task is introduced, involving a diversion of attention from the event, competition ensues between the neural resources devoted to the secondary task and those resources necessary for processing the audiovisual speech stimuli. This, in turn, may reduce the automaticity of the congruency detection, and consequently, of the multisensory integration process.

It is important to distinguish these above-described top-down attentional effects from some earlier behavioral [55-59] and electrophysiological [60-62] work on supramodal attention effects. In particular, earlier studies showed that top-down spatial attention in one modality not only induces enhanced processing of stimuli in that modality at the attended location, but also of stimuli in other (task-irrelevant) modalities at that location – for example, attending visually to a location in space results in the processing of auditory stimuli at that location also being enhanced. Other studies have shown that salient stimuli in one modality in a particular location can capture spatial attention and affect the processing of stimuli in another modality that occur shortly afterward at that same location [56, 59]. These latter findings suggest that spatial attention tends to be coordinated across modalities, both when voluntarily directed and when induced in a bottom-up cueing fashion. In both cases, however, the stimuli in the different modalities do not need to be presented concurrently to be enhanced – rather, they just need to occur in the spatially attended location. Thus, although such effects reflect fundamental mechanisms by which top-down factors modulate processing across modalities, they contrast with the studies mentioned above in which top-down attention modulates the actual integration of multisensory inputs that occur aligned in time [63]).

Cross-modal Spreading of Attention

Another way that attention can interactively influence the processing of stimulus inputs in different sensory modalities that do occur at the same time has been termed cross-modal spreading of attention, which has been found to occur even when the visual and auditory stimuli arise from different spatial locations [37]. More specifically, it has been shown that a task-irrelevant, centrally presented, auditory stimulus can elicit a different brain response when paired in time with an attended versus an unattended visual stimulus presented laterally (Figure 4) [37]. This effect appeared as a prolonged, negative-polarity, fronto-central ERP deflection, beginning at ∼200 ms post-stimulus and lasting for hundreds of milliseconds, as well as a corresponding enhancement of fMRI activity in auditory cortex. The ERP effect resembled an activation known as the “late processing negativity,” a hallmark of the enhanced processing of attended unisensory auditory inputs [64]. This auditory enhanced processing effect was particularly striking because it was induced solely by whether a simultaneously occurring visual stimulus in a different spatial location was attended versus unattended. Such a result is thus consistent with a spread of attention across modalities and space to the temporally coincident auditory stimulus, occurring via a systems-level cascade of top-down and bottom-up influences. This cascade begins with top-down visuospatial attention that determines which stimulus in the visual modality is to be selectively processed. Then, presumably by means of an automatic (bottom-up) linking mechanism derived from the temporal coincidence of the multisensory components, attention spreads across modalities to encompass the auditory component, despite it being completely task-irrelevant and not even in the same spatial location.

Figure 4. Spreading of attention across a multisensory object.

(a) Experimental Design. Visual stimuli were flashed successively, and in random order, at left and right hemifields. On half of the trials, a task-irrelevant central tone was presented synchronously with the lateral visual stimulus. Participants were instructed to visually attend selectively to only one of the two locations. (b) ERP subtraction procedure and key results. ERPs elicited by attended visual stimuli that occurred alone were subtracted from ERPs elicited by attended visual stimuli that were accompanied by a tone, yielding the extracted ERP response to the central tones when they occurred in the context of an attended (lateral) visual stimulus. A similar subtraction procedure was applied to unattended-visual trials in order to yield the extracted ERP response to the central tones when they occurred in the context of an unattended (lateral) visual stimulus. An overlay of these two extracted ERPs showed that tones presented in the context of an attended visual stimulus elicited a prolonged negativity over fronto-central scalp areas beginning at around 200 ms poststimulus. This effect resembles the late auditory processing negativity, which is a hallmark neural effect elicited during intramodal auditory attention. (c) fMRI results from the same paradigm. These results, extracted using an analogous contrast logic, showed that auditory stimuli presented in the context of an attended auditory stimulus yielded an increase in brain activity in the auditory cortex, compared to the activation elicited by the same tone when it was presented in the context of an unattended visual stimulus. The enhanced processing of task-irrelevant auditory stimuli that occur simultaneously with an attended visual stimulus, even one occurring in a different location, suggests that the visual attention has spread across the components of the multisensory object to encompass the auditory part. Adapted, with permission, from ref [37]. Copyright © 2005, The National Academy of Sciences.

Variants of this effect have recently been observed in other studies [38, 41, 65]. Moreover, a distinction has been made between the object-based multisensory spread of attention that occurs when stimuli are associated simply due to their co-occurrence in time, and a representation-based spread of attention for naturally associated stimulus pairs [38]. On the other hand, in some circumstances a task-irrelevant auditory stimulus that is semantically or representationally in conflict with a task-relevant visual stimulus can serve as an attention capturing distractor and increase attentional spread [66, 67]. Regardless, the currently available data suggest that the spatial-temporal linking of visual and auditory stimulation is initially organized at an early processing stage [37, 41, 68], but that higher-level cognitive representations can also modulate the cross-modal spreading of attention [38, 66, 67, 69].

In some ways, the spread of attention [37] can be considered to be a reversed version of the sound-induced (i.e., bottom-up) visual pop-out effect described earlier [24]. In the spread-of-attention case [37], it is visuospatial attention that determines what becomes more salient in the auditory world (Figure 1b, case 3). In the sound-induced visual pop-out case [24], an auditory stimulus determines what becomes more salient in the visual world (Figure 1b, case 1). Regardless, stimuli in one modality, or attention to stimuli in one modality, selectively influence the processing of co-occurring stimuli in another modality. Thus, these studies underscore the multifaceted --and bi-directional-- ways by which attention and multisensory processing can interact [70].

Concluding remarks and future directions

While it is generally acknowledged that multisensory integration processes can influence the bottom-up orienting of attention to salient stimuli [23], the role of top-down attention and the interplay between bottom-up and top-down mechanisms during multisensory integration processes remains a matter of ongoing debate. On the one hand, there is relatively little effect of top-down attention on multisensory integration under conditions of low competition between stimuli [33]. On the other hand, robust top-down effects on multisensory integration processes have been observed under higher degrees of competition between successive or concurrent inputs to different modalities [40, 46]. Accordingly, we suggest that the degree of competition between neural representations of stimulus inputs is a major determinant for the necessity of top-down attention, thereby helping to explain the contradictory findings in the literature.

As such, we identify several important directions and challenges for future research (see Box 3). One promising direction will be to develop multisensory attention paradigms that systematically modulate the degree of competition between stimuli within an experiment, for instance by manipulating perceptual load [71] or stimulus delivery rates, while simultaneously manipulating top-down attentional factors. Such an approach would likely be helpful for determining more precisely which aspects of multisensory integration may be affected by the ongoing interplay between sensory input competition and top-down attentional influence, and the mechanisms by which these interactions occur.

Box 3: Questions for Future Research.

• What is the effect of stimulus competition on the need for top-down attention in multisensory integration?

• Is there a differential influence of spatial and non-spatial attention on multisensory integration, and if so, which stimulus properties are affected?

• What are the neural mechanisms by which competition influences multisensory integration and how can attention influence these mechanisms?

• Does the complexity of a perceptual task influence the requirement of attention in multisensory integration, or is the attentional requirement only a function of the complexity of the stimulus input array regardless of the task.

• What are the circumstances and mechanisms that control whether the processing of a task-relevant stimulus in one modality is enhanced versus diminished by a task-irrelevant conflicting stimulus in a second modality?

• What is the relationship between the spread of attention and sound-induced visual pop-out effects?

• What role does attention play in cross-modal processing deficits in populations with neuropsychological or psychiatric disorders, such as autism and schizophrenia?

• How does the relationship between attention and multisensory integration change across the lifespan?

Another challenge lies in more fully and precisely delineating the circumstances in which attention affects multisensory integration, and the mechanisms by which it does so. For instance, top-down visuo-spatial attention can have a strong impact on the processing of concurrent auditory stimuli, as shown by the visual-to-auditory spreading-of-attention effect. However, it appears that spatial attention in the auditory modality has a much less pronounced effect on the integration of visual stimuli that are not presented at the same location. Auditory inputs, and attention to them, can nevertheless affect the processing of visual stimuli, but particularly in terms of their temporal characteristics. This tendency for temporal dominance of the auditory modality, presumed due to the importance of temporal processing in auditory processing, is reflected by the sound-induced double-flash illusion [44] (see Box 1), as well as other data suggesting that auditory attention tends to temporally align the processing of concurrent visual stimuli [72].

We note that the vast majority of studies on multisensory processing and attention reviewed here have used audio-visual stimulus material. While vision and audition have by far been the most frequently investigated senses in multisensory research, it will be important in the future to delineate which aspects of the findings and framework presented here can be generalized to other combinations of sensory modalities (e.g. touch, smell, taste, or proprioception).

Finally, research on the relationship between multisensory processing and attention may also have critical implications for our understanding of multisensory integration deficits in neurological and psychiatric populations, such as in those with hemispatial neglect, schizophrenia [73], and autism [74, 75], as well as multisensory integration decrements in the aging population [76, 77]. For example, many clinical symptoms in schizophrenia seem to be the result of a pathological connectivity between cortical networks, including between sensory processing networks, during cognitive tasks [78]. It is therefore possible, perhaps even likely, that an altered interplay between attention and multisensory processes will be found in this group of patients [79]. Thus, the study of these interactions in clinical populations is important for the understanding of deficits in these populations, as well as for advancing our understanding of the underlying neural mechanisms.

We live in a multisensory world in which we are continually deluged with stimulus input through multiple sensory pathways. For effective cognitive functioning, we must continually select and appropriately integrate together those inputs that are the most relevant to our behavioral goals from moment to moment. Thus, the dynamic and bi-directional interplay between attentional selection and multisensory processing is fundamental to successful behavior. This review has aimed at organizing the present state of the research in this important area of cognitive science, and at developing a conceptual framework that we hope will be helpful for future research.

Glossary

Attentional Orienting

Attention involves mechanisms whereby processing resources are preferentially allocated toward particular locations, features, or objects. Attentional orienting refers to the process responsible for moving the focus of attention from one location, feature, or object, to another. Orienting can occur covertly, that is, in the absence of movements of the eyes or the other the sensory receptor surfaces (e.g., eyes, ears), as well as overtly, where the shift is accompanied by a reorienting of the sensory receptors (e.g., by a head turn) to the newly attended location or object.

Bottom-up

A form of stimulus-driven selection that is mainly determined by the ability of sensory events in the environment to summon processing resources. This type of selection is invoked relatively independently of voluntary control; rather, stimulus salience (see entry, below) is the driving factor. Particularly salient stimuli (i.e., sudden motion in an otherwise still visual scene or loud sounds in an otherwise quiet room), or other stimuli for which an individual has a low detection threshold (e.g. one's own name), attract attention in a bottom-up fashion.

McGurk effect

Audiovisual illusion, where an auditory phoneme (i.e., /b/) dubbed onto incongruent visual lip movements (i.e., [g]) tends to lead to illusory auditory percepts that are typically intermediate between the actual visual and auditory inputs (i.e., /d/), completely dominated by the visual input (i.e., /g/), or a combination of the two (i.e., /bg/). The McGurk effect occurs in the context of isolated syllables, words or even whole sentences.

Modality Appropriateness

This framework about multisensory integration is based on the fact that some stimulus characteristics are processed more accurately in one sensory modality than in another. For instance, vision in general has a higher spatial resolution than audition, whereas audition has a higher temporal resolution than vision. According to this framework, information from visual stimuli tends to dominate the perceptual outcome of the spatial characteristics of audiovisual events (sometimes causing a shift of the apparent location of an auditory stimulus toward the location of the visual event). Conversely, the perceived temporal characteristics of an audiovisual event tend to be dominated by those of the auditory component.

Pop-out

A perceptual phenomenon whereby a stimulus with a particularly distinctive feature relative to its surrounding background triggers quick attentional orienting and leads to rapid detection. It is often used to describe the fact that finding such particularly distinctive objects within a visual display is highly efficient and not affected by the amount of distractor elements in the scene. High stimulus salience (see entry below) leads to pop-out.

Resolving

The process of extracting relevant information from an attended stimulus.

Retinotopic

Spatial organization of a group of neurons based on a topographical arrangement whereby their responses map stimulus locations in the retina in a more or less orderly fashion across a brain area. In a retinotopically organized brain area, neurons involved in processing adjacent parts of the visual field are also located adjacently. This organization is most clearly seen in early (i.e., lower-level) areas of the visual pathway, but many higher-order cortical areas involved in processing visual information also show rough retinotopic organization.

Salience

Refers to a characteristic of an object or event which makes it stand out from its context. Visual objects are said to be highly salient when they have a particularly distinctive feature with respect to the neighboring items and the background, or if they occur suddenly. A bright light spot within an otherwise empty, dark context has a high saliency. Salience is often associated with being more likely to capture attention (see Bottom-up entry).

Sound-induced double-flash illusion

An audiovisual illusion in which a single flash of light, presented concurrently with a train of various (two or three) short tone-pips, is perceived as two (or more) flashes. This phenomenon is an example of the tendency of auditory stimulus to dominate in the perception of the temporal characteristics of an audiovisual event.

Stimulus congruence

The match of one or more features across two stimuli, stimulus components, or stimulus modalities. Congruence can be defined in terms of temporal characteristics, spatial characteristics, or higher-level informational content (such as semantics). In audiovisual speech perception, congruency typically refers to the matching or mismatching of a sequence of auditory speech sounds with respect to lip movements being concurrently presented. Incongruence is at the base of some multisensory phenomena, such as the McGurk illusion and the ventriloquist effects (see Box 1).

Stimulus-driven

Is a defining feature of Bottom-up processing (see entry above), and refers to a process that is triggered or dominated by current sensory input.

Top-down

A mode of attentional orienting whereby processing resources are allocated according to internal goals or states of the observer. It is often used to refer to selective processing and attentional orienting triggered in a voluntary fashion.

Ventriloquism effect

An audiovisual illusion in which the apparent location of an auditory stimulus is perceived as closer to the location of a concurrent visual cue than is actually the case.