Dissecting neural circuits for multisensory integration and crossmodal processing

Abstract

We rely on rich and complex sensory information to perceive and understand our environment. Our multisensory experience of the world depends on the brain's remarkable ability to combine signals across sensory systems. Behavioural, neurophysiological and neuroimaging experiments have established principles of multisensory integration and candidate neural mechanisms. Here we review how targeted manipulation of neural activity using invasive and non-invasive neuromodulation techniques have advanced our understanding of multisensory processing. Neuromodulation studies have provided detailed characterizations of brain networks causally involved in multisensory integration. Despite substantial progress, important questions regarding multisensory networks remain unanswered. Critically, experimental approaches will need to be combined with theory in order to understand how distributed activity across multisensory networks collectively supports perception.

1. Introduction

Much of our knowledge of how networks of neurons contribute to perception and cognition is based on the ability of systems neuroscientists to correlate the activity of neurons with both the sensory stimulus and behavioural outcomes. For example, trial-by-trial correlations of the responses of sensory neurons with perceptual decisions have been used to probe for functional roles in perception [1,2]. However, a crucial test of any model relating neural activity to behaviour involves interrogating how behaviour changes upon causal manipulation of these networks. Large strides have been made in understanding basic neural circuits using such neuromodulatory techniques in recent years. Although studies of sensory processing have traditionally been performed one sense at a time, there is also a long and parallel history of multisensory research.

In this review, we highlight some of the most important discoveries in the field of multisensory and crossmodal processing derived from causal manipulations of brain activity. Multisensory integration refers to the process by which information from different sensory modalities is combined by the nervous system to form a stable and coherent percept of the world. Multisensory integration enhances our ability to perceive and understand our environment, enabling us to move and interact with objects in our surroundings. Crossmodal processing, as used in this review, refers to the influence of stimulation in one sensory modality on neural processing mechanisms supporting a different modality (e.g. auditory stimulation influencing neural activity in visual cortex).

Progress to date has been based on a multitude of techniques, including psychophysics, neurophysiology and neuroimaging. Here we will provide a brief synopsis of how neuromodulation approaches (figures 1 and 4) have built on these techniques to (i) provide a systems-level understanding of the areas involved in multisensory integration and the areas providing sensory inputs to these integrators; (ii) demonstrate that multisensory integration can be a distributed property across cortical networks; (iii) show that sensory areas traditionally considered to be dedicated to a single modality can be multimodal, with crossmodal responses being modulatory or serving explicit and specific functions; and (iv) demonstrate how multisensory integration can depend on microcircuit interactions between particular cell types.

Figure 1.

Non-invasive brain stimulation techniques. (a) With transcranial magnetic stimulation (TMS), current passed through wire loops (red arrows) creates a brief (100–200 µs), yet strong magnetic field (peak strengths of 1–2 T, blue dashed lines) that can induce electrical currents (yellow patch) non-invasively in nearby brain tissue [3]. TMS spatial specificity and penetration depth vary depending on coil size and geometry—commonly used coils achieve spatial resolutions on the order of 1 cm² on the brain surface with field strength decaying exponentially with distance from the coil. TMS is used in ‘online’ modes to acutely disrupt neural activity patterns in the targeted brain region. TMS is used in ‘offline’ modes to modulate cortical excitability—the modulatory effect depends on stimulation parameters such as the temporal rate and pattern. (b) With transcranial direct current stimulation (tDCS), low-amplitude (typically 1–2 mA) electrical currents (yellow arrow) passing between surface electrodes (red and blue boxes) modulate excitability of brain tissue residing beneath the electrodes [4]. Prolonged anodal or cathodal tDCS administration can induce persistent potentiation or depression of cortical excitability, respectively. Current spread depends on a variety of factors including electrode size (typically 10–40 cm²) and arrangement.

Figure 4.

Invasive neuromodulation techniques. (a) With cooling deactivation, cortical tissue temperature is reduced to block neural activity temporarily [71]. (b) With chemical deactivation, pharmacological agents are used to block neural activity temporarily [72]. (c) With electrical microstimulation, low-amplitude electrical currents are delivered through microelectrodes to drive neural activity in small groups of neurons [73]. Microstimulation effects can depend strongly on stimulation parameters like amplitude, pulse width, pulse train duration and pulse train frequency. (d) With optogenetics, light is used to drive activity in neuron populations that have been genetically sensitized to light through the expression of light-sensitive proteins [74]. Optogenetics can be used to excite or inhibit specific cell types or neuron populations generally.

Because neuromodulatory techniques differ between animal and human models of multisensory function, we first outline recent discoveries in multisensory processing in humans (see also, [5]), followed by a brief review of parallel studies in animals.

2. Summary of advances using neuromodulatory techniques in human studies

This section focuses exclusively on studies that have used two non-invasive brain stimulation (NIBS) methods: transcranial magnetic stimulation (TMS) and transcranial direct current stimulation (tDCS). Here we provide a brief overview of these methods and interested readers are directed to detailed descriptions of TMS, tDCS and related NIBS methods [3,4,6,7]. The sections following the TMS and tDCS overview then review studies employing these methods to investigate the multisensory and crossmodal properties of (i) higher-order association regions like the posterior parietal cortex (PPC) and the superior temporal sulcus (STS), and (ii) sensory cortex.

TMS (figure 1a) is used to modulate neural activity via electromagnetic induction. A high-voltage current is briefly passed through a coil to generate a rapidly changing magnetic field—this time-varying field induces electrical current in the brain (according to Farraday's induction principle) when the coil is positioned against the scalp. TMS-induced currents interact with the targeted brain region's endogenous neural activity to temporarily alter neural processing. Accordingly, in an ‘online’ mode of use, single-pulse TMS (spTMS) or brief trains of repetitive TMS (rTMS) are used to induce acute changes (most often perturbations) in motor, perceptual or cognitive performance in order to demonstrate causally the targeted brain region's functional involvement in the task. Beyond acutely influencing neural activity, prolonged application of rTMS can also induce changes in neuronal excitability that persist after TMS cessation. Depending on the temporal rates or patterning of the stimulation trains, rTMS can induce long-term potentiation- and long-term depression-like changes in cortical excitability. For instance, with regularly patterned rTMS, pulse frequency generally determines the direction of neuroplastic change: low frequency rTMS (less than 1 Hz) typically results in decreased cortical excitability while high frequency rTMS (greater than 1 Hz) leads to increased cortical excitability. Because cortical excitability changes can influence (by enhancing or impairing) perception and behaviour, rTMS paradigms are used in ‘offline’ TMS studies to characterize the functions of different brain regions.

With tDCS (figure 1b), electric currents can be used directly and non-invasively to stimulate neural tissue. tDCS involves delivering continuous and prolonged low-amplitude electric current (typically 1–2 mA) to brain tissue via surface electrodes. Although much of the current is shunted through the scalp and cranium, substantial currents are passed to cortical tissue residing immediately beneath the electrodes and these currents can alter the membrane potential of neuron cell bodies or axon terminals to influence signal transmission. Unlike TMS, tDCS does not initiate action potentials, but instead changes spontaneous and stimulus-evoked neural activity. These neural excitability changes can persist temporarily beyond tDCS cessation. Crucially, the neuromodulatory influences of tDCS depend on current polarity: simplistically, cortical excitability of tissue underneath the anode is enhanced while excitability of tissue underneath the cathode is depressed. These cortical excitability changes can modulate neural processing sufficiently to impact motor, perceptual and cognitive function. Accordingly, studies have used tDCS as a tool for examining the contributions of targeted brain areas to particular functions.

(a). Multisensory functions of posterior parietal cortex

(i). Attention and stimulus binding

NIBS studies support the view that PPC plays a critical role in functions related to attention allocation for unimodal and multisensory processing. Indeed, anodal tDCS, which enhances cortical excitability [4,7], speeds reaction times for detecting auditory, visual and bimodal auditory–visual targets when applied over right PPC [8]. While reflexive attention shifts can be advantageous, automatic orienting may distract and impede performance in particular contexts. For instance, reflexive orienting to a non-informative visual stimulus can impair localization of a subsequently presented tactile target. Online TMS targeting the right angular gyrus (AG) and right supramarginal gyrus (SMG) impairs suppression of non-informative visual and tactile distractors in the context of tactile target localization [9]. This pattern indicates that PPC contributes to controlling reflexive spatial attention shifts across and within modalities. Because TMS targeting the right AG also impairs the temporal binding of auditory and visual stimulation [10], multisensory PPC circuits may be involved in attentional control over space and time. Although PPC may function as a general filter and integrator of sensory information within and across modalities, different PPC subregions residing in the inferior parietal lobule and intraparietal sulcus (IPS) probably subserve attention functions for particular modalities over others [11]. Given the involvement of PPC in unimodal and multisensory attention functions, this region offers great potential as a neuromodulation target for learning and attention training interventions [12].

(ii). Reference frame remapping

Because sensory information conveyed by each modality may be represented in particular spatial reference frames (e.g. tactile information in body-centred coordinates), integration of sensory inputs, in particular to guide actions, may require remapping neural representations into crossmodally consistent coordinate systems. NIBS studies in humans indicate that the remapping of sensory information from body-centred to external coordinates relies on the PPC. Bolognini & Maravita [13] explored the role of PPC in aligning tactile and visual reference frames by exploiting the fact that a touch felt on the hand enhances visual cortex excitability only at visual field locations coinciding with the stimulated hand's position in visual space. Using phosphenes induced by TMS [14] as a measure of visual cortex excitability, Bolognini & Maravita [13] showed that phosphene sensitivity increases in the right hemifield only when the touched right hand occupies the same position as the reported phosphene location. Moreover, if the arms are crossed and the left hand is positioned to the right of body midline, touch to the left hand, but not the right, increases phosphene sensitivity in the right hemifield. This pattern reveals that the tactile influences on visual cortex excitability depend on the location of touch in external coordinates: touch location remaps to external space based on proprioceptive cues. (Note that touch location could have been remapped to world-centred, body-centred or head-centred external coordinate frames, but this was not tested in [13].) Critically, following 1-Hz rTMS perturbation of right PPC, only tactile stimulation of the right hand influenced visual cortex excitability [13], with and without arm crossing. This result reveals that PPC mediates the remapping of touch to an external reference frame. Importantly, TMS targeting the right PPC also disrupts participants' ability to localize and compare touched body locations in external coordinates [15], in the absence of visual inputs. Thus, PPC appears to be involved in remapping touch from egocentric to allocentric coordinates. In the context of multisensory stimulation, reference frame remapping by the PPC may be critical for aligning inputs to facilitate integration.

(iii). Mediating crossmodal interactions in sensory cortex

In traditional hierarchical sensory processing models, PPC has been viewed as a higher-order sensory association area that integrates modality-specific sensory representations [16]. The propensity of multisensory neurons and the clear demonstration of multisensory processing in PPC support this view [17,18]; however, PPC need not be viewed exclusively as the recipient of inputs processed by the sensory cortices. PPC is also ideally situated to mediate interactions between the sensory systems by shaping processing in primary sensory areas via feedback projections [19]. For instance, just as touch can modulate visual cortex activity, vision can also modulate activity in somatosensory cortex. Indeed, visual feedback is tremendously helpful for guiding haptic processing by providing information about what objects may be touching the body, where to expect touch and when contact may occur.

Results from a growing body of NIBS studies implicate PPC as a mediator of the rich communications between vision and touch. For instance, concurrently viewing a hand being touched on video improves the speed with which participants identify which of their own fingers is actually touched in a localization task. Online TMS to PPC abolishes this facilitation and reduces video-supported performance to unisensory levels [20]. Furthermore, viewing a picture of a hand immediately before engaging in a tactile grating discrimination task improves tactile performance [21,22]. Online TMS delivered to anterior IPS immediately after visual stimulus presentation but prior to touch onset disrupts visual enhancement of touch [22]. Importantly, TMS to IPS that is concurrent with touch has no effect, but TMS targeting primary somatosensory cortex (S1) at the time of tactile stimulation effectively disrupts tactile processing [22]. This spatio-temporal pattern is consistent with a visually driven PPC exerting modulatory influences on somatosensory cortex activity, and this top-down signal may specifically affect S1 rather than secondary somatosensory cortex circuits [21]. Prolonged visual stimulation can even serve to recalibrate tactile sensitivity, and this recalibration relies on an intact PPC [23]. Finally, auditory stimulation that is localized near the hand can similarly enhance touch, and this facilitation is also abolished by rTMS targeting PPC [24]. Together, these findings imply that PPC mediates crossmodal influences on primary sensory cortex activity. For visual and auditory influences on somatosensory processing, in particular, PPC may be necessary for maintaining stable and useful multisensory representations of peripersonal space.

(b). Multisensory functions of the superior temporal sulcus

We rely on auditory and visual information to communicate and it is unsurprising that these senses interact in speech perception. In the McGurk effect, the perception of an auditory syllable depends on whether it is accompanied by a visual depiction of the speaker producing the same or a different syllable [25]. In this robust phenomenon, incongruent auditory and visual stimuli can result in the perception of illusory syllables not presented in either modality. Neuroimaging studies have consistently identified the STS as a region involved in auditory–visual speech processing [26] and, indeed, the homologous area in macaque monkeys contains neurons tuned for communication-related auditory and visual inputs [27]. To establish a causal contribution of the STS to auditory–visual integration for the McGurk effect, Beauchamp et al. [28] applied spTMS over STS (figure 2a) while participants reported their perception of auditory and auditory–visual speech stimuli. With spTMS over STS, participants report significantly fewer occurrences of the McGurk illusion compared with trials without TMS or with TMS applied over a control site located approximately 4 cm dorsal and posterior to the STS (figure 2b) [28]. Crucially, TMS targeting STS does not disrupt performance on control trials containing auditory-only and congruent auditory–visual stimuli, implying that STS is specifically involved in the generation of the McGurk illusion with incongruent auditory–visual stimuli. To establish the time course of STS involvement in the McGurk effect, Beauchamp et al. additionally conducted a chronometric TMS experiment (figure 2c) and found that TMS to STS disrupts the McGurk effect only when pulses were delivered during an interval spanning 100 ms before the onset of the auditory stimulus to 100 ms after stimulus onset [28]. These experiments collectively provide strong support for the role of the STS in auditory–visual integration for speech. The causal contributions of the STS to the McGurk effect have recently been corroborated by a study using tDCS [29]—cathodal stimulation, which depresses cortical excitability, decreases the likelihood of McGurk responses when applied over the STS. Interestingly, the authors also reported that anodal tDCS over PPC increases the likelihood of McGurk responses [29]. This result is consistent with neuroimaging studies reporting PPC activations related to auditory–visual speech processing and the McGurk effect [30,31]. Together, these findings imply that multiple brain areas contribute to multisensory speech perception, and future studies are needed to establish the functional relationships between regions like the STS and PPC.

Figure 2.

fMRI-guided TMS-disruption of the McGurk illusion (from Beauchamp et al. [28]). (a) Auditory–visual speech stimuli evoke strong activations in the left superior temporal sulcus (STS; red voxels) and other brain regions (green voxels). The dashed black line shows the fundus of the posterior STS. (b) Participants are highly likely to report experiencing the McGurk illusion when presented with incongruent auditory–visual speech stimuli without concurrent TMS (black) or with TMS delivered to a control site near the STS (blue). For the same auditory–visual speech stimuli, TMS targeting STS significantly reduces the likelihood of the McGurk percept (red). (c) Single-pulse TMS targeting STS impacts McGurk illusion generation only when TMS is delivered within a specific temporal window relative to the onset of the auditory speech stimulus.

NIBS studies indicate that the STS may be generally involved in binding auditory and visual inputs, regardless of whether they contain speech or biological motion. For example, tDCS applied over the STS influences the sound-induced fission illusion [32], a phenomenon in which a single flash appears to flicker when paired with multiple sounds [33]. Neuromodulation of the fission illusion is polarity-specific: anodal tDCS over STS increases fission while cathodal tDCS over STS decreases fission [32]. These results support the notion that the STS also serves as a locus for the fission illusion. Alternatively, the STS may be part of a network that supports auditory–visual binding and tDCS applied over STS may remotely modify excitability across this distributed network. Indeed, tDCS over primary visual cortex, whose activity has been shown to correlate with the strength of fission illusion across individuals [34,35], can also modulate fission strength in the sound-induced illusion in a polarity-dependent manner. These considerations exemplify the potential challenges involved in interpreting NIBS results, but the collective results from these studies reveal the STS' clear involvement in the binding of auditory–visual information.

(c). Multimodality of sensory cortex

In traditional models of sensory cortex organization, sensory information processing for each modality is initially segregated in modality-specific sensory areas before subsequent integration in higher-order association areas. This organizational scheme has been challenged recently by the demonstration of multimodal responses in primary and secondary sensory areas [16,36]. NIBS has been tremendously helpful for testing the modality-specificity of sensory cortex functions. For clarity, here we define the modality that is traditionally associated with a given sensory region as ‘preferred’ (i.e. visual inputs are preferred by visual cortex). Accordingly, the other modalities are defined as ‘non-preferred’ with respect to the same region (i.e. auditory and somatosensory inputs are non-preferred by visual cortex). In the following section, we review two types of multimodal responses in sensory cortex that have been revealed by NIBS. In the first case, NIBS reveals the contributions of traditionally-defined sensory areas to perceptual processing of non-preferred modalities. In the second case, NIBS reveals modulation of sensory cortex excitability by non-preferred sensory inputs.

(i). Crossmodal recruitment of sensory cortex

Sensory brain regions can be characterized according to their preferred input modalities and according to the functions and computations they perform. For instance, early processing stages in the ventral visual pathway are known to represent simple visual shape information like orientation and curvature, while higher-order ventral pathway areas support object processing [37,38]. Rather than being strictly dedicated to visual processing, these regions also respond to tactile inputs [39–42] in the absence of concurrent visual stimulation if these signals convey spatial or shape information. This response selectivity is consistent with the notion that visual cortex serves as a general operator for processing spatial or shape information [43]. Supporting this hypothesis, rTMS disrupting visual cortex processing in sighted individuals impairs Braille reading [42] and fine tactile dot spacing judgements [44]. Yau et al. [45] recently demonstrated that anodal tDCS applied over visual cortex, which enhances contrast sensitivity to visual grating patterns [46], also temporarily enhances tactile spatial acuity [45]. Participants performed a discrimination task in which they identified the orientation of tactile gratings that varied in grating width. Following anodal tDCS over visual cortex (figure 3a), but not sham stimulation over visual cortex or anodal tDCS over auditory cortex, participants were better able to discriminate the orientation of tactile gratings at smaller grating width values [45]. This pattern implies that visual cortex (probably V1/V2) contributes to tactile orientation perception, corroborating an earlier report showing that spTMS delivered to occipital cortex impairs participants' ability to discriminate tactile grating orientation [47]. Zangaladze et al. [47] employed a chronometric analysis to reveal that TMS disruption occurs for pulses delivered 180 ms after touch [47]. This temporal specificity indicates that tactile recruitment of visual cortex occurs relatively late in the processing stream.

Figure 3.

Crossmodal recruitment for spatial and temporal touch (from Yau et al. [45]). (a) Participants performed a tactile grating orientation discrimination task during sessions in which they received sham tDCS (grey-hued traces) or anodal tDCS (red-hued traces) over visual cortex (yellow crosshair in left panel). Performance scaled according to grating width and widths corresponding to 75% correct performance define the grating orientation threshold, a measure of tactile spatial acuity. Inset: grating orientation (GO) thresholds estimated for different test intervals during each session (Pre, Stim, Post1 and Post2) reveal a significant enhancement of tactile spatial acuity following anodal tDCS. (b) Participants performed a tactile frequency discrimination task during sessions in which they received sham tDCS (grey-hued traces) or anodal tDCS (red-hued traces) over auditory cortex (yellow crosshair in left panel). Psychometric plots indicate the probability that participants judged a comparison frequency (f_c) as being higher in perceived frequency compared with a 300-Hz standard stimulus. Inset: sensitivity estimates for different test intervals during each session (Pre, Stim, Post1 and Post2) reveal a significant enhancement of tactile frequency sensitivity following anodal tDCS.

Involvement of visual cortex in tactile shape processing is not restricted to the perception of low-level shape features: rTMS targeting the lateral occipital complex (LOC), a higher-order visual area that also responds to tactile shapes [48], disrupts visual, tactile and visual–tactile instances of the Muller-Lyer illusion [49], a length illusion that requires integration of multiple line stimulus segments. Crucially, dorsal pathway visual areas also appear to support processing for non-preferred modalities. For instance, middle temporal area (area MT), which is well studied given its prominent role in visual motion processing, also activates in response to tactile motion [50–52]. rTMS targeting area MT impairs tactile motion speed judgements [53]. Collectively, these NIBS studies support the emerging view that brain regions which have been traditionally ascribed visual functions may also contribute to non-visual perception.

Auditory cortex also contributes to processing for non-preferred modalities. Tactile stimulation alone evokes responses in auditory cortex [54–58]. Because auditory cortex comprises neural circuits specialized for processing temporal information like frequency and timing, tactile responses in auditory cortex may reflect functional recruitment for tactile temporal processing. Indeed, anodal tDCS over auditory cortex, which modulates auditory frequency sensitivity [59], improves performance on a tactile frequency discrimination task by enhancing frequency sensitivity or perceptual learning in the frequency domain [45] (figure 3b). Additionally, spTMS targeting the superior temporal gyrus (STG) impairs performance on a tactile duration discrimination task [60,61]. In individuals with normal hearing, TMS targeting STG impairs tactile performance only when TMS pulses are presented 180 ms after touch; however, TMS delivered to STG 60 ms after touch impairs tactile performance in congenitally deaf individuals [61]. These studies reveal that tactile recruitment of auditory cortex may be ubiquitous in hearing and deaf individuals, and the degree and nature of crossmodal processing is shaped by auditory experience.

Just as regions in visual and auditory cortex can be defined by the functions they support (i.e. spatial form and temporal frequency processing, respectively), rather than by modality per se, somatosensory cortex appears to contribute to processing of body-related information independent of input modality. This perspective, also articulated in theories of embodied cognition [62], proposes that visual perception of facial expressions recruits face-processing circuits in sensorimotor brain areas in addition to those in visual cortex. Supporting this view, Pitcher et al. [63] demonstrated that online rTMS targeting the somatosensory face region, but not the adjacent hand region, impairs participants' ability to discriminate visually presented facial expressions [63]. This result is consistent with a previous study demonstrating that spTMS targeting primary somatosensory cortex affects facial emotion judgements, but not eye-gaze direction [64]. The magnitude of S1-TMS disruption is comparable to the effect of delivering TMS to the occipital face area (OFA). Notably, the time course of TMS-induced perturbations differs between the two targets: TMS targeting OFA at 60 ms after stimulus onset maximally disrupts performance while the largest effects of S1-TMS occur with pulses delivered 130 ms after visual stimulus onset [63]. Visual processing related to non-face body parts also recruits somatosensory cortex, as online rTMS targeting S1 impairs discrimination performance in a task requiring participants to judge whether a video depicts a hand being touched [65]. Thus, somatosensory cortex may be specialized for processing body-related information across sensory modalities.

(ii). Crossmodal modulation of sensory cortex excitability

As discussed previously, a single TMS pulse delivered to occipital cortex can evoke acute visual sensations (i.e. phosphenes) in restricted portions of the visual field [14]. Because the likelihood of producing phosphenes depends on the excitability state of the visual cortex [66] in addition to TMS intensity, phosphene generation provides a unique tool for probing crossmodal influences on visual cortex excitability. Furthermore, just as chronometric TMS experiments track the time course of a brain region's involvement in a particular function (e.g. figure 2c), the time course of crossmodal influences on visual cortex excitability can be profiled. Ramos-Estebanez et al. [67] demonstrated that subthreshold tactile stimulation (electrocutaneous stimulation that was imperceptible) on the right hand increases phosphene occurrence in the right visual field when occipital TMS is delivered 60 ms after tactile stimulation [67]. This result reveals that the crossmodal somatosensory influences on visual cortex excitability are highly specific in space and time. Additionally, the fact that subthreshold tactile stimuli modulate visual cortex excitability implies crossmodal influences operate via automatic, pre-attentional mechanisms. Interestingly, Ramos-Estebanez et al. found that subthreshold stimulation of the right hand increases phosphene occurrence in the right visual field even if the arms are crossed and the hand occupies space in the left hemifield. Recall that Bolognini & Maravita [13] reported that a stimulated right hand positioned to the left of body midline did not modulate excitability in the right hemifield—the authors interpreted this as evidence for the tactile reference frame remapping function of PPC [13]. These contradictory results may be reconciled by considering that Bolognini and Maravita used suprathreshold tactile stimulation while the tactile stimulus used by Ramos-Estebanez et al. was subthreshold; conceivably, reference frame remapping specifically occurs for detectable or attended stimuli, consistent with PPC's ascribed role in attention allocation.

TMS induction of phosphenes has also been used to demonstrate that auditory stimulation modulates visual cortex excitability. Auditory modulation of visual cortex occurs with a temporal profile matching tactile-mediated excitability changes [68]. Furthermore, auditory modulation of visual cortex can depend on sound characteristics—looming sounds exert greater influences on visual cortex excitability compared with receding or stationary sounds [69]. Looming stimuli preferentially drive visual cortex excitability even for sound durations short enough to abolish participants' ability to discriminate sound characteristics. These results indicate that auditory modulation of visual cortex is pre-perceptual and feature-specific [69].

NIBS has been used not only to reveal crossmodal influences on visual cortex excitability, but also to identify potential cortical origins of crossmodal modulatory signals. Anodal tDCS targeting auditory cortex enhances the influence of looming sounds on visual cortex excitability [70]. Likewise, anodal tDCS targeting parietal cortex enhances the tactile influences on visual cortex excitability [70]. That auditory stimulation can combine with tactile stimulation to induce larger excitability changes in visual cortex than can be achieved by either modality alone [70] suggests that common mechanisms exist for crossmodal modulation of visual cortex. Together, these results imply that visual cortex excitability changes are shaped by the activity of auditory and somatosensory cortex. Because visual cortex excitability can be influenced by stimuli that are subthreshold [67] and pre-perceptual [69], interactions between sensory cortical systems appear to be automatic and nearly ubiquitous.

3. Summary of advances using neuromodulatory techniques in animal studies

In the following section, we summarize studies that characterize multisensory neural mechanisms using neuromodulatory methods in animal models. Animal models permit use of invasive techniques like electrical microstimulation, chemical and cryogenic deactivation, and more recently, optogenetics (figure 4). These approaches, combined with psychophysics and neurophysiology, have led to important advances in our understanding of the brain systems and mechanisms that support multisensory functions. Here we synthesize the relevant literature according to four general topics: (i) multisensory network interactions supporting attention and orienting behaviour, (ii) crossmodal activity in putative unisensory regions, (iii) optimal cue integration, and (iv) neural and synaptic architecture for multisensory integration.

(a). Multisensory networks for orienting behaviours and attention

(i). Multisensory integration in cat superior colliculus

The superior colliculus (SC) is a brainstem structure involved in spatial orienting. SC neurons display a wide range of unisensory and multisensory responses to visual, auditory and somatosensory inputs [75–78]. Because of these complex interactions, pioneering work on multisensory processing focused on SC neurophysiology. Much of the work performed in SC builds upon the observation that SC unit responses to bimodal stimuli often exceed the response to the most effective unimodal component stimuli. Such facilitation has been used to define multisensory enhancement (MSE) in SC [79,80] and this definition has been extended to quantify multisensory integration in other brain structures [81,82]. MSE strongly depends on the spatial and temporal relationships between the integrated sensory inputs. In general, MSE is found to be reduced when stimuli from two modalities are separated in either space or time [80]. MSE is also evident in behaviour, as animals trained to orient to visual, auditory or bimodal spatial cues perform best when bimodal cues are spatially and temporally congruent [77]. Importantly, an intact SC appears to be required for MSE effects in cat orienting behaviour because excitotoxic SC lesions abolish the MSE in behaviour and reduce performance to unimodal levels [83]. The well-defined neurophysiological substrate of orienting behaviour in the SC of cats has provided a rich model system for testing multisensory integration mechanisms using neuromodulatory methods.

Substantial efforts have focused on characterizing the inputs that drive multisensory responses in cat SC. For instance, microstimulation applied to the frontal eye fields (FEF) preferentially activates multisensory, not unisensory, neurons in SC [84]. This suggests that frontotectal projections may serve specific functions in SC multisensory processing. Additionally, evidence from many studies using a variety of neuromodulation techniques highlights the importance of sensory cortex in sculpting multisensory interactions in SC (figure 5). In particular, electrical microstimulation has revealed that the SC receives projections from multiple sensory cortical areas including the lateral suprasylvian visual area (LS) and the anterior ectosylvian area (AES), the latter of which comprises an auditory field AES (FAES), a visual region (AEV) and the fourth somatosensory area (SIV). Electrical microstimulation delivered to each of these cortical regions produces orthodromic activation of multisensory SC units [86]. Furthermore, microstimulation of sensory cortex produces modality-specific SC responses that are consistent with activity evoked by actual sensory stimulation [86]. Because sensory cortex contains both unisensory and multisensory neurons, either population (or both) could project to SC; however, microstimulation targeting SC only evokes antidromic responses in unisensory cortical neurons. This result suggests that unisensory, not multisensory, cortical neurons make corticotectal projections.

Figure 5.

Putative circuitry underlying multisensory integration in cat superior colliculus (from Rowland et al. [85]). Multisensory neurons in superior colliculus receive ascending and descending inputs from multiple sources. Cortico-collicular projections from different sensory cortex regions in the anterior ectosylvian sulcus (AES) target electrotonically coupled areas of the dendrite. Afferents also project to interneurons that inhibit multisensory output neurons. Cell labelled ‘I’ is an inhibitory interneuron; PAG, periaqueductal grey area.

Consistent with the notion that sensory cortex provides critical inputs that shape multisensory integration in the SC, chemical deactivation of sensory cortex selectively impacts orienting and approach behaviours that are based on bimodal cues [87]. Specifically, lidocaine injections in AES reduce both the behavioural enhancements that arise from congruent bimodal cues and the performance impairments that are caused by incongruent bimodal cues [87]. This demonstrates that modifying sensory cortex responses directly influences SC activity. Indeed, reversible cryogenic deactivation of AES and LS selectively impacts the multisensory responses of many SC neurons. Following cooling, SC responses to bimodal cues are reduced to levels that match the maximum unisensory responses [88]. This indicates that multisensory integration in the SC requires intact and functional sensory cortex. Importantly, these cooling experiments also reveal that SC neurons display different patterns of dependencies on sensory cortex: some SC neurons maintained MSE as long as either AES or LS remained functional, but MSE in a majority of SC neurons required functional integrity of both AES and LS [88]. Multisensory integration in SC thus appears to rely on interactions between auditory and visual sensory cortical areas (figure 5), and this ‘corticotectal synergism’ is also reflected in SC dependencies on the separate auditory and visual subfields contained within AES [89].

Because intact sensory cortical function is required for multisensory integration in SC neurons, it is unsurprising that sensory cortex also plays a major role in shaping the maturation of multisensory properties in SC neurons during development. Rowland et al. [90] recently demonstrated that prolonged, but temporary, deactivation of AES and LS, using muscimol-releasing packets implanted at postnatal week 3, impairs the development of MSE in SC neurons. Chemical deactivation of sensory cortex early in life led to an absence of MSE in behaviour and neurophysiological tests in 1-year old animals [90]. Importantly, MSE appeared by age 4 in these same animals. Thus, maturation of SC multisensory properties that are delayed by early cortical deactivation can recover later in life, assuming sensory cortex integrity.

(ii). Spatial attention mechanisms

Sensory responses can be modulated by top-down attentional signals [91]. In macaque monkeys, electrical microstimulation of FEF, a forebrain area involved in gaze control, increases both behavioural sensitivity and the responsiveness of V4 neurons to stimuli presented within the movement field of the stimulated FEF neurons [92,93]. In this manner, microstimulation of FEF is thought to mimic the neural and behavioural effects seen with endogenous visual spatial attention. A top-down mechanism for spatial attention has also been reported in the barn owl [94]. Microstimulation applied to the arcopallial gaze fields (AGFs) in the barn owl modulates spatially-tuned auditory responses in the optic tectum (OT), suggesting that AGF serves an analogous function as mammalian FEF. Critically, the spatially-tuned neurons in barn owl OT respond to visual as well as auditory stimulation, so the OT can be considered a multisensory structure. Winkowski & Knudsen [95] demonstrated that electrical stimulation of AGF enhances the auditory and visual responsiveness and stimulus discriminability of OT neurons that are driven by the location in space represented by the AGF microstimulation site [95]. Conversely, AGF microstimulation suppresses OT responses to sensory inputs presented outside of the AGF receptive field (RF) [95].

These results indicate that microstimulation-induced AGF modulation of OT, which mimics endogenous spatial attention, depends on the spatial alignment of AGF and OT RFs. Furthermore, the ability of AGF microstimulation to produce these endogenous spatial attention effects depends on the integrity of a particular inhibitory circuit in the barn owl midbrain tegmentum, the nucleus isthmi pars magnocellularis (Imc). Chemical inactivation of Imc abolishes endogenous competitive suppression caused by AGF microstimulation [96]. Imc blockade also disrupts the suppressive influences of distractor stimuli on OT activity [96]. Thus, a common inhibitory circuit appears to mediate exogenous and endogenous attention effects on OT activity, which potentially impacts multisensory responses across multiple processing levels in the tectofugal pathway [97]. Together, these studies use neuromodulation to reveal how top-down control mechanisms influence multimodal spatial processing circuits in the barn owl.

(b). Multimodal processing in auditory brainstem and cortex

(i). Somatosensory activation of the dorsal cochlear nucleus

Based on its anatomical and physiological characteristics, the dorsal cochlear nucleus (DCN) is traditionally considered to be an auditory brainstem structure [98]. However, the DCN also receives a projection from the somatosensory dorsal column and spinal trigeminal nuclei (collectively termed the medullary somatosensory nuclei; MSN) [99]. This connectivity implies that tactile inputs also access DCN circuits. Neuromodulation has been used to characterize the nature of somatosensory inputs to DCN. Electrical microstimulation of MSN strongly modulates DCN activity in cats [99] through a mixture of inhibitory and excitatory effects that depend on cell type [100]. Microstimulation of the trigeminal nuclei also produces complex interaction patterns in the guinea pig DCN [101]. These results clearly indicate that the DCN does not exclusively respond to auditory stimulation.

The observation that mechanical stimulation of the pinna and vibrissae, but not other body regions, drives DCN activity [99] implies that somatosensory inputs to DCN may serve specific functions. In fact, muscle-related somatosensory stimulation of the pinna drives DCN activity most robustly [102]. These patterns are consistent with the hypothesis that somatosensory inputs to DCN modulate auditory responses in behaviourally meaningful ways: excitatory interactions that amplify auditory signals can result from pinna movements related to spatial orienting while inhibitory interactions can suppress sound processing related to self-generated movements [98,103]. Importantly, because the DCN occupies a relatively early processing stage in the auditory pathway, somatosensory information accessing DCN circuits necessarily propagates to all levels of the auditory system that receive direct or indirect inputs from DCN [99].

(ii). Crossmodal recruitment of auditory cortex

Neuromodulation studies have provided clear evidence for functional visual recruitment of auditory cortex in congenital and early-deafened cats. Lomber et al. [104] used cryogenic deactivation techniques to establish the neural mechanism mediating the supranormal visual capacities of deaf cats [104]. Specifically, deaf cats have superior visual localization and lower visual movement detection thresholds in the peripheral field compared with hearing cats. Consistent with the hypothesis that auditory cortex in deaf animals is recruited to process visual inputs, substantially more visually responsive neurons are found in the FAES of deaf compared with hearing cats [105]. To demonstrate that FAES causally supports vision, Lomber and colleagues deactivated all or parts of FAES in cats performing visual tasks. Cooling all of FAES reduces supranormal visual performance to typical levels in all domains that are enhanced by deafness [104,105]. Moreover, deactivating only the posterior auditory field (PAF) selectively impairs visual localization and deactivating only the dorsal zone of auditory cortex (area DZ) selectively impairs visual motion detection [104]. This double dissociation indicates that distinct auditory cortical regions support separate visual functions in the deaf. This highly specific crossmodal recruitment of sensory cortex, which pairs particular visual functions with specific auditory cortical loci, may also account for the superior visual sensibilities in deaf humans [106].

(c). Optimal cue integration

(i). Visual and vestibular cue integration for perceiving self-motion

A popular theoretical framework for considering multisensory cue integration is based on maximum-likelihood estimation (MLE). According to the MLE framework, redundant sensory cues are combined linearly in a statistically optimal manner such that (i) the variance of perceptual estimates based on the combined cues is minimized and (ii) the perceptual weights assigned to each cue are proportional to the relative reliabilities of the cues, such that more reliable cues are weighted more heavily [107,108]. Near-optimal cue integration properties have been observed for a broad variety of sensory modalities and tasks [109–114], and substantial progress has been made in establishing the neurophysiological mechanisms of optimal cue integration [115]. Recent studies have used neuromodulation techniques to investigate the cortical circuits that support optimal cue integration.

Visual and vestibular cues combine near-optimally to inform our perception of heading, which is the direction of translational self-motion [116]. Neurons in the dorsal medial superior temporal area (MSTd) exhibit directional tuning for visual and vestibular heading signals, making this region a prime candidate for mediating multisensory heading perception. If visual and/or vestibular signals in MSTd are used by the monkey to judge heading, electrical microstimulation targeting neuron clusters with similar tuning preferences is expected to add a net signal to the local circuit, thus biasing behaviour. Specifically, microstimulation is expected to shift the psychometric function toward the preferred heading of neural activity recorded at the stimulation site, with little or no effect on the slope of the function (figure 6a). By contrast, reversible chemical inactivation using muscimol, a GABA-A receptor agonist, is expected to flatten the psychometric function if a large enough volume is used to suppress activity in populations of neurons with diverse tuning preferences. In this case, chemical inactivation would be expected to increase heading thresholds, rather than inducing a choice bias (figure 6b).

Figure 6.

Hypothesized and actual changes in heading perception (from Gu et al. [117]). (a) Predicted effects of microstimulation on heading discrimination. If multi-unit (MU) activity at the stimulation site has a leftward heading preference, microstimulation should lead to more leftward choices such that the psychometric function is shifted to the right. By contrast, stimulating sites with a rightward heading preference should cause a leftward shift of the psychometric function. PSE, point of subjective equality. (b) Predicted effects of reversible inactivation on heading discrimination performance. Muscimol injection suppresses neural activity (inset) in a large region and is expected to reduce the precision of heading discrimination, leading to a shallower psychometric function. (c) Example microstimulation experiment at a site with a leftward heading preference. Psychometric functions for stimulated (dashed curves) and non-stimulated (solid curves) trials in response to vestibular (black), low-coherence visual (red) and combined (green) stimuli. (d) Example inactivation experiment. Visual psychometric functions were collected during four separate sessions: a control block before muscimol injection (‘Pre’, dashed curves and cross symbols), a block immediately after inactivation (‘0 h’, orange), a block 12 h after inactivation (‘12 h’, red) and a recovery block 36 h after inactivation (‘36 h’, black solid curves and open symbols).

In a series of experiments, Gu et al. [117] used electrical microstimulation and muscimol inactivation methods to test the causal involvement of area MSTd in multisensory heading perception [117,118]. Electrical microstimulation targeted to clusters of MSTd neurons with consistent heading tuning was found to induce systematic biases in heading percepts. Although electrical microstimulation greatly affected performance on vision-only trials, performance on vestibular-only and multisensory trials was minimally influenced (figure 6c). The weak effects on vestibular heading perception were probably due to the fact that tuning of MSTd neurons for heading based on vestibular signals is relatively weak. Because of this weak vestibular tuning, microstimulation may have failed to activate sufficiently large clusters of MSTd neurons with consistent vestibular heading preferences.

To counter some of the limitations of microstimulation, Gu et al. [117] also injected muscimol into MSTd to test how inactivation of this area impacts heading perception. Suppressing MSTd activity using large bilateral injections of muscimol significantly impaired heading perception and increased behavioural thresholds (figure 6d). The threshold changes were largest for the visual condition, intermediate for the multisensory condition and weakest for the vestibular condition, again revealing that visual perception is more strongly impacted by neuromodulation of MSTd. The consistently weaker effects on vestibular-only trials suggested that other brain regions also support vestibular heading perception. Candidate regions include the parieto-insular vestibular cortex [119], the visual posterior sylvian area [120] and the ventral intraparietal area (VIP) [121,122], all of which contain neurons with responses that are either dominated by vestibular inputs or have more balanced representations of visual and vestibular signals than MSTd, which is visually dominated. Macaque area VIP has also been shown to respond to tactile and auditory stimuli, as well as visual inputs [123–125]. Electrical microstimulation of VIP evokes behaviourally relevant motor actions, including defensive postures or avoidance movements [126].

Interestingly, even though microstimulation and inactivation of area MSTd biased and impaired heading perception, respectively, behavioural cue integration continued to be near-optimal [117]. These results, supported by model simulations, suggest that optimal cue integration for heading perception is a distributed operation that involves additional populations of heading selective neurons located outside of MSTd. Similarly, a recent study reported that reversible chemical deactivation of PPC impairs visual processing without affecting auditory sensitivity or auditory–visual integration in an event-counting task [127]. Thus, from these collective results, cue integration appears to be supported by distributed multisensory networks.

(ii). Optimal integration of intracortical microstimulation and sensory inputs

Remarkably, even arbitrary patterns of electrical microstimulation conveying sensory information can be optimally integrated with natural sensory cues, so long as the relationship between the microstimulation patterns and sensory information can be learned [128]. Macaque monkeys initially trained to perform centre-out reaches in a virtual reality environment based on visual cues can learn to perform reaches based solely on microstimulation feedback after a period of multisensory training that pairs the microstimulation patterns with visual cues. This learning is possible even if the microstimulation activates somatosensory neurons in an arbitrary fashion that ignores the tuning preferences of the stimulated cortical site, so long as the microstimulation is spatio-temporally correlated with the visual cues during training. Following training, the microstimulation-evoked somatosensory activity and the visual signals provide redundant feedback information and reaching performance with either cue alone is characterized by particular bias and variance levels. Critically, the visual and microstimulation cues combine near-optimally such that performance variance with bimodal cues is smaller than with either cue separately and the weight ascribed to each cue scaled according to its relative reliability. This study highlights not only the potential for electrical microstimulation to provide sensory inputs in neuroprosthetic applications but also the fundamental and ubiquitous nature of optimal cue integration.

(d). Neural and synaptic architecture for multisensory integration

Although our understanding of the brain areas and networks involved in multisensory integration has advanced tremendously, much less is known regarding mechanisms of multisensory integration at the cellular and synaptic levels. A recent study used optogenetics to characterize cortical microcircuits mediating multisensory integration [129]. Olcese et al. [129] focused their experiments on the visuotactile area RL, a region of parietal cortex in the mouse that lies between the visual and somatosensory cortices and projects to motor areas. According to measured post-synaptic potentials and action potentials, area RL neurons respond to visual, tactile and visual–tactile stimulation. In a critical set of experiments, Olcese et al. first established that both pyramidal cells and Parvalbumin-positive interneurons (Pv-INs) respond to visual, tactile and combined inputs. Thus, both excitatory and inhibitory neural circuit components potentially process multisensory inputs. Notably, compared with pyramidal cells, Pv-INs were more likely to be bimodal (i.e. responsive to both visual and tactile inputs), but they displayed less MSE.

To test whether the multisensory properties of Pv-INs influence MSE in pyramidal cells, Olcese et al. selectively activated Pv-INs using optogenics [129]. Pv-INs were activated by light while unimodal and bimodal sensory stimuli were presented. Photoactivation increased the spiking activity of Pv-INs on multisensory trials to a greater extent than on unisensory trials. Consequently, Pv-IN photoactivation resulted in larger reductions in the synaptic responses and spiking activity of pyramidal cells on multisensory compared with unisensory trials. Thus, artificially augmenting MSE in Pv-INs reduced MSE in pyramidal cells. This experiment demonstrates a causal relationship linking the multisensory properties of excitatory and inhibitory cell types in cortical microcircuits.

4. Discussion

As summarized here, the addition of causal neuromodulatory techniques to behavioural, neurophysiological and neuroimaging methods has greatly advanced our understanding of the neural mechanisms underlying multisensory integration. In particular, neuromodulation has helped to elucidate the constellations of brain structures that contribute to multisensory integration and perception. Multisensory processing clearly relies on coordinated activity over networks that span multiple scales: multisensory integration depends on complex interactions between neural populations in cortical microcircuits as well as interactions between cortical and subcortical brain structures. Despite recent progress, many unanswered questions remain. For example, why are there multiple areas with similar multisensory properties? What is the functional relationship between areas that comprise multisensory networks? Experiments that combine stimulation approaches with simultaneous recording and imaging will be critical for addressing these questions. In humans, multimodal non-invasive approaches like concurrent TMS-fMRI [130,131] and concurrent TMS-EEG [132] enable direct characterizations of functional connectivity. In animal models, the pairing of electrical microstimulation or optogenetic stimulation with electrode recordings or optical imaging offers even more powerful tools for interrogating multisensory networks, especially in systems that permit genetic control. Recent advances in such multimodal approaches position the field to explore not only the architecture of multisensory networks, but also the functional properties of these circuits.

A particularly important question regarding multisensory networks that neuromodulation methods may address involves the coupling of network nodes: how do distributed brain areas collaborate to support integration and perception? The last decade has brought a growing appreciation for cortical oscillations and the range of cognitive functions they potentially support, which includes the binding of neuronal ensembles [133]. Indeed, recent evidence suggests that coupled oscillatory activity across distributed sensory brain regions can serve to bind information across the senses [134]. Multisensory integration and attention-driven stimulus binding and selection may similarly rely on mechanisms based on coherence in cortical oscillations [135,136]. Accordingly, some instances of crossmodal processing in sensory areas may be explained in part by attention-based mechanisms. Because neuromodulation can be used to entrain or shape cortical oscillations [137], stimulation methods may be particularly well suited for establishing a causal role of oscillations in multisensory integration.

Crucially, neuromodulation techniques suffer from some limitations. It is important to note that neuromodulation approaches can have limited spatial and temporal resolution and lack cell-type specificity. NIBS methods certainly are limited in their spatial and cell-type specificity, although clever experimental manipulations exploiting state-dependent stimulation effects have been developed for more targeted interventions [138]. Among invasive neuromodulation approaches, although the older microstimulation, cooling, and chemical inactivation techniques have limited cell-type specificity, the development of optogenetics has largely overcome this challenge in rodents. In primates and larger animals, optogenetic specificity remains lacking (although continued developments are promising [139–143]; however, as shown by the study of Olcese et al. [129], investigating multisensory integration in the mouse offers superb advantages. Of course, all neuromodulatory techniques, including optogenetics, have the caveat that rapid plasticity mechanisms may alter network properties.

Finally, and perhaps most importantly, there is a great need to couple experimental approaches with theory. For example, it is generally assumed that a lack of behavioural deficits following inactivation of a brain area implies that the area does not make a causal contribution to behaviour. However, a recent theoretical analysis of the relationships between neural population codes and inactivation effects suggests that this assumption may not always hold [144]. Specifically, as summarized above, heading information is known to be distributed across at least two areas in the macaque cortex—MSTd and VIP. Surprisingly, although area VIP contains neurons with stronger choice-related activity than MSTd, inactivating VIP had no observable effect on discrimination thresholds [144], whereas MSTd inactivation did impair performance (figure 6).

Thus, with the development of better neuromodulatory techniques comes a greater need for theoretical advances in order to appropriately interpret experimental findings. As discussed, understanding the functional organization of multisensory networks remains a primary challenge and focusing on the roles of feed-forward and feedback signalling in sensory processing may be particularly insightful in this endeavour. Indeed, neuroanatomical feedback represents a hallmark of brain connectivity; yet, how feedback affects early sensory properties remains mostly unexplored and theoretical principles governing such feedback networks are largely lacking. Such theoretical developments would be especially important for exploring multisensory interactions, for which the interconnections across levels of processing circuits are necessarily more extensive.

Competing interests

We declare we have no competing interests.

Funding

This work was supported by DC00720 and EY017866 (D.E.A.) and EY016178 (G.C.D.).