Multisensory integration in the mammalian brain: diversity and flexibility in health and disease

Abstract

Multisensory integration (MSI) occurs in a variety of brain areas, spanning cortical and subcortical regions. In traditional studies on sensory processing, the sensory cortices have been considered for processing sensory information in a modality-specific manner. The sensory cortices, however, send the information to other cortical and subcortical areas, including the higher association cortices and the other sensory cortices, where the multiple modality inputs converge and integrate to generate a meaningful percept. This integration process is neither simple nor fixed because these brain areas interact with each other via complicated circuits, which can be modulated by numerous internal and external conditions. As a result, dynamic MSI makes multisensory decisions flexible and adaptive in behaving animals. Impairments in MSI occur in many psychiatric disorders, which may result in an altered perception of the multisensory stimuli and an abnormal reaction to them. This review discusses the diversity and flexibility of MSI in mammals, including humans, primates and rodents, as well as the brain areas involved. It further explains how such flexibility influences perceptual experiences in behaving animals in both health and disease.

This article is part of the theme issue ‘Decision and control processes in multisensory perception’.

1. Introduction

In the mammalian brain, cortical areas interact with each other and communicate with subcortical areas through mutual projections. These complex circuits process a variety of sensory inputs and meaningfully integrate them to animals better understand complicated sensory information. How multisensory inputs are integrated in the brain, a.k.a. multisensory integration (MSI), is critical for an animal to make optimal and flexible decisions in a sensory world that is constantly changing. So far, many brain areas have been identified as polymodal areas that respond to a variety of different modality inputs. MSI, however, is beyond simple responses to multiple stimuli. It explains why neuronal responses to spatially and temporally co-presented multisensory stimuli differ from those to unisensory stimuli. MSI shapes not only neural responses but also the perceptual experiences of an animal by integrating one modality with another in stimuli given together. Which parts of the brain are engaged in this processing? In this review article, we investigated the important brain areas implicated with MSI (table 1). We particularly focused on audiovisual MSI (integration of auditory and visual stimuli) since both types of stimuli are physically discrete and separable when presented to animals at different times and in different spaces.

Table 1.

Summary table of the references listed in this paper. References were classified according to the examined brain areas and the multisensory integration (MSI) types in health and diseases. ASD, autism spectrum disorder; SC, superior colliculus; PPC, posterior parietal cortex; STS, superior temporal sulcus; PFC, prefrontal cortex; MPC, medial prefrontal cortex; IFG, inferior frontal gyrus; ACC, anterior cingulate cortex; V1/VC, (primary) visual cortex; V1/AC, (primary) auditory cortex.

factor	brain
		association cortex					sensory cortex		subcortical
	SC	PPC	STS	insula cortex	PFC (MPC, IFG)	ACC	V1/VC	A1/AC	thalamus	amygdala
simple MSI	[1–18]	[19–21]	[22,23]				[24–29]	[26,28,30–33]	[34–38]	[39–41]
temporal MSI			[23,42]				[42,43]	[32,42,44,45]
spatial MSI		[46]	[47,48]					[49]	[35]
object MSI		[50–52]	[51,53–58]		[51,52,56,59,60,61]	[59,61]
speech MSI	[62]		[55,62–67]		[68–71]		[62]
ASD	[72]		[73]	[74]			[73]	[73]
schizophrenia			[75]		[75]		[76]

MSI may happen in various circumstances where animals have different knowledge of the stimuli from previous experiences. Therefore, it is important to understand the types of MSI we are discussing. We classified MSI into five separate categories based on the parameters of the stimuli given to the animal (figure 1). First, many studies recorded brain activities or neural responses to simple multisensory stimuli in order to investigate the differences in responses evoked by multisensory stimuli compared to unisensory stimuli. We note these as simple MSI (figure 1a). Second, MSI can occur in the temporal dimension, which we note as temporal MSI (figure 1b). It requires the integration of audiovisual stimuli that are given at the same or different times. Third, we discuss spatial MSI, which requires the integration of audiovisual stimuli at different locations in space (figure 1c). These three types of MSIs happen even when the stimuli are novel to the animals. On the other hand, animals may integrate meanings in the stimuli that they learned from previous experiences. We examine two types of semantic MSI: those linked to the meanings of objects (object MSI; figure 1d) and those related to speech (speech MSI; figure 1e).

Figure 1.

Task paradigms used for studying different types of multisensory integration (MSI). (a) Schematic illustration of simple MSI measured in subjects (either humans or animals) presented with random audiovisual stimuli simultaneously. (b) Temporal MSI as shown in the simultaneity judgement task. Red bold text and dashed lines, visual; blue regular weight text and continuous lines, auditory. (c) Spatial MSI as shown in the ventriloquism effect. (d) Object MSI by using congruent or incongruent multisensory stimuli. (e) Schematic illustration of speech MSI as shown in the McGurk effect.

MSI varies subjectively, as individuals report different experiences even from the same multisensory stimuli. In this paper, we describe not only five types of MSI and the brain areas involved but also how MSI may change flexibly based on the states of an animal (mostly human cases). In §3, we discuss how MSI alters flexibly, proposing potential factors that affect MSI measured both behaviourally and physiologically. To gain more insights into the importance of such flexible MSI in normal brain functions, we extend our discussion on how neuropsychiatric disorders, such as autism spectrum disorder (ASD) and schizophrenia, influence MSI (see §4). We explain how each disease affects MSI in brain areas such as the sensory cortex, superior colliculus (SC), superior temporal sulcus (STS), insula cortex and prefrontal cortex (PFC), which results in abnormal MSI behaviours (table 2). In each section, we attempt to describe human studies together with relevant animal research.

Table 2.

Abnormal MSI in different types of audiovisual integration tasks shown in patients with autism spectrum disorder (ASD) or schizophrenia.

task type	disease type	autism spectrum disorder (ASD)	schizophrenia
temporal MSI	simultaneity judgement task	temporal window ↑ ASD [77,78]; schizophrenia [79,80]
	sound-induced flash illusion task	flash-beep illusion ↑ [81]	flash-beep illusion ↓ [82]
	temporal order judgement task	temporal window ↑ sensitivity ↓ [83,84]	sound → visual temporal sensitivity ↑ [85]
spatial MSI	spatial ventriloquist task	n.a.	no difference [86]
object MSI	object integration task	no difference [87]	n.a.
speech MSI	McGurk effect task	McGurk effect ↓ ASD [88–91]; schizophrenia [92–94]

2. Brain regions involved in multisensory integration

On the way to deciphering the neural basis of MSI, anatomical tracing, electrophysiology and neuroimaging experiments revealed several key areas in the mammalian brain (figure 2). The first brain area where MSI was identified is the SC. Early studies in cats and primates identified the role of the SC in the accurate orientation of the multisensory stimuli in space. Not only the SC but several cortical areas were also identified as hotspots for MSI. One is the association cortex, which receives convergent inputs from multiple sensory cortices. These association cortical areas are scattered across the parietal, temporal and frontal lobes. For example, the parietal areas, the ventrolateral prefrontal cortex (vlPFC) and the STS receive visual and auditory inputs directly [95–97]. Furthermore, the primary sensory cortices have been shown to respond to other modality information through mutual interactions [30,98]. Beyond the cortex, even sensory thalamic nuclei were involved in processing multisensory information [34]. Therefore, MSI occurs in multiple brain areas in parallel, and this might be important for the flexible integration of sensory stimuli that constantly bombard an animal with distinct features in space and time. In this section, we tried to cover not all but most of the important brain areas that are known to be involved in MSI.

Figure 2.

Multisensory brain areas. (a–c) Schematic overview of brain areas that process multisensory information. The equivalent cortical and subcortical multisensory areas were labelled in human (a), macaque monkey (b) and mouse (c) brains. These areas were selected based on data from electrophysiological recordings, functional imaging and anatomical tracing studies in these animals. Purple, frontal areas; MPC, medial prefrontal cortex; IFG, inferior frontal gyrus; ACC, anterior cingulate cortex; vlPFC, ventrolateral prefrontal cortex. Magenta, parietal areas; PPC, posterior parietal cortex; SPL, superior parietal lobule; IPS, intraparietal sulcus; VIP, ventral intraparietal area; LIP, lateral intraparietal area. Cyan, temporal area; STS, superior temporal sulcus. Red, primary visual cortex (V1); blue, primary auditory cortex (A1). Subcortical structures: orange, thalamus; green, superior colliculus (SC); yellow, amygdala. Areas residing inside the cortical surface (thalamus, amygdala, SC, MPC and ACC) are coloured transparently. (Online version in colour.)

(a) . Superior colliculus

The first brain area systemically examined as a potential hub for MSI is the SC [99]. The SC is a subcortical midbrain structure largely known for its importance in gaze orientation towards visual stimuli [100]. The pioneering work to identify multisensory neurons in the brain was performed in the SC of anaesthetized cats using single-unit recordings [1], and similar multisensory neurons were found later in the SC of rhesus monkeys [2,3] and alert cats [4] (figure 1a). While neurons in the superficial layers of the SC were primarily involved in visual responses, multisensory neurons were in the deep layers of the SC [1,3,5]. These neurons changed their responses (either enhanced or depressed) to the multisensory stimuli compared with their unisensory responses depending on the configuration of the stimuli (e.g. stimulus combination, timing or physical properties). One important MSI property of the SC neurons was nonlinear multisensory responses, following the principle of inverse effectiveness, which explains that the multisensory responses of the SC neurons were more robust when the unisensory stimuli that evoked less response were combined [1,4,6,7]. Another interesting observation was that the multisensory activities of deep-layer SC neurons were highly dependent on spatial and temporal coincidence (see §3c below) [3,4,7,8]. The inverse effectiveness and the spatial matching principles of MSI in the SC were also demonstrated in mice through large-scale electrophysiology and modelling of a neural population with nonlinear audiovisual responses [9].

SC receives auditory and visual inputs from a variety of cortical (both sensory and association) and subcortical (such as the brachium of the inferior colliculus) areas [101], and these converging inputs may be critical for shaping the multisensory responses of the SC neurons. For example, two cortical areas projecting onto the SC, the anterior ectosylvian sulcus (AES) [102,103] and the rostral lateral suprasylvian sulcus (rLS) [104,105], were found to influence the multisensory properties of the cat SC. A reversible deactivation of these areas disturbed multisensory orientation behaviours in cats [106,107] and abolished the multisensory activities in the SC [10–12]. Furthermore, the multisensory activities of the SC did not develop if these cortical areas were removed in the early development of cats [13]. The SC is an output area that projects directly to the motor and premotor neurons in the brainstem or the spinal cord, and a majority of these descending projections were more active in response to the multisensory stimuli [5]. The SC multisensory neurons were critical for the multisensory behaviours, as lesions of the SC in cats demolished enhancement and depression of orientation behaviours by multimodal stimuli, although the modality-specific orientation behaviours were maintained [108]. Therefore, the multisensory activity of the SC neurons is potentially important for accurate orientation responses of animals in the context of real-life multisensory environments.

(b) . Parietal cortex

Areas of the parietal cortex have been found to be involved in MSI. Diffusion tensor imaging has shown that individuals with higher white matter connectivity of the superior parietal lobule (SPL) and the intraparietal sulcus (IPS) with early sensory areas displayed faster detection of the multisensory stimuli [109]. Furthermore, intracranial recordings in human SPL found nonlinear multisensory interactions between simple auditory and visual stimuli [19]. Through fMRI in humans, the SPL and the IPS were also found to be involved in integrating multisensory motion stimuli, their responses being stronger to congruent audiovisual motion [110]. Moreover, the IPS was shown to be involved in both visual and auditory motion discrimination, and its activity was enhanced during crossmodal comparison of motion speeds [111]. MSI in these areas was not straightforward but further modulated by stimulus or behavioural conditions. For example, Regenbogen et al. have demonstrated that the IPS is more involved in processing noise-rich audiovisual stimuli, while noise-free stimuli were integrated at early sensory cortices, suggesting a specific role of the IPS in more difficult MSI [50]. Other studies have shown that the IPS weighted sensory signals according to the sensory reliability and the task relevance in a ventriloquism-like spatial localization task [46] and that super/sub-additivity of the IPS response to multisensory stimuli predicted the accuracy of subjects' performance in a semantic object categorization task [51].

The intraparietal cortices, such as the ventral intraparietal area (VIP) and the lateral intraparietal area (LIP), were also shown to be candidate areas for MSI in primates. The VIP was found to display visual–tactile MSI [112–113]. Spatial and temporal congruency between visual and tactile stimuli was important for the MSI effects on the VIP responses, which followed the principle of inverse effectiveness [112]. Not only visual–tactile but auditory responses were observed in the VIP. The receptive fields of visual and auditory stimuli were found to be highly overlapping in space across the VIP neurons, and the VIP neurons represent modality-invariant external space [114]. Furthermore, similar areas within the borders of VIP were found to be activated in response to both visual and auditory moving stimuli, although the visual activation was much more robust [115]. Visual and auditory responses were also found in the LIP. In monkeys performing memory-guided saccades to unimodal visual or auditory cues, spatially tuned bimodal neurons in the LIP showed activity in both sensory-guided tasks [116,117]. However, so far, studies have just found that neurons in the VIP and the LIP show their bimodal responses to visual and auditory stimuli, not their roles in audiovisual MSI or any changes in their responses to audiovisual multisensory stimuli. Future studies are required to understand the audiovisual MSI in these areas more clearly.

Due to its connectivity with sensory, motor and association areas, the posterior parietal cortex (PPC) is also regarded as a potential MSI region in rodents [118,119]. Song et al. found that feedforward inhibition in the mouse PPC is responsible for auditory dominance in go/no-go decisions under audiovisual conflicts [20]. However, in rats performing categorization-based left–right decisions, Raposo et al. showed that inactivation of the PPC did not affect the multisensory decisions enhanced by congruent audiovisual stimuli but only impaired the visually guided performance [120]. In another study, mice made left–right choices when they detected changes in visual or auditory stimuli. Inactivation of the PPC did not affect their behavioural choices in response to changes in either auditory, visual or audiovisual stimuli [121]. Therefore, the rodent PPC may primarily process visual information for making decisions. This process can be inhibited by incongruent auditory stimuli presented together with visual stimuli that lead to go/no-go action decisions [20,21].

(c) . Temporal cortex

Another well-known cortical area that displays MSI is the temporal cortex [122]. Within this region, the STS has been recognized as a critical area involved in MSI. In vivo single-unit recordings in macaque monkeys revealed multisensory neurons in the STS [22,23]. For example, the STS neurons that showed visual responses were further modulated by sound stimuli congruent with the visual stimulus [22]. In humans, fMRI and PET imaging also revealed activation of the STS during semantic integration of audiovisual stimuli [63–65]. In the year 2000, Calvert and colleagues reported fMRI experiments with semantically matching and non-matching audiovisual stimuli. They found that the matching audiovisual stimuli evoked supra-additive enhancement, and the non-matching stimuli evoked sub-additive suppression of the activity in the left STS [63]. Moreover, other lines of studies showed that higher activity in the left posterior STS was responsible for a stronger McGurk illusion, an illusory speech perception induced by a mismatch between the sound and the lip shape saying a syllable, suggesting a role of the STS in generating the McGurk illusion of the conflicting audiovisual stimuli [64,65] (figure 1e). Supporting this, disruption of the STS by fMRI-guided transcranial magnetic stimulation caused a reduction of the McGurk effect [66].

Although many human brain imaging studies have examined audiovisual integration in the STS for speech perception (figure 1e), some studies found that the STS is also involved in MSI for perceiving non-speech objects [53,54] (figure 1d). In fMRI experiments by Beauchamp and colleagues, subjects were presented with an animal image, a man-made object or meaningless scrambled visual stimuli. In some trials, the visual stimuli were presented together with sound stimuli related to the visual objects or meaningless ripple sounds, either congruently or incongruently. Interestingly, posterior STS showed higher responses to meaningful unimodal stimuli and congruent audiovisual stimuli compared to meaningless ones and incongruent combinations [53]. Furthermore, Berger and colleagues found through an fMRI study that the STS was also active during the integration of an imaginary visual object and real auditory stimulus in a space, which evoked a ventriloquist illusion—an illusory sound localization to the place where the visual cue was present [47] (figure 1c). Like other multisensory areas previously described, multisensory response in the STS also showed the inverse effectiveness in both the speech and the object audiovisual integration conditions [55].

(d) . Frontal cortex

The areas in the frontal lobe, particularly the premotor and prefrontal cortices, have been suggested as potential areas for MSI due to their rich connections with other cortical and subcortical areas. For example, the frontal cortex was found to be connected with STS and the superior temporal gyrus (STG), which are well-known multisensory areas [123,124]. By making single-unit recordings in macaque monkeys receiving unimodal and multimodal face/non-face visual and corresponding vocalization/non-vocalization auditory stimuli, Sugihara and colleagues have shown that the neurons in the vlPFC respond not only to unimodal visual and auditory stimuli [125,126] but also to multimodal stimuli [68]. Multisensory neurons that showed both enhancement and suppression in their multisensory response were found in the vlPFC, although multisensory suppression was more prominent [68]. Unlike the parietal or temporal association areas, the vlPFC showed more selective responses to the face/vocalization multisensory stimuli rather than non-face/non-vocalization, demonstrating for the first time that single prefrontal neurons can integrate audiovisual communication information [68]. In a human fMRI study using linguistic visual and auditory stimuli consisting of words, phrases and sentences, the activity in the ventral region of the left inferior frontal gyrus (IFG) was important for the integration of audiovisual inputs during sentence comprehension [69]. In another human fMRI study focusing on the processing of the phonetic features of audiovisual speech using phonetically matching or conflicting vowel stimuli (with no McGurk effect), the inferior frontal region—specifically Broca's area—was found to increase activity in response to conflicting stimuli [70].

Another frontal cortex area that is potentially important for MSI is the anterior cingulate cortex (ACC). One human fMRI study used objects or scenes for the audiovisual stimuli and found that the ACC and the medial prefrontal cortex (MPC) showed strong multisensory responses to congruent audiovisual stimuli [59]. Another human fMRI study showed that the ACC was activated by both auditory and visual unimodal motion stimuli and displayed activity enhancement in an audiovisual speed comparison test, suggesting the role of ACC in MSI [111]. In mice, ACC showed a reciprocal connection with different sensory cortices [119]. Therefore, it is possible that the ACC is a potential candidate for MSI and processes and integrates sensory information across modalities. Future studies are required to understand the role of ACC in MSI.

(e) . Primary sensory cortices

Although many multisensory studies have focused on the higher association cortices, more recent studies have found that lower-level sensory areas, previously thought to be strictly unimodal, also contribute to MSI. Anatomical connectivity between the sensory cortices and other known multisensory areas has been observed [119,127–135]. For example, tracing experiments in non-human primates [129,130] and rodents [119,132,134] have shown that the primary visual cortex (V1) is connected to low-level auditory areas, including the primary auditory cortex (A1). In humans, connectivity between visual and auditory cortices was also demonstrated through diffusion tensor imaging [127]. However, the effect of cross-modal inputs in sensory cortices can be additive or suppressive depending on the recipient modality and the stimulus configurations. Here, we describe some examples reported so far.

In vivo neural recordings in rodents [24,44,98] and non-human primates [31,32] revealed that sensory inputs from other modalities modulated neuronal activity in the primary sensory cortices. Recordings in the A1 of awake Mongolian gerbils presented with combined flash and sound burst stimuli found enhancement or suppression of auditory responses in more than a quarter of the recorded neurons, and this modulation was more prominent when the two stimuli were temporally closer and when the visual stimulus is given first [44]. Similar results were found by Kayser and colleagues in the A1 of rhesus monkeys presented with naturalistic audiovisual stimuli [32], and they showed that this multisensory modulation of A1 neurons was dependent on the congruency of the audiovisual stimuli and followed the principle of inverse effectiveness [31,32]. Furthermore, Atilgan et al. found that temporally matching visual stimuli facilitated a more accurate representation of the auditory information in the A1 via direct inputs from the visual cortex (VC) [30].

Through various electrophysiological recordings in mice, Iurilli and colleagues showed that activation of the A1 inflicted GABAergic inhibition on the V1, and visual responses were decreased in response to audiovisual stimuli [24]. Moreover, they showed a reflection of this on behaviour, as mice showed reduced response to a conditioned visual stimulus when the visual stimulus was temporally coupled with an auditory stimulus [24]. Suppression of visual responses in V1 of mice by long-range projections from auditory cortex (AC) was also demonstrated by Garner and Keller, by coupling preceding unimodal audio cues to unimodal visual stimuli in a behaviourally relevant context [98]. This effect was found to be experience-dependent and occurred only after learning [98]. Furthermore, a two-photon calcium imaging of V1 neurons in awake mice showed that their responses were modulated (either enhanced or suppressed) by audiovisual stimuli depending on the configuration of the multisensory stimuli such as composition and contrast, as well as temporal congruency [25].

Audiovisual MSI in low-level sensory cortices has also been investigated through haemodynamic and electromagnetic brain imaging methods. In both V1 and A1, early cross-modal activations and audiovisual interactions were observed through MEG and fMRI in humans [26]. Phase-modulation of visual responses in the VC areas, including the V1, was enhanced by the addition of the same-phase auditory stimuli as measured in humans by EEG [43] and subdural recordings [27]. Similarly, auditory responses in the AC areas, including the A1, were found to be enhanced by visual stimuli through an fMRI study in macaque monkeys [33] as well as an EEG experiment in humans [45].

(f) . Thalamus

The sensory cortices are connected not only to other cortical areas but also to subcortical structures like the thalamus [132]. Sensory thalamic nuclei project not only to the cortical region of their own modality but also to the other primary sensory areas, showing a potential role of MSI [132,136]. Single-neuron recordings in rats performing an auditory spatial discrimination task with or without visual cues found that auditory thalamic neurons responded to not only auditory but also audiovisual stimuli and that the early phase of auditory thalamic responses was driven by visual signals [35]. Multisensory activity in the sensory thalamus was also observed in human subjects. In an fMRI study on human subjects judging whether two bars passed or collided with each other, a collision sound presented together with visual stimuli in some trials evoked more activity in the posterior thalamus when subjects perceived collision in audiovisual trials [36]. Another fMRI study by Noesselt et al. has shown that visual and auditory thalamic nuclei are involved in the enhanced detection of low-contrast visual stimuli by auditory co-stimulation [34]. Furthermore, van den Brink and colleagues found that the connectivity in the auditory stream from the cochlear nucleus to the auditory thalamus and to the A1, measured by diffusion tensor imaging, can predict a human's ability to combine visual and auditory information during an audiovisual integration task [37]. Collectively, these studies support the idea that sensory thalamic nuclei not only relay specific modality information to their corresponding cortices but are also directly involved in MSI processing.

Due to its connectivity to sensory, multisensory and premotor areas [137–140], the medial pulvinar nucleus of the thalamus has also been suggested as a potential MSI region (see more details in the review paper by Froesel et al. [141]). Auditory [142] and visual [143] responsive neurons were found in the pulvinar in separate studies. More recently, single-unit recordings in macaque monkeys have revealed multisensory neurons that show suppressive responses to multisensory stimuli [38]. In this study, they also identified ‘complex multisensory’ neurons, whose activity depended on the categories of the stimuli [38]. Therefore, the pulvinar might be one of the thalamic areas that play a critical role in MSI.

(g) . Amygdala

Animals must make appropriate reactions for survival in an environment with dangerous stimuli. The pathways processing threatening stimuli from each sensory modality have been shown to project onto the amygdala [144–146], a core midbrain structure that is important for emotional processing and fear responses [147]. Human studies have shown that lesions of the amygdala decrease the auditory recognition of fear and anger intonations [148] and the visual detection of angry, threatening faces [149]. Information conveying threats rarely comes from a single modality, making MSI in the amygdala potentially valuable for survival. Very recently, Shan et al. have investigated the active participation of the amygdala and its close regions in audiovisual MSI through multi-array recordings in macaque monkeys by presenting them with auditory, visual, and audiovisual looming and receding stimuli [39]. Both positive and negative modulation of sensory responses (mostly auditory responses) was observed in the amygdala neurons when the audiovisual stimuli were presented [39]. Human fMRI studies also showed that the amygdala is involved in the facilitation of fear perception in the face when the visual stimulus is paired with an emotionally congruent rather than an incongruent fearful auditory stimulus [40]. Furthermore, the amygdala showed stronger responses to audiovisual (face–voice) emotional stimuli compared with unimodal emotional stimuli, independent of the types of emotion [41]. Further research is required to understand the role of the amygdala in the MSI of emotional stimuli in more detail.

3. The flexibility of multisensory integration and decision makings

In a rapidly changing environment, one of the most important issues for animals would be adapting their behaviours according to given sensory inputs that vary in time and space across modalities. Integration of different modality inputs is important for animals to make optimal decisions in a multisensory environment. Therefore, the integration process needs to be flexible in a multisensory environment, according to the animal's state. So far, we have described brain areas spanning the cortical regions, from the higher association cortex to the lower sensory cortex and subcortical areas, including the colliculus and thalamus, and the unique or similar principles of MSI. Previous studies in humans, however, found huge variations in multisensory perception of the McGurk illusion and Colavita visual dominance during the detection of audiovisual stimuli across individuals and experimental conditions [150–153]. A human twin study revealed that shared genetic background was not sufficient to explain all individual differences in McGurk effects, as the authors observed wide variations of the McGurk effect even in twins [154]. These variations naturally suggest that MSI is not uniformly developed across individuals. Rather, it can be modified during development as well as by experience and learning throughout life [155]. The flexible change in MSI can happen even in a faster time window with state changes in individuals [156]. However, the neural mechanisms underlying the flexibility in MSI are still not fully understood.

This section introduces the three categories that can induce flexible changes in MSI: development/ageing (figure 3a), experience/learning (figure 3b,c) and attention/internal states (figure 3d). We have mainly described the flexibility of MSI found in humans, with an emphasis on the temporal, spatial and semantic (both object and speech) MSI (figure 1). We have also introduced animal studies, which pioneered a path for understanding the neural circuit mechanisms of flexible MSI by using state-of-the-art methods.

Figure 3.

Factors that cause flexible changes in MSI. (a) Left, age-dependent change of McGurk illusory perception [157]. Note that there is a gradual increase in illusory perception with increasing age. Right, the multisensory response of young adults (black) and older adults (grey) to audiovisual stimuli [52]. (b) Spatio-temporal modulation of MSI by experience and learning during the task. Left, spatial information of each modality before and after experience or training [158–161]. Right, the proportion of simultaneity reports before and after the training at different stimulus-onset asynchronies (SOAs) [42,162,163]. Arrows indicate temporal binding window (TBW). Blue regular weight text and continuous lines, auditory stimuli; red bold text and dashed lines, visual stimuli. (c) Modulation of MSI by semantic experience and learning. Top, multisensory responses in the STS, IFG and IPS during audiovisual category learning [56]. Bottom, multisensory responses in the STS to a congruent (black) or an incongruent (yellow) speech [63]. (d) Top, effects of the top-down spatial attention on multisensory responses in the brain [164–167]. Grey, multisensory response to an unattended stimulus; black, multisensory response to an attended stimulus. Bottom, multisensory responses to expected multisensory stimuli after lever press (grey) or unexpected onset of stimuli (black) [168]. stim, stimulus.

(a) . Development/ageing

As multisensory organisms, animals already develop their multisensory processing ability in the prenatal stage [169]. Along with the gradual development of unisensory systems [14,170,171], multisensory systems also develop postnatally [14,99]. Pioneering studies on the development of multisensory processing found age-dependent changes in MSI in cat brains [14,99]. For example, Wallace and colleagues conducted electrophysiological recordings in the cat SC throughout postnatal development. There was a gradual increase in the number of multisensory neurons, and the multisensory responses in the SC neurons increased as cats became older [14]. A similar result was also found in the SC of the monkey [15]. Furthermore, early visual deprivation in cats by raising them in a dark room when they were young caused a significant decrease in multisensory responses in the SC and AES [16,172]. The AC of human patients with congenital cataracts at an early age also showed a reduced multisensory response, while abnormal visual suppression by co-presented sound was observed in the VC [28]. Patients suffering cataracts at an early age also showed deficits in MSI, as their visual perception was less affected by a distractor tone than in normal people [173]. These studies provided direct evidence of the gradual emergence of multisensory neurons in the brain and the importance of sensory experiences during postnatal development.

Several studies measured the behavioural reactions of human infants to multisensory stimuli and established a canonical perspective on how the ability of MSI develops in the brain [155]. Multisensory perceptual narrowing theory explains the development of MSI as occurring in three stages: immature, broadly tuned and narrowly tuned stages [155]. During the immature stage, non-verbal infants (less than one month old) have preferential reactions to multisensory stimuli [174,175]. In this stage, infants preferentially respond to multisensory stimuli based on intensity-matching or temporal synchrony [175,176], although the dependency of their multisensory preference on the spatio-temporal synchrony of multisensory stimuli was relatively low in this stage [174,177–179]. Moreover, human infants in the immature stage tend to react equally to unnatural multisensory stimuli (e.g. monkey faces with arbitrary tones) [174]. These results indicate that MSI is an innate feature of the brain and non-specifically happens in the early developmental period regardless of spatio-temporal asynchrony between modalities. With exposure to stochastically frequent multisensory stimuli that are spatio-temporally coherent and possess natural meanings between modalities, human infants develop some degree of specificity in MSI. As a result, human infants in the broadly tuned stage (five to eight months old) showed a narrower temporal binding window [177,178] as well as increased attention to naturally coherent multisensory stimuli [180,181]. Moreover, infants at this stage develop an audiovisual perceptual ability to be more likely to respond to coherent multisensory stimuli and experience the McGurk effect [182]. In the narrowly tuned stage (approx. 12 months old), babies develop more narrowed MSI, preferentially perceiving and responding to coherent native speech [183,184] (figure 3a, left). Overall, human infants show flexible changes in MSI during postnatal development, such as perceptual narrowing in MSI by binding more specific and coherent multisensory stimuli. Further studies are required to help our understanding of the neural mechanisms underpinning such development.

Humans continue to show changes in MSI as they get older and become adults. For example, Robinson and Sloutsky showed age-dependent changes in the dominant modality for decision, as the children in their study (who were approx. 4 years old) were much more responsive to the auditory stimuli and usually showed auditory-based prediction for upcoming events [185]. Similarly, the McGurk effect and Colavita visual dominance effect were less prominent in the children, as visual information was less effective in manipulating their auditory perception [157,186]. These results indicate that humans develop more visually dominant MSI as they age. Even in simple MSI conditions, humans show more enhancement in their responses to multisensory stimuli than to unisensory stimuli with ageing. This improvement in MSI effects occurs even during the adolescent period, as human subjects showed faster reactions to multisensory stimuli than to unisensory stimuli as they aged during adolescence, and reached their maximum performance at the age of 14 [187]. These reports suggest that developmental maturation of the brain during adolescence is critical for the maturation of MSI.

There are also differences in MSI in fully grown adults according to age. For example, the PPC and the MPC of older adults responded more prominently to multisensory stimuli than to unisensory stimuli, whereas sensory areas showed weaker responses than in the younger group [52] (figure 3a, right). Moreover, only the older adults showed a significant correlation between the level of PPC and MPC activities and the ability to perceive multisensory stimuli [52]. These results suggest that the brain develops an adapted strategy with ageing to maximize the benefits of MSI [52,188]. However, older participants with minor hearing loss tended to show less auditory perception and showed smaller MSI for the spatial localization of the audiovisual stimuli [189] (figure 1c) while increasing the weight of visual modality for MSI [52]. Therefore, changes in MSI with ageing may result from changes in unisensory systems. Regardless, developmental maturation and ageing are critical factors for long-term adaptation and lifespan flexibility of MSI processing in the brain.

(b) . Experience/learning

Animals face numerous multisensory events from the environment throughout their life. From the perspective of the brain as a Bayesian receiver, the brain undergoes dynamic perceptual recalibration of the external world and internal priors according to experience [190–192]. A causal inference explained audiovisual spatial integration in humans by using maximum-likelihood estimation (MLE) about unisensory perception and an individual's prior beliefs [193,194] (but see [195] for inconsistency about the MLE). Therefore, prior experience would be one of the major factors that can cause flexible MSI. This section discusses how experience and flexible learning shape MSI in the brain. We divided the cases into three categories: non-instructive exposure to the stimuli without any feedback or rewards, instructive exposure with feedback during learning and effect of prior knowledge.

Repetitive exposure to multisensory events without any associated rewards can shape spatial MSI responses in humans [155]. For example, repetitive exposure to ventriloquism-inducing audiovisual stimuli without any feedback induced humans to perceive more illusory localization of the sound stimuli to the visual cues, which is called the ventriloquism aftereffect [158,159,196] (figure 3b, left). Hong and colleagues also found that repetitive exposure to spatio-temporally congruent or incongruent stimuli enhanced or suppressed the common cause prior (an assumption that different modality stimuli are from the same object) of human participants during spatial MSI [159]. Furthermore, multisensory exposure attenuated auditory sensitivity during spatial integration in the STG [48]. These studies indicate that multisensory exposures enhance the weighting of visual information during spatial MSI.

Other studies showed that the association of random auditory cues with visual objects disrupts visual processing in the brain [17,57,58,197,198]. For example, meaningless sound stimuli co-presented with visual objects induced robust multisensory responses in the human middle temporal gyrus (MTG) and STG [57,58]. Interestingly, the robustness of multisensory responses in the MTG is negatively correlated with the performance of subjects in the following visual discrimination and object categorization after the multisensory exposure, suggesting that multisensory exposure dampens visual recognition [198]. In other studies, repetitive audiovisual exposure for visual-to-auditory substitution switched visual-preferring activity to auditory-preferring activity in the MTG [197]. Changes in MSI in the brain were observed not only humans but also in other animals. For example, experiencing spatio-temporally overlapping multisensory stimuli but not the non-overlapping stimuli increased multisensory responses in the cat SC [17]. Recent rodent studies have also showed that repeated multisensory exposure drives a multisensory response in the VC [29]. However, it is still obscure how other higher brain areas are affected by multisensory exposure in animals.

Animals learn the meaning of the stimuli rapidly, not only as a result of simple exposure to the stimuli, but also after experiencing stimuli associated with specific outcomes makes. Such learning induces changes in MSI as the values of specific sensory information change in distinct sensory modalities. For example, when human and animal subjects learned to associate specific stimuli (either auditory or visual) with designated reward values, they recalibrated the weights of the stimuli according to the values during spatial integration in the ventriloquism task [160,161] (figure 3b, left). Similarly, learning the simultaneity judgement task (figure 1b) increased correct judgements of the timing in visual and auditory stimuli and sharpened the temporal binding window, during which the subject reported that visual and auditory stimuli are presented at the same time and integrated as a multisensory stimulus [42,162,163] (figure 3b, right). The enhancement of functional connectivity accompanied the sharpening of the temporal binding window among the STS, VC and AC [42]. Interestingly, such sharpening was not observed if the subject was not informed about the correctness of their judgements [42,162]. These results strongly suggest that it was not the simple exposure but the learning of the task that induced changes in MSI.

Semantic learning also changed MSI. One study in humans tested an audiovisual object association task, in which subjects needed to report whether or not the presented audiovisual stimuli were congruent while their brain activity was measured [56] (figure 1d). The brain develops multisensory responses in the STS, IFG and IPS during learning of the audiovisual category [56] (figure 3c, top). In the case of animal studies, the modulation of primary sensory areas by multisensory learning has been examined in rodents. Interestingly, when mice were trained to discriminate paired audiovisual stimuli, perceptual learning enhanced multisensory decisions and responses in both the AC [199,200] and the VC [98]. More animal studies are required to understand how learning changes MSI properties in the association cortex.

Lastly, another considerable aspect of the flexibility of MSI is that our brain processes the multisensory information depending on our prior knowledge. For example, it has been shown that working memory of a previous multisensory experience influences the multisensory responses of PFC neurons [60]. Multiple studies have conducted experiments on how the congruency between auditory and visual stimuli affects human multisensory perception (figure 1d,e). Interestingly, congruent and incongruent stimuli induced different responses in certain brain areas, and they are processed in different brain areas, including the STS and frontal areas [56,201]. A previous study showed that temporally and contextually congruent audiovisual speech enhanced cortical representation of speech [71]. For example, during speech perception, the STS area shows multisensory enhancement in response to congruent speech stimuli while exhibiting depressed multisensory responses to incongruent stimuli [63] (figure 3c, bottom). Moreover, in a series of studies, it has been established that congruent stimuli are mainly processed in the PFC and the STS, while incongruent stimuli induce strong responses in the ACC [56,61,63]. These data support the idea that the PFC and the ACC in the frontal area have roles in the semantic categorization of congruent and incongruent multisensory stimuli [56,61]. Meanwhile, the PPC processed both the congruent and incongruent stimuli equally [61]. Because the congruency between modalities is based on the subject's knowledge, it is highly plausible that MSI can flexibly change in the brain according to prior knowledge.

(c) . Attention/internal brain state

MSI is also affected by subjects' internal states. In the case of the McGurk illusory perception, some studies revealed that the level of illusory perception is highly correlated with pre-stimulus activity of the STS in humans [67,152]. The pre-stimulus activity of the STS affected the functional connectivity between the STS and the auditory areas, which influenced how much auditory information contributes to multisensory perception [67]. In general, pre-stimulus activity is highly associated with ongoing changes in internal states, which reflect attention, expectation, arousal and self-generated movements. Therefore, any factors that can change the internal states can strongly affect and modulate MSI.

Attention is a fundamental cognitive state of the brain formed by complex neural mechanisms, which select sensory elements for further processing among multiple elements presented [202]. From the beginning of the study of sensation and perception, attention has been considered one of the most decisive factors that control sensory processing. Attention can either be driven by external stimuli through bottom-up modulation or consciously directed in a top-down fashion [164]. For example, salient multisensory stimuli drove the bottom-up attention of the observer and improved the perception of individuals in [203,204]. Interestingly, spatio-temporally coherent multisensory stimuli attracted stronger bottom-up attention accompanied by saccadic movement and induced stronger responses of SC neurons to the stimuli than the incoherent stimuli [8,18,205,206]. Therefore, the attention recruited by stimuli in a bottom-up fashion can modulate MSI, and it is a mechanism that can selectively integrate important and relevant stimuli for the survival of an animal in a complicated environment with numerous sensory inputs.

It is well known that spatial attention reshapes the receptive-field properties in the lower sensory areas, including the VC and AC [62,207–209], which may enhance MSI in higher cortical areas. Supporting this idea, a series of studies have found how the instructed top-down spatial attention modulates MSI [164–166,210]. If multimodal stimuli were presented at the spatially attended area, the multimodal brain areas, such as the SC and the STS, showed robust integrative responses (figure 3d, top). Attention guided to specific sensory modalities can also affect MSI. For example, effective MSI was only observed when participants were guided to attend to both modalities [167,211,212]. Moreover, the level of attention to the multisensory stimuli showed a significant correlation with the enhancement in the perception of the multisensory information [211,213]. Attention to the given multisensory stimuli can be degraded if subjects are guided to attend to other tasks irrelevant to MSI [213,214]. For instance, human subjects experienced less sound-induced illusory flash perception (SIFI) if they were instructed to attend to other tasks, such as remembering letters given before the test stimuli [213]. In another study, both congruent and incongruent audiovisual stimuli accelerated reaction times if participants were instructed to report the onset of any stimuli (multisensory detection task) [214]. However, the incongruent stimuli caused an even slower reaction time than unisensory stimuli during the discrimination task, during which participants should report only the correct stimuli [214]. These studies show that MSI is highly influenced by the level of attention given to the stimuli or task paradigms.

On the other hand, one modality can be more potent than the other depending on which modality information is more attended to within the multisensory stimuli. For example, several studies have examined how the attention given to specific features of the stimuli, such as temporal information or spatial information, determines prepotent modality during MSI. When participants were asked to focus on temporal information, they showed more auditory dominance in perceiving audiovisual stimuli [165,215]. Conversely, attention to spatial information caused visually dominant perception in participants through modulation of neural response in contralateral AC and STG from the direction of biased perception [48,49,165,216]. These results support the ‘modality appropriateness’ theory, which explains how an animal uses modality-specific information to appropriately process specific features in the multimodal sensory information [217]. These results also show how top-down attention flexibly changes MSI in the brain.

In the spatial localization task (figure 1c), human subjects showed less audiovisual integration if the stimuli were generated by themselves [168] (figure 3d, bottom). Therefore, multisensory perception may be affected by a corollary discharge signal in the brain, which refers to the signal from motor areas that evoke self-generated movements [218]. Similarly, the decisions of mice under audiovisual conflicts, in which auditory and visual stimuli have opposite meanings, were measured in two different behavioural states: a head-restrained setup in a stationary state and a freely navigating setup in the T-maze [20,21]. Even though the perceptual discrimination performances were similar between the auditory and the visual stimuli, the mouse followed the auditory rules for the decisions in the head-restrained state while they followed the visual rules during navigation in the T-maze [20,21]. Therefore, corollary discharge in the brain may contribute to the different multisensory decisions depending on the locomotion state.

4. Changes in multisensory integration in neurodevelopmental disorders

Deficits in MSI are shown in a number of neurodevelopmental disorders, such as ASD [219], schizophrenia [220], developmental dyslexia [221] and attention-deficit/hyperactivity disorder (ADHD) [222]. In particular, ASD and schizophrenia are the most prevalent neurodevelopmental disorders accompanied by multiple behavioural abnormalities. Patients with ASD and schizophrenia show typical abnormalities in their behaviours. According to the Diagnostic and Statistical Manual of Mental Disorders-IV (DSM-IV) criteria [223], ASD patients show common symptoms such as persistent impairment in reciprocal social communication and interaction and restricted and repetitive behaviours [223,224]. On the other hand, schizophrenia patients show distinct symptoms such as delusions, hallucinations and disorganized speech [223,225]. Overall, both types of patients show deficits in sensory processing. ASD patients show hyper- or hyposensitivity to auditory, visual, tactile and olfactory stimuli, and these symptoms appear in various combinations across patients [226–230]. Patients with schizophrenia also often show hyper- or hyposensitivity to auditory and visual stimuli, and they sometimes experience distorted auditory and visual perception and hallucinations [231–235]. With such backgrounds, ASD and schizophrenia patients show similar or distinct impairments in sensory processing and perception (table 2). In this section, we describe abnormal MSI behaviours that appeared in these patients and neural mechanisms of such changes in MSI found in human patients and animal models with ASD and schizophrenia.

(a) . MSI deficits in autism

ASD patients not only have dysfunction in unisensory perception but they also have impairments in MSI. Many human studies have demonstrated abnormal audiovisual integration in ASD patients [73,77,81,87–91,219,236,237]. For instance, ASD children and teenagers (8–17 years old) who performed SIFI tasks showed stronger flash-beep illusions than normal children with typical development (TD) [81]. In the temporal order judgement task, during which the subject reports which one is presented first when visual and auditory stimuli were randomly given at intervals of 0 to hundreds of milliseconds, ASD children showed a wider temporal binding window and less sensitivity to timing differences [83,84]. Similarly, in the simultaneity judgement task (figure 1b), during which the subject determines whether the visual and the auditory stimuli were presented simultaneously or not, ASD children also showed a wider temporal binding window than TD children [77,78]. ASD patients also showed impairments in MSI during speech perception, and the McGurk effect was significantly reduced in ASD children compared to TD children [87–91,237].

Interestingly, unlike ASD children, older ASD patients (teenager to adult) showed a tendency to recover normal behaviours in these temporal or speech MSI tasks [219,236,238–240]. For example, multisensory speech recognition was impaired when ASD children (5–12 years old) were presented with random monosyllabic words spoken by a female speaker, but ASD teenagers (13–15 years old) showed normal performance in the same task [236]. In other types of MSI, however, such as the object integration task (figure 1d), ASD children and teenagers (8–19 years old) showed normal accuracies [87]. In another study using a causal inference model, ASD patients (15–17 years old) showed impaired causal inference of multisensory stimuli with a tendency to attribute auditory and visual stimuli from separate locations to a common source, although some compensatory mechanisms recovered their reports during an explicit task [241]. Therefore, not all MSI or MSI-related behaviours were disrupted in ASD, and this might be due to disruption in specific brain areas of MSI in ASD patients.

Although many human studies showed MSI changes only in their behavioural phenotypes in ASD patients, some studies have tried to find the underlying neural mechanisms of the MSI impairments [72,73,242]. For example, in a computational modelling study, researchers made a network model that explained the connection between the sensory cortex and the posterior STS, and they trained the network with the behavioural data from the McGurk experience of TD and ASD children [73]. They found that the decreased connectivity between the sensory cortex and the posterior STS cortex, due to fewer sensory inputs to the sensory cortex, explained the abnormal audiovisual integration behaviour in ASD patients [73]. Moreover, in a human EEG study, task-related α power (8–12 Hz), which increases when a subject suppresses the task-irrelevant sensory information with attention, was reduced in the parieto-occipital scalp of ASD patients [242]. Therefore, weakening in the connection between sensory cortices and the association cortex might cause impairments of MSI in ASD patients.

To reveal the neural mechanisms of ASD deficits in MSI, researchers examined several mouse models of ASD. For example, the serotonin transporter (SERT) mutant mice, known as an animal model of ASD, showed an impaired perceptual ability to detect audiovisual cues [72]. This might be due to impaired serotonergic secretion from the dorsal raphe nucleus to the SC [243], one of the core areas for multisensory processing [1]. Another example is the Cntnap2 knockout rats, known as ASD models, and they showed less sensitivity to temporal asynchrony in the temporal order judgement task [244]. However, the neural mechanism underlying this MSI change is still unclear. Furthermore, various mouse models of ASD, including BTBR T + tf/J, GAD65, Shank3 and Mecp2 knockout mice, developed abnormal E/I balance in the insular cortex, which showed reduced responses to the audio-tactile stimuli [74]. Future study is needed to determine if audiovisual integration is also altered in the insular cortex of these animals.

(b) . MSI deficits in schizophrenia

Schizophrenia patients also showed impairments in audiovisual integration [75,79,86,92,93,220,245,,246]. For example, during the speech perception test, schizophrenia patients rarely experienced the McGurk illusion [92–94]. They also experienced lower fusion illusions in the SIFI task [82]. Interestingly, schizophrenic patients showed normal MSI in spatial integration of audiovisual stimuli as tested by the ventriloquism effect test [86] but clear deficits in temporal MSI of audiovisual stimuli during the temporal order judgement and the simultaneity judgement tasks [79,80,85]. Therefore, schizophrenic patients show impairments in different types of MSI, which are particularly sensitive to the temporal aspects, compared with ASD patients (table 2). Some studies examined the ketamine-treated rodents as an animal model of schizophrenia and found MSI deficits in tactile–visual or olfactory–visual integrations [247,248]. Future research is needed to determine whether audiovisual integration is similarly disturbed in the animal model of schizophrenia.

Neural mechanisms of MSI deficits in schizophrenia are largely unknown. However, some human EEG and fMRI studies figured out the key brain areas that showed abnormal neural activities in schizophrenia patients when they performed audiovisual integration tasks [75,76,93,249]. These studies found that the fronto-temporal regions, including the IFG, the MTG and the STS [75], the medio-central scalp regions [93] and the occipital–parietal scalp regions [249] are potentially key regions that harbour MSI deficits in schizophrenia patients. Moreover, beta–gamma band power was reduced in the occipital region of schizophrenia patients during the SIFI test [76].

In summary, ASD and schizophrenia patients and animal models showed similar or distinct impairments in MSI processing. Both ASD and schizophrenia patients showed abnormal MSI and perception during the McGurk effect, SIFI, temporal order judgement and simultaneity judgement tasks (table 2). Behavioural phenotype in McGurk effect and simultaneity judgement tasks is similar between ASD and schizophrenia patients, but there are still abnormal but different results in SIFI and temporal order judgement tasks (table 2). In the object integration task and the spatial ventriloquism task, ASD and schizophrenia patients show normal behaviour, although not all tasks were tested (table 2). The common brain areas impaired in ASD and schizophrenia were the STS and STG [73,75]. Neural mechanisms of MSI changes in ASD and schizophrenia are still elusive, and further research is necessary to discover them.

5. Conclusion and future directions

In this review, we overviewed the multisensory areas in the mammalian brain (figure 1). We reviewed these areas based on their involvement in the five types of MSI measured in mammals, including humans, non-human primates and rodents (table 1). In awake and behaving animals, MSI can flexibly change depending on long- and short-term changes in the internal states of an animal as well as in the conditions of developing psychiatric disorders. Future studies are required to understand the neural circuit mechanism of each change in MSI and how the altered MSI in disease conditions has an impact on patients suffering from abnormal perception in complicated sensory environments.

Ethics

This work did not require ethical approval from a human subject or animal welfare committee.

Data accessibility

This article has no additional data.

Authors' contributions

I.C.: conceptualization, writing—original draft, writing—review and editing; I.D.: conceptualization, writing—original draft, writing—review and editing; S.O.: conceptualization, writing—original draft, writing—review and editing; S.-H.L.: conceptualization, supervision, writing—original draft, writing—review and editing.

All authors gave final approval for publication and agreed to be held accountable for the work performed therein.

Conflict of interest declaration

We declare we have no competing interests.

Funding

This study was funded by grants to S.-H.L. from the National Research Foundation of Korea, funded by the Ministry of Science & ICT (NRF-2021R1A2C3012159) and the Institute for Basic Science (IBS-R002-A2).