Development of multisensory integration from the perspective of the individual neuron

Abstract

The ability to use cues from multiple senses in concert is a fundamental aspect of brain function. It maximizes the brain’s use of the information available to it at any given moment and enhances the physiological salience of external events. Because each sense conveys a unique perspective of the external world, synthesizing information across senses affords computational benefits that cannot otherwise be achieved. Multisensory integration not only has substantial survival value but can also create unique experiences that emerge when signals from different sensory channels are bound together. However, neurons in a newborn’s brain are not capable of multisensory integration, and studies in the midbrain have shown that the development of this process is not predetermined. Rather, its emergence and maturation critically depend on cross-modal experiences that alter the underlying neural circuit in such a way that optimizes multisensory integrative capabilities for the environment in which the animal will function.

The environment is rife with events that emit multiple types of energy (for example, electromagnetic radiation and pressure waves), but all can contain information about food, shelter, mates and/or danger. Even small enhancements in the ability to detect such signals and evaluate these biologically significant events can have a major impact on the survival of a species. Thus, it is endlessly fascinating to speculate about how selective pressures have spurred the evolution and aggregation of different sensory systems that maximize information gathering within different ecological niches. Each of an organism’s sensory systems is tuned to a different form of energy, and they can compensate for one another when necessary, as when hearing and touch compensate for vision under conditions of darkness. Given the diversity of possible ecological niches, it is perhaps not surprising that evolution has produced animals with widely divergent appearances, senses and sensory capabilities. However, no matter how exotic these variants may seem, they share a common innovation, one that was likely presaged by our single-celled progenitor: the ability to use their senses synergistically¹.

Biologically significant events are often registered by more than one sense. Because each sense independently derives and transmits a report of the event, more accurate perceptual evaluations of the event and behavioural decisions can be made through the synthesis of their different sensory signals^2–4. This process, called multisensory integration, increases the collective impact of biologically significant signals on the brain and enables the organism to achieve performance capabilities that it could not otherwise realize. Consequently, multisensory integration has enormous survival value and has undoubtedly played a far more important part in the evolutionary histories of extant species than is currently recognized. It is also a ubiquitous process that profoundly affects how we perceive the world and the decisions we make to best meet its challenges. However, we are nearly always unaware of this process. More often than not, the judgements we think we are making based on information from a single sense, such as vision, are strongly influenced by seemingly irrelevant but informative cues from other senses such as hearing and touch. Sensory judgements are rarely exclusive to a single sense because multiple sensory channels converge on and share the use of the neural processes that mediate perception and action. It is little wonder that interest in the operational features of multisensory integration has become so widespread.

How a developing nervous system creates this capability to use the senses interactively is even less well understood than the various functional domains in which it will ultimately be expressed. Although behavioural studies have shown that neonates can detect certain cross-modal correspondences very early in life⁵, physiological studies indicate that the capacity to integrate information across the senses is not an inherent feature of the newborn’s brain. Rather, as is discussed in detail here using information derived from interactions between the visual, auditory and somatosensory systems, this capability develops in an experience-dependent manner during early postnatal life. During this window of time, the operational parameters of multisensory integration are customized to the features of the local environment⁶; that is, multisensory processing rules are configured to efficiently process the cross-modal signal relationships that the organism encounters. This adaptation occurs alongside the maturation of the contributing unisensory systems. However, multisensory development does not require that each individual sense reaches its maturational end point, and the functional development of an individual sense does not require its interaction with another. Rather, multisensory development and unisensory development are interconnected but parallel processes that, at the level of the circuit, often have different computational targets and constraints. It may not even be appropriate to think of multisensory ‘development’ as strictly an early-life process. These issues are discussed in this Review.

Defining multisensory integration

Multisensory integration refers to the process by which inputs from two or more senses are combined to form a product that is distinct from, and thus cannot be easily ‘deconstructed’ to reconstitute the components from which it is created⁷. Whether considering neural signals or behavioural performance, this is defined operationally as a statistically significant difference between the response evoked by a cross-modal combination of stimuli and that evoked by the most effective of its components individually¹. With respect to single-neuron physiology, this comparison is made between the total number of impulses or firing rates evoked by stimuli and their combinations^1,7–9. This could result in response enhancement or response depression. These physiological changes produce alterations in sensation and perception, as well as the behaviours dependent on them. Multisensory enhancement, which is the most reliable index of multisensory integration (and will be discussed here most extensively) may reflect computations that yield response magnitudes that are equal to, less than or greater than the sum of the responses to the individual component stimuli⁷. In behaviour, performance enhancements are often quantified by evaluating differences in the accuracy and speed of detection, localization and/or identification of stimuli^3,4,10–26. In short, multisensory integration refers to a broad class of computations involving multiple sensory modalities in which information is integrated to produce an enhanced (or degraded) response. Other computations involving multisensory processing, such as comparing the features of a stimulus (for example, its shape) across modalities or detecting certain cross-modal correspondences in timing or rhythm²⁷, require that the comparators maintain their identity rather than being fused into a single product. As a consequence, they will probably have different, albeit currently unknown, underlying mechanisms and developmental time courses.

The multisensory superior colliculus neuron

It was in the 1970s that systematic efforts were first begun to understand the underlying neural circuits through which multisensory integration is achieved, the behavioural manifestations that reflect its operation and how this capability becomes instantiated in individual neurons during early life. These studies used the multisensory neuron in the cat superior colliculus as a model. Since that time, multisensory neurons have been identified in many brain areas and species^1,28–39, but most of the information about their development has been derived from studies of the cat superior colliculus. This is, in part, because it seems to be an excellent model.

Neurons in the deep layers of the superior colliculus are primary sites of multisensory convergence (neurons in overlying superficial layers are purely visual), thereby affording the opportunity to gain insight into the circuit requirements and initial processes that are involved in integrating cross-modal inputs before the resultant products are shared with other brain areas. Its utility as a model is also facilitated by the presence of many such neurons and their involvement in the well-defined behavioural roles of the superior colliculus: mediating the animal’s detection and localization of external events and its orientation to them^40–49, which are behaviours that gradually mature during postnatal life^50,51. As there is a comparatively high incidence of multisensory superior colliculus neurons, it is practical to target them in electrophysiological experiments, and as most of them are also output neurons that project to motor regions of the brainstem and spinal cord⁵², it is possible to evaluate their properties in the context of their influence on overt behaviour^1,53–55. The computations that describe these relationships and the factors affecting them have also been described using signal detection and Bayesian frameworks^56–58 that can inform future empirical studies. Furthermore, as the processing capabilities of superior colliculus neurons at birth are immature^59,60, their development can be followed during postnatal life, when many factors affecting their ultimate functional capability can be experimentally manipulated.

The overarching functional development of the superior colliculus, of which its multisensory integration capability is one important facet, reflects an architecture that facilitates efficient sensorimotor transduction. Each of its three sensory representations for sight, hearing and touch (that is, visual, auditory and somatosensory) develops so that it is laid out in a map-like manner^61–65, with all of the maps in overlapping spatial register with one another. This is a general feature of the superior colliculus that seems to be independent of species^66–74. The different maps are constituted, in large part, by multisensory neurons that have modality-specific receptive fields in register with one another. The sensory maps, in turn, overlap with a common motor map, which also involves many of the same multisensory neurons and through which orientation of the eyes, ears, head and limbs can be initiated. It is through this elegantly simple design that a salient environmental event excites a localized region of the map (or maps) to register an appropriate sensory report and initiate a coordinated overt response (FIG. 1) regardless of which sense or combination of senses was activated¹.

Figure 1. The organization and development of the multisensory superior colliculus.

a | The cut-away diagram shows the location of the superior colliculus (SC) in the midbrain of the cat and the association cortex (anterior ectosylvian sulcus (AES) and rostrolateral suprasylvian sulcus (rLS)), from which the SC receives crucial cortico-collicular inputs. b | The three sensory representations (visual, auditory and somatosensory; shown at the top) in the SC are organized into an overlapping multisensory topographic map, as shown below (grey map). In each individual map, the horizontal meridian runs roughly rostral–caudal and the vertical meridian runs medial– lateral. Thus, forward or central space is represented rostrally, rearward or peripheral space is represented caudally, superior space is represented medially and inferior space is represented laterally. The multisensory map shows the topographic correspondence among the three maps, with the purple regions encompassing the variations in the two meridians that exist among the three maps. External events, such as the presence of the rodent, are often registered by multiple senses (in this case, vision and audition) and relayed via converging cross-modal afferents onto common multisensory target neurons in the map, which are exemplified by crosses in the maps. In adult animals, this leads to enhancements in neuronal activity (that is, physiological salience) and, behaviourally, to a higher probability of detecting the event, localizing in space and orienting to it. c | The basic developmental chronology of sensory responsiveness within the deep layers of the cat SC is shown. Some neurons are already responsive to touch (somatosensation) prenatally. Hearing (audition) becomes effective in activating some SC neurons before the end of the first week of age and sight (vision) at approximately 3 weeks. Despite the convergence of inputs that produces multisensory neurons early in life, these neonatal multisensory neurons cannot yet integrate their cross-modal inputs. This capability for multisensory integration does not appear until approximately 4 weeks of age and gradually matures until the adult-like condition is achieved after several months. Part a is adapted with permission from REF. 101, The American Physiological Society. Part b is adapted with permission from Stein, Barry E., and M. Alex Meredith., The Merging of the Senses, Figure 8.1, © 1993 Massachusetts Institute of Technology, by permission of The MIT Press.

Convergence does not guarantee integration

It is important to note that although the capability of multisensory neurons to integrate their inputs will markedly facilitate the sensorimotor role of the superior colliculus, this capability does not inevitably result from the convergence of two or more sensory inputs onto a common neuron. Multisensory convergence develops in young animals long before those target neurons are capable of integrating their inputs. When an event is registered by more than one sense in such animals, the default operation of their (naïve) multisensory neurons is to respond as if the cross-modal signals were not complementary: the response to a cross-modal stimulus is no better than it is to the most effective of its individual component stimuli and is often a weighted average of those responses, and thus less than the most robust unisensory response^9,75–80. This may seem counterintuitive if one expects that the default mode of a naïve neuron is to sum its cross-modal inputs. However, the empirical data reinforce the need to compare a neuron’s multisensory response with the most effective of its unisensory component responses in order to examine the development and expression of multisensory integration. There are also several circumstances in which cross-modal stimulus configurations are not integrated in the mature animal. Once again, in each of these cases, neurons yield responses no greater than those to one of the component stimuli. Thus, although most multisensory superior colliculus neurons ultimately develop the ability to integrate their different sensory inputs and must be multisensory to do so, these two neuronal characteristics are not inextricably bound. In fact, their maturational asynchrony enables their independent evaluation.

Guiding principles of multisensory integration

Studies of adult cat superior colliculus neurons have yielded three general operating principles, or ‘rules of thumb’, for multisensory integration. The first two involve space and time. Cross-modal (for example, visual–auditory) cues that are in close spatial and temporal register generally enhance the responses of multisensory neurons, whereas those that are spatially or temporally disparate often elicit response depression or fail to be integrated^81,82. The third principle, that of ‘inverse effectiveness’ (REF. 83) (BOX 1), describes the observation that proportionately greater effects of cross-modal cues are obtained when those individual cues are weakly effective. Thus, the magnitude of multisensory integration is inversely related to the efficacy of the stimuli being integrated⁷.

Box 1. Enhancement, inverse effectiveness and superadditivity.

The primary function of the superior colliculus is to guide orienting behaviour towards salient external stimuli. Given that an organism can orient to but one stimulus at a time, it is reasonable to view the sensory environment at any given moment as consisting of a myriad of sensory-specific competitors, each vying to be the goal of the next orienting movement. With this in mind, the phenomena of multisensory enhancement and multisensory depression are readily understood as means towards resolving competition between mutually exclusive alternatives. In other words, stimuli from different modalities that are spatially congruent enhance the physiological salience of their commonly held spatial location, whereas those emanating from disparate locations mutually degrade. With respect to the activity of single superior colliculus multisensory neurons, multisensory enhancement is always defined as a response to a cross-modal stimulus that exceeds the response to either of its modality-specific components; however, it is important to note that it is not a uniform phenomenon. More specifically, the magnitude of the resultant enhancement, which is expressed as proportion of the best unisensory response, is inversely proportional to the efficacies of the modality-specific component stimuli — a relationship dubbed ‘inverse effectiveness’. When the enhanced multisensory response of a neuron is instead referenced to the sum of its responses to the modality-specific stimuli, inverse effectiveness typically manifests as a transition from superadditivity (more than the sum of the unisensory responses) to subadditivity, as the modality-specific stimuli themselves become more potent^7,169. However, regardless of how it is quantified, inverse effectiveness, and particularly superadditivity, suggests that multisensory integration is of most value for detecting stimuli that are weakly effective on their own. Along with appealing to intuition, modelling studies suggest that this particular feature of multisensory integration may be part of an optimal solution to the problem of detecting stimuli in the face of both sensory and neural noise^57,170, something that is very much in line with the primary function of the superior colliculus. As such, it is worth noting that inverse effectiveness may not generalize to, nor be particularly advantageous for, multisensory computations that contribute to functions beyond stimulus detection^13,171.

These principles are consistent with the presumptive ‘benefit’ of multisensory integration in this context: improving the ability to detect an external event, localizing it in space and using it as a target for a superior colliculus-mediated orientation response. The spatial and temporal components relate to the idea that because cross-modal stimuli are proximate in space and time, they are most likely to be derived from the same event. Enhancing the physiological impact of the initiating event through multisensory integration, especially when it provides only very weak cues, increases the probability that it will generate a superior colliculus-mediated response. By contrast, cross-modal stimuli that are disparate are more likely to belong to unrelated or competing events and will either fail to interact or will interact competitively, thereby producing response depression^83–85. Building this organizational framework during early life involves multiple steps. Afferents carrying signals that refer to common regions of sensory space are first routed onto common target neurons in topographically appropriate patterns. Then, the circuit configures its internal computations so that convergent cross-modal signals that are most probably derived from the same event can interact in complementary ways, whereas others either fail to interact or compete with one another. As shown below, the elaboration and refinement of this framework are dependent on several postnatal factors.

Development of multisensory integration

The superior colliculus of a newborn cat, the principal source of what we know about the development of multisensory integration, has no functional multisensory neurons and no multisensory integration capabilities. In late fetal stages, and for a few days following birth, the only active sensory-responsive neurons are those that respond to tactile cues (FIG. 1). Their receptive fields are on or around the mouth, nose and whisker pads⁵⁹, and help the kitten to process cues obtained from sweeping its face across its mother’s fur in search of the nipple. When the perioral region is anaesthetized with topical lidocaine, the kitten continues to sweep its face through the mother’s fur but does not find the nipple; however, when the animal’s mouth is placed on the nipple, it begins to suckle. Olfactory cues are of limited value in this context: they help the kitten to find the mother but are not of primary use in finding the nipple⁸⁶.

The animal’s other sensory systems are still poorly developed at birth. Its eyes and ear canals remain closed, so it is blind and deaf. The ear canals require several postnatal days to open, and auditory-responsive superior colliculus neurons become evident soon thereafter. This is followed by the appearance of the first multisensory neurons (somatosensory–auditory neurons) at about 10 days of age. However, the eyelids do not open until postnatal days 7–11, and visual multisensory neurons (the most common multisensory neuron in this visually dominant structure) do not appear until 3 weeks after birth^59,77. The response latencies of immature superior colliculus neurons are exceedingly long^59,60, and the delay in visual responsiveness in the multisensory layers of the superior colliculus is in striking contrast to the appearance of visually responsive neurons elsewhere in the nervous system and even in the overlying superficial layers, in which visual responsiveness begins before the end of the first postnatal week^87,88. This distinction in the maturation of unisensory versus multisensory responsiveness of superior colliculus neurons underscores the protracted developmental time course of multisensory processes. Even after some multisensory neurons first appear, their incidence increases towards the full complement only gradually over the next 2–3 months (FIG. 2). Their information-processing capabilities require an even longer developmental period.

Figure 2. Developmental profile of multisensory integration in the cat superior colliculus.

Multisensory neurons (shown in red) first appear in the second postnatal week and steadily increase in number thereafter, nearing adult levels by postnatal week 20. However, the first neurons with multisensory integration capabilities (shown in green) are not seen until the fourth week of life. Thereafter, their incidence increases roughly in parallel with the total incidence of multisensory neurons. The multisensory responses of neurons with multisensory integration capabilities are nearly always depressed by deactivating the association cortex, from which descending cortico-collicular afferents originate (shown in blue). Although the timing of these three developmental trajectories is parallel, the delay in the development of multisensory integration is consistent with the idea that the ability to respond to multiple modalities and the ability to integrate the information they provide are different phenomena, which are mediated by related but not identical developmental processes. Inset bar graphs provide sample responses from individual multisensory neurons at three different ages, one before the development of multisensory integration capabilities (left), one after this development (middle) and one from an adult animal. The responses from the adult neuron are also displayed as ‘impulse rasters’ in which each dot represents a single impulse and each row (ordered from bottom to top) represents the response to a single stimulus presentation. The grey bars show multisensory enhancement (ME). A, auditory; S, somatosensory; V, visual; VA, visual–auditory; VS, visual–somatosensory. Republished with permission of Society for Neuroscience, from Development of multisensory neurons and multisensory integration in cat superior colliculus. Wallace, M. T. & Stein, B. E. 17, 1997; permission conveyed through Copyright Clearance Center, Inc.

The most relevant factor in this context is the initial inability of these multisensory neurons to integrate their different sensory inputs. As noted earlier, they generate no more impulses to the concordant combination of cross-modal cues than they do to the most effective of them alone. The capability of superior colliculus neurons to integrate cross-modal signals does not appear until weeks after the first multisensory neurons have appeared. Even then, the incidence of neurons with this capability is quite low and steadily increases as development progresses, not reaching adult status until the animal is several months old⁷⁷ (FIG. 2). This protracted time course parallels that of the development of inputs from regions of the association cortex (see below).

The maturational period required for creating the adult-like complement of superior colliculus neurons capable of multisensory integration seems to be surprisingly long. It is not due to a paucity of effective sensory inputs, and it does not seem to relate to the vigour of these inputs. The disconnection between the maturity of the unisensory inputs and that of multisensory neurons and the immaturity of their multisensory integration capability is particularly apparent in the newborn rhesus monkey, a precocial species that, like humans, is born with its eyes and ears open. Because the rhesus monkey undergoes substantially more prenatal maturation than the cat, its sensory capacities at birth are more advanced, a feature that is particularly evident in the responses of its visual neurons in both the cortex and superior colliculus^89–92. It can see and hear very well, and it already has many multisensory superior colliculus neurons. However, although these neurons vigorously respond to their different sensory inputs and seem to be much more mature than their counterparts in the cat, they too fail to integrate their cross-modal inputs. As in the cat superior colliculus, their responses to coincident cross-modal sensory stimuli are no greater than responses to the most effective of those stimuli individually^92,93.

For a reasonable period of early postnatal development, many of the unisensory and multisensory properties of cat superior colliculus neurons develop in parallel. Receptive fields in each modality are initially very large and gradually contract during maturation, progressively improving the spatial fidelity of their individual representations, the alignment among the three sensory topographies and the register of the different receptive fields of individual multisensory neurons. This increasing spatial registration is most obvious in visual–auditory neurons because when the eyes are centred within the head, coordinates in visual space (eye-centred) align with those in auditory space (head-centred) and thus the two receptive fields can be mapped in the same exteroceptive reference frame.

At the point at which a neuron’s receptive fields have contracted in size to approximately 150% of the adult average, its probability of showing multisensory integration capabilities becomes quite high⁷⁷. This rule of thumb can be useful but is only correlative. It is not causative, and one can have large receptive fields in neurons capable of multisensory integration⁷⁹. The shrinking of these neurons’ receptive fields seems to be most dependent on exposure to discrete modality-specific stimuli, whereas the development of their multisensory integration capabilities seems to require exposure to particular configurations of cross-modal stimuli — both of which usually follow similar time courses. How cross-modal experience is integrated by the underlying circuit to develop multisensory integration capabilities in individual neurons is a critical dynamic that is discussed in detail below. However, in order to describe this relationship properly, the nature of the circuit underlying superior colliculus multisensory integration at maturity, especially the importance of influences descending to the superior colliculus from the association cortex, is discussed first.

Developmental role of cortical inputs

The adult superior colliculus receives inputs from a host of subcortical and cortical sources that represent different levels of processing within their unisensory hierarchies. These include ‘lower-order’ subcortical areas such as the retina and pretectum (visual), the inferior colliculus (auditory) and the trigeminal nucleus (somatosensory), as well as ‘higher-order’ association cortices^62,94–98. These are only a select few examples of the rich set of unisensory afferents that innervate the superior colliculus and from which it constructs its three sensory representations. Each of these representations is formed from multiple projection sources, giving the circuit the appearance of significant afferent redundancy. Some of the cortical inputs in this circuit have been shown to develop gradually during postnatal life and, as shown below, provide a key to the maturation of the integrative capability of superior colliculus neurons. Of particular concern in this context are the inputs derived from the association cortex (in cats, these include the anterior ectosylvian sulcus (AES) and rostrolateral suprasylvian sulcus (rLS)). Selective deactivation of these inputs in the adult animal (and neonate) (FIG. 2) has revealed that at least some afferents make unique contributions to the information-processing capabilities of superior colliculus neurons^99–106.

The AES, the region that is most important in constructing the multisensory properties of superior colliculus neurons, contains modality-specific subregions in which most neurons are responsive to a given sensory input: namely, the anterior ectosylvian visual area (AEV), the auditory field of the AES (FAES) and somatosensory area IV (SIV)^107–109. Although there are multisensory neurons scattered within these AES subregions, multisensory neurons are concentrated at the margins, or transitional areas, between them. Nevertheless, neither those multisensory neurons within the largely unisensory regions nor those concentrated at their margins project to multisensory superior colliculus neurons^52,110. It is the unisensory AES neurons that project to them, and they do so in sensory combinations that match the convergence patterns they derive from other sources^99,110. A neuron that receives visual and auditory inputs from sources other than the AES, for example, will receive inputs from the AEV and the FAES but not from SIV.

In the mature adult, these descending inputs play a crucial part in superior colliculus multisensory integration, and their deactivation eliminates this superior colliculus capability. Although most superior colliculus neurons depend on inputs from the AES for this capability, some require inputs from both the AES and rLS¹⁰¹. In the absence of this essential input, the responses of a superior colliculus neuron are either equivalent to those elicited by the most effective stimulus of the cross-modal pair or approximate a weighted average¹¹¹. The striking shift downward from response enhancement underscores the powerful and selective effect of these cortico-collicular inputs on superior colliculus multisensory integration (FIG. 3) and renders the multisensory responses of these neurons equivalent to those exhibited during neonatal stages. In both cases, these neurons continue to provide common access to the motor machinery of the superior colliculus, but their activity is not enhanced by the coincident action of multiple sensory inputs.

Figure 3. A synergy between unisensory subregions of the association cortex drives multisensory integration capabilities in the mature superior colliculus.

Depicted is the degree of multisensory enhancement (ME) above the largest unisensory response in a typical visual–auditory superior colliculus neuron when the auditory subregion (the auditory field of the anterior ectosylvian sulcus (FAES)) and/or visual subregion (anterior ectosylvian visual area (AEV)) of the AES was reversibly deactivated. In the control condition, the average multisensory response exceeded the largest average unisensory component response (shown as 100% response on the graph). However, deactivation of either cortical subregion alone eliminated this multisensory enhancement, rendering the multisensory response statistically indistinguishable (denoted by NS) from the largest unisensory response. Subsequent reactivation after each deactivation series restored the neuron’s functional capabilities. Asterisks indicate statistical significance.

To better understand the impact of the converging inputs from AES subregions to superior colliculus neurons in the mature circuit, and to determine whether their functional maturation is relevant to the development of multisensory integration capabilities, the visual (AEV) and auditory (FAES) subregions were deactivated individually as well as collectively while recording from their target visual–auditory superior colliculus neurons⁹⁹ (FIG. 3). It is interesting to note that focal deactivation of either area alone had the same deleterious effect of eliminating superior colliculus multisensory integration as did deactivating them together. Apparently, synergy is a key feature of descending influences from these subregions. What is not yet clear is where this functional synergy is exerted or how it is created. Nevertheless, it seems likely that the key is in the synaptic configurations they form on their superior colliculus target neurons either directly or via interneurons¹¹².

Unfortunately, although the projection patterns have begun to be evaluated^113,114, little is known about the morphology of the cortico-collicular synaptic configurations on multisensory neurons or how and when they develop. Cortico-collicular inputs from at least some regions of the association cortex are already present within days of birth¹¹⁵ and presumably during late embryonic stages, but their synaptic inputs are likely to be unformed or non-functional. The gradual postnatal maturation of this input is one of the likely reasons for the protracted developmental time course of superior colliculus multisensory integration⁷⁸. As shown below, it is the maturation of this pathway that is crucial for both the expression of multisensory integration in adult superior colliculus neurons and for its acquisition during development. Removal of the AES and rLS early in life precludes the acquisition of multisensory integration capability⁷⁶ and the performance benefits they normally provide in superior colliculus-mediated orientation tasks¹¹⁶. Similar multisensory deficits are also observed when these areas are deactivated in adult animals¹⁰¹. However, there is a clear difference in the compensatory capacity of the neonate and adult in this context. In adults, loss of the influences from only one of these regions of the association cortex produces striking multisensory deficits, but this is not the case after removing only one of these areas in early life. Neonates can compensate for the loss of either sub-area, presumably by an enhancement of the influences (for example, the projections) from the remaining area⁷⁶. These two regions are unique in this early compensatory capability for multisensory integration, as no other brain region is capable of substituting them in this functional role¹¹⁷.

The necessity of cross-modal experience

The prolonged postnatal period of superior colliculus multisensory development not only allows the crucial cortical inputs from the association cortex to develop but also allows the multisensory circuit to obtain considerable experience with sensory cues. Presumably, this period is sufficiently long to sculpt a circuit that distinguishes those cross-modal relationships that signal the same biologically significant event from those that do not. This is not a simple task, and extracting the statistical regularities that are required to reveal such relationships in any environment is likely to be compromised by the profusion of unrelated stimuli. In order to examine this assumption, several experimental strategies have been used to test the effect of restricting and manipulating an animal’s early experience with cross-modal stimuli. These included rearing animals in various conditions in which some cross-modal experiences are eliminated or in which these experiences are systematically altered. The first strategy tested the importance of these experiences, and the second examined whether their specifics were incorporated into the principles that would later govern multisensory integration. There are several ways to eliminate specific cross-modal experiences, the simplest of which is to rear animals in the dark, thereby precluding visually contingent cross-modal experience.

Rearing in darkness

The high incidence of visually responsive multisensory superior colliculus neurons (especially visual–auditory), the ease of restricting visual experiences by dark rearing and the information available from the extensive use of visual deprivation in early studies of visual system development^118,119 made this an obvious first choice of manipulations. Litters were placed in the dark with their mothers within several days of birth, and superior colliculus neurons were studied when they reached adulthood (that is, when they reached >6 months of age). The normal categories of unisensory and multisensory neurons were found, and visual–auditory neurons were very well represented. However, these multisensory neurons were immature. Their receptive fields were extremely large, resembling those of much younger animals, but, more importantly in the current context, they were unable to engage in multisensory integration. Although superior colliculus neurons responded quite well to visual and to auditory stimuli, their responses to spatiotemporally concordant visual–auditory stimuli were no greater than those to the most effective of these component stimuli alone¹²⁰ (FIG. 4). Presumably, this failure to integrate resulted from the lack of visual–auditory experience needed to form links between these senses via associative learning principles¹²¹. Alternatively, however, this failure may have been due to the lack of visual experience itself rather than the lack of cross-modal experience. Certainly, the absence of the spatial references that visual inputs provide, and the general increase in activity that they produce, has widespread consequences on the brain^71,122–130. In a visually dominant structure such as the superior colliculus, the loss of visual input might reduce afferent activity to such an extent that the architecture needed to support multisensory integration cannot be created.

Figure 4. The development of multisensory integration depends on concordant experience with cross-modal cues.

Depicted are the responses of exemplar neurons that illustrate the most common outcomes of four different rearing conditions. Shown for each exemplar are summary histograms describing the visual (V), auditory (A) and multisensory (VA) responses as the average number of impulses elicited by each stimulus. Also indicated (dashed line) is the sum of the average V and A responses in each condition, as well as the percentage increase elicited by their combined presentation (horizontal lines above each bar indicate the SEM, NS indicates not significantly different from the greatest unisensory response in that condition and asterisks indicate a statistically significant difference). Rearing with visual experience but with degraded auditory experience (that is, noise rearing, left), with auditory experience but without visual experience (that is, dark rearing, right) or with random independent visual and auditory experience (bottom) yields multisensory superior colliculus (SC) neurons lacking multisensory integration capabilities, as shown by the lack of significant difference between the bars representing the multisensory (VA) response and the strongest unisensory (V or A) response. However, rearing with concordant VA experience (top middle) allows SC neurons to develop their multisensory integration capabilities, as shown by the significant increase in the VA response, which in many cases exceeded even the sum of a neuron’s unisensory responses. Republished with permission of Society for Neuroscience, from Incorporating cross-modal statistics in the development and maintenance of multisensory integration. Xu, J., Yu, L., Rowland, B. A., Stanford, T. R. & Stein, B. E. 32, 2012; permission conveyed through Copyright Clearance Center, Inc.

Rearing with masking noise

If visual input has only a permissive role in this developmental process, restricting cross-modal experience without limiting vision itself should not compromise multisensory development. This was the thinking behind an experiment in which animals were reared in an illuminated room with multiple speakers arranged around the home cage to provide omni-directional broadband masking noise. Within this ‘noise room’, the blanket of sound effectively masked all but the loudest transient auditory stimuli^131,132. Some superior colliculus auditory receptive fields did not contract normally during development, but many others appeared to do so. Nevertheless, even these contracted receptive fields did not align well with their visual counterparts (which had contracted to normal size), and the neurons themselves lacked the capacity to engage in multisensory integration (FIG. 4). Instead, they responded to cross-modal cues in the same way as do neurons in dark-reared and neonatal animals¹³³. This result rules out the idea that a given sensory modality (for example, the visual modality) is uniquely critical and suggests instead that there is something specific about the nature of cross-modal exposure that determines whether multisensory integration capabilities mature. Both dark-rearing and noise-rearing studies were based on sensory exclusion, albeit very different forms of it. However, both rearing conditions precluded, or seriously degraded, experience with patterned stimulation in one of the relevant senses. The possibility that the development of superior colliculus multisensory integration might simply require patterned experience in each modality, rather than explicit exposure to their cross-modal combination, still cannot be eliminated.

Rearing with random sensory cues

The possibility that the multisensory deficit would not be obtained in animals with patterned experience in both senses, even if they had no relevant cross-modal experience, was examined directly. Animals reared in the dark were periodically exposed to visual and auditory cues, the timing and location of which were randomized. In this way, each sensory input activated multisensory superior colliculus neurons but did so independently of one another. Once again, the receptive fields of multisensory superior colliculus neurons remained larger than normal, although the neurons developed robust visual and auditory responses. However, these neurons did not develop the capacity to engage in multisensory integration⁷⁹ (FIG. 4). A separate cohort of animals reared in the same room but periodically exposed to the same stimuli in spatiotemporal concordance did develop this integrative capability and, in this regard, their multisensory superior colliculus neurons functioned very much like their counterparts in normally reared animals. However, the sizes of their receptive fields were also very much like those of immature and dark-reared animals. Together, the results underscore the idea that multisensory experience (that is, experience with cross-modal stimuli) triggers changes in the underlying circuitry that lead to the ability of superior colliculus neurons to integrate these different sensory inputs and that this process is not disrupted by some immature unisensory properties (for example, incomplete contraction of receptive fields).

The results also seem to suggest that this developmental requirement for multisensory integration favours a system that adapts to those cross-modal relationships that are experienced in the rearing environment. Presumably that is also the environment in which the animals will later live, so animals will now be armed with a system best suited to those particular events. The alternative is that the underlying circuit achieves the same computational end point regardless of the specific features of the cross-modal stimuli experienced. This would create a generalized system that is broadly useful across environments. As space and time are broadly invariant across environments, such a developmental plan could also be successful. Attempts to examine these alternatives involved exposing animals to ‘anomalous’ cross-modal stimuli. If the specifics of the stimulus experience were encoded, they should be reflected in the products of multisensory integration.

Rearing with anomalous cross-modal experience

Animals reared in the dark room were periodically exposed to synchronous visual–auditory stimuli that were always spatially disparate. This type of cross-modal event was considered ‘anomalous’ only because it is unlikely that two such cues would co-occur with great regularity in non-laboratory conditions and would be inconsistent with a single target for a superior colliculus-mediated orientation response. When the animals had matured, it was evident that although most of their superior colliculus neurons had not developed multisensory integration capabilities, and responded no differently than did those in neonatal, noise-reared or dark-reared animals, a significant minority of them exhibited an ‘abnormality’ that reflected their rearing condition. Their visual and auditory receptive fields had contracted (albeit not necessarily to normal size) but were displaced laterally to one another just like the visual and auditory stimuli in the rearing condition. Some neurons showed no receptive field overlap, making it impossible to present a coincident visual– auditory stimulus that would fall within both of their receptive fields. However, when the visual stimulus was placed in the visual receptive field and the auditory stimulus in the auditory receptive field, the neurons showed clear multisensory integration. The spatial principle had been altered: enhanced responses were now obtained only with spatially disparate stimuli¹³⁴, a finding consistent with the idea that the system adapts to the specific cross-modal relationships it experiences^61,135–138.

However, the fact that only a minority of neurons (29%) developed this capability rather than the majority (75–85%), as is the case in animals experiencing concordant visual–auditory stimuli, suggests that the inherent flexibility of the system is limited. The maturational outcome seems not to be solely determined by the cross-modal relationships encountered in the environment but also by a native bias, or selective filter, that alters the access of different cross-modal configurations to the learning process. This bias is probably derived from the overlapping sensory topographies of the superior colliculus, which reflect an unequal density of afferents tuned to spatiotemporally concordant cross-modal cues. As a result of this preferential selectivity, the statistical relationships that become encoded in the system are not veridical but biased towards concordance. The benefit of such a predisposition would be to prioritize stimulus configurations that are likely to refer to singular events, which can be targets for superior colliculus-mediated gaze shifts. Whether this bias extends beyond the spatial and temporal relationships among cross-modal stimuli to the particular features of the stimuli is not yet clear. However, studies in the cortices of human subjects^139–142 (see also REF. 143) and non-human primates^34,144 suggest that the semantic concordance of stimuli is a relevant factor that determines their integrative product. Whether this derives from the brain’s early experiences with such concordant stimuli or another inherent bias, and whether this is applicable to a structure such as the superior colliculus, which is primarily concerned with detecting and locating events, is not yet known.

Experience and cortical inputs

The two major factors needed for superior colliculus multisensory integration — cross-modal experience and influences from the association cortex — are unlikely to be independent developmental events. If, as expected, the cross-modal experience alters influences from the association cortex in ways that facilitate the development and manifestation of this superior colliculus property, this would help to explain why association cortex lesions preclude the maturation of superior colliculus multisensory integration, why association cortex inputs to a superior colliculus neuron are functional at or before the time it develops multisensory integration capability and why this capability is lost in neonatal (FIG. 2) and adult animals when the association cortex is deactivated^{99,101,102,106}.

That cross-modal experience is actually capable of altering the functional nature of cortico-collicular inputs to facilitate the maturation of multisensory integration has been inferred from experiments in which visual–auditory experience was precluded by dark rearing, as discussed above. Instead of developing selective enhancement of multisensory superior colliculus responses that are crucial for this capacity, the association cortex has been found to exert a non-selective facilitation of responses to each modality-specific stimulus as well as their cross-modal combination. Thus, in the absence of cross-modal experience, the cortico-collicular inputs do develop but fail to provide the specific influences that are necessary for the development of multisensory integration¹⁴⁵.

Further supporting evidence for the interdependent effects of association cortex influences on superior colliculus neurons and cross-modal experience comes from studies using pharmacological deactivation of the cortex during early development. Implanting a polymer infused with an inhibitory agent (that is, muscimol) over the association cortex silenced its neurons during the period (postnatal weeks 4–12) in which cross-modal sensory experience is being encoded and superior colliculus multisensory integration capability is first being expressed¹⁴⁶. Even a year or more after the cortical inputs had been reactivated, these animals were unable to use visual and auditory cues synergistically to enhance their performance in a standard superior colliculus-mediated detection and localization task (FIG. 5). Furthermore, their superior colliculus neurons failed to integrate these cross-modal stimuli. Apparently, when these cortical neurons are not privy to cross-modal events, superior colliculus multisensory integration capability does not develop even though the superior colliculus itself is not deprived of access to this information.

Figure 5. Deactivating the association cortex in early life delays the development of multisensory integration.

Unilateral muscimol-infused implants were used to deactivate neurons in the association cortex during the period in early life when superior colliculus (SC) multisensory integration capabilities are first being instantiated. During this time period (not shown), these cortical neurons were unable to process (contralateral) sensory information and were unable to influence their ipsilateral SC target neurons (which also respond to contralateral stimuli). a | When the animals had matured to 1.5 years of age, they were tested on their ability to locate events in space. Their performance was significantly impaired. They were unable to show the normal enhanced localization ability to events in contralateral space that had both visual and auditory components. However, these performance benefits were normal when the events were in ipsilateral space. This behavioural deficit was paralleled by a physiological deficit in ipsilateral SC neurons (left inset). Most failed to produce a better response to the visual–auditory combination than to the most effective of these stimuli individually (multisensory enhancement (ME)). b | However, when animals were re-tested on the same task at 4 years of age, behavioural performance on both sides of space was equivalent, and the SC physiological deficits seemed to have been resolved as well (inset). Apparently, the circuit was still able to acquire the experience needed to develop its multisensory integration capability during adulthood (albeit much more slowly).

All of the current observations point to the association cortex as the portal through which cross-modal experience affects the circuit underlying superior colliculus multisensory integration and as the site that is crucial for its expression throughout life. How experience with cross-modal events produces the refinements in this projection that render it capable of facilitating superior colliculus multisensory integration remains to be determined, and there are multiple models that may prove to be helpful in this regard. For example, the process that induces superior colliculus neurons to integrate cross-modal inputs seems to be highly consistent with the operation of an associative learning rule such as spike-timing-dependent plasticity (STDP)^121,147,148. This algorithm provides a method for selectively potentiating cortico-collicular connections that have impulse times that match the statistics of cross-modal experience. The encoding of these statistics is believed to be the computational foundation on which later multisensory integration is built, as hypothesized by several neural network models^149,150 (BOX 2).

Box 2. Modelling the development of multisensory integration.

The empirical results suggest that the development of superior colliculus multisensory integration capabilities is synchronized to the maturation of cortico-collicular afferents from unisensory regions of the association cortex. If these cortical inputs are removed early in development or deactivated in the mature adult, superior colliculus neurons lose the ability to integrate signals across the senses: they resemble the neonatal state. Cuppini et al.¹⁴⁹ proposed a neural network model that accounts for these observations. In the model, naive neonatal superior colliculus neurons are primarily controlled by their subcortical and primary cortical inputs. Excitatory and inhibitory influences produced by these inputs implement a competitive dynamic, so that a neuron’s response to a simultaneous pair of cross-modal stimuli reverts to the most robust response evoked by an individual stimulus in the pair. According to the model, afferents from the association cortex mature during the developmental period and establish their own excitatory and inhibitory dynamics. Unlike the interactions between the non-association cortex inputs, these dynamics facilitate synergistic interactions between concordant cross-modal stimuli and restrict competitive interactions to discordant stimuli. The inputs from the association cortex also suppress the input from subcortical and primary cortical sources. Thus, although the original connections from subcortical and primary cortical sources are never lost, their impact on the overt superior colliculus responses becomes minimized, because the inputs from the association cortex have effectively subsumed their role in directing superior colliculus responses. This results in a stable system that can appropriately integrate signals in a manner that is consistent with the animal’s cross-modal experience acquired in the postnatal period.

Ongoing plasticity

The functional plasticity of neonatal superior colliculus neurons is likely to be characteristic of the midbrain multisensory circuits of many, if not all, species. Furthermore, it is unlikely to be eliminated when the underlying circuit achieves its mature operational principles. Indeed, the plasticity of this circuit seems to continue well into adulthood^6,137, suggesting that it may be appropriate to think of its development as an ongoing process.

That such plasticity exists in multisensory superior colliculus neurons in the adult cat became evident when observing their reactions to a train of sequential stimuli. For example, a visual–auditory neuron exposed to a sequence of visual and auditory stimuli with a temporal offset just beyond its window for multisensory integration (that is, each stimulus elicited a well-defined unisensory response) began to show multisensory interactions after only a few trials¹⁵¹. The first ‘unisensory’ response in the pair increased in magnitude and duration, and the latency of the second decreased, as if the two unisensory responses were becoming fused into a single multisensory response despite the stimulus conditions remaining unchanged (FIG. 6a). This sort of change is consistent with the same STDP learning rules that are likely to have led to the acquisition of multisensory integration capability in the first place¹⁵². Because presynaptic activity promotes potentiation in synaptic weights when it precedes postsynaptic activity, the activity initiated by the first response produced substantially more potentiation than would be expected if the second stimulus were not presented.

Figure 6. Adult plasticity in multisensory integration.

Illustrated are three conditions in which multisensory (visual–auditory (VA)) plasticity has been shown. a | A raster display (each dot represents one impulse and each row represents the response to a single stimulus presentation, ordered bottom to top) shows the effect of repeated presentations of sequentially arranged spatially concordant VA stimuli on a multisensory superior colliculus (SC) neuron. Square waves atop the display represent the stimuli. Their repetition increased the magnitude and duration of this characteristic neuron’s response to the first stimulus in the sequence and decreased the latency of the response to the second, leading to the minimization of the temporal gap between the response trains. The changes can be seen by comparing the first set of trials in the grey zone at the bottom of the raster, with the last set of trials, also in grey, at the top. b | The visual and auditory receptive fields of an exemplar SC neuron are shown on the left on a polar plot of VA space (each concentric circle is 10 degrees). Repeated presentation of spatiotemporally concordant VA stimuli increased both the multisensory and unisensory responses of this characteristic neuron. The results of preliminary tests illustrated on the pre-exposure graph show that there was no auditory response (A) and the average multisensory response (VA) was 76% greater than the largest unisensory (V) response (and thus greater than their sum (dashed line)). After repeated exposure to the VA stimulus, magnitudes of all responses (post-exposure graph) were enhanced, and the previously subthreshold (auditory) input was ‘exposed’. c | Multisensory neurons in adult dark-reared animals initially do not integrate cross-modal stimuli but can be rapidly trained to do so by repeated exposure to spatiotemporally concordant VA stimuli. The visual and auditory receptive fields of three exemplar neurons are shown on the left on a polar plot of VA space (each concentric circle is 10 degrees). At the bottom are the numbers of cross-modal exposures provided to each (3,600–50,400). In the middle are the raster displays in response to V, A and AV stimuli, and to the right are graphs of the average impulse counts in each condition. The receptive fields retain immature (that is, large) sizes (compared with that shown in part b for example) with this impoverished sensory experience despite developing their integrative capability, and the magnitude of the integrated multisensory response is larger in neurons with more cross-modal experience (exposure). The horizontal line above each bar represents the SEM. Asterisks indicate statistical significance. Part a is republished with permission of Society for Neuroscience, from Adult plasticity in multisensory neurons: short-term experience-dependent changes in the superior colliculus. Yu, L., Stein, B. E. & Rowland, B. A. 29, 2009; permission conveyed through Copyright Clearance Center, Inc. Part b is reprinted with permission from REF. 145, The American Physiological Society. Part c is republished with permission of Society for Neuroscience, from Initiating the development of multisensory integration by manipulating sensory experience. Yu, L., Rowland, B. A. & Stein, B. E. 30, 2010; permission conveyed through Copyright Clearance Center, Inc.

A change in multisensory responses has also been documented when the same spatiotemporally concordant cross-modal stimulus is repeatedly presented, as would likely occur when an organism continues to interact with its source. This was particularly evident in neurons that overtly responded to only one of their modality-specific inputs (for example, auditory) but not to the other (for example, visual). The second input was ‘covert’, but its influence was revealed through its multisensory interaction with the first. After only a few repetitions of the cross-modal stimulus, not only were the multisensory responses more robust but the covert unisensory input became overt (FIG. 6b). The ‘exposure’ of the previously covert channel lasted for a considerable period, but without further exposure to the cross-modal stimulus it gradually degraded to its previous state¹⁴⁵. There are likely to be many examples of such plasticity and, although few other examples have been examined in detail thus far, one — the ability to use adult experience to develop what is normally instantiated during early life — deserves special mention.

When early experience is not sufficient for the acquisition of multisensory integration, as is the case when it has been restricted, this capability can still be acquired later in life (FIG. 6c). How much of this adult plasticity is due to an extension of the sensitive period because of early sensory restriction and how much is due to the normal inherent plasticity of the adult multisensory circuit is not yet known. However, the system is so sensitive to experience that superior colliculus neurons in dark-reared animals were able to acquire this capability after comparatively few sessions in which a single visual–auditory stimulus was repeatedly presented. This occurred even when no overt responses to the stimulus were required and in the absence of any of the reinforcement contingencies that are normally associated with learning. It even occurred when cross-modal experience was provided only when the animal was anaesthetized^79,153,154. Furthermore, nearly the same proportion of superior colliculus neurons acquired multisensory integration capability in these conditions, and did so with nearly the same enhancement magnitudes, as in normal rearing conditions. The capability was also retained in the absence of continued experience with the relevant cues and generalized to other cross-modal stimulus combinations. It seems that what was acquired in these conditions was the general principle that concordant visual and auditory cues should be bound together. Presumably, greater specificity could have been learned if some stimulus features were paired and others were not, as happens under natural circumstances, but this remains to be explicitly demonstrated. In short, as long as the animal experiences a consistent relationship between the cross-modal stimulus components, the capability to integrate those cues develops rapidly even in adulthood.

Why then did this capability for multisensory integration not develop rapidly in animals whose cortices were deactivated briefly during early development, as reported by Rowland et al.¹⁴⁶? Superior colliculus neurons in these animals were unable to integrate cross-modal cues even after more than a year of experience in a normal environment. Although this capacity was ultimately acquired, it took an additional 1.5 to 4 years — far longer than expected based on the results of cross-modal training experiments (FIG. 5). It is possible that the cortex was unable to respond properly to these stimuli until that time, but this seems unlikely given the rapid functional return of cortical activity following the deactivation process¹⁵⁵. Although it seems counterintuitive that the richness of the normal environment would be a poorer condition for acquiring multisensory integration capability than the impoverished training environment, the ambiguity in the former circumstance may prove to be the problem. In contrast to the training environment, in which there is only a single cross-modal stimulus configuration and few competing stimuli, the normal environment contains a great deal of complexity. There are not only a host of visual and auditory events present that are unrelated to the events providing cross-modal stimuli, but there is extensive variability in those that are. Cross-modal cues derived from the same events will vary in their relative intensities, will be experienced in very different background conditions and can occur at different distances or angles to the animal, thereby varying in their relative timing and spatial alignment. Presumably such variability increases the time needed to learn cross-modal associations but also increases tolerance for their variability. Deriving the cross-modal relationships in these challenging circumstances does not seem to be a significant problem for the neonate but may be more problematic for the adult, whose brain is no longer equivalent to that of the neonate. Its functional sensory systems would have colonized areas that are normally devoted to the missing or compromised sense, and all of its sensory areas would have continued developing based on intrinsic factors and the sensory inputs to which they were responsive. Perhaps this is why many human patients with early visual or auditory deficits that are later corrected fail to fully recover their visual–auditory integration deficits despite years of cross-modal experience in a normal environment^156–158 (but see REF. 159).

That superior colliculus neurons in young animals master the specifics of the cross-modal stimulus components in the training environment is evident from the observations that only when a neuron has both of its receptive fields encroaching on the exposure site does it develop multisensory integration capabilities¹⁵³ (FIG. 7), and that these neurons develop ‘anomalies’ reflecting the relationship of the component stimuli⁷⁹. Superior colliculus neurons trained with spatially and temporally concordant visual–auditory stimuli exhibit preferences for a cross-modal stimulus with that configuration. Their responses are progressively degraded as the component stimuli are separated from one another in space and/or time, a systematic bias that is not evident in normal animals^81,82. Multisensory response magnitude in normal animals has no systematic relationship with the location of the visual and auditory stimuli within their overlapping receptive fields, and these neurons prefer that the visual stimulus precede the auditory by 50–100 ms.

Figure 7. Learning to integrate requires a neuron to experience both cues simultaneously.

Shown on the left are schematics of the receptive fields (RFs) of three visual–auditory (VA) neurons. The visual (icon of a light bulb) and auditory (icon of a speaker) stimuli fell into one or both RFs of a given neuron and were repeatedly presented in close temporal proximity. As shown in the summary histograms to the right, only the neuron in which both stimuli were in their respective RFs ultimately developed the capability to integrate those inputs and showed a significantly enhanced multisensory response (indicated by the asterisks). The horizontal line above each bar represents the SEM. NS, not statistically significant. Republished with permission of Society for Neuroscience, from Initiating the development of multisensory integration by manipulating sensory experience. Yu, L., Rowland, B. A. & Stein, B. E. 30, 2010; permission conveyed through Copyright Clearance Center, Inc.

Summary and concluding remarks

It seems that the brain develops the capacity to integrate information from different senses only after it obtains considerable experience with their cross-modal combinations. For cat superior colliculus neurons, this acquisition period lasts for several postnatal months. It is during this time that these neurons master the cross-modal statistics that typify common detectable events and, through learning mechanisms, presumably craft the principles that will determine how these cues are integrated. This ensures that the system adapts to the environment in which it will be used, in most cases resulting in enhanced responses to cross-modal stimuli that occur in close spatiotemporal register (that is, those derived from the same event) and a corresponding facilitation of superior colliculus-mediated orientation.

The association cortex seems to be critical for this process, not only because it functions as a portal for experience to access the relevant neural circuit but because its descending influences are required for superior colliculus multisensory integration to occur. Although the appropriate circuit dynamics for this process are normally achieved over the first few months of postnatal life when cross-modal experiences can effectively influence the system, multisensory plasticity is retained into adulthood. Thus, appropriate experiences in adulthood can compensate for their absence earlier in life, albeit with lowered efficiency.

Nevertheless, regardless of age, the likely substrates for these experience-based changes are the synapses that convey these cortical influences through their direct and indirect projections to the superior colliculus. Unfortunately, at present, little information is available about the morphological development of these synapses or the microcircuits in which they are embedded. Such information would be of considerable help in understanding how this process is instantiated at any stage of life.

At present, the primary source of information about multisensory integration at the level of the single neuron comes from the cat superior colliculus, and many of the lessons that we have learned from this model are likely to apply to other species and even to other brain circuits. Indeed, similar experiential requirements and developmental chronologies, albeit even more delayed, have been noted in multisensory neurons in the cat association cortex¹⁶⁰. These observations are also consistent with the gradual maturation of many higher-order multisensory perceptual capabilities in humans^{156,161–165}, disruptions of which may underlie the deficits in multisensory integration that have been noted in several human developmental disorders^166–168.

However, some caution must be exercised here in generalizing from the observations discussed above. Multisensory integration takes place in many brain areas of many species with different evolutionary histories, different sensory capabilities, different experiences and facing different ecological challenges. It is unlikely that a single template is used in all such circumstances or that all integrated multisensory responses are manifested as a simple increase in the number of stimulus-evoked impulses. In addition, developing the capacity to integrate cues for taste and smell or infrared and visual cues in species that depend most on such processes may involve very different time courses than those for the visual, auditory and somatosensory inputs to the cat superior colliculus. Whether, and how, species-specific and region-specific adaptations of multisensory integration facilitate the functional role of different neuronal populations represents one of the forefronts of this field, one that could benefit from the combined expertise of neuroscientists, psychologists and ethologists.

Glossary

Multisensory

A process (behaviour) or entity (neuron or circuit) that incorporates information derived from more than one sensory modality.

Multisensory enhancement

The response to a cross-modal stimulus is significantly greater than its responses to either of the component stimuli.

Bayesian frameworks

Statistical frameworks used to model perception in which a feature of the world is inferred based on acquired sensory evidence.

Receptive fields

Regions of external space or location on the body in which stimuli will reliably elicit responses from a given neuron.

Cross-modal stimulus

A stimulus that activates two or more senses.

Spatiotemporal concordance

Closely aligned in space and time.

Spike-timing-dependent plasticity (STDP)

A principle by which synaptic efficacy is strengthened when the presynaptic neuron reliably generates an action potential before the postsynaptic neuron generates an action potential, but is weakened when the reverse relationship occurs or when the activity patterns are decorrelated.