Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2010 Dec 1.
Published in final edited form as: Hear Res. 2009 Apr 2;258(1-2):4–15. doi: 10.1016/j.heares.2009.03.012

The Neural Basis of Multisensory Integration in the Midbrain: Its Organization and Maturation

Barry E Stein 1, Terrence R Stanford 1, Benjamin A Rowland 1
PMCID: PMC2787841  NIHMSID: NIHMS107515  PMID: 19345256

Abstract

Multisensory Integration describes a process by which information from different sensory systems is combined to influence perception, decisions, and overt behavior. Despite a widespread appreciation of its utility in the adult, its developmental antecedents have received relatively little attention. Here we review what is known about the development of multisensory integration, with a focus on the circuitry and experiential antecedents of its development in the model system of the multisensory (i.e., deep) layers of the superior colliculus. Of particular interest here are two sets of experimental observations: 1) cortical influences appear essential for multisensory integration in the SC, and 2) postnatal experience guides its maturation. The current belief is that the experience normally gained during early life is instantiated in the cortico-SC projection, and that this is the primary route by which ecological pressures adapt SC multisensory integration to the particular environment in which it will be used.

Multisensory Integration describes a process by which information from different sensory systems is combined to influence perception, decisions, and overt behavior. All brains engage this strategy at multiple levels of the neuraxis (Calvert et al., 2004), and its impact on cognition and behavior has been repeatedly demonstrated. Multisensory integration has been shown to enhance and speed the detection, localization, and reaction to biologically significant events (Corneil and Munoz, 1996;Frens and Van Opstal, 1995;Hughes et al., 1994;Stein et al., 1989;Marks, 2004;Newell, 2004;Woods and Recanzone, 2004a;Shams et al., 2004;Sathian and Prather, 2004;Stein and Meredith, 1993). It is also a potent asset in signal disambiguation, including signals involving human speech and animal communication (Senkowski et al., 2007;Busse et al., 2005;Woldorff et al., 2004;Sathian, 2005;Grant et al., 2000;Sathian, 2000;Lakatos et al., 2007; Recanzone, 1998;King and Palmer, 1985;Schroeder and Foxe, 2004; Recanzone, 1998;Ghazanfar and Schroeder, 2006;Sathian and Prather, 2004;Weisser et al., 2005; Zangaladze et al., 1999;Liotti et al., 1998;Talsma et al., 2006a;Talsma et al., 2006b;Corneil and Munoz, 1996;Frens and Van Opstal, 1995;Hughes et al., 1994;Stein et al., 1989;Marks, 2004;Newell, 2004;Woods and Recanzone, 2004b;Shams et al., 2004;Wallace et al., 1996;Sumby and Pollack, 1954;Massaro, 2004;Ghazanfar et al., 2005;Partan, 2004;Bernstein et al., 2004;Sugihara et al., 2006). The impact of this evolutionary strategy is difficult to overestimate. Yet, despite the widespread and enthusiastic appreciation of its utility in the adult (see Calvert and Lewis, 2004;Stein and Meredith, 1993;Spence and Driver, 2004;Ghazanfar and Schroeder, 2006), far less attention has been devoted to examining the development of these processes and the impact of experience in crafting them during maturation. This may be due to the view among some that these neural processes are, in large part, already specified before birth so that the brain of the newborn is already capable of multisensory integration (Bower, 1974;Gibson J.J., 1966;Gibson, 1979;Gibson, 1969;Werner, 1973;Marks, 1978;Lickliter and Bahrick, 2004;Lewkowicz and Kraebel, 2004) and any later postnatal elaboration would not alter its fundamental principles.

However, neurophysiological experiments in cat and monkey (Wallace and Stein, 2007;Wallace and Stein, 1997;Wallace et al., 2006;Wallace et al., 1993) and recent behavioral studies in human infants (see Neil et al., 2006;Gori et al., 2008;Putzar et al., 2007) point to a very different understanding of how this capability develops and matures, one in which the neonatal brain is not yet capable of multisensory integration. Rather, multisensory processes appear to develop gradually during postnatal life with a time course distinct from unisensory processes, even within the same circuit. Recent data suggest that multisensory integration in the adult is crafted by early life experiences, adapting its properties according to the configurations of cross-modal stimuli present in the neonatal environment. This developmental principle seems quite simple on its face. However, the development of multisensory integration and its expression in the adult reflect complex processes: the fact that afferents from different senses converge onto a particular neuron does not guarantee that it will develop the ability to integrate cross-modal information, nor does it specify how the information is to be integrated (Jiang et al., 2001;Jiang and Stein, 2006;Wallace and Stein, 1997;Stein and Meredith, 1993).

These issues will be reviewed here in the context of the development of the multisensory (i.e., deep) layers of the SC. This is an excellent model system for this purpose for a variety of reasons. The involvement of SC neurons in a well-defined behavior (orientation and localization) allows one to relate physiology to behavior and there is already a good deal of information available regarding the maturation of their unisensory properties (Kao et al., 1994;Stein, 1984b). The SC is also a primary site at which inputs from different senses converge (Wallace et al., 1993;Stein and Meredith, 1993). Indeed, much of what we know about the maturation and expression of multisensory integration has been derived from this model (Stein, 1984b;Peck, 1987;Stein and Arigbede, 1972;Stein et al., 1976;Stein and Meredith, 1993;Stein et al., 1993;Wallace, 2004;Stein and Clamann, 1981;Stein et al., 1973; but see also Gutfreund and Knudsen, 2004;Lakatos et al., 2007;Senkowski et al., 2007;Sathian and Prather, 2004;Woods and Recanzone, 2004a;Groh and Sparks, 1996b;Groh and Sparks, 1996a;Jay and Sparks, 1987a;Jay and Sparks, 1987b;King et al., 2004;Barth and Brett-Green, 2004;Calvert and Lewis, 2004). Of particular interest here are two sets of experimental observations: 1) cortical influences appear essential for multisensory integration in the SC, and 2) postnatal experience guides its maturation. These observations are unlikely to be coincidental, especially in light of the fact that the cortical afferents are immature at birth, and therefore are likely to be highly plastic. The current belief is that the experience normally gained during early life is instantiated in the cortico-SC projection, and that this is the primary route by which ecological pressures adapt SC multisensory integration to the particular environment in which it will be used. However, before examining the details of its maturation and plasticity, it is necessary to review the expression of multisensory integration in the adult.

SC Multisensory Organization and Integration in the Mature Brain

Functionally the SC has been divided into two regions (though interconnections between them are known to exist). The superficial layers (I–III) are unisensory and neurons there are exclusively responsive to visual stimuli. The deeper layers (IV–VII) are multisensory in that they contain a variety of unisensory (visual, auditory and somatosensory) neurons, as well as groups of multisensory neurons that represent each possible combination of these senses. It is the neurons in this portion of the SC that are of interest here.

Each sensory representation in SC is laid out in a topographic (i.e., map-like) fashion with the different maps sharing the same basic plan so that they are in spatial register with one another (Stein et al., 1993;Meredith et al., 1991;Meredith and Stein, 1990; Middlebrooks and Knudsen, 1984;Stein et al., 1981;Stein et al., 1976). It is as if the maps of visual space, auditory space and the body surface were created on the same template and extended vertically (albeit with some geometric distortions in each) throughout the depths of the SC. As a result, the neurons in a vertical column of tissue represent the same general region of visual, auditory, and somatosensory space. Because the sensory maps are also in register with a premotor map, this arrangement is a particularly effective way to match incoming sensory information about the event with the outgoing “motor” signals required to orient to it (Grantyn and Grantyn, 1982;Sparks and Nelson, 1987;Guitton and Munoz, 1991;Harris, 1980;Jay and Sparks, 1987b;Jay and Sparks, 1987a;Jay and Sparks, 1984;Munoz and Wurtz, 1993a;Munoz and Wurtz, 1993b;Peck, 1987;Sparks, 1986;Stein and Clamann, 1981;Wurtz and Albano, 1980;Wurtz and Goldberg, 1971;Groh and Sparks, 1996a;Groh and Sparks, 1996b).

Thus, within each map there are multisensory SC neurons whose multiple receptive fields (one for each sensory modality to which they respond) are in spatial register with one another (King et al., 1996;Meredith et al., 1991;Meredith and Stein, 1990;Meredith et al., 1992). Many have excitatory regions bordered by inhibitory regions. So the impact of a cross-modal combination of stimuli is determined, in large part, by how such stimuli relate to a neuron’s receptive field regions (Stein and Meredith, 1993). Spatiotemporally concordant cross-modal stimuli evoke responses that can have significantly more impulses than is evoked by the strongest of the individual stimuli (multisensory enhancement, see Fig. 1). Spatially disparate stimuli can evoke responses with fewer impulses than the strongest individual stimulus (multisensory depression). Multisensory enhancement has not only been observed in the SC of the cat (Meredith and Stein, 1986a;Meredith and Stein, 1996;Meredith and Stein, 1986b;Stein et al., 1993;Wallace et al., 1993;Wallace et al., 1998;Wallace and Stein, 1994), but also in the hamster (Meredith and Stein, 1983), guinea pig (King and Palmer, 1985), and monkey (Wallace et al., 1996), as well as in the cortex of cat (Wallace et al., 1992), rat (Wallace et al., 2004b;Barth and Brett-Green, 2004), monkey (Ghazanfar and Schroeder, 2006;Lakatos et al., 2007;Schroeder and Foxe, 2002;Schroeder et al., 2001;Schroeder and Foxe, 2004), and human (Calvert and Lewis, 2004;Stein et al., 1996;Fort and Giard, 2004;Macaluso and Driver, 2004;Gelder et al., 2004;Sathian and Prather, 2004;Laurienti et al., 2002;Lovelace et al., 2003).

Figure 1.

Figure 1

Open in a new tab

Multisensory enhancement. This multisensory neuron responds to visual and auditory stimuli and exhibits a significant overlap in the visual and auditory receptive fields (shaded regions, top). Responses were recorded to visual, auditory, and spatiotemporally concordant (see maps and stimulus traces on top) visual-auditory stimulus pairs. The effectiveness level of the visual stimulus was altered by adjusting its intensity (effectiveness increases from top to bottom). The impulse rasters, peri-stimulus time histograms, and summary figures showing mean response magnitudes are provided for each stimulus type (visual=white, auditory=gray, multisensory=black). When stimulus effectiveness levels are low, the multisensory responses evidence relatively large enhancements, larger than the sum of the unisensory responses (superadditive). However, as stimulus effectiveness levels are increased, the proportional response enhancement evidenced by the multisensory response decreases, and at the highest levels of effectiveness is not distinguishable from the sum of the unisensory responses. This phenomenon is known as “inverse effectiveness.” From Alvarado et al., 2007b.

When different sensory stimuli are only weakly effective, but are spatiotemporally aligned, their combined presentation can elicit a response exceeding their sum. Superadditive computations (primarily at the lower end of a neuron’s dynamic range) transition to additive or subadditive computations when stimulus efficacy is increased (Alvarado et al., 2007b;Stanford et al., 2005;Stanford and Stein, 2007;Perrault, Jr. et al., 2005;Perrault, Jr. et al., 2003). These findings were consistent with the concept of inverse effectiveness (Meredith and Stein, 1986b), wherein multisensory enhancement is greatest when component stimuli are weakest. This principle is simple and intuitive. The benefit of multisensory integration is greatest when the cues associated with an event are weakest; as strongly effective cues from individual senses are, by definition, easy to detect and often easy to locate and identify. By the same principle, if one evaluates the a multisensory response as it evolves over time, it is apparent that the enhancement of its magnitude (relative to the best unisensory response) is greatest near its onset when the comparator component responses would be just beginning and at their weakest. In fact, multisensory integration appears to take place in real-time, as soon as the inputs arrive (Rowland et al., 2007;Rowland and Stein, 2007) (see Fig. 2).

Figure 2.

Figure 2

Open in a new tab

The temporal profile of multisensory enhancement. Left: Impulse rasters illustrating the responses of a multisensory SC neuron to visual (V), auditory (A), or combined visual-auditory (VA) stimulation. Right: Two different measures of the response show the same basic principle of initial response enhancement: multisensory responses appear enhanced from their very onset (i.e., as soon as inputs arrive). On top the measure is the mean stimulus-driven cumulative impulse count (qsum), on the bottom it is an instantaneous measure of response efficacy referred to as the event estimate. From Rowland and Stein 2008.

The integration of information across the senses is computationally different from integrating information from the same sense, because inputs derived from different senses provide independent estimates of the same initiating event while inputs from the same sense can exhibit substantial noise covariance (Ernst and Banks, 2002). Consequently, one might expect that two spatiotemporally concordant visual stimuli would not yield enhancements equivalent to what is obtained with a visual and an auditory stimulus. Another possibility, reflecting a very different assumption, is that both multisensory and unisensory integration would yield equivalent results because multisensory response enhancement simply reflects the presence of more environmental energy or multiple, redundant stimuli instead of any “special” computation (Miller, 1982;Gondan et al., 2005;Lippert et al., 2007;Leo et al., 2008;Sinnett et al., 2008). This issue was recently explored by Alvarado and colleagues (Fig. 3) and it was found that multisensory integration yielded far greater products than within-modal integration, reflecting very different underlying neural computations (Alvarado et al., 2007b). Within-modal stimulus pairs rarely exhibited response enhancement and typically reflected a subadditive computation. The same differences were found in cats tested on a detection and localization task in response to cross-modal (visual-auditory) and within-modal (visual-visual or auditory-auditory) stimulus combinations (Gingras et al., 2009) (Fig. 4).

Figure 3.

Figure 3

Open in a new tab

Comparisons between multisensory and unisensory integration in physiology. A: The magnitude of response evoked by a cross-modal stimulus (y-axis) is plotted against the magnitude of the largest response evoked by the component unisensory stimuli when presented alone (x-axis). Most of the observations evidence multisensory enhancement (positive deviation from the solid line of unity). B: The same cannot be said for response magnitudes evoked by two within-modal stimuli. Here, the evoked response is typically not statistically better than the largest evoked by one of the component stimuli individually. This observation is consistent for both multisensory and unisensory neurons (insets on right). From Alvarado et al. 2007b.

Figure 4.

Figure 4

Open in a new tab

Comparisons between multisensory and unisensory integration in behavior. Animals were trained in a spatial localization task and their localization accuracy was tested with brief (50ms) visual and auditory stimuli. There were two potential visual targets (V1 and V2) at each location. They were presented either alone, individually with the auditory stimulus at the same location to assay multisensory integration, or together to assay unisensory integration. A: The mean accuracy of the animals in localizing the auditory (A1) stimulus, the visual stimuli at each location (V1 and V2), the visual stimuli when simultaneously presented (V1V2), and conditions where a visual stimulus was paired with the auditory stimulus (V1A1). While both multisensory and unisensory integration yielded enhancements, multisensory integration was far more efficacious. B: Responses are collapsed across locations and decomposed by type: correct responses (C), no-go responses (NG) where the animal did not move from the starting position, and wrong responses (W) where the animal oriented to the wrong location. While multisensory integration yields robust decreases in both types of errors, unisensory integration decreased only the numbers of NG errors. C: Multisensory and unisensory integration follow parallel but significantly trends of “inverse effectiveness.” Each point on this plot of % enhancement vs. the best unisensory accuracy represents a different test location. From Gingras et al., 2009. In Press.

As noted earlier, the SC is a primary site of converging inputs from different senses (Meredith and Stein, 1986a;Stein and Meredith, 1993;Wallace et al., 1993) so that its integrative products are the result of processes that take place within the SC itself. The SC receives sensory inputs from many unisensory structures (Huerta and Harting, 1984;Edwards et al., 1979;Wallace et al., 1993;Stein and Meredith, 1993) and its multisensory neurons are mostly output neurons that project to motor related areas of the brainstem and spinal cord involved in the control of orientation behaviors (Moschovakis and Karabelas, 1985;Peck, 1987;Stein and Meredith, 1993). Consequently, the principles that govern the multisensory integration at the level of the single SC neuron also govern SC-mediated orientation behaviors (Jiang and Stein, 2002; Stein et al., 1989; Wilkinson et al., 1996; Jiang et al., 2007a).

SC Multisensory Integration Depends on Influences from Cortex

SC neurons are rendered multisensory by virtue of the convergence of unisensory inputs from many different sources but are unable to integrate these different sensory inputs in the absence of inputs from one particular source: association cortex. This conclusion was drawn from a number of studies in cat in which reversibly deactivating these cortico-collicular inputs disrupted the multisensory integrative capabilities of their target neurons in the SC, but not their capacity to respond to stimuli from different senses (see example in Fig. 5). This means that, in the absence of cortex, the response evoked in the SC by a cross-modal stimulus complex was equivalent in magnitude to a single stimulus presented in isolation (Jiang and Stein, 2003;Jiang et al., 2002;Stein et al., 2002;Alvarado et al., 2007a;Burnett et al., 2007). The critical regions of cortex include the anterior ectosylvian sulcus (AES) and an adjacent area, the rostral aspect of the lateral suprasylvian sulcus (rLS). The evidence collected thus far indicates that these two areas uniquely serve this purpose and other areas of the brain appear incapable of taking on this role if AES and rLS are damaged during early life (Jiang et al., 2007b;Jiang and Stein, 2003;Jiang et al., 2006;Burnett et al., 2004;Wilkinson et al., 1996). Although they sometimes function cooperatively in facilitating multisensory integration in SC neurons, the most important of these areas is AES, as far more SC neurons depend on this region than on rLS (Jiang et al., 2001). It has also been the source of most of our understanding of the role of cortex in this context. Just what mechanism is employed by these cortical inputs to make this integration possible in SC neurons, even those rendered multisensory via non-association cortex inputs, is not yet fully understood.

Figure 5.

Figure 5

Open in a new tab

Multisensory integration in the SC depends on association cortex. SC responses to auditory (A), visual (V), and multisensory (AV) stimuli are recorded before (left) and after (right) the deactivation of association cortex. The visual stimulus is tested at multiple (5) levels of effectiveness. The graphs at the top of the figure provide stimulus traces, impulse rasters, and peri-stimulus time histograms for each response. The bottom graphs provide summaries of the mean response levels (lines) and the percent multisensory enhancement (bars) observed for each of the stimulus pairings. Prior to the deactivation of cortex, the responses evidence the characteristic “inverse effectiveness,” with larger unisensory responses associated with smaller multisensory enhancements. However, after cortical deactivation (shaded region of inset), multisensory enhancements are significantly attenuated and even eliminated at each of the stimulus effectiveness levels tested. From Jiang, et al., 2001.

Cortico-collicular inputs from AES are unisensory. They converge from its different subdivisions (visual, AEV; auditory, FAES; and somatosensory, SIV) to match the modality-convergence profile of their SC target neurons (Wallace et al., 1992;Fuentes-Santamaria et al., 2008). For example, an SC neuron that receives auditory and somatosensory inputs from non-AES sources will often also receive convergent inputs from SIV and FAES. An example of converging cortico-collicular projections from SIV and FAES is shown in Fig. 6. Although there is an assumption that a similar arrangement is present in the human brain, no direct evidence is yet available to support this idea. The functional homologue of AES and rLS in the human brain has not yet been identified.

Figure 6.

Figure 6

Open in a new tab

SC neurons receive converging input from different sensory subregions of association cortex. Neurons in the auditory (FAES) and somatosensory (SIV) subregions of the anterior ectosylvian sulcus were injected with florescent tracers to identify their projections into the deep layers of the SC. It was found that these inputs often converged and crossed one another in the SC, with areas of thickening representing presumptive locations of contact (arrows). From Fuentes et al., 2008.

The Development of SC Multisensory Integration

The properties described above do not characterize the SC of the neonate. In fact, the differences between the sensory properties of the neonatal and mature cat SC are quite striking and reflect the general immaturity of its sensory information processing capabilities during early life. The cat is a particularly good subject for examining the development of sensory information processing because, unlike more precocial species such as most ungulates and primates, it is born at a very early stage of development. At birth its eyes and ear canals are still closed and many SC neurons are unresponsive to sensory stimuli. Those SC neurons that can be driven by sensory stimuli, even those in its deep layers, are unisensory. Initially they respond only to tactile stimuli, and this capability is already noted in late fetal stages (Stein et al., 1973). Most notable in this regard are neurons whose receptive fields encompass portions of the perioral region, a region that is of critical importance in helping the hungry neonate locate the nipple and initiate suckling (Larson and Stein, 1984). Auditory-responsive neurons appear in the deep layers at approximately 5 postnatal days, and neighboring visual-responsive neurons do not appear here until at about 3 postnatal weeks, long after the eyes have opened, and long after neurons in the earlier developing overlying superficial layers have become responsive to visual stimuli (Stein et al., 1973;Stein, 1984a;Kao et al., 1994;Wallace and Stein, 1997).

The normal ontogeny of multisensory neurons is also gradual compared to their neighboring unisensory counterparts. Perhaps this is to be expected, given the greater complexity of their requirements for signal processing. Unlike unisensory neurons, multisensory neurons must acquire information from different sensory channels and synthesize this cross-modal information in ways that are quite distinct from those used by unisensory neurons to synthesize within-modal information (see above). This makes it difficult to generalize from the maturation of one population or process to the other.

The first category of multisensory neuron, somatosensory-auditory, is evident soon after the appearance of auditory neurons (at P10). Visual-nonvisual neurons appear as soon as visual responsiveness is evident in the multisensory layers. However, these multisensory neurons appear incapable of integrating concordant cross-modal signals to generate enhanced responses. This capacity requires months of maturation, a period in which the functional maturation of inputs from AES are also formed (Wallace and Stein, 1997;Wallace and Stein, 2000;Stein et al., 2002). The prolonged nature of this postnatal period suggests that experience with cross-modal signals may be important in structuring the operational principles that would best adapt multisensory integration to the environment in which it would be used. This possibility was tested by rearing animals without the possibility of experience with certain cross-modal cues (e.g., visual-nonvisual) and also by perturbing the typical spatial configurations of these cues encountered by the young animal.

Precluding Early Visual-Nonvisual Experience

By raising animals in the dark until 6 months of age (the physiological properties of SC neurons are normally mature by this time), the possibility of early visual-nonvisual experience is precluded. This allows one to examine how the absence of experience with these cross-modal cues affects the maturation of multisensory integration (Wallace et al., 2001;Wallace et al., 2004a). These experiments demonstrated that such experience was not necessary for the SC to develop a near normal complement of visual, auditory or somatosensory neurons, all of which responded well to physiologically appropriate stimuli. A near normal complement of multisensory neurons also developed. However, these multisensory neurons were clearly atypical. Their very large receptive fields were more characteristic of the neonate than of a mature animal. They were also incapable of multisensory integration (Fig. 7), just as are neonatal SC neurons and the SC neurons of adults who have had neonatal ablation of association cortex (Jiang et al., 2006). The data suggested several possibilities, most provocative of which was that experience with the senses working together is essential for neurons to develop their capacity to engage in multisensory integration.

Figure 7.

Figure 7

Open in a new tab

Dark-rearing prevents the normal development of multisensory integration and the normal contraction of SC RFs. Illustrative receptive field overlap is shown for one neuron (upper) as is its lack of multisensory integration (8% change, NS, t-test). Population analyses showed that although multisensory neurons were common in these animals (pie chart, lower left) evidence of multisensory integration was very rare (4% vs. 81% in normals). The effect of dark rearing on RF size (they contract very little during development) is shown in the lower right where visual, auditory and somatosensory RF sizes are compared to those in normal populations. Adapted from Wallace et al., 2004a.

Providing Anomalous Early Visual-Auditory Experience

The dark rearing results showed that early experience is critical for the maturation of multisensory integration. It also suggested that during this period the SC might adapt its response properties to reflect the statistical regularities of the spatiotemporal relationships between cross-modal cues. In normal conditions, for example, the visual and auditory cues associated with a given event generally occur at the same time and in the same place. With enough experience, the link between these cross-modal stimuli can be encoded, so that later instances of concordant cross-modal stimuli yield enhanced detection, localization, and identification. Presumably then, altering the normal concordance of the cross-modal stimuli during early life would initiate developmental changes that adapt the principles of multisensory integration to an atypical environment. To test that possibility, cats were raised from birth to 6 months in a dark room in which simultaneous, but spatially disparate visual-auditory cues were periodically presented (Wallace and Stein, 2007). The stimuli were generated by speakers and light emitting diodes that were fixed to different locations on the animals’ cage wall.

Many visual-auditory SC neurons of these animals had some properties similar to those in animals without visual-nonvisual experience (see above): their receptive fields showed little evidence of developmental contraction and neurons could not engage in multisensory integration. This was not surprising given that the visual-auditory stimuli had no consequences associated with them. However, a significant proportion of SC neurons in these “disparity reared” animals did appear to be affected. They developed poorly aligned visual-auditory receptive fields, some of which were completely out of spatial register with one another (see Fig. 8). This is almost never seen in normal or dark reared cats, but clearly does reflect the animal’s rearing condition. Now only when the visual and auditory stimuli were spatially disparate could they simultaneously fall within their respective receptive fields. When such cross-modal stimuli were presented, the response magnitude of these neurons was significantly enhanced (Fig. 9). Apparently, the temporal concordance of these cues was sufficient to link them and reverse the “normal” spatial principle governing multisensory integration. As a result spatially disparate stimuli produced response enhancement, a cross-modal stimulus configuration that produces response depression (or has no effect) in multisensory SC neurons of normally reared animals (Kadunce et al., 2001;Meredith and Stein, 1996). These observations provide strong support for the hypothesis stated above.

Figure 8.

Figure 8

Open in a new tab

SC sampling and RF overlap in normal and V-A spatial disparity reared animals. Top: Normal sampling of all SC quadrants yielded visual (red) and auditory (green) receptive field (RF) center distributions. Bottom: % RF overlap in control animals (yellow) was often 91–100%; far exceeding that seen in spatial-disparity reared animals (red, often <10%). The physiology of a characteristic neuron is shown in Figure 9. Adapted from Wallace and Stein 2007.

Figure 9.

Figure 9

Open in a new tab

Rearing with visual-auditory spatial disparity yields anomalous multisensory integration: When the visual (red) and auditory (white) stimuli are spatially coincident in either the visual (blue) or auditory (green) RF of this typical neuron (top and middle), they produce no multisensory integration. But when spatially disparate and in their RFs (bottom) they produce significant enhancement - a striking reversal of the normal condition, but one consistent with the animal’s abnormal multisensory experience. Adapted from Wallace and Stein 2007.

Cortical deactivation during development

It was not immediately apparent where in the circuit underlying multisensory integration early cross-modal experience exerted its effects. However, given the well known sensitivity of cortex to early experience, association cortex seemed like a good bet. To examine this possibility, AES and rLS were rendered unable to gather early cross-modal experiences during the temporal window in which multisensory integration develops (25–81 dpn (Wallace and Stein, 1997)) by covering them with a polymer infused with muscimol, a GABAa receptor agonist. Once the implant was removed or depleted of its muscimol stores, cortex was reactivated. Preliminary observations revealed that there were significant impairments of multisensory integration in neurons of the ipsilateral SC (Rowland et al., 2006;Stein et al., 2008). They were unable to fuse their inputs from different sensory channels to enhance responses. The same deficits were evident in behavior (see Fig. 10). These observations strongly suggest that the cortico-collicular projection is a critical site for incorporating early experience with cross-modal cues.

Figure 10.

Figure 10

Open in a new tab

Reversible cortical deactivation using muscimol-impregnated Elvax. AES and rLS were unilaterally deactivated in these animals during a period of early life (postnatal weeks 4–12) during which multisensory integration normally develops in the SC. The cortex reactivated after this time. Animals were then assessed behaviorally in a spatial localization task (top) or physiologically (bottom) at one year of age. Animals show normal multisensory enhancement in their localization of visual (V) or multisensory (VA) stimuli on the side of space ipsilateral to the deactivation (green), but significantly attenuated enhancement on the contralateral side (red). The auditory stimulus was neutral in this task and did not evoke responses on its own. Similarly, a sample of neurons from the ipsilateral SC of these animals shows virtually no multisensory enhancement (bottom left), in contrast to a control sample (bottom right).

The Development of Cortical Multisensory Integration

The midbrain is believed to develop its physiological properties earlier than does the cortex. Thus, the absence of multisensory integration in the newborn’s SC strongly suggested that multisensory integration in cortex would appear even later in development. This possibility was explored by studying multisensory neurons in AES. As noted earlier (see above), AES neurons projecting to the SC are unisensory. However, multisensory neurons are also found in this cortical area, primarily clustered at the borders between its three largely unisensory regions. Though the multisensory AES circuit is independent of the SC, and the targets of these neurons are not yet known, they exhibit multisensory characteristics, including integration properties, much like those of SC neurons (Wallace et al., 1992). Thus, they can serve as an excellent developmental comparator to multisensory neurons in the SC.

The results of developmental studies of multisensory processes in AES showed that, as expected, they too lack the ability to engage in multisensory integration at birth. They develop this capability only gradually during postnatal life, but do so with a time course that begins and ends later in ontogeny than does the time course in SC neurons (Wallace et al., 2006). These observations are not only consistent with the expected developmental time lag between cortex and SC, but with the general notion that multisensory integration capabilities are crafted over a considerable period of early postnatal life. Using the dark rearing approach, it was once again demonstrated that visual and visual-nonvisual experience was not necessary to develop a near normal complement of visual and multisensory neurons. However, it was essential for the maturation of visual-nonvisual multisensory integration (Carriere et al., 2007). These data suggest that the same general developmental principles of multisensory integration are operative at different levels of the cat’s nervous system (see also Wallace et al., this volume).

The Development of Multisensory Integration in the Primate Brain

The SC of the adult Rhesus monkey also contains many multisensory neurons, and they too are capable of multisensory integration. In fact, their physiological properties are very similar to those seen in the cat SC (Wallace et al., 1996). To examine whether or not the absence of multisensory integration capabilities in neurons of the midbrain and cortex of the neonatal cat is an idiosyncratic feature of the brain of this species, the physiological properties of the newborn SC of Rhesus monkey was also explored. The monkey is far more precocial at birth than is the cat and, unlike the cat, already has multisensory neurons in its SC at this time. However, these neurons, like their counterparts in the young cat, proved to be incapable of multisensory integration (Wallace and Stein 2001). They responded no better to combinations of concordant cross-modal cues than they did to the best of these cues individually. At what point such neurons obtain the ability to synthesize their different sensory inputs, and whether they require experience to do so, remains to be determined. However, given their similar fundamental properties, and their absence in newborns of both species, it does not seem unreasonable to expect that SC neurons in both species have similar developmental antecedents and that this reflects a general mammalian plan.

Given the general assumption that midbrain processes develop before higher-order cortical processes, it is reasonable to assume that the absence of multisensory integration in the SC indicates the absence of such capabilities in cortex. Furthermore, the absence of such capabilities in both the altricial cat and the precocial monkey suggest the absence of such capabilities in the newborn human.

Although this certainly remains to be determined empirically, some recent observations in humans are consistent with such a postulate. Human subjects whose early vision was obscured by dense congenital cataracts were studied many years after those cataracts had been removed. In a striking parallel to the animal studies described above, the vision of these subjects seemed normal, but their ability to integrate visual-nonvisual information was impaired. This was the observation when challenged with both speech and non-speech tasks (Putzar et al., 2008). Although it is not yet clear whether the human SC, like the cat SC and monkey SC, is incapable of multisensory integration at birth, this also seems quite likely, and recent studies have shown that infants do poorly in integrating visual-auditory cues for the purpose of localization prior to 8 months of age (Neil et al., 2007). Apparently, they also do relatively poorly on integrating visual and haptic information prior to 8 years of age (Gori et al., 2008) suggesting that the maturation of multisensory capabilities is particularly long in humans. If so, this would be consistent with the comparatively long postnatal time period required for human brain maturation. It would also not be surprising if the human brain also requires experience with cross-modal cues over an appreciable period in order to develop this capacity, and that it may never form properly if the requisite experience for its instantiation is compromised. It will therefore be of substantial interest to determine the auditory-nonauditory capabilities of people born with such severe visual or auditory impairments that visual (e.g., retinal) or auditory (e.g., cochlear implants) prostheses are required to allow them to process this sensory information.

But it is also of import to determine whether there are commonalities among the circuitries that mediate multisensory integration in the brains of different species. This would not only help determine the utility of different models, but might provide insight into the alternative mechanisms for instantiating this remarkable capability. The present discussion focused on the cat SC and the implications of this model during development and adulthood for processing of multisensory information. Examining whether the cortico-collicular input is as critical for SC multisensory integration in the human brain as in the cat, and whether the primary site for the incorporation of this experience is (as suggested here) cortex, is one obvious starting point for such an examination. The utility of TMS as a noninvasive means of functionally disrupting human cortex may provide one mechanism of testing this possibility (e.g., see Bolognini and Maravita 2007). The challenge, of course, is first determining the regions of human cortex to target. Assuming the generality of the model as a starting point, it would seem most productive to search for likely functional homologues of AES in the human brain. Those that meet the most basic features in the cat, such as the presence of cortico-collicular projections, and the confluence of inputs from different sensory modalities, would be the most obvious initial areas of interest.

Acknowledgments

Portions of the research described here were supported by NIH grants NS36916 and EY016716.

Abbreviations

SC

superior colliculus

AES

anterior ectosylvian sulcus

rLS

rostral portion of the lateral suprasylvian sulcus

AEV

anterior ectosylvian visual area

FAES

anterior ectosylvian auditory area

SIV

anterior ectosylvian somatosensory area

P10

10th postnatal day

RF

receptive field

Dpn

days postnatal

TMS

transcranial magnetic stimulation

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Reference List

  1. Alvarado JC, Stanford TR, Vaughan JW, Stein BE. Cortex mediates multisensory but not unisensory integration in superior colliculus. J Neurosci. 2007a;27:12775–12786. doi: 10.1523/JNEUROSCI.3524-07.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Alvarado JC, Vaughan JW, Stanford TR, Stein BE. Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J Neurophysiol. 2007b;97:3193–3205. doi: 10.1152/jn.00018.2007. [DOI] [PubMed] [Google Scholar]
  3. Barth DS, Brett-Green B. Multisensory -evoked potentials in rat cortex. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004. pp. 357–370. [Google Scholar]
  4. Bernstein LE, Auer ET, Moore JK. Audiovisual speech binding: convergence or association. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004. pp. 203–224. [Google Scholar]
  5. Bolognini N, Maravita A. Proprioceptive alignment of visual and somatosensory maps in the posterior parietal cortex. Curr Biol. 2007;17:1890–1895. doi: 10.1016/j.cub.2007.09.057. [DOI] [PubMed] [Google Scholar]
  6. Bower TGR. The evolution of sensory systems. In: Macleod RGPHL Jr, editor. Perception: Essays in Honor of James J Gibson. Ithaca: Cornell Univ. Press; 1974. pp. 141–165. [Google Scholar]
  7. Burnett LR, Stein BE, Chaponis D, Wallace MT. Superior colliculus lesions preferentially disrupt multisensory orientation. Neuroscience. 2004;124:535–547. doi: 10.1016/j.neuroscience.2003.12.026. [DOI] [PubMed] [Google Scholar]
  8. Burnett LR, Stein BE, Perrault TJ, Jr, Wallace MT. Excitotoxic lesions of the superior colliculus preferentially impact multisensory neurons and multisensory integration. Exp Brain Res. 2007;179:325–338. doi: 10.1007/s00221-006-0789-8. [DOI] [PubMed] [Google Scholar]
  9. Busse L, Roberts KC, Crist RE, Weissman DH, Woldorff MG. The spread of attention across modalities and space in a multisensory object. Proc Natl Acad Sci U S A. 2005;102:18751–18756. doi: 10.1073/pnas.0507704102. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Calvert GA, Lewis JW. Hemodynamic studies of audiovisual interactions. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004. pp. 483–502. [Google Scholar]
  11. Calvert GA, Spence C, Stein BE. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004. [Google Scholar]
  12. Corneil BD, Munoz DP. The influence of auditory and visual distractors on human orienting gaze shifts. J Neurosci. 1996;16:8193–8207. doi: 10.1523/JNEUROSCI.16-24-08193.1996. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Edwards SB, Ginsburgh CL, Henkel CK, Stein BE. Sources of subcortical projections to the superior colliculus in the cat. J Comp Neurol. 1979;184:309–330. doi: 10.1002/cne.901840207. [DOI] [PubMed] [Google Scholar]
  14. Ernst MO, Banks MS. Humans integrate visual and haptic information in a statistically optimal fashion. Nature. 2002;415:429–433. doi: 10.1038/415429a. [DOI] [PubMed] [Google Scholar]
  15. Fort A, Giard MH. Multiple electrophysiological mechanisms of audiovisual integration in human perception. In: Calvert G, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004. pp. 503–514. [Google Scholar]
  16. Frens MA, Van Opstal AJ. A quantitative study of auditory-evoked saccadic eye movements in two dimensions. Exp Brain Res. 1995;107:103–117. doi: 10.1007/BF00228022. [DOI] [PubMed] [Google Scholar]
  17. Fuentes-Santamaria V, Alvarado JC, Stein BE, McHaffie JG. Cortex contacts both output neurons and nitrergic interneurons in the superior colliculus: direct and indirect routes for multisensory integration. Cereb Cortex. 2008;18:1640–1652. doi: 10.1093/cercor/bhm192. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Gelder B, Vroomen J, Pourtois G. Multisensory perception of emotion, its time course aned its neural basis. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004. pp. 581–596. [Google Scholar]
  19. Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci. 2005;25:5004–5012. doi: 10.1523/JNEUROSCI.0799-05.2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Ghazanfar AA, Schroeder CE. Is neocortex essentially multisensory? Trends Cogn Sci. 2006;10:278–285. doi: 10.1016/j.tics.2006.04.008. [DOI] [PubMed] [Google Scholar]
  21. Gibson JJ. The Senses Considered as Perceptual Systems. Boston, MA: Houghton Mifflin.; 1966. [Google Scholar]
  22. Gibson EJ. Principles of perceptual learning and development. Englewood Cliffs, N.J: Princtice Hall; 1969. [Google Scholar]
  23. Gibson JJ. An ecological approach to perception. Boston: Houghton Mifflin; 1979. [Google Scholar]
  24. Gingras G, Rowland BA, Stein BE. The differing impact of multisensory and unisensory integration on behavior. J Neurosci. 2009 doi: 10.1523/JNEUROSCI.4120-08.2009. In Press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Gondan M, Niederhaus B, Rosler F, Roder B. Multisensory processing in the redundant-target effect: a behavioral and event-related potential study. Percept Psychophys. 2005;67:713–726. doi: 10.3758/bf03193527. [DOI] [PubMed] [Google Scholar]
  26. Gori M, Del VM, Sandini G, Burr DC. Young children do not integrate visual and haptic form information. Curr Biol. 2008;18:694–698. doi: 10.1016/j.cub.2008.04.036. [DOI] [PubMed] [Google Scholar]
  27. Grant AC, Thiagarajah MC, Sathian K. Tactile perception in blind Braille readers: a psychophysical study of acuity and hyperacuity using gratings and dot patterns. Percept Psychophys. 2000;62:301–312. doi: 10.3758/bf03205550. [DOI] [PubMed] [Google Scholar]
  28. Grantyn A, Grantyn R. Axonal patterns and sites of termination of cat superior colliculus neurons projecting in the tecto-bulbo-spinal tract . Exp Brain Res. 1982:243–265. doi: 10.1007/BF00237182. [DOI] [PubMed] [Google Scholar]
  29. Groh JM, Sparks DL. Saccades to somatosensory targets. III. eye-position-dependent somatosensory activity in primate superior colliculus. J Neurophysiol. 1996b;75:439–453. doi: 10.1152/jn.1996.75.1.439. [DOI] [PubMed] [Google Scholar]
  30. Groh JM, Sparks DL. Saccades to somatosensory targets. II. motor convergence in primate superior colliculus. J Neurophysiol. 1996a;75:428–438. doi: 10.1152/jn.1996.75.1.428. [DOI] [PubMed] [Google Scholar]
  31. Guitton D, Munoz DP. Control of orienting gaze shifts by the tectoreticulospinal system in the head-free cat. I. Identification, localization, and effects of behavior on sensory responses. J Neurophysiol. 1991;66:1605–1623. doi: 10.1152/jn.1991.66.5.1605. [DOI] [PubMed] [Google Scholar]
  32. Gutfreund Y, Knudsen EI. Visual instruction of the auditory space map in the midbrain. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Calvert, G.A.; Spence, C.; Stein B.E: MIT Press; 2004. pp. 613–624. [Google Scholar]
  33. Harris LR. The superior colliculus and movements of the head and eyes in cats. J Physiol (Lond) 1980;300:367–391. doi: 10.1113/jphysiol.1980.sp013167. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Huerta MF, Harting JK. The Mammalian Superior Colliculus: Studies of Its Morphology and Connections. In: Vanegas H, editor. Comparative Neurology of the Optic Tectum. Plenum Publishing Corporation; 1984. pp. 687–773. [Google Scholar]
  35. Hughes HC, Reuter-Lorenz PA, Nozawa G, Fendrich R. Visual-auditory interactions in sensorimotor processing: saccades versus manual responses. J Exp Psychol Hum Percept Perform. 1994;20:131–153. doi: 10.1037//0096-1523.20.1.131. [DOI] [PubMed] [Google Scholar]
  36. Jay MF, Sparks DL. Auditory receptive fields in primate superior colliculus shift with changes in eye position. Nature. 1984;309:345–347. doi: 10.1038/309345a0. [DOI] [PubMed] [Google Scholar]
  37. Jay MF, Sparks DL. Sensorimotor integration in the primate superior colliculus. II. Coordinates of auditory signals. J Neurophysiol. 1987a;57:35–55. doi: 10.1152/jn.1987.57.1.35. [DOI] [PubMed] [Google Scholar]
  38. Jay MF, Sparks DL. Sensorimotor integration in the primate superior colliculus. I. Motor convergence. J Neurophysiol. 1987b;57:22–34. doi: 10.1152/jn.1987.57.1.22. [DOI] [PubMed] [Google Scholar]
  39. Jiang W, Stein BE. Neonatal cortical ablation disrupts development in superior colliculus. J Neurophysiol. 2006;95:1380–1396. doi: 10.1152/jn.00880.2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Jiang W, Jiang H, Rowland BA, Stein BE. Multisensory orientation behavior is disrupted by neonatal cortical ablation. J Neurophysiol. 2007b;97:557–562. doi: 10.1152/jn.00591.2006. [DOI] [PubMed] [Google Scholar]
  41. Jiang W, Jiang H, Rowland BA, Stein BE. Multisensory orientation behavior is disrupted by neonatal cortical ablation. J Neurophysiol. 2007a;97:557–562. doi: 10.1152/jn.00591.2006. [DOI] [PubMed] [Google Scholar]
  42. Jiang W, Jiang H, Stein BE. Two corticotectal areas facilitate multisensory orientation behavior. J Cogn Neurosci. 2002;14:1240–1255. doi: 10.1162/089892902760807230. [DOI] [PubMed] [Google Scholar]
  43. Jiang W, Jiang H, Stein BE. Neonatal cortical ablation disrupts multisensory development in superior colliculus. J Neurophysiol. 2006;95:1380–1396. doi: 10.1152/jn.00880.2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Jiang W, Stein BE. Effect of early ablation of anterior ectosylvian sulcus (AES) and rostral lateral suprasylvian (rLS) cortex on multisensory processes in the superior colliculus (SC) Soc Neurosci Abstr. 2002:560.6. [Google Scholar]
  45. Jiang W, Stein BE. Cortex controls multisensory depression in superior colliculus. J Neurophysiol. 2003;90:2123–2135. doi: 10.1152/jn.00369.2003. [DOI] [PubMed] [Google Scholar]
  46. Jiang W, Wallace MT, Jiang H, Vaughan JW, Stein BE. Two cortical areas mediate multisensory integration in superior colliculus neurons. J Neurophysiol. 2001;85:506–522. doi: 10.1152/jn.2001.85.2.506. [DOI] [PubMed] [Google Scholar]
  47. Kadunce DC, Vaughan JW, Wallace MT, Stein BE. The influence of visual and auditory receptive field organization on multisensory integration in the superior colliculus 1. Exp Brain Res. 2001;139:303–310. doi: 10.1007/s002210100772. [DOI] [PubMed] [Google Scholar]
  48. Kao CQ, Stein BE, Coulter DA. Postnatal development of excitatory synaptic function in deep layers of superior colliculus. Soc Neurosci Abstr. 1994a;20:1186. [Google Scholar]
  49. King AJ, Palmer AR. Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res. 1985;60:492–500. doi: 10.1007/BF00236934. [DOI] [PubMed] [Google Scholar]
  50. King AJ, Schnupp JW, Carlile S, Smith AL, Thompson ID. The development of topographically-aligned maps of visual and auditory space in the superior colliculus. Prog Brain Res. 1996;112:335–350. doi: 10.1016/s0079-6123(08)63340-3. [DOI] [PubMed] [Google Scholar]
  51. King AJ, Doubell TP, Skaliora I. Epigenetic factors that align visual and auditory mas in the ferret midbrain. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004. pp. 613–624. [Google Scholar]
  52. Lakatos P, Chen CM, O’Connell MN, Mills A, Schroeder CE. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron. 2007a;53:279–292. doi: 10.1016/j.neuron.2006.12.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Larson MA, Stein BE. The use of tactile and olfactory cues in neonatal orientation and localization of the nipple. Dev Psychobiol. 1984;17:423–436. doi: 10.1002/dev.420170408. [DOI] [PubMed] [Google Scholar]
  54. Laurienti PJ, Burdette JH, Wallace MT, Yen YF, Field AS, Stein BE. Deactivation of sensory-specific cortex by cross-modal stimuli. J Cogn Neurosci. 2002;14:420–429. doi: 10.1162/089892902317361930. [DOI] [PubMed] [Google Scholar]
  55. Leo F, Bolognini N, Passamonti C, Stein BE, Ladavas E. Cross-modal localization in hemianopia: new insights on multisensory integration. Brain. 2008;131:855–865. doi: 10.1093/brain/awn003. [DOI] [PubMed] [Google Scholar]
  56. Lewkowicz DJ, Kraebel KS. The value of multisensory redundancy in the developmemt of intersensory perception. In: Calvert G, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004. pp. 655–678. [Google Scholar]
  57. Lickliter R, Bahrick LE. Perceptual development and the origins of multisensory responsiveness. In: Calvert G, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004. pp. 643–654. [Google Scholar]
  58. Liotti M, Ryder K, Woldorff MG. Auditory attention in the congenitally blind: where, when and what gets reorganized? Neuroreport. 1998;9:1007–1012. doi: 10.1097/00001756-199804200-00010. [DOI] [PubMed] [Google Scholar]
  59. Lippert M, Logothetis NK, Kayser C. Improvement of visual contrast detection by a simultaneous sound. Brain Res. 2007;1173:102–109. doi: 10.1016/j.brainres.2007.07.050. [DOI] [PubMed] [Google Scholar]
  60. Lovelace CT, Stein BE, Wallace MT. An Irrelevant Light Enhances Auditory Detection in Humans: a Psychophysical Analysis of Multisensory Integration in Stimulus Detection. Brain Research. 2003 doi: 10.1016/s0926-6410(03)00160-5. [DOI] [PubMed] [Google Scholar]
  61. Macaluso E, Driver J. Neuroimaging studies of cross-modal integration for emotion. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004. pp. 529–548. [Google Scholar]
  62. Marks LE. Cross-modal interactions in speeded classification. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004. pp. 85–106. [Google Scholar]
  63. Marks LE. The unity of the senses: Interrelations among the modalities. New York, NY: Academic Press.; 1978. [Google Scholar]
  64. Massaro DW. From multisensory integration to talking heads and language learning. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press.; 2004. pp. 153–176. [Google Scholar]
  65. Meredith MA, Clemo HR, Stein BE. Somatotopic component of the multisensory map in the deep laminae of the cat superior colliculus. J Comp Neurol. 1991;312:353–370. doi: 10.1002/cne.903120304. [DOI] [PubMed] [Google Scholar]
  66. Meredith MA, Stein BE. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol. 1986a;56:640–662. doi: 10.1152/jn.1986.56.3.640. [DOI] [PubMed] [Google Scholar]
  67. Meredith MA, Stein BE. Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res. 1986b;365:350–354. doi: 10.1016/0006-8993(86)91648-3. [DOI] [PubMed] [Google Scholar]
  68. Meredith MA, Stein BE. Interactions among converging sensory inputs in the superior colliculus. Science. 1983;221:389–391. doi: 10.1126/science.6867718. [DOI] [PubMed] [Google Scholar]
  69. Meredith MA, Stein BE. The visuotopic component of the multisensory map in the deep laminae of the cat superior colliculus. J Neurosci. 1990;10:3727–3742. doi: 10.1523/JNEUROSCI.10-11-03727.1990. [DOI] [PMC free article] [PubMed] [Google Scholar]
  70. Meredith MA, Stein BE. Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol. 1996;75:1843–1857. doi: 10.1152/jn.1996.75.5.1843. [DOI] [PubMed] [Google Scholar]
  71. Meredith MA, Wallace MT, Stein BE. Visual, auditory and somatosensory convergence in output neurons of the cat superior colliculus: multisensory properties of the tecto-reticulo- spinal projection. Exp Brain Res. 1992;88:181–186. doi: 10.1007/BF02259139. [DOI] [PubMed] [Google Scholar]
  72. Middlebrooks JC, Knudsen EI. A neural code for auditory space in the cat’s superior colliculus. J Neurosci. 1984;4:2621–2634. doi: 10.1523/JNEUROSCI.04-10-02621.1984. [DOI] [PMC free article] [PubMed] [Google Scholar]
  73. Miller J. Divided attention: evidence for coactivation with redundant signals. Cogn Psychol. 1982;14:247–279. doi: 10.1016/0010-0285(82)90010-x. [DOI] [PubMed] [Google Scholar]
  74. Moschovakis AK, Karabelas AB. Observations on the somatodendritic morphology and axonal trajectory of intracellularly HRP-labeled efferent neurons located in the deeper layers of the superior colliculus of the cat. J Comp Neurol. 1985;239:276–308. doi: 10.1002/cne.902390304. [DOI] [PubMed] [Google Scholar]
  75. Munoz DP, Wurtz RH. Fixation cells in monkey superior colliculus. I. Characteristics of cell discharge. J Neurophysiol. 1993a;70:559–575. doi: 10.1152/jn.1993.70.2.559. [DOI] [PubMed] [Google Scholar]
  76. Munoz DP, Wurtz RH. Fixation cells in monkey superior colliculus. II. Reversible activation and deactivation. J Neurophysiol. 1993b;70:576–589. doi: 10.1152/jn.1993.70.2.576. [DOI] [PubMed] [Google Scholar]
  77. Neil PA, Chee-Ruiter C, Scheier C, Lewkowicz DJ, Shimojo S. Development of multisensory spatial integration and perception in humans. Dev Sci. 2006;9:454–464. doi: 10.1111/j.1467-7687.2006.00512.x. [DOI] [PubMed] [Google Scholar]
  78. Newell FN. Cross-modal object recognition. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press.; 2004. pp. 123–140. [Google Scholar]
  79. Partan SR. Multisensory animal communication. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press.; 2004. pp. 225–242. [Google Scholar]
  80. Peck CK. Visual-auditory interactions in cat superior colliculus: their role in the control of gaze. Brain Res. 1987;420:162–166. doi: 10.1016/0006-8993(87)90253-8. [DOI] [PubMed] [Google Scholar]
  81. Perrault TJ, Jr, Vaughan JW, Stein BE, Wallace MT. Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J Neurophysiol. 2005;93:2575–2586. doi: 10.1152/jn.00926.2004. [DOI] [PubMed] [Google Scholar]
  82. Perrault TJ, Jr, Vaughan JW, Stein BE, Wallace MT. Neuron-specific response characteristics predict the magnitude of multisensory integration. J Neurophysiol. 2003;90:4022–4026. doi: 10.1152/jn.00494.2003. [DOI] [PubMed] [Google Scholar]
  83. Putzar L, Goerendt I, Lange K, Rosler F, Roder B. Early visual deprivation impairs multisensory interactions in humans. Nat Neurosci. 2007;10:1243–1245. doi: 10.1038/nn1978. [DOI] [PubMed] [Google Scholar]
  84. Recanzone GH. Rapidly induced auditory plasticity: the ventriloquism aftereffect. Proc Natl Acad Sci U S A. 1998;95:869–875. doi: 10.1073/pnas.95.3.869. [DOI] [PMC free article] [PubMed] [Google Scholar]
  85. Rowland BA, Jiang W, Stein BE. Long-term plasticity in multisensory integration. Soc Neurosci Abstr. 2006:505.08. [Google Scholar]
  86. Rowland BA, Quessy S, Stanford TR, Stein BE. Multisensory integration shortens physiological response latencies. J Neurosci. 2007;27:5879–5884. doi: 10.1523/JNEUROSCI.4986-06.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  87. Rowland BA, Stein BE. Multisensory integration produces an initial response enhancement. Front Integr Neurosci. 2007;1:4. doi: 10.3389/neuro.07.004.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  88. Rowland BA, Stein BE. Temporal profiles in multisensory integration. Frontiers in Integrative Neurosci. 2008 doi: 10.3389/neuro.01.033.2008. In Press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  89. Sathian K. Practice makes perfect: sharper tactile perception in the blind. Neurology. 2000;54:2203–2204. doi: 10.1212/wnl.54.12.2203. [DOI] [PubMed] [Google Scholar]
  90. Sathian K. Visual cortical activity during tactile perception in the sighted and the visually deprived. Dev Psychobiol. 2005;46:279–286. doi: 10.1002/dev.20056. [DOI] [PubMed] [Google Scholar]
  91. Sathian K, Prather SCZM. Visual cortical involvement in normal tactile perception. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004. pp. 703–710. [Google Scholar]
  92. Schroeder CE, Foxe JJ. Multisensory convergence in early cortical processing. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004. pp. 295–310. [Google Scholar]
  93. Schroeder CE, Foxe JJ. The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Res Cogn Brain Res. 2002;14:187–198. doi: 10.1016/s0926-6410(02)00073-3. [DOI] [PubMed] [Google Scholar]
  94. Schroeder CE, Lindsley RW, Specht C, Marcovici A, Smiley JF, Javitt DC. Somatosensory input to auditory association cortex in the macaque monkey. J Neurophysiol. 2001;85:1322–1327. doi: 10.1152/jn.2001.85.3.1322. [DOI] [PubMed] [Google Scholar]
  95. Senkowski D, Talsma D, Grigutsch M, Herrmann CS, Woldorff MG. Good times for multisensory integration: Effects of the precision of temporal synchrony as revealed by gamma-band oscillations. Neuropsychologia. 2007;45:561–571. doi: 10.1016/j.neuropsychologia.2006.01.013. [DOI] [PubMed] [Google Scholar]
  96. Shams L, Kamitani Y, Shimojo S. Modulations of visual perception by sound. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press.; 2004. pp. 27–34. [Google Scholar]
  97. Sinnett S, Soto-Faraco S, Spence C. The co-occurrence of multisensory competition and facilitation. Acta Psychol (Amst) 2008;128:153–161. doi: 10.1016/j.actpsy.2007.12.002. [DOI] [PubMed] [Google Scholar]
  98. Sparks DL. Translation of sensory signals into commands for control of saccadic eye movements: role of primate superior colliculus. Physiol Rev. 1986;66:118–171. doi: 10.1152/physrev.1986.66.1.118. [DOI] [PubMed] [Google Scholar]
  99. Sparks DL, Nelson JS. Sensory and Motor Maps in the Mammalian Superior Colliculus. Trends in Neuroscience. 1987:312–317. [Google Scholar]
  100. Spence C, Driver J. Crossmodal Space and Crossmodal Attention. Oxford: Oxford Press; 2004. [Google Scholar]
  101. Stanford TR, Quessy S, Stein BE. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J Neurosci. 2005;25:6499–6508. doi: 10.1523/JNEUROSCI.5095-04.2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  102. Stanford TR, Stein BE. Superadditivity in multisensory integration: putting the computation in context. Neuroreport. 2007;18:787–792. doi: 10.1097/WNR.0b013e3280c1e315. [DOI] [PubMed] [Google Scholar]
  103. Stein BE. Development of the superior colliculus. Ann Rev Neurosci. 1984b;7:95–125. doi: 10.1146/annurev.ne.07.030184.000523. [DOI] [PubMed] [Google Scholar]
  104. Stein BE. Annual Review of Neuroscience. Palo Alto, CA: Annual Reviews, Inc.; 1984a. Development of the superior colliculus; pp. 95–125. [DOI] [PubMed] [Google Scholar]
  105. Stein BE, Arigbede MO. Unimodal and multimodal response properties of neurons in the cat’s superior colliculus. Exp Neurol. 1972;36:179–196. doi: 10.1016/0014-4886(72)90145-8. [DOI] [PubMed] [Google Scholar]
  106. Stein BE, Clamann HP. Control of pinna movements and sensorimotor register in cat superior colliculus. Brain Behav Evol. 1981;19:180–192. doi: 10.1159/000121641. [DOI] [PubMed] [Google Scholar]
  107. Stein BE, Gallagher HL. Maturation of cortical control over superior colliculus cells in cat. Brain Res. 1981;223:429–435. doi: 10.1016/0006-8993(81)91160-4. [DOI] [PubMed] [Google Scholar]
  108. Stein BE, Labos E, Kruger L. Sequence of changes in properties of neurons of superior colliculus of the kitten during maturation. J Neurophysiol. 1973;36:667–679. doi: 10.1152/jn.1973.36.4.667. [DOI] [PubMed] [Google Scholar]
  109. Stein BE, London N, Wilkinson LK, Price DD. Enhancement of Perceived Visual Intensity by Auditory Stimuli: A Psychophysical Analysis. Journal of Cognitive Neuroscience. 1996;8:497–506. doi: 10.1162/jocn.1996.8.6.497. [DOI] [PubMed] [Google Scholar]
  110. Stein BE, Magalhaes-Castro B, Kruger L. Relationship between visual and tactile representations in cat superior colliculus. J Neurophysiol. 1976;39:401–419. doi: 10.1152/jn.1976.39.2.401. [DOI] [PubMed] [Google Scholar]
  111. Stein BE, Meredith MA, Huneycutt WS, McDade L. Behavioral indices of multisensory integration: orientation to visual cues is affected by auditory stimuli. Journal of Cognitive Neuroscience. 1989;1:12–24. doi: 10.1162/jocn.1989.1.1.12. [DOI] [PubMed] [Google Scholar]
  112. Stein BE, Meredith MA. The merging of the senses. Cambridge, Mass: MIT Press.; 1993. [Google Scholar]
  113. Stein BE, Meredith MA, Wallace MT. The visually responsive neuron and beyond: multisensory integration in cat and monkey. Prog Brain Res. 1993;95:79–90. doi: 10.1016/s0079-6123(08)60359-3. [DOI] [PubMed] [Google Scholar]
  114. Stein BE, Perrault TJ, Jr , Vaughan JW, Rowland BA. Long-term plasticity of multisensory neurons in the superior colliculus. Soc Neurosci Abstr. 2008:457.14. [Google Scholar]
  115. Stein BE, Stanford TR, Wallace MT. Neural Mechanisms of Cross-Modal Synthesis. In: Goldstein EB, editor. Blackwell’s Handbook of Perception. Oxford; U.K: 2001a. pp. 709–736. [Google Scholar]
  116. Stein BE, Wallace MT, Stanford TR. Cross-Modal (Multi-Sensory) Integration: Psychological and Neural Aspects. In: McClelland JL, Thompson R, editors. International Encyclopedia of the Social and Behavioral Sciences. Elsevier; 2001b. pp. 3008–3015. [Google Scholar]
  117. Stein BE, Wallace MW, Stanford TR, Jiang W. Cortex governs multisensory integration in the midbrain. Neuroscientist. 2002;8:306–314. doi: 10.1177/107385840200800406. [DOI] [PubMed] [Google Scholar]
  118. Sugihara T, Diltz MD, Averbeck BB, Romanski LM. Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. J Neurosci. 2006;26:11138–11147. doi: 10.1523/JNEUROSCI.3550-06.2006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  119. Sumby WH, Pollack I. Visual contribution to speech intelligibility in noise. J Acoust Soc Am. 1954;26:212–215. [Google Scholar]
  120. Talsma D, Doty TJ, Strowd R, Woldorff MG. Attentional capacity for processing concurrent stimuli is larger across sensory modalities than within a modality. Psychophysiology. 2006a;43:541–549. doi: 10.1111/j.1469-8986.2006.00452.x. [DOI] [PubMed] [Google Scholar]
  121. Talsma D, Doty TJ, Woldorff MG. Selective Attention and Audiovisual Integration: Is Attending to Both Modalities a Prerequisite for Early Integration? Cereb Cortex. 2006b doi: 10.1093/cercor/bhk016. [DOI] [PubMed] [Google Scholar]
  122. Wallace MT. The development of multisensory integration. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004. pp. 625–642. [Google Scholar]
  123. Wallace MT, Carriere BN, Perrault TJ, Jr, Vaughan JW, Stein BE. The development of cortical multisensory integration. J Neurosci. 2006;26:11844–11849. doi: 10.1523/JNEUROSCI.3295-06.2006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  124. Wallace MT, Hairston WD, Stein BE. Long-term effects of dark-rearing on multisensory processing. Soc Neurosci Abstracts. 2001:511.6. [Google Scholar]
  125. Wallace MT, Meredith MA, Stein BE. Integration of multiple sensory modalities in cat cortex. Exp Brain Res. 1992;91:484–488. doi: 10.1007/BF00227844. [DOI] [PubMed] [Google Scholar]
  126. Wallace MT, Meredith MA, Stein BE. Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. J Neurophysiol. 1993;69:1797–1809. doi: 10.1152/jn.1993.69.6.1797. [DOI] [PubMed] [Google Scholar]
  127. Wallace MT, Meredith MA, Stein BE. Multisensory integration in the superior colliculus of the alert cat. J Neurophysiol. 1998;80:1006–1010. doi: 10.1152/jn.1998.80.2.1006. [DOI] [PubMed] [Google Scholar]
  128. Wallace MT, Perrault TJ, Jr, Hairston WD, Stein BE. Visual experience is necessary for the development of multisensory integration 1. J Neurosci. 2004a;24:9580–9584. doi: 10.1523/JNEUROSCI.2535-04.2004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  129. Wallace MT, Ramachandran R, Stein BE. A revised view of sensory cortical parcellation. Proc Natl Acad Sci U S A. 2004b;101:2167–2172. doi: 10.1073/pnas.0305697101. [DOI] [PMC free article] [PubMed] [Google Scholar]
  130. Wallace MT, Stein BE. Onset of cross-modal synthesis in the neonatal superior colliculus is gated by the development of cortical influences. J Neurophysiol. 2000;83:3578–3582. doi: 10.1152/jn.2000.83.6.3578. [DOI] [PubMed] [Google Scholar]
  131. Wallace MT, Stein BE. Early experience determines how the senses will interact. J Neurophysiol. 2007;97:921–926. doi: 10.1152/jn.00497.2006. [DOI] [PubMed] [Google Scholar]
  132. Wallace MT, Stein BE. Cross-modal synthesis in the midbrain depends on input from cortex. J Neurophysiol. 1994;71:429–432. doi: 10.1152/jn.1994.71.1.429. [DOI] [PubMed] [Google Scholar]
  133. Wallace MT, Stein BE. Development of multisensory neurons and multisensory integration in cat superior colliculus. J Neurosci. 1997;17:2429–2444. doi: 10.1523/JNEUROSCI.17-07-02429.1997. [DOI] [PMC free article] [PubMed] [Google Scholar]
  134. Wallace MT, Wilkinson LK, Stein BE. Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol. 1996;76:1246–1266. doi: 10.1152/jn.1996.76.2.1246. [DOI] [PubMed] [Google Scholar]
  135. Weisser V, Peltier S, Hu X, Sathian K. Short-term visual deprivation alters neural processing of tactile form. Exp Brain Res. 2005;166:572–582. doi: 10.1007/s00221-005-2397-4. [DOI] [PubMed] [Google Scholar]
  136. Werner H. Comparative psychology of mental development. New York, NY: International Universities Press; 1973. [Google Scholar]
  137. Wilkinson LK, Meredith MA, Stein BE. The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp Brain Res. 1996;112:1–10. doi: 10.1007/BF00227172. [DOI] [PubMed] [Google Scholar]
  138. Woldorff MG, Hazlett CJ, Fichtenholtz HM, Weissman DH, Dale AM, Song AW. Functional parcellation of attentional control regions of the brain. J Cogn Neurosci. 2004;16:149–165. doi: 10.1162/089892904322755638. [DOI] [PubMed] [Google Scholar]
  139. Woods TM, Recanzone GH. Cross-modal interactions evidenced by the ventriloquism effect in humans and monkeys. In: Calvert GA, Spence C, Stein BE, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004a. pp. 35–48. [Google Scholar]
  140. Woods TM, Recanzone GH. Visually induced plasticity of auditory spatial perception in macaques. Curr Biol. 2004b;14:1559–1564. doi: 10.1016/j.cub.2004.08.059. [DOI] [PubMed] [Google Scholar]
  141. Wurtz RH, Albano JE. Two visual systems: brain mechanisms for localization and discrimination are dissociated by tectal and cortical lesions. Annu Rev Neurosci. 1980;3:189–226. doi: 10.1146/annurev.ne.03.030180.001201. [DOI] [PubMed] [Google Scholar]
  142. Wurtz RH, Goldberg ME. Superior colliculus cell responses related to eye movements in awake monkeys. Science. 1971;171:82–84. doi: 10.1126/science.171.3966.82. [DOI] [PubMed] [Google Scholar]
  143. Zangaladze A, Epstein CM, Grafton ST, Sathian K. Involvement of visual cortex in tactile discrimination of orientation. Nature. 1999;401:587–590. doi: 10.1038/44139. [DOI] [PubMed] [Google Scholar]

RESOURCES