Incorporating Cross-Modal Statistics in the Development and Maintenance of Multisensory Integration

Abstract

Development of multisensory integration capabilities in superior colliculus (SC) neurons was examined in cats whose visual–auditory experience was restricted to a circumscribed period during early life (postnatal day 30–8 months). Animals were periodically exposed to visual and auditory stimuli appearing either randomly in space and time, or always in spatiotemporal concordance. At all other times animals were maintained in darkness. Physiological testing was initiated at ∼2 years of age. Exposure to random visual and auditory stimuli proved insufficient to spur maturation of the ability to integrate cross-modal stimuli, but exposure to spatiotemporally concordant cross-modal stimuli was highly effective. The multisensory integration capabilities of neurons in the latter group resembled those of normal animals and were retained for >16 months in the absence of subsequent visual–auditory experience. Furthermore, the neurons were capable of integrating stimuli having physical properties differing significantly from those in the exposure set. These observations suggest that acquiring the rudiments of multisensory integration requires little more than exposure to consistent relationships between the modality-specific components of a cross-modal event, and that continued experience with such events is not necessary for their maintenance. Apparently, the statistics of cross-modal experience early in life define the spatial and temporal filters that determine whether the components of cross-modal stimuli are to be integrated or treated as independent events, a crucial developmental process that determines the spatial and temporal rules by which cross-modal stimuli are integrated to enhance both sensory salience and the likelihood of eliciting an SC-mediated motor response.

Introduction

Neurons in the superior colliculus (SC) receive inputs from multiple sensory modalities, and their ability to integrate information across them has been linked to enhancements in stimulus detection and localization (for review, see Stein and Stanford, 2008) (Hartline et al., 1978; Gaither and Stein, 1979; Zahar et al., 2009). However, this capability is not yet present at birth (Wallace and Stein, 1997, 2001). Its maturation has been examined most closely in the cat, an altricial species whose neonatal SC neurons are unisensory (Stein et al., 1973) but gradually acquire multisensory response properties over many postnatal weeks (Wallace and Stein, 1997). The first stage involves a transition to responding to more than one sensory input (i.e., becoming “multisensory”), and then, after extensive experience with cross-modal cues, acquiring the ability to integrate these signals to produce enhanced multisensory responses.

This period of multisensory maturation normally takes place over the first few months of life when the brain is highly plastic (Wiesel and Hubel, 1965; Simons and Land, 1987; Buonomano and Merzenich, 1998; Feldman, 2001; Rauschecker, 2002; Li et al., 2006; Sanes and Bao, 2009). The first neuron seen to exhibit multisensory integration (as measured by multisensory enhancement) appears no earlier than 4 postnatal weeks, and the normal complement of such neurons does not develop for several months. Eliminating cross-modal experience by rearing animals in darkness disrupts this process in neurons responsive to multiple modalities; for example, although exposure to visual stimuli is not required for neurons to become visually responsive or visual–auditory, these neurons are not capable of multisensory integration (Wallace et al., 2004). Nevertheless, some multisensory plasticity remains in adulthood (Yu et al., 2009), and neurons deprived of early experience may still acquire this capability with appropriate experience later in life (Yu et al., 2010). Interestingly, these experiential effects can be induced in these adult animals while they are anesthetized, indicating that they need neither to evoke overt responses nor be coupled to any of the reinforcement contingencies normally associated with learning.

These observations raise a number of questions concerning the development of multisensory integration, and whether it reflects the encoding of specific stimulus relationships or the general statistics of sensory experience. Of particular interest here is whether it is constrained by the features of the stimuli that were encountered; whether once acquired, this capability is permanent or requires continued exposure to cross-modal stimuli for maintenance; and whether the impact of these experiences in anesthetized preparations is in any way equivalent to those initiated by the same stimuli in alert animals. The present experiments were designed to explore these questions by rearing cats in darkness, exposing some of them to concordant visual–auditory stimuli and others to random modality-specific visual and auditory stimuli during a limited developmental time frame (postnatal day 30–8 months), and then maintaining them in darkness for an additional 1.3 years before examining the multisensory integration capabilities of their visual–auditory SC neurons.

Materials and Methods

All protocols were used in accordance with the Guide for the Care and Use of Laboratory Animals (National Institutes of Health Publication 86–23) and were approved by the Animal Care and Use Committee of Wake Forest University School of Medicine, an Association for the Assessment and Accreditation of Laboratory Animal Care International accredited institution.

Rearing conditions.

A total of eight cats were used here: four were reared in standard housing conditions and formed the control group from which the normal control neurons were drawn. An additional four animals were raised from birth in a dark room and exposed to light and sound in a cylindrical arena for 5 h/d, 5 d/week between postnatal day 30 and 8 months of age. The arena was 1 m in diameter, 40 cm in height, and contained an array of 12 colocalized light emitting diodes (LEDs) and speakers mounted with even spacing (30°) along its inside perimeter at 10 cm of elevation (Fig. 1). Ambient background noise in the arena was 51–52 dB. At all other times the animals were maintained in complete darkness. At 8 months of age, the exposure procedures were terminated and the animals were housed in the dark without additional visual experience for an additional 16 months before electrophysiological testing began.

Figure 1.

Four cats were raised from birth in darkness, but were exposed daily to periodic visual and/or auditory stimuli in a cylindrical enclosure using two different stimulus exposure paradigms. The animals were free to move anywhere within the enclosure, and the positions and responses shown are illustrative only. In most cases, animals did not react to the stimuli. A, In the coincident exposure paradigm, two animals were exposed to visual–auditory (cross-modal) stimuli that were in spatiotemporal coincidence, but could occur at any location around the circumference of the enclosure. The stimuli were presented at random intervals between 50 and 4000 ms. B, In the random exposure paradigm, two other animals received visual or auditory (modality-specific) stimuli that were presented randomly in space and time with the same frequency. Both groups received 5 h of exposure/d 5 d/week from postnatal day 30 to 8 months of age. Stimuli consisted of flashes of light from LEDs and broadband noise burst from speakers embedded in the wall of the enclosure. At all other times the animals were maintained in darkness.

The visual stimuli presented in the arena were brief (100 ms) flashes of bright light and the auditory stimuli were brief (100 ms) highly audible (65 dB) bursts of broadband noise (20–20,000 Hz). Stimuli occurred at random locations with an interstimulus interval randomly selected between 50 and 4000 ms. Animals were divided into two groups. The first group (n = 2) was only exposed to spatiotemporally concordant cross-modal visual–auditory stimulus pairs (i.e., a single random process determined the timing of activation of both channels with randomly-determined location). The second group (n = 2) was exposed to visual and auditory stimuli that were randomly distributed in space and time (i.e., the timing of activation of the visual and auditory stimuli were determined by two independent random processes). In both groups the exposure stimuli occurred independent of any responses on the part of the animals and could be assessed (or ignored) from a variety of angles in the apparatus. Daily observation showed that animals showed no specific reactions to the stimuli.

Daily animal care was facilitated by the use of binocular infrared goggles. In addition, an infrared closed-circuit television system allowed animals to be observed from an adjacent room at any time. All routine veterinary procedures that required removing the animals from the dark were conducted after the animals were anesthetized and blindfolded in the dark room.

Surgical and electrophysiological recording procedures.

At approximately two years of age, animals were surgically implanted with recording chambers and electrophysiological recordings commenced. Removal from the dark room for implantation surgery and all electrophysiological procedures was conducted only after animals were anesthetized and blindfolded, and the transport cage was masked.

Surgery.

Animals were anesthetized in the dark room with ketamine hydrochloride (20 mg/kg, i.m.) and acepromazine maleate (0.1 mg/kg, i.m.), then intubated and placed in a stereotaxic apparatus in the surgical suite. Surgical anesthesia was then induced with isoflurane (2–4%) and maintained with isoflurane (1.5–2%). During surgery, expiratory CO₂ (carbon dioxide), SpO₂ (peripheral oxygen saturation), blood pressure, and heart rate were continuously monitored using a digital vital signs monitor (VetSpecs VSM7), and body temperature was maintained at ∼37–38°C with a heating pad. A stainless steel recording chamber was placed over a craniotomy to provide access to the SC via the overlying cortex and attached to the skull with stainless steel screws and dental acrylic (McHaffie and Stein, 1983). After surgery, postsurgical analgesics (ketoprofen, 2 mg/kg, i.m.) were administered as needed, and antibiotics (ceftriaxone, 20 mg/kg, i.m.) were administered twice a day for 7 d.

Electrophysiological recording.

Weekly recording sessions began after a postsurgical recovery period of at least 7 d. The animal was anesthetized with ketamine hydrochloride (20 mg/kg, i.m.) and acepromazine maleate (0.1 mg/kg, i.m.), intubated, and ventilated mechanically. It rested comfortably in a recumbent position and its head orientation was fixed by two horizontal stainless steel bars attached to the recording chamber. Respiratory rate and volume were adjusted to keep the end-tidal CO₂ at ∼4.0%. Expiratory CO₂, SpO₂, heart rate, and blood pressure were monitored continuously to ensure the maintenance of anesthesia. Paralysis was induced with an injection of pancuronium bromide (0.1 mg/kg, i.v.) to prevent ocular drift. Anesthesia, paralysis, and hydration were maintained by continuous intravenous infusion of ketamine hydrochloride (5–10 mg kg⁻¹ h⁻¹), pancuronium bromide (0.05 mg kg⁻¹ h⁻¹), and 5% dextrose in sterile saline (2.4–3.6 ml/h). Body temperature was kept at ∼37–38°C. Conventional methods were used for single-neuron electrophysiological recording. A glass-coated tungsten electrode (tip diameter: 1–3 μm, impedance: 1–3 MΩ at 1 kHz) was positioned and lowered to the superficial layers of the SC. The electrode was advanced by a hydraulic microdrive to search for single neurons in the multisensory (i.e., deep) layers. Once a single neuron was identified, its responses (i.e., impulse activity) to the sensory stimuli were recorded, amplified, and routed to an oscilloscope, audio monitor, and computer for on-line and off-line analyses as in the past. The anesthesia and paralysis were terminated at the end of the recording session, and the animal was returned to its home cage once stable respiration and locomotion returned. Saline (40–50 ml) was given subcutaneously as needed.

Sensory-responsive neurons were identified using a variety of visual and auditory search stimuli. Visual search stimuli consisted of a moving bar of light or flash of light back-projected from an LC 4445 Philips projector onto a tangent screen located 45 cm from the front of the animal. Auditory stimuli consisted of broadband (20–20,000 Hz) noise bursts and tones delivered via one of 15 stationary speakers positioned around the animal at 15° intervals on a hoop whose axis of rotation was in line with the animal's interaural axis. Stimuli were controlled using custom software operating a NIDAQ digital controller (National Instruments) connected to a personal computer. When a visual–auditory neuron was isolated, its modality-specific visual and auditory receptive fields were mapped as in the past (Alvarado et al., 2008). Custom software acquired raw data waveforms and impulses from single neurons (after analog-to-digital conversion) identified using a threshold criterion of three times elevation of the action potential amplitude above background noise.

Once a neuron was identified as responding to more than one sensory modality and was properly isolated, its sensory responses were quantified using randomly interleaved modality-specific and cross-modal stimulus pairs at intertrial intervals of 5–7 s. Stimulus intensities were adjusted so that unisensory responses were weak in magnitude to better expose the impact of multisensory integration, as measured by multisensory enhancement, according to the “principle of inverse effectiveness” (Meredith and Stein, 1986). Multisensory enhancement has proved to be the most reliable index of multisensory integration as multisensory depression is found only in a subset of neurons exhibiting this capability (Kadunce et al., 1997). Some of the neurons that proved capable of multisensory integration were also tested with additional stimulus sets. In one stimulus set, stimuli having different physical properties (e.g., size, shape, orientation, frequency) were presented to assess whether there was response generalization across these feature domains. In two other sets the spatial or temporal properties of the stimuli were manipulated to determine the impact of the rearing condition on the spatial and temporal principles of multisensory integration.

Impulse times were recorded for each trial at 1 ms resolution, and analyzed off-line. The response window was defined using a geometric algorithm based on the cumulative impulse count described in earlier studies (Rowland and Stein, 2007, 2008). The mean spontaneous firing rate for each condition was calculated in the 500 ms window preceding the stimulus. The magnitude of each response was identified as the mean number of impulses occurring in the response window minus the expected number given the spontaneous firing rate. The mean response to the stimulus combination was then statistically compared with the response to the most effective single-modality component stimulus (Student's t test, p < 0.05). Multisensory enhancement (i.e., positive multisensory integration) was defined as a significant increase in the number of impulses to the combined stimuli compared with the most effective single-modality stimulus. The magnitude of multisensory response enhancement was represented by the multisensory integration index (MSI): MSI = [(CM − SM_max)/SM_max] × 100, where CM represents the mean magnitude of the multisensory response, and SM_max represents the magnitude of the response evoked by the more effective modality-specific stimulus (Meredith and Stein, 1983). The utility and underlying rationale of this measure of multisensory integration have been discussed in detail previously (Stanford and Stein, 2007; Stein et al., 2009b).

Results

A total of 320 visual–auditory multisensory neurons were studied in the present experiments: 140 from animals exposed to coincident cross-modal visual–auditory stimuli (the coincident exposure group) during early life, 94 from animals exposed to the component modality-specific stimuli that were randomly distributed in space and time (the random exposure group) during early life, and 86 from normal control animals.

Coincident versus random exposure

A substantial proportion (64%, 89/140) of the neurons in the coincident exposure group developed and retained multisensory integration capabilities as measured by their enhanced responses to cross-modal stimuli (see Materials and Methods). However, comparatively few (24%; 23/94) did so in the random exposure group, and those that did evidenced substantially less robust enhancement than neurons in the coincident exposure group. The responses of typical neurons from the coincident and random exposure groups are shown in Figure 2. In the examples from the coincident exposure group, the responses to the cross-modal stimulus were significantly larger than those to the most effective component stimulus, and in these particular cases far exceeded the sum of the unisensory responses as is often the result when weakly effective stimuli are combined. Such superadditivity is a major contributor to the phenomenon of inverse effectiveness (Meredith and Stein, 1986) and partly the manifestation of threshold nonlinearities at the low end of a neuron's dynamic range (Perrault et al., 2005; Stanford et al., 2005; Stanford and Stein, 2007). In contrast, as shown in the examples from the random exposure group, the magnitude of multisensory responses often did not significantly exceed those of the most robust unisensory responses.

Figure 2.

The development of multisensory integration capabilities was noted after cross-modal exposure, but not after random exposure. A, Coincident exposure produced the normal expression of multisensory integration as revealed by significant multisensory enhancement. Left, Receptive fields are delimited within schematics of visual–auditory space in which each circle represents 10° (black, visual receptive field; gray, auditory receptive field). Icons (V, A) show stimulus locations. Middle, Rasters (ordered bottom-top) illustrate responses to V, A, and V–A stimuli. Right, Summary bar graphs illustrate the mean responses elicited by each stimulus condition, as well as the resultant MSI, a measure of multisensory enhancement. Note that the multisensory response exceeded both unisensory responses and also exceeded their sum, yielding high MSIs (**p < 0.001). B, Random modality-specific exposure failed to produce multisensory integration capabilities in either of the exemplar neurons illustrated. Note the absence of significant multisensory enhancement in both cases, with MSIs between −20% and +22%. Error bars represent the SEM. Receptive fields are contralateral to the SC of the recorded neurons.

Both the incidence of significant multisensory enhancement (χ² test, df = 1, χ² = 32.911, p ≤ 0.001) and its mean magnitude (i.e., MSI) were significantly higher in the coincident than in the random exposure group (86.82 ± 7.35% vs 20.22 ± 4.79%, Mann–Whitney U Statistic = 2961.00; p ≤ 0.001) (Fig. 3), despite no significant variation in their best unisensory (comparator) response magnitudes (coincident group mean = 5.29 ± 0.33 impulses/trial; random exposure group mean = 5.73 ± 0.38 impulses/trial; Mann–Whitney U Statistic = 5947.50; p = 0.213). Although the incidence of multisensory enhancement in the coincident exposure group is somewhat lower than the normal control group (coincident exposure: 64% vs 78%, χ² test, df = 1, χ² = 4.47, p = 0.034), there was no significant difference in MSI between those groups (coincident exposure = 86.82 ± 7.35%; normal = 90.81 ± 7.59%; Mann–Whitney U Statistic = 5588; p = 0.366).

Figure 3.

The populations of neurons exhibiting multisensory integration following either of two exposure paradigms and normal control. A, MSIs are plotted for multisensory neurons from normally reared control animals. B, A high proportion of neurons in the coincident exposure group exhibited multisensory integration. Although fewer of them developed than in normal animals, their MSIs were within the normal range. C, However, few neurons in the random exposure group developed multisensory integration, and those that did had very low MSIs. These group differences are also evident in the summary bar graphs of D and E, which compare the incidence of multisensory integration and the mean MSI magnitude (black bar, normal; gray bar, coincident; white bar, random). *p < 0.05, **p < 0.001.

That the coincident exposure group contained such a high complement of neurons capable of integrating cross-modal cues (Meredith and Stein, 1986; Jiang et al., 2001; Alvarado et al., 2007) suggests that the brief daily periods of cross-modal exposure were sufficient to instantiate this capability (the cats were awake and alert during the exposure). That this capability was retained despite no additional visual–nonvisual experience for the subsequent 16 months also indicates that it does not require constant experiential updates for maintenance. Rather, it appears that once the capability for multisensory enhancement is acquired, it is retained in the absence of conflicting cross-modal experience.

It is interesting to note that a small but nontrivial proportion (i.e., 24%) of neurons in the random exposure group also developed the capacity for multisensory enhancement. However, this likely occurred because the random nature of the exposure to the auditory and visual stimuli provided some opportunities for the cross-modal stimuli to be in spatial concordance. Experience with visual–auditory pairings was limited to circumstances in which the visual stimulus was within the animal's range of sight, and there was a 3/12 (i.e., 25%) chance that an auditory stimulus would be within 30° of the visual stimulus (1/12) and an 8.3% chance of same location, regardless of their respective timing.

Generalization

Although the neurons capable of multisensory enhancement generally responded best to stimuli similar to those in the exposure arena (a small flashing spot and a broadband noise burst), they also showed similarly enhanced responses to combinations of stimuli having different stimulus properties. Although an extensive stimulus set was not used to examine the scope of this effect, its presence became evident in the neurons tested with several additional combinations of visual and auditory stimuli (n = 37). Each neuron was tested with the six visual–auditory combinations that were possible among three different visual (flashing light spots of two sizes, and a moving bar of light) and two different auditory (a broadband noise burst and a pure tone) stimuli. Despite apparent differences in the integrated multisensory response magnitudes that were elicited by the different cross-modal stimulus combinations, they were not statistically significant. Figure 4 shows the data acquired from a typical neuron. The responses to all six stimulus sets are displayed here, with each row illustrating responses to one set. In each case the multisensory responses far exceeded the unisensory responses to the component stimuli. Yet when one stimulus (either the visual or the auditory) differed from that presented in the exposure configuration, the magnitude of the multisensory response was slightly degraded. When both stimuli differed, the multisensory response was degraded even further, but in neither case did the degraded response differ significantly from the response to stimuli matching the exposure configuration.

Figure 4.

Neurons that acquired multisensory integration capabilities during the cross-modal exposure period generalized it to configurations that had not been experienced. Shown is an exemplar neuron. Responses to the exposure set are provided in the first row (shaded). The responses to six stimuli sets are displayed here, with each row representing one set. Note changes in the nature of the visual and auditory stimuli as indicated by the icons or labels. In each case, responses to the cross-modal stimuli were far greater than those to either modality-specific component stimuli, even when the test stimuli did not match those presented during the exposure period. All conventions are the same as in Figure 2.

The population data are summarized in Figure 5, which shows the mean MSI for each neuron's responses to the exposure combination and each of the other combinations. As was the case in the individual example provided in Figure 4, when the cross-modal combination tested was stepped away from the exposure combination so that one or two of the cross-modal stimulus components differed, there was a stepwise decline in the mean MSI. Nevertheless, the distribution of points is quite similar across the different combinations, and responses to the exposure combination were not significantly different from any of the other nonexposure combinations (Mann–Whitney rank sum test, p > 0.05 in all comparisons, Fig. 5B).

Figure 5.

Population data revealing that the magnitude of multisensory integration did not differ between exposure and nonexposure stimulus configurations. A, Plotted is the MSI as a function of the best unisensory responses for each of six stimuli sets, respectively, from all sampled neurons. Tests of the exposure configuration are in the top left. Note that most neurons were able to integrate all six stimuli sets. B, The mean MSI of the exposure combination (for the population of neurons studied) was not significantly different from any of the other nonexposure combinations, although there was a stepwise decline in the mean MSI when the cross-modal combination tested was stepped away (see text) from the exposure combination (C). Note that there are also no significant differences in the best unisensory responses among the groups (D).

As shown in Figure 5D, the relatively constant MSI was accompanied by a relative constancy in the largest unisensory component response. This reflects the adjustment of stimuli to maintain the relative strength of the unisensory responses across the population of neurons studied to maximize their potential for generating multisensory response enhancement (see Materials and Methods). It also avoids contaminating comparisons via the effects of inverse effectiveness, wherein higher MSIs are produced by combinations of weakly effective cross-modal stimuli than by combinations of strongly effective cross-modal stimuli.

The principles governing multisensory integration

Despite the simplicity and specificity of the cross-modal coincident exposure condition, neurons acquiring the capacity for multisensory integration were governed by the same rules that operate in normal animals. Spatially coincident cross-modal stimuli were necessary for multisensory enhancement, and spatially disparate stimuli resulted in either no multisensory enhancement or response depression. This spatial principle of multisensory integration (Meredith and Stein, 1986) was evident in the neuron illustrated in Figure 6. The neuron exhibited significant multisensory response enhancement when each of the cross-modal component stimuli was located within its overlapping modality-specific receptive field (VA1), but when one of those stimuli was outside its receptive field and spatially disparate from the other component stimulus, multisensory response depression resulted (VA2). Within the sample tested, the majority of the neurons (75%, 18/24) showed this enhancement/depression result with spatial coincidence/disparity, whereas the remainder showed enhancement/no significant change. In addition, the temporal principle of multisensory integration was evident, so that a neuron's optimal MSI was elicited when the onsets of its cross-modal inputs were set so that there was overlap in the individual discharge trains they elicited (but see below).

Figure 6.

The spatial principle of multisensory integration was acquired despite exposure to only spatially concordant cross-modal stimuli. During the cross-modal exposure period the stimuli were always in spatial concordance. However, both multisensory enhancement (to concordant stimuli, A1), and multisensory depression (to discordant stimuli, A2) capabilities developed in this exemplar neuron. Conventions are the same as in previous figures.

The principle of inverse effectiveness was evident when stimulus effectiveness was manipulated, as shown in an example neuron in Figure 7. Although the multisensory response of this neuron was significantly higher than the best unisensory response in each condition, the proportionate magnitude of that difference was inversely related to the robustness of its unisensory comparator responses. Thus, cross-modal combinations of poorly effective component stimuli led to proportionately greater multisensory enhancement than combinations of highly effective stimuli.

Figure 7.

The principle of inverse effectiveness was acquired despite exposure being confined to only one invariant pair of cross-modal stimuli. Shown are the responses of this neuron to three relative intensities (low, medium, high) of modality-specific and cross-modal stimuli. As stimulus effectiveness (i.e., intensity) increased, so did unisensory and multisensory responses. However, the former increased proportionately more than did the latter, thereby decreasing MSI. Conventions are the same as in prior figures.

Despite the fact that the normal overarching principles of multisensory integration were evident in these neurons following cross-modal exposure, a more detailed analysis showed that some of their properties were not typical of normal animals. Rather, they represented variants in the normal relationships between the cross-modal component stimuli that appeared to derive from the specifics of the exposure condition. These exposure conditions in the coincident exposure group were such that the cross-modal stimuli were presented with no variation in their relative location or timing. They always occurred at the same location in space and their onsets were simultaneous. To determine whether this invariance had any impact on the spatial and temporal cross-modal relationships that produced the highest MSIs, these relationships were systematically varied during testing.

In normal animals, the spatial principle depends on the stimuli being present in their respective receptive fields, but within this constraint there is no systematic relationship between their absolute proximity and the magnitude of the enhanced response they evoke (Kadunce et al., 2001). In short, there is no reason to expect that two within-field cross-modal stimuli will produce reliably different proportionate changes as a function of their relative disparity. However, in the present sample tested (n = 21), a proximity effect was evident following cross-modal exposure, and the results from one characteristic example are provided in Figure 8A. In this test condition the visual stimulus was always presented at the same location in the neuron's visual receptive field, whereas the auditory stimulus was presented at either of two within-field positions: one overlapping (VA1) and one disparate (VA2) from the visual. Multisensory enhancement was evident in both configurations, but was significantly greater when the cross-modal stimuli overlapped as they did in the exposure condition. In the matching condition, the multisensory response far exceeded the sum of the unisensory responses. In contrast, the non-overlapping (nonmatching) combination evoked a response that was less than the predicted sum. This occurred despite near equivalent unisensory responses in the different conditions. The population data show that this was the typical result, and the data are summarized in Figure 8C. The MSI was significantly higher when the visual and auditory stimuli had disparities <15° (128.64 ± 14.84%) than when their disparities were >45° (68.08 ± 9.11%) (paired t test, p ≤ 0.001). In each case this occurred despite the presence of near-equivalent best modality-specific (i.e., auditory) responses, and near-identical predicted sums of responses to the individual component stimuli.

Figure 8.

A spatial “proximity effect” was induced by the cross-modal exposure experience. A, This exemplar neuron exhibited significantly greater multisensory responses to “aligned” cross-modal stimuli that matched the exposure configuration (VA1) than to those that were nonaligned (VA2) despite still being within their respective receptive fields. B, Whereas the former configuration produced responses that were larger than the predicted sum of the V and A responses (i.e., superadditive), the latter produced only additive responses. C, This was generally the case in the population of neurons studied. Results from this population are shown by comparing average MSIs in aligned and unaligned conditions. Conventions are the same as in prior figures.

In normal control animals the temporal offset between the cross-modal stimuli (i.e., the stimulus onset asynchrony, or SOA) that elicits the largest MSI is usually 50–100 ms [light (V) before sound (A)] (Meredith et al., 1987). A typical example is shown in Figure 9A. In the present experiment, the influence of exposure with only temporally aligned cross-modal stimuli yielded a different optimum SOA as shown in the exemplar neuron of Figure 9D. For this neuron the largest MSI occurred for an SOA of 0 ms and fell off rapidly as SOAs shifted away from simultaneity when either the visual or auditory stimulus led. These differences were also evident at the level of the population, with the exposure group having a significantly different best SOA than normal, one far closer to simultaneity and thereby matching the conditions of the exposure paradigm (Mann–Whitney rank sum test, Mann–Whitney U statistic = 211.500, p ≤ 0.001; normal: n = 30, mean value of best SOA = 79.0 ± 7.76 ms; coincident exposure group: n = 32, mean value of best SOA = 31.6 ± 7.96 ms) (Fig. 9E).

Figure 9.

A temporal “proximity effect” was induced by the cross-modal exposure experience. A–B, Shown are exemplar neurons from animals in the normal control group and from those in the coincident cross-modal exposure group. At the top are shown rasters of their unisensory visual and auditory responses. Just below are rasters showing their responses to stimuli at different SOAs (stimulus onset asynchronies). C–D, The MSI of each neuron in each group is plotted as a function of SOA, fitted with a Gaussian function. A200 V = A preceding V by 200 ms, and V = A represents simultaneity. Note that the optimal SOA for neurons in normal animals is close to 100 ms, whereas for neurons in the coincident cross-modal exposure group the optimal is near simultaneity. E, The distribution of best SOAs (predicted by the peak positions of the Gaussian fit), bin width = 15 ms. Note that in coincident exposure animals the best SOAs are to the left of those from normal animals. Conventions are the same as in prior figures.

Discussion

The ability to integrate signals across multiple sensory channels is a powerful strategy used by the SC to facilitate the detection and localization of environmental events, but it is not an innate feature of its constituent neurons. Although multisensory neurons appear shortly after birth, they are incapable of integrating their multiple sensory inputs in a synergistic way without the acquisition of the requisite sensory experience (Stein et al., 2009a). As shown here, mere exposure to individual stimuli from multiple sensory modalities is insufficient to spur the development of multisensory integration capabilities in these neurons. Rather, the system requires experience with their cross-modal combinations, presumably to adapt to their statistics. The majority of the multisensory SC neurons in these young animals repeatedly exposed to spatiotemporally concordant cross-modal stimuli grew to integrate signals having this configuration. This occurred despite the innocuous nature of the stimuli and the absence of any apparent behavioral or reward contingencies associated with them (see Materials and Methods). It also appeared to be stable.

These results are consistent with those obtained recently in animals that were reared to maturity in the dark, then anesthetized and repeatedly exposed to spatiotemporally concordant cross-modal stimuli (Yu et al., 2010). The initial inability of their multisensory SC neurons to integrate cross-modal inputs was rapidly replaced by seemingly normal abilities to engage in this process.

Together, these findings suggest that the experience-dependent maturation of the neural circuits that instantiated the spatial and temporal principles of multisensory integration are neither substantially debilitated nor facilitated by anesthesia, by engagement in goal-directed behavior, or by the presence or absence of reward; nor are they compromised by extended periods in which no cross-modal experience is available. This is not meant to imply that all the characteristics of this process are normal in animals raised in these stimulus-impoverished circumstances. Indeed, several anomalies in this process were noted (see below). Nevertheless, acquiring the fundamental multisensory integration capabilities appears to require little more than consistent relationships between the modality-specific components of the cross-modal experience: joint probabilities of occurrence in space and time that are re-encoded into filters that result in multisensory enhancement and depression when cross-modal stimuli are encountered later in life. Accordingly, this cannot be achieved by experience with the visual and auditory component stimuli that lack the requisite probabilistic cross-modal relationships.

Once acquired, multisensory integration capabilities were not limited to the specific features of the exposure configuration that induced them. Although bound by the spatial and temporal features of these stimuli (see below), cross-modal test stimuli having substantially different physical features from those in the exposure configuration yielded similar magnitudes of multisensory enhancement. Any marginal advantage of the cross-modal exposure configuration over the other cross-modal stimulus configurations did not achieve statistical significance. However, the extent of this generalization phenomenon is not yet known as an extensive parametric manipulation of stimulus properties was not conducted here.

The results of the present experiments also revealed that once multisensory integration capabilities (and the principles guiding them) were acquired, they were maintained without further experience for more than one year, and may have been maintained for the life of the animal. Although there is no way of knowing whether some neurons had acquired and then lost this capacity, and whether others had their integrative product degraded during the intervening period of no visual–auditory experience, there is little reason to suppose this to have been the case. The incidence of multisensory integration was high, and was only slightly lower than the range (74–93%) reported in normally reared animals (Meredith and Stein, 1986; Jiang et al., 2001; Perrault et al., 2003; Alvarado et al., 2007). In addition, the relative magnitude of the integrated responses appeared comparable to that of normally reared animals. A more parsimonious explanation of the results is that once acquired, there is no passive decay of multisensory integration capabilities or the principles that govern them. Presumably, however, these principles can be altered by cross-modal experiences that conflict with those originally used to instantiate them, but this supposition remains to be examined empirically.

Consistent with the idea that the statistics of the cross-modal event was a driving factor in the maturation of the circuitry underlying SC multisensory integration are the two anomalies in this process that became evident here. The specifics of the spatial and the temporal preferences for these SC neurons were not entirely consonant with what has been found in normal animals. In the spatial domain, these neurons exhibited a clear “proximity effect,” wherein there was a systematic relationship between the spatial concordance of the stimuli and the magnitude of the multisensory response: those cross-modal stimuli that were closer to one another elicited significantly larger multisensory responses than those that were further away, even though the latter were still within their respective receptive fields. In normal animals, stimuli within their respective overlapping receptive fields exhibit no systematic relationship between stimulus proximity and response magnitude (Kadunce et al., 2001). In the temporal domain, neurons in these animals preferred a 0 ms SOA, whereas the preferred SOAs in normal animals are more variable, with a majority preferring 50–150 ms offsets (V before A) (Meredith et al., 1987).

These deviations from “normal” make intuitive sense if one considers the statistics of cross-modal events to be a driving force underlying the maturation of multisensory integration capabilities. Under normal conditions the same events can have substantial spatial and temporal variability in the cross-modal cues they provide. The relative timing of the component stimuli and their relative locations vary with distance and angle and variations in the conduction and reflective properties of the medium through which they move (e.g., air, water, glass). All of these factors contribute variability to the cross-modal experience, and it is likely they all are relevant to how the neural circuit expresses multisensory integration and how individual neurons become specialized. In general, normal experiences should produce neurons having more “tolerance” for spatial and temporal disparity, as well as neurons having different preferences. In contrast, the present exposure condition provided no variability in the spatial or temporal properties of the cross-modal stimuli, and yielded little variability in the resulting neural product. Based on this hypothesis, lack of tolerance for cross-modal disparity would be a reasonable expectation. So, too, is the expectation that raising animals with spatially disparate visual–auditory stimuli would lead to the requirement of such disparity to engage in multisensory integration, a result that has empirical validation (Wallace and Stein, 2007).

Adaptation and plasticity in neural processes have both benefits and costs: a highly plastic system can rapidly accommodate to changing environmental circumstances, yet it is also sensitive to perturbations based on noise. Given that multisensory SC neurons maintain some cross-modal plasticity during adulthood (Yu et al., 2009, 2010), it is of considerable interest to determine how these neurons use experience to calculate the relationships among cross-modal stimuli to maintain or change their requirements for integration. This is also of considerable interest for enhancing rehabilitative strategies directed toward overcoming early deficits in multisensory integration. Whether extensive experience in the normal environment would enable SC neurons in animals reared with random visual and auditory stimuli to acquire multisensory integration capabilities, and even correct the spatial and temporal “anomalies” that developed in animals in the coincident exposure group, is open to question. However, it would provide a metric of the ongoing plasticity of multisensory integration.

Human patients whose early sensory problems (e.g., congenital cataracts) are corrected early in life appear never to fully overcome perceptual deficits in multisensory integration despite living for many years in a normal environment (Putzar et al., 2007). Similarly, SC neurons in cats whose association cortices have been deactivated for a brief period during early development and then are placed in a normal environment have long-lasting deficiencies in multisensory integration (Rowland et al., 2005). The deficits in both cases are likely due to the adult brain being less able to deal with the cross-modal ambiguities inherent in natural cross-modal events. Perhaps training with more invariant cross-modal stimuli (Yu et al., 2010) is a way to best tap the adult plasticity in this system and thereby enhance the effectiveness of rehabilitative strategies. This multisensory training strategy has proved to be effective in sensory rehabilitation after large cortical lesions in both experimental animals (Jiang et al., 2008; McHaffie et al., 2008, 2009, 2010) and in human patients (Bolognini et al., 2005).