Initiating the Development of Multisensory Integration by Manipulating Sensory Experience

Abstract

The multisensory integration capabilities of superior colliculus neurons emerge gradually during early postnatal life as a consequence of experience with cross-modal stimuli. Without such experience neurons become responsive to multiple sensory modalities but are unable to integrate their inputs. The present study demonstrates that neurons retain sensitivity to cross-modal experience well past the normal developmental period for acquiring multisensory integration capabilities. Experience surprisingly late in life was found to rapidly initiate the development of multisensory integration, even more rapidly than expected based on its normal developmental time course. Furthermore, the requisite experience was acquired by the anesthetized brain and in the absence of any of the stimulus–response contingencies generally associated with learning. The key experiential factor was repeated exposure to the relevant stimuli, and this required that the multiple receptive fields of a multisensory neuron encompassed the cross-modal exposure site. Simple exposure to the individual components of a cross-modal stimulus was ineffective in this regard. Furthermore, once a neuron acquired multisensory integration capabilities at the exposure site, it generalized this experience to other locations, albeit with lowered effectiveness. These observations suggest that the prolonged period during which multisensory integration normally appears is due to developmental factors in neural circuitry in addition to those required for incorporating the statistics of cross-modal events; that neurons learn a multisensory principle based on the specifics of experience and can then apply it to other stimulus conditions; and that the incorporation of this multisensory information does not depend on an alert brain.

Introduction

The capability to integrate information from multiple senses is a characteristic feature of superior colliculus (SC) neurons. It is through this process that the spatiotemporally concordant cross-modal stimuli that are derived from the same event produce markedly enhanced multisensory responses and an increased likelihood of SC-mediated orientation responses to it. This phenomenon is referred to as “multisensory enhancement,” which is a specific case of the more general phenomenon of multisensory integration (Stein and Stanford, 2008).

This ability to integrate information from different senses is not restricted to any given species or brain structure. Rather, it is found in all species that have been examined and in neurons at many locations within their nervous systems (Stein and Meredith, 1993; Zahar et al., 2009). The ubiquity of multisensory integration speaks to its broad adaptive significance for a variety of functions in a host of organisms that face very different ecological challenges. Nevertheless, and despite the substantial survival challenges faced by the newborn, extant data suggest that this capability appears comparatively late in postnatal development. Thus, for example, SC neurons in the newborn are incapable of multisensory integration because they have not yet become responsive to multiple sensory inputs (Stein et al., 1973) or because they cannot yet synthesize those cross-modal inputs (Wallace and Stein, 1997, 2001).

It is only after considerable experience during early life with the statistics of cross-modal events that the brain is able to craft the spatial, temporal, and other principles that are characteristic of multisensory integration (Wallace and Stein, 2007). When, for example, visual–auditory experience is precluded by rearing cats in the dark, their SC neurons do not express their signature ability to integrate visual–nonvisual inputs (Wallace et al., 2004). Normally, this capacity appears within the first few months of postnatal life, a time during which rapid changes in neural architecture are taking place in many interrelated brain structures (Katz, 1999; Hensch, 2004). It is often referred to as a “sensitive” period, because the absence of relevant experience during this time is difficult or impossible to compensate for later in life, and many normal functions are delayed or precluded. For example, if not corrected soon enough, early blindness or deafness can have long-standing detrimental effects on unisensory and/or multisensory functions (Hubel, 1978; Anooshian and Bryan, 1979; Carlson et al., 1987; Welsh et al., 1996; Putzar et al., 2007).

The present studies were initiated to determine whether the cross-modal sensory experience required for instantiating multisensory integration in SC neurons could be acquired at maturity, far later in life than is normally the case. The results reveal that the adult brain is surprisingly efficient in incorporating these experiences and in developing this capacity, and that these processes do not require an alert brain, an associated reward, or any of the stimulus–response contingencies normally associated with learning.

Materials and Methods

All protocols were used in accordance with the Guide for the Care and Use of Laboratory Animals (National Institutes of Health Publication 86-23) and were approved by the Animal Care and Use Committee of Wake Forest University School of Medicine, an Association for Assessment and Accreditation for Laboratory Animal Care International-accredited institution.

Housing environment and surgical preparation.

Five cats were deprived of visual and visual–nonvisual experience by rearing them in a light-tight housing environment (Wallace et al., 2004). Auditory experience, and all other sensory experience, was not constrained. Daily care was accomplished with the aid of binocular infrared goggles, and an infrared closed-circuit television system allowed animals to be observed in their home environment from an adjacent room. Surgical implantation, routine veterinary procedures, and transport to the electrophysiological recording room within a tightly masked carrier were conducted only after animals were anesthetized and either blindfolded or had their eyelids taped shut with paper tape in the dark room. When the animals matured (7–12 months), they were implanted with stainless-steel recording chambers using previously described techniques (McHaffie and Stein, 1983). Briefly, each animal was first anesthetized with a mixture of ketamine hydrochloride (20 mg/kg, i.m.) and acepromazine maleate (0.4 mg/kg, i.m.), fitted with opaque contact lenses or the eyelid tape, intubated through the mouth, and then placed in a stereotaxic apparatus. Surgical anesthesia was maintained with isoflurane (1.5–3%). During surgery, expiratory CO₂, blood pressure, and heart rate were continuously monitored using a digital vital signs monitor (VetSpecs VSM7), and body temperature was maintained at ∼37–38°C with a heating pad. A recording chamber was stereotaxically placed over a craniotomy to provide access to the SC via the overlying cortex and attached to the skull with stainless steel screws and dental acrylic. After surgery, postsurgical analgesics (butorphanol tartrate, 0.1–0.2 mg/kg; ketoprofen, 1 mg/kg) were administered as needed, and antibiotics (cefazolin sodium, 25 mg/kg) were administered twice a day for 7 d.

Electrophysiological recording procedures.

Weekly recording sessions began after a postsurgical recovery period of at least 7 d. In the very first experiments, multisensory neurons were isolated and tested with combinations of visual, auditory, and visual–auditory stimuli to assess the multisensory integration capabilities of these neurons in their “naive” state. However, each subsequent recording session began with an “exposure” period followed by a “testing” period (see below). For each session, the animal was anesthetized with a combination of ketamine hydrochloride (20 mg/kg, i.m.) and acepromazine maleate (0.4 mg/kg, i.m.), intubated, and ventilated mechanically. The animal rested comfortably in a recumbent position while its head was fixed on a stereotaxic apparatus through two horizontal stainless steel bars that were inserted into the openings in the recording chamber. Respiratory rate and volume were modulated to keep the end-tidal CO₂ at ∼4.0%. CO₂, heart rate, and blood pressure were monitored continuously to ensure the maintenance of a continued state of clinical anesthesia. Paralysis was induced with an injection of pancuronium bromide (0.06 mg/kg, i.v.) to prevent ocular drift. The pupil of the right eye was dilated with ophthalmic atropine sulfate (1%) and fitted with a contact lens to correct for refraction. The left eye was occluded with an opaque contact lens. The optic disc of the right eye was mapped using an ophthalmoscope and reverse projection onto a tangent screen. Anesthesia, paralysis, and hydration were maintained by continuous intravenous infusion of ketamine hydrochloride (4–6 mg kg⁻¹ h⁻¹), pancuronium bromide (0.15–0.2 mg kg⁻¹ h⁻¹), and 5% dextrose in sterile physiological saline (3–6 ml/h). Body temperature was kept at 37∼38°C. Conventional methods were used for single-neuron electrophysiological recording. A glass-coated tungsten electrode (tip diameter, 1–3 μm; impedance, 1–3 MΩ at 1 kHz) was positioned and lowered to the superficial layers of the SC via a hydraulic microdrive. The daily exposure trials were then conducted (see below), after which the electrode was advanced to search for single neurons in the multisensory (i.e., deep) layers. Once identified, single units were presented with test stimuli (see below). Single-unit neural activity was recorded, amplified, and routed to an oscilloscope, audio monitor, and computer for on-line and off-line analyses as in the past (see below). At the end of the recording session, the anesthesia and paralysis were terminated. The eyes were covered, and the animal was returned to its home cage in the dark room once stable respiration and locomotion were reinstated.

Stimulus exposure procedure.

At the beginning of each recording day, each animal was exposed to repeated presentations of sensory stimuli (1800/d). Different animals were exposed to different stimulus configurations. In two animals, a pair of spatiotemporally congruent visual and auditory stimuli (i.e., cross-modal stimuli) were presented unilaterally in either macular (0°, cat 1) or contralateral extramacular (30°, cat 2) visual space, as shown in Figure 1, at a 6 s interstimulus interval (3 h/d). Two other animals were exposed to these same visual and auditory component stimuli, in these same spatial locations, and for the same number of iterations, but these modality-specific stimuli were presented independently in an interleaved manner (cat 3 at 0°, cat 4 at 30°) with a 4 s interstimulus interval (4 h/d). A fifth animal (cat 5) was also exposed to these modality-specific stimuli, but in random order and at random locations in space. The auditory stimulus consisted of a brief (100 ms) 65 dB burst of broadband noise (20–20,000 Hz) against the ambient background noise of 51.2–52.0 dB. The visual stimulus was a moving bar (width × height, 10° × 2°) of light (13.67 cd/m² on a background luminance of 0.16 cd/m²), with a speed of 100°/s and a duration of 100 ms.

Figure 1.

The two experimental exposure sites included one macular (left) and one extramacular (right). During exposure trials, four animals experienced the same visual and auditory stimuli in one of two possible stimulus configurations and at one of the two exposure sites (macular, 0°; extramacular, 30°) shown on the schematics above. The schematics represent the central 60° of visual–auditory space, and each concentric circle represents 10°. The stimulus configurations included a cross-modal (visual–auditory) and a modality-specific (interleaved visual and auditory) paradigm. The visual stimulus was a moving bar of light, represented by the icon and arrow, and the auditory stimulus was a broadband noise burst, represented by the star. One additional animal received random presentation of the modality-specific stimuli at random locations (see Materials and Methods for details).

Search strategy and receptive field mapping.

Sensory-responsive neurons were identified using a variety of visual and auditory search stimuli. Visual search stimuli consisted of moving bars of light back-projected from an LC 4445 Philips projector onto a tangent screen located 45 cm from the front of the animal. Auditory stimuli consisted of broadband (20–20,000 Hz) noise bursts, hisses, clicks, and taps. Stimuli were controlled using custom software operating a NIDAQ digital controller (National Instruments) connected to a personal computer. When a visual–auditory neuron was isolated, its modality-specific visual and auditory receptive fields were mapped as in the past (Alvarado et al., 2008). The visual receptive field was mapped with moving light bars, and the auditory receptive field was mapped with broadband noise bursts from any of 16 hoop-mounted speakers placed 15° apart and 15 cm from the head on a rotating hoop that permitted adjustments in elevation. Custom software acquired raw data waveforms and impulses from single neurons (after analog-to-digital conversion) identified using a threshold criterion of three times elevation of the action potential amplitude above background noise.

Testing procedure.

Once a multisensory neuron was identified, a short series (15–20 trials) of pretests was conducted using the visual, auditory, and spatiotemporally concordant visual–auditory stimuli identified above. The purpose of these tests was to choose stimuli that would maximize the likelihood of exposing the multisensory integration capabilities of the neuron under study. Thus, individual modality-specific component stimuli were chosen to be weakly effective for the particular neuron under study, and were presented in spatial congruence (Stanford et al., 2005) within the most responsive (i.e., “best”) areas of its visual and auditory receptive fields (Kadunce et al., 2001). These regions were identified whenever apparent and identified as the receptive fields in subsequent text. Last, although many naive neurons preferred simultaneous visual and auditory stimuli (Wallace and Stein, 1997), neurons in normal adults often preferred an offset between these two stimuli (Meredith et al., 1987). In all the neurons studied, these pretest trials were used to determine the best stimulus offset asynchrony to be used in subsequent tests of multisensory integration.

Data acquisition and analysis.

Impulse times (1 ms resolution) were recorded for each trial and analyzed off-line. The response window was defined using the algorithm identified in earlier studies (Rowland and Stein, 2007, 2008). The mean spontaneous firing rate for each condition was calculated in the 500 ms window preceding the stimulus. The magnitude of each response was identified as the mean number of impulses occurring in the response window minus the expected number given the spontaneous firing rate. The response to the combined cross-modal stimuli was compared with the response evoked by the more effective modality-specific stimulus to create the multisensory integration index (MSI): MSI = [(CM – SM_max)/SM_max] × 100%, where CM represents the mean magnitude of multisensory response, and SM_max represents the magnitude of the response evoked by the more effective modality-specific stimulus. Data were compared statistically to determine significant differences using SPSS 11.5, and t tests, Mann–Whitney U tests, ANOVA, and χ² tests where appropriate. All of the data were expressed as mean ± SD and the minimum criterion for statistical significance was p < 0.05.

Results

A total of 434 multisensory (visual–auditory) neurons were studied in the present experiments. Of these, 26 neurons were identified as naive and were studied in initial experiments (i.e., before the experiments containing exposure trials) to provide a baseline for comparison. These naive neurons had response properties very similar to those previously reported in earlier dark-rearing studies (Wallace et al., 2004). Their receptive fields were large, and thus the visual and auditory maps they formed likely had lower than normal spatial resolution. Although the neurons were responsive to visual and auditory stimuli, they were incapable of integrating spatiotemporally concordant visual–auditory stimuli to produce the enhanced multisensory response that characterizes their counterparts in the SC of animals raised in normal environments (Meredith and Stein, 1986; Jiang et al., 2001; Kadunce et al., 2001; Alvarado et al., 2007b). The unisensory and multisensory responses of a typical naive multisensory neuron from a dark-reared animal and a typical multisensory neuron from a normally reared animal are shown in Figure 2.

Figure 2.

Multisensory integration did not develop in the neurons of dark-reared animals in the absence of cross-modal experience. A, B, The receptive fields and responses of a typical neuron from a dark-reared animal, recorded before cross-modal exposure trials (A), are shown above those of a typical neuron from an animal reared in normal illumination (B). Left, Shown are the neuron's visual (black) and auditory (gray) receptive fields (shading) as well as the stimuli presented. Middle, Ramps and square waves represent the visual (V), auditory (A), and visual–auditory (VA) stimuli. The cross-modal and the modality-specific stimuli were randomly interleaved during testing. Responses were resorted by stimulus condition and displayed in rasters below the traces. Each dot in a raster represents one impulse, and each row represents the response to one stimulus presentation. Trials are ordered from bottom to top, and all trials, even those without responses, are included. Right, Summary bar graphs show the mean response (number of impulses) for each stimulus condition in the rasters. Numbers above the bars represent the percentage multisensory response enhancement (MSI). Note that the neuron from the naive animal shows no significant difference between its response to the cross-modal stimuli and its response to the most effective component stimulus (V). However, marked multisensory enhancement is evident in the response of the animal reared in an illuminated environment. **p < 0.001. Conventions are the same as in Figure 1.

Following these initial experiments, animals were exposed to the controlled repeated presentation of different sensory stimuli in the experimental room (i.e., “exposure trials”) at the beginning of each recording session. It is important to note that the animals were otherwise kept in their dark housing facility so that the experimental circumstance was their sole source of experience with cross-modal stimuli, and that they could accumulate exposure trials over multiple experimental sessions.

A total of 408 multisensory neurons were studied in this manner: 186 in animals provided spatiotemporally coincident cross-modal exposure trials (cat 1, n = 98; cat 2, n = 88); 147 in animals given interleaved modality-specific exposure trials (cat 3, n = 75; cat 4, n = 72); and 75 in the animal (cat 5) exposed to randomly interleaved modality-specific stimuli at random locations. In the first experimental sessions containing exposure trials, neuronal response properties closely resembled those of the naive population studied before the exposure trials; that is, they responded to both visual and auditory stimuli but did not produce enhanced responses to these stimuli when presented together in spatiotemporal register. However, over the course of multiple experimental sessions in which the animals were provided repeated exposure to these spatiotemporally concordant cross-modal stimuli (i.e., cats 1 and 2), there was a striking increase in the probability of identifying neurons capable of multisensory integration.

The visual, auditory, and multisensory responses recorded from three different neurons after different numbers of cross-modal exposure trials are shown in Figure 3. The first two neurons were recorded from the animal with cross-modal exposure in extramacular space, the third from the animal with exposure in macular space. In each case, the visual and auditory receptive fields of each neuron encompassed the exposure site. This was the first neuron identified that was capable of multisensory integration. It was encountered after only 3600 cross-modal exposure trials, which approximately equated to 6 h of experience with controlled cross-modal stimuli. Although the neuron's multisensory response was comparatively modest, its magnitude exceeded that of its response to the most effective unisensory component stimulus (visual) by a significant margin. This was not apparent in any of the previously identified neurons. The neurons depicted in Figure 3, B and C, were encountered, respectively, after 37,800 and 50,400 exposure trials. In these cases, their multisensory responses were comparatively larger and exceeded the sum of their unisensory component responses. Such superadditive responses are characteristic of multisensory responses to weak modality-specific stimuli (Stanford et al., 2005), but also appeared to reflect the additional cross-modal experience they obtained (see below). It should be noted, however, that given the nature of the sampling procedure it was possible to determine only that a given neuron required fewer than the number of exposure trials received at the time of recording.

Figure 3.

Exposure of dark-reared animals to repeated presentations of cross-modal stimuli led to the development of multisensory integration. A–C, Three exemplar neurons are shown. Left, Each neuron's receptive fields and stimulus positions are shown in schematics using the conventions provided in Figures 1 and 2. Below each schematic is the number of cross-modal exposure trials presented before recording from the neuron, and the exposure site (0° or 30°). The exposure site is shown on each schematic as a transparent gray square. Note that the receptive fields of each neuron encompassed the exposure site. Middle, In each case responses to the cross-modal stimuli were far greater than those to either modality-specific component stimulus. Right, Summary bar graphs show that these neurons with cross-modal experience typically had a multisensory response that was significantly more robust than the response to the most effective component stimulus. All conventions are the same as in Figure 2. *p < 0.05, **p < 0.001.

Repeated exposure to equivalent amounts of randomly interleaved modality-specific visual and auditory stimuli in the same locations did not have the same effect and did not yield a substantial proportion of neurons with the capacity for multisensory integration. Instead, the multisensory response properties of neurons in these animals closely resembled those recorded from neurons before the exposure trials. Figure 4 provides three example neurons recorded after different numbers of modality-specific exposure trials, and unlike the neurons shown in Figure 3, none show enhanced responses to spatiotemporally concordant stimuli presented within its receptive fields. Presumably, the absence of temporal concordance eliminated the link between the cross-modal stimuli, and thus did not provide the essential cross-modal experience necessary for the development of this capacity.

Figure 4.

Interleaved modality-specific (visual, auditory) exposure did not initiate the development of multisensory integration. Three exemplar neurons are provided after low, intermediate, and high levels of stimulus exposure. In each case, the modality-specific visual and auditory components of the cross-modal stimulus pair were presented independently and repeatedly in an interleaved sequence at 0°. The physical properties of the stimuli and their iterative rates were equivalent to those used in the cross-modal exposure trials shown in Figures 2 and 3, and each neuron's visual and auditory receptive fields encompassed the exposure site. Unlike exposure to the cross-modal stimulus combination, exposure to its independent modality-specific components failed to initiate the development of a neuron's ability to integrate cross-modal inputs and produce multisensory enhancement. All conventions are the same as in Figure 3.

As shown in Figure 5, the increases in multisensory response magnitude with exposure were not due to a concomitant decrease in the comparator unisensory responses (the response to the most effective of the component stimuli). This is an important factor as it eliminates the possibility that smaller comparator responses could have inadvertently been used in later samples. Had this happened, proportionately larger multisensory responses could have been obtained, not as a consequence of increased cross-modal exposure, but because of the principle of inverse effectiveness (Stein et al., 2009). The relative constancy of the unisensory comparator responses was due to the fact that stimulus efficacies were adjusted for each neuron to maximize its potential for generating an enhanced multisensory response (see Materials and Methods). However, despite the constancy of the unisensory comparator response magnitudes, there were obvious increases in the multisensory response magnitudes after repeated exposure to spatiotemporally concordant cross-modal stimuli (Fig. 5A,B). Similar increases were not seen in neurons recorded from animals that were given modality-specific exposure (Fig. 5C–E). Their multisensory and comparator unisensory responses were equivalent regardless of the amount of sensory exposure.

Figure 5.

The proportionate vigor of multisensory responses gradually increased with increasing cross-modal exposure. A, B, Data are shown for multisensory neurons from the animals dark reared and then exposed to coincident cross-modal stimuli in either macular (0°) (A) or extramacular (30°) (B) space. Note that the unisensory comparator responses (open circles) were not appreciably different as a function of exposure, but there was a gradual increase in the magnitude of their multisensory responses (filled circles) regardless of the site of cross-modal exposure. The solid and dotted lines respectively denote the lines of regression for multisensory responses and unisensory responses. C–E, Dark-reared littermates were exposed to equivalent numbers of the component visual and auditory stimuli in an interleaved sequence in macular (left) or extramacular (middle) space, or to a random sequence of these stimuli at random locations (right). Note that there was very little change in the proportionate magnitude of their multisensory responses. The receptive fields of the neurons studied in A–D encompassed the exposure site, and all neurons encountered were sampled in E.

After only hours of experience with spatiotemporally concordant cross-modal stimuli, these multisensory responses reached the criterion for multisensory enhancement (i.e., multisensory integration) as shown in Figure 6,A and B. This was true regardless of whether the cross-modal exposure occurred in macular or extramacular space, as long as the exposure site was encompassed by the neuron's receptive fields. Indeed, there were no significant differences in the data based on exposure site, although the first few neurons exhibiting multisensory integration were obtained in the animal with extramacular exposure. The first neuron exhibiting multisensory integration was obtained after ∼6 h of cross-modal exposure, and the incidence of these neurons increased rapidly thereafter. There was also a rapid increase in the magnitude of the multisensory enhancement in these neurons, as quantified by the MSI. This was contrary to our expectation based on observations in normally reared animals (Wallace and Stein, 1997), because it indicated that multisensory integration had not achieved an adult-like profile as soon as it appeared. Rather, the magnitude of the observed multisensory enhancements increased with increasing numbers of exposure trials.

Figure 6.

Cross-modal exposure led to the development of an ability to integrate cross-modal inputs and produce multisensory enhancement. A, B, Increasing proportions of neurons in dark-reared animals, with subsequent macular or extramacular cross-modal exposure, were capable of integrating their cross-modal inputs (black circles, neurons showing multisensory enhancement; white circles, neurons showing no multisensory enhancement as measured by MSI). After ∼27,000 cross-modal exposures, the majority of neurons were capable of multisensory integration. MSI also increased in magnitude with increasing cross-modal exposure. C–E, Dark-reared littermates were exposed to equivalent numbers of the component visual and auditory stimuli in an interleaved sequence in macular (left) or extramacular (middle) space, or to a random sequence of these stimuli in random locations (right). Their neurons did not develop the capability to integrate cross-modal stimuli and induce multisensory enhancement. Each solid line in the graphs denotes the line of regression. Dotted line denotes 0 MSI. All conventions are the same as in Figure 5.

The maximum incidence of exposure-induced multisensory integration was achieved after ∼27,000 trials (15 experimental sessions), which equates to ∼47 h of experience with these cross-modal stimuli. At this point, the incidence of neurons with multisensory integration capabilities and their MSIs were now no lower than that found in normally reared adults (Meredith and Stein, 1986; Jiang et al., 2001; Perrault et al., 2003; Stanford et al., 2005; Alvarado et al., 2007a). Yet, as noted above, neurons in animals exposed to the modality-specific stimulus components independently did not develop multisensory integration (Fig. 6C–E). Instead, their multisensory responses were rarely significantly above their unisensory comparator response, and there were no significant changes in this pattern as a result of increasing numbers of modality-specific exposure trials.

To better summarize these developmental trends, the experimental series was divided and the data grouped into three stages (Fig. 7): a first stage of <10,800 exposure trials; a second stage of 10,800–27,000 exposure trials; and a third and final stage of >27,000 exposure trials. The data were combined across animals based on whether their exposure consisted of cross-modal or modality-specific exposure. In the first stage, it was only the rare neuron (13.4%, 4/28) that manifested the ability to integrate its cross-modal inputs and produce an enhanced response. Thus, the mean MSIs in the cross-modal (15.93 ± 28.27%) and modality-specific (10.91 ± 20.24%) exposure groups were still quite similar. But in the second stage, neurons exhibiting multisensory integration in the cross-modal group increased to more than half the population of neurons sampled (59.4%, 19/32), a significant increase over the first stage (χ² test, df = 1, χ² = 12.84, p < 0.001), and higher than that (7.0%, 4/66) obtained from the modality-specific exposure groups at the same stage (χ² test, df = 1, χ² = 34.11, p < 0.001). Similarly, the mean MSI (50.61 ± 60.27%) had significantly increased (Mann–Whitney U statistic = 304.000; p = 0.033) and was higher than the second stage (15.55 ± 30.17%) in the modality-specific group (Mann–Whitney U statistic = 592.000; p = 0.006).

Figure 7.

Stages in the development of multisensory integration following sensory exposure. A, There was a rapid and significant increase in the incidence of neurons capable of multisensory integration when their receptive fields encompassed the cross-modal exposure site (black bars). B, These neurons also showed a significant increase in the magnitude of their integrated responses, as indicated by the MSI. However, this was not the case when exposure consisted of interleaved or random presentation of the individual component stimuli at the same locations (white bars). These differences between the exposure paradigms reached significance at the second and third stages of development. All conventions are the same as in previous figures.

In the third stage, SC multisensory integration capabilities in the cross-modal exposure group had reached the mature level, comprising 90% (63/70) of the sampled population and having a mean MSI of 124.2 ± 70.2%, which represents a significant increment above the second stage in both incidence (χ² test, df = 1, χ² = 13.07, p < 0.001) and magnitude (Mann–Whitney U statistic = 450.50; p ⩽ 0.001). In contrast, only 9.6% (8/83) of multisensory neurons in animals with modality-specific exposure paradigms became capable of integrating visual–auditory inputs in the third stage (χ² test, df = 1, χ² = 98.60, p < 0.001), and their mean MSI was very low (18.27 ± 32.80%).

For the cross-modal experience to initiate the maturation of multisensory integration capabilities in any given neuron, the neuron's visual and auditory receptive fields had to encompass the exposure site (Fig. 8). These neurons were categorized as type I. Neurons in which one or both of its receptive fields did not encompass the exposure site were combined (there was no difference between these groups) and categorized as type II (10 had one receptive field encompassing the exposure site, 18 had neither). Whereas the vast majority (90%, 63/70) of type I neurons developed multisensory integration capabilities (Fig. 8E), very few (14.3% 4/28) type II neurons developed this capability. Similarly, the mean MSI of type I neurons (124.2 ± 70.2%) was significantly higher than that of the type II neurons (10.2 ± 35.3%) (Mann–Whitney U statistic = 78.50, p < 0.001). Thus, it appeared that the simple exposure of an animal to cross-modal stimuli at a given spatial location was insufficient to initiate the development of multisensory integration in the entire population of SC neurons. Rather, a given SC neuron generally had to have both its receptive fields encompassing the exposure site to have access to the cross-modal experience.

Figure 8.

To initiate multisensory integration maturation, both cross-modal component stimuli had to fall within their respective receptive fields. A–C, Data from three characteristic neurons are illustrated. In A, only the visual receptive field of this neuron contained the cross-modal exposure site, in B only the auditory receptive field of this neuron contained the exposure site, and in C neither receptive field of this neuron contained the exposure site. No multisensory integration was apparent in any of these cases despite high numbers of exposure trials. Conventions are the same as in previous figures. D, This summary figure shows the percentage of neurons developing multisensory integration when the cross-modal exposure site was encompassed by their two receptive fields (type I, black bars) and those in which it was outside one or both receptive fields (type II, white bars). The data were collected after 27,000–54,000 cross-modal exposure trials. Note that many type I neurons exhibited multisensory integration, whereas this was comparatively rare in type II neurons. E, F, The average MSI (E) and the number of neurons showing high MSI (F) were far greater for type I than for type II neurons (error bars denote SEM).

However, once a neuron acquired the ability to engage in multisensory integration as a consequence of cross-modal exposure at one location within its receptive fields, it appeared to generalize this capacity to other receptive field sites. This was the case in nearly all of the neurons under study (94%, 31/33). Two typical examples are provided in Figure 9. In each case, multisensory integration capabilities were acquired after cross-modal exposure at a single site (0° or 30°), but were also evident at a nonexposure test site, albeit on average with lower magnitude. The data for the population are shown in Figure 10. Here, it is apparent that in nearly all cases (91%, 30/33) the MSI was higher at the exposure than the nonexposure site. This was the case regardless of the location of the exposure site, and the group means at these sites were significantly larger than those evoked from their respective nonexposure sites (paired t test, p < 0.001). This difference held regardless of whether the exposure site was macular and the nonexposure site was extramacular (125.8 ± 125.0% vs 76.0 ± 74.0%) (Fig. 10A) or the reverse (126.1 ± 103.9% vs 90.1 ± 78.1%) (Fig. 10B).

Figure 9.

Once a neuron acquired multisensory integration at one receptive field location, it was generalized to other locations. The responses of two neurons are displayed here, each in two rows. A, The neuron represented in the top two rows not only showed multisensory integration at the macular exposure site (top), but also when the stimuli were ∼30° peripheral to that site (bottom). B, The neuron represented in the bottom two rows showed multisensory integration at its extramacular exposure site (top) and also at a site 30° central to it (bottom). All conventions are the same as in prior figures.

Figure 10.

MSI was higher at exposure than at nonexposure sites. Neurons at the two different exposure sites (A = 0°; B = 30°) were recorded after 27,000–54,000 cross-modal exposure trials. A gray line connects the MSI obtained at the exposure site (filled circle) and the nonexposure (unfilled circle) site within each neuron's receptive fields. The dark horizontal bar represents the mean MSI for each group's data. Note that despite the fact that most neurons developed multisensory integration capabilities at the nonexposure site, the population mean MSIs at each nonexposure site were lower than at their comparator exposure site.

Discussion

Presumably, the normal period for multisensory development in SC neurons is protracted so that the brain can accommodate to the statistics of cross-modal events, then use this information to link the modalities to one another and organize the principles by which their information will be integrated. Normally, this takes place very early in life (Stein et al., 1973; Wallace and Stein, 1997, 2001, 2007) when brain plasticity is believed to be greatest (Schoop et al., 1997; Katz, 1999; Berardi et al., 2000; Hensch, 2004; de Villers-Sidani et al., 2007; Galtrey and Fawcett, 2007). However, in the present experiments these cross-modal experiences were delayed until long after the capability for multisensory integration is normally acquired in SC neurons. That these neurons were still able to acquire this capability indicates that the brain retains the ability to use experiences with cross-modal cues to initiate its development far longer than previously suspected, an observation that is consonant with previous speculations (Wallace et al., 2004).

That cross-modal experiences were critical for the development of multisensory integration was apparent from its absence in dark-reared animals and the inability of equivalent exposure to the component stimuli, either individually or in an interleaved manner, to produce comparable results. An important factor was that both modality-specific components of the cross-modal stimulus be contained within a neuron's receptive field. Only then would a neuron be directly privy to that cross-modal information. The development of multisensory integration appeared to involve several stages. On first appearance, the relative magnitude of integrated responses was far lower than is characteristic of normal animals, but gradually increased with continued cross-modal exposure. Although it appears likely that the continued accumulation of cross-modal experience was instrumental in this process, the possibility that, once initiated, multisensory integration capabilities can continue to develop in the absence of further experience cannot yet be excluded.

One can also infer that the acquisition of multisensory integration is initially closely linked to the physical properties of the stimuli but is then generalized to other stimulus configurations. This inference is based on two observations. First, all the neurons that acquired multisensory integration capabilities did so at the receptive field position corresponding to the site of cross-modal exposure, but not all of these neurons did so at a nonexposure site. Second, those that did exhibit the capacity for multisensory integration at a nonexposure site exhibited lower magnitudes of response enhancement there than at the actual exposure site. Although these observations are consistent with the idea that a multisensory principle has been learned (i.e., the visual and auditory cues are linked at a given receptive field location) and then generalized, support for this concept would be strengthened considerably if future observations can demonstrate a temporal interval between these two presumptive stages and/or show that changes in the multisensory responses to other, nonspatial properties of the cross-modal stimulus exhibit a similar pattern.

It is particularly interesting to note that the cross-modal experiences that led to the development of multisensory integration in SC neurons were acquired during anesthesia and in the absence of any specific reward contingency. Apparently, exposure to the spatiotemporal coincidence of the stimuli was sufficient for them to become bound in the anesthetized brain so that they were then able to elicit an enhanced multisensory response in SC neurons. The experience in this case is unlike that usually referred to by this term, as it did not appear to involve higher order cognitive processes. Relatively simple Hebbian learning rules such as spike-timing-dependent plasticity (STDP) (Gerstner et al., 1996; Markram and Tsodyks, 1996; Fu et al., 2002; Dan and Poo, 2004; Young et al., 2007; Dahmen et al., 2008) have been shown to be operative in other areas of the brain, and even in brain slices, and are likely to be sufficient to encode the physical correlations between the two inputs in the anesthetized brain in a manner that would replicate the present observations (Yu et al., 2009). According to this principle, presynaptic (i.e., tectopetal) afferents whose activity repeatedly precedes postsynaptic spiking are strengthened. If operative in circumstances in which spatiotemporally concordant visual and auditory stimuli are presented, STDP should lead to the preferential strengthening of the afferents activated at approximately the same time, because they cooperate in activating the postsynaptic (SC) neuron. Consequently, the strongest inputs to a given SC neuron should be those that respond to visual and auditory stimuli that are approximately aligned in time and space.

The exceptional sensitivity of SC neurons to cross-modal stimulus exposure was reflected in the speed with which they used this experience to develop the capacity to integrate those inputs. Indeed, SC neurons accomplished this feat far more rapidly than would have been predicted from their normal developmental chronology. The first SC neuron capable of multisensory integration was encountered after several hours of visual–auditory experience rather than after many weeks, as in the normally developing animal (Wallace and Stein, 1997). Furthermore, experience with this cross-modal stimulus configuration was sufficient to achieve a normal complement of integrating multisensory neurons. Presumably, this seemingly speeded acquisition reflects the high iterative rate and constancy of the cross-modal experiences that were provided here, as well as the presence of a neural substrate that had become mature enough to incorporate these experiences. Although it is assumed that the developmental principles observed here will generalize to other cross-modal stimulus combinations, at present only the visual–auditory combination has been studied.

That surprisingly little cross-modal exposure can have a substantial impact on the acquisition of multisensory integration capabilities, and that this high degree of experiential sensitivity is retained during early adulthood, suggest that multisensory integration is subject to continual modification throughout life. Although this possibility requires far more empirical support, it is consistent with a recent observation that in normal adult animals the responses of multisensory SC neurons to sequential visual and auditory stimuli change substantially after only minutes of exposure (Yu et al., 2009), that dark-reared animals can manifest multisensory integration at the behavioral level after comparatively short training periods (Gingras et al., 2007), and that intense training with cross-modal stimuli can reverse hemineglect in cats (Jiang et al., 2008; McHaffie et al., 2008) and hemianopia in humans (Bolognini et al., 2005). It is also likely to be of substantial import in the context of the current rehabilitative efforts directed at inducing hearing or sight via cochlear or retinal prostheses or other intervention strategies such as the removal of congenital cataracts (Putzar et al., 2007). Although these strategies are generally directed specifically toward the compromised sensory modality, providing the experiences with cross-modal stimuli that would be necessary to develop an ability to use these senses in concert may require little additional effort.