Cross-Modal Competition: The Default Computation for Multisensory Processing

Abstract

Mature multisensory superior colliculus (SC) neurons integrate information across the senses to enhance their responses to spatiotemporally congruent cross-modal stimuli. The development of this neurotypic feature of SC neurons requires experience with cross-modal cues. In the absence of such experience the response of an SC neuron to congruent cross-modal cues is no more robust than its response to the most effective component cue. This “default” or “naive” state is believed to be one in which cross-modal signals do not interact. The present results challenge this characterization by identifying interactions between visual-auditory signals in male and female cats reared without visual-auditory experience. By manipulating the relative effectiveness of the visual and auditory cross-modal cues that were presented to each of these naive neurons, an active competition between cross-modal signals was revealed. Although contrary to current expectations, this result is explained by a neuro-computational model in which the default interaction is mutual inhibition. These findings suggest that multisensory neurons at all maturational stages are capable of some form of multisensory integration, and use experience with cross-modal stimuli to transition from their initial state of competition to their mature state of cooperation. By doing so, they develop the ability to enhance the physiological salience of cross-modal events thereby increasing their impact on the sensorimotor circuitry of the SC, and the likelihood that biologically significant events will elicit SC-mediated overt behaviors.

SIGNIFICANCE STATEMENT The present results demonstrate that the default mode of multisensory processing in the superior colliculus is competition, not non-integration as previously characterized. A neuro-computational model explains how these competitive dynamics can be implemented via mutual inhibition, and how this default mode is superseded by the emergence of cooperative interactions during development.

Introduction

Multisensory integration has been a subject of considerable interest, not only because of its obvious survival value (for review, see Stein, 2012), but because of conflicting views about the mechanisms underlying its computation (Anastasio et al., 2000; Patton and Anastasio, 2003; Knill and Pouget, 2004; Rowland et al., 2007b; Alvarado et al., 2008; Cuppini et al., 2011; Ohshiro et al., 2011; Ursino et al., 2014; Miller et al., 2017). Its neural basis and its postnatal development have been studied most extensively in the cat superior colliculus (SC; Wallace and Stein, 1997; Yu et al., 2010, 2013; Xu et al., 2012, 2014, 2015, 2017; Stein et al., 2014). In the typical adult, the responses of multisensory SC neurons to spatiotemporally congruent cross-modal (e.g., visual-auditory) cues are significantly more robust than their responses to the most effective individual component cue, thereby increasing the likelihood of SC-mediated behavioral responses to the generating event (Stein et al., 1989; Burnett et al., 2004, 2007). However, developing this capacity for multisensory enhancement requires experience with covariant cross-modal cues; for example, multisensory enhancement is not observed when early visual-auditory experience is compromised by rearing animals in darkness or omnidirectional masking sound (Wallace et al., 2004; Carriere et al., 2007; Yu et al., 2010; Xu et al., 2014). Related defects have been seen in patients with congenital cataracts or hearing deficits (Putzar et al., 2007; but see Rouger et al., 2007; Nava et al., 2014; Gori, 2015).

In these “multisensory-naive” or “default” states, the topography, incidence, and cross-modal convergence patterns of SC neurons are only marginally different from those in the normal adult. However, their responses to congruent cross-modal cues are very different: they are no greater than those to the most effective modality-specific component cue (Wallace et al., 2004; Yu et al., 2010; Xu et al., 2014). The inference that has been drawn from these observations is that cross-modal signals do not interact in the default state, so that responses are simply controlled by the strongest unisensory input. This inference has shaped the way that the development of multisensory integration is conceptualized: that it proceeds from absence to presence (Bremner et al., 2012). In this way, multisensory development has been conceptualized similarly to the development of other perceptual and cognitive capabilities; e.g., the development of neuronal preferences for elementary visual and auditory features (Wiesel, 1982; Sanes and Bao, 2009), executive functions (Kolb et al., 2012), and the emergence of language and social communication capabilities (Pinker, 2010).

Although there is logic behind the inference that the default multisensory computation reflects “non-integration”, recent theoretical perspectives suggest an alternative possibility. Within the functional organization of the SC, singular salient targets are normally selected via a competition between spatially noncongruent cues (Meredith and Stein, 1986; Kadunce et al., 1997, 2001; Pluta et al., 2011; Mysore and Knudsen, 2013). Here we hypothesize that this competition is the native default mechanism for all convergent cross-modal inputs. If this is correct, the maturation of multisensory enhancement capabilities should not be thought of as the transition from their absence to their presence, but as a more radical transition from the default competitive mechanism that depresses multisensory responses to the cooperative mechanism that enhances them. This possibility was explored here in adult SC neurons that were naive to visual-auditory events.

Materials and Methods

All protocols used were in accordance with the Guide for the Care and Use of Laboratory Animals, Ed 8 (NRC, 2011) and were approved by the Animal Care and Use Committee of Wake Forest Medical School, an AAALAC-accredited institution.

Housing facilities.

Normally-reared animals (“normal” cohort) were raised from birth until the time of study in a standard housing facility. Dark-reared (“naive” cohort) animals were housed from birth in a light-tight facility (“dark room”) that precluded experience with visual and visual-auditory stimuli. They remained housed in this facility for the duration of the experiment. The dark room did not impede the animals' exposure to auditory and other nonvisual sensory modalities, but ensured that animals were naive to visual-nonvisual stimulus combinations before experimentation began. Animal care staff wore infrared goggles during daily care and an infrared CCTV system allowed viewing of the animals from outside of the dark room. Animals were anesthetized in the dark room, blindfolded, and transported in a masked carrier for all procedures (e.g., surgery and electrophysiological recording).

Surgery.

All animals were implanted before their initial study with recording well/head-holder devices positioned over the SC to permit electrophysiological recordings. These devices allowed the head to be held in place without wounds or pressure points (McHaffie and Stein, 1983). Briefly, each animal was anesthetized with ketamine hydrochloride (20 mg/kg, i.m.) and acepromazine maleate (0.1 mg/kg, i.m.) in its home cage, transported to the surgical suite, intubated, and then deeply anesthetized to a surgical plane of anesthesia with isoflurane (3%). It was then placed in a stereotaxic frame. Anesthesia was maintained with isoflurane (1–3%) while vital signs (expiratory CO₂, blood pressure, and heart rate) were continuously monitored on a VetSpecs VSM7. Core temperature was maintained at ∼37–38°C via hot water circulating through a heating pad. A craniotomy was performed over the SC and the recording well/head-holder was secured over it using stainless steel screws and dental acrylic. At the termination of the procedure, the animal was extubated, recovered, and returned to its home cage. Analgesics (buprenorphine, 0.005–0.01 mg/kg, i.m.; ketoprofen, 1 mg/kg, i.m.) were provided as needed and antibiotics (either ceftriaxone, 20 mg/kg bid, i.m., or cefazolin sodium, 25 mg/kg bid, i.m.) were administered for 7 d.

Electrophysiological recordings.

Recording experiments began after at least 7 d of recovery following surgery, and were typically conducted with a frequency of one session per animal per week. Before each session, each animal was anesthetized in its home cage with a combination of ketamine hydrochloride (20 mg/kg, i.m.) and acepromazine maleate (0.1 mg/kg, i.m.), transported to the laboratory, intubated, and connected to a quiet ventilator (Ugo Basile, model 6025). The head was fixed to a stereotaxic apparatus via stainless steel rods which connected to the recording well/head-holder, and the animal was placed in a relaxed recumbent posture. Heart rate, CO₂, respiratory rate and volume were monitored continuously, and blood pressure every 15 min, to ensure an adequate state of anesthesia for the entire experiment. The saphenous vein was catheterized, and paralysis was induced with pancuronium bromide (0.1 mg/kg, i.v.) to prevent ocular drift. The right eye was dilated with atropine sulfate (1%) and fitted with a contact lens to correct for refraction, whereas the left eye was occluded with an opaque lens. The optic disc of the right eye was reverse-projected onto a tangent screen using an ophthalmoscope and mapped. Anesthesia, paralysis, and hydration were maintained by continuous intravenous infusion of ketamine hydrochloride (6–8 mg kg⁻¹·h⁻¹), pancuronium bromide (0.05 mg kg⁻¹·h⁻¹), and 5% dextrose in sterile saline (3–6 ml/h). Body temperature was kept at 37∼38°C. End tidal CO₂ was maintained at ∼4%.

Once the animal was prepared and stable, recording procedures were initiated. A glass-coated tungsten electrode (tip diameter, 1–3 μm; impedance, 1–3 MΩ at 1 kHz) was positioned and lowered to the superficial layers of the SC. From there, the electrode was slowly advanced via a hydraulic microdrive into and through the intermediate and deep (i.e., the multisensory) layers of the SC while a variety of visual and auditory search stimuli (e.g., flashes, moving bars, clicks, beeps, and snaps) were presented. Individual neurons were identified and isolated based on a criterion of 3× impulse amplitude over background. Once a neuron was isolated, its visual and auditory receptive fields (RFs) were mapped using standard procedures (Alvarado et al., 2008). Briefly, the visual RF was mapped with moving light bars projected from a LC 4445 Philips projector onto a tangent screen located 45 cm from the front of the animal, and the auditory RF was mapped with broadband noise bursts from any of 16 hoop-mounted speakers placed 15° apart and 15 cm from the head on a rotating hoop that permitted adjustments in elevation.

Each neuron was tested with a battery of “standard” visual and auditory stimuli presented individually and together at the same location within the region of visual-auditory RF overlap. Visual stimuli were moving bars of light (10° × 2°) with average intensity of 13.67 cd/m² projected against a background luminance ∼ 0.16 cd/m²), with a speed of 100°/s in the neuron's preferred direction if one was apparent, and with a duration of 100 ms. Auditory stimuli were brief (100 ms) bursts of broadband noise (20–20,000 Hz) with average intensity of 65 dB against the ambient background noise of ∼45 dB. In addition to the standard stimulus tests, some neurons were tested with a larger and more diverse set of stimuli designed to elicit different response levels. When time permitted, some neurons were also tested with a stimulus battery of spatially incongruent cues. In these, the visual stimulus remained within its excitatory RF, but the auditory stimulus moved outside of the excitatory auditory RF, and they were presented alone or in temporal concordance.

Custom software operating on a NIDAQ digital controller (National Instruments) connected to a personal computer was used to control stimulus presentation and record electrophysiological data. Individual impulse waveforms were visualized on an oscilloscope and sorted both on-line and off-line using custom software.

At the end of the recording session, the anesthesia and paralysis were terminated. Once stable locomotion was reinstated, animals' eyes were covered and they were returned to their home cages.

Exposure paradigm.

After the initial study, five of the seven dark-reared animals were given repeated exposure to spatiotemporally congruent visual-auditory cues in a paradigm previously noted to be sufficient to instantiate (i.e., “train”) visual-auditory enhancement capabilities (Yu et al., 2010). After this training their multisensory neurons were again tested to determine how this exposure had changed the multisensory computation.

Exposure training took place once a week. Each animal was anesthetized, transported, and stabilized in the same way as described above for electrophysiological recordings experiments, but received cross-modal exposure trials instead (1800–2000 trials). The cross-modal exposure stimulus was a pair of spatiotemporally congruent visual and auditory cues presented at one of two locations: either in macular space at coordinates (X = 0°, Y = 0°) relative to the area centralis, or extra-macular space at coordinates (X = 30°, Y = 0°). Afterward, they were recovered and returned to their home cages in the dark room. After completion of the exposure paradigm (a total of 26,000 trials/animal), the second set of electrophysiological recordings was conducted in the (contralateral) SC of the now “trained” cohort. Electrode penetrations were made in regions of the SC representing the locations of the exposure sites, and only neurons with visual and auditory RFs overlapping at least one of the exposure sites were studied in this second round of tests.

Experimental design and statistical analysis.

Eight normally-reared “control” animals (5 male, 3 female) and seven dark-reared (4 male, 3 female) adult cats were used in these experiments (all at least 7 months old). The impulses for each stimulus trial for each test condition (20–30 trials/condition) were transformed into an impulse raster with 1 ms resolution. The window containing the stimulus-elicited response was determined by a geometric method used in earlier studies (Rowland et al., 2007a; Rowland and Stein, 2008). The magnitude of the response for each test condition was identified as the number of impulses occurring in the response window minus the expected spontaneous number, estimated using the spontaneous firing rate estimated in the 500 ms window preceding stimulus onset. The neuron classified as responsive to the stimulus if response magnitude was significantly >0 (Wilcoxon signed rank test).

Visual and auditory RF sizes were quantified along the horizontal axis. Additionally, for neurons responsive to both modalities, their percentage overlap was quantified (Jiang et al., 2006). The effectiveness of multisensory integration was quantified using several traditional metrics of multisensory enhancement and superadditivity (Stein et al., 2009). The metric of multisensory enhancement (ME) quantified the percentage difference between the response to a cross-modal pair (VA) and the largest response to one of its modality-specific (V, visual; A, auditory) components [max(V,A)]: ME = 100 × [VA − max(V,A)]/max(V,A). The additivity index (AI) quantified the percentage difference between the multisensory response and the sum of the component unisensory responses: AI = 100 × [VA − (V+A)]/(V+A).

An important response feature in the present context was the relative efficacy, or “balance”, of the visual and auditory response magnitudes elicited in a test battery. This was quantified as unisensory imbalance (UI): UI = 100 × (|V − A|)/(V+A) (where |x| indicates the absolute value of x). UI has a minimum of 0 when the visual and auditory responses are equal magnitude and a maximum of 100 when one of the responses is nonexistent. The metric of unisensory imbalance has shown to be one of the predictive factors of multisensory enhancement magnitude in neurotypic populations (Otto et al., 2013; Miller et al., 2015). However, it also represents a measure of the dissimilarity of the two unisensory response magnitudes. In circumstances in which a computation yields multisensory response magnitudes that are between the two unisensory response magnitudes (e.g., reflects an average of them) rather than equal to one or the other, this difference will not be visible when UI is low.

Multisensory and unisensory response magnitudes and other statistics were compared using Mann–Whitney U and Wilcoxon signed rank tests where appropriate, and evaluated as indicated (α = 0.05). Neurons were categorized as “overt multisensory” if they overtly responded to both visual and auditory modalities, and “covert multisensory” if they overtly responded to only one modality, but their response to a visual-auditory pair altered that response significantly. Otherwise, the neuron was categorized as “unisensory”. For the purposes of comparison to other literature, multisensory responses were categorized as “enhanced” when significantly more robust than the most robust component unisensory response, “depressed” if significantly less robust, or “nonsignificant” otherwise. The incidences of these different response types were compared within and across cohorts using χ² tests. Linear regression was used to determine the significance of trends using F tests.

Neuro-computational model.

To facilitate interpretation, a neuro-computational model was implemented to determine whether the empirical data could be accurately predicted by the competitive and associative dynamics that were hypothesized. To this end, we evolved a model we had previously developed to explain the pairwise-specific nature of the maturation of multisensory capabilities in this structure (Cuppini et al., 2018; Fig. 1). Below we provide a brief summary of the key features of the model, followed by its specific equations and parameters.

Figure 1.

A neuro-computational model of SC multisensory integration. External visual and auditory inputs to the multisensory layers of the SC are abstractly represented as derived from either the visual (AEV) or auditory (FAES) subdivisions of the anterior ectosylvian sulcus (AES) or non-AES sources (“Visual area”, “Auditory area”). Each input region contacts principal (excitatory) neurons in the SC through competitive or cooperative pathways: the projections from non-AES sources are arranged to be functionally competitive across the modalities. This is implemented via independent excitatory projections (A and V) as well as reciprocal inhibitory synapses. A, This computation dominates the default, or naïve state that exists without covariant cross-modal experience. B, Projections from AES are arranged to be functionally cooperative (AV), moreover these input regions can suppress the native competitive mechanism through dedicated inhibitory synapses. Excitatory and inhibitory AES projections are strengthened by covariant cross-modal experience and instantiate multisensory enhancement capabilities in the neurotypic adult.

Circuits running through the SC are represented abstractly by five interconnected regions including the SC and four unisensory input regions (each containing 100 U). The input regions represent topographic maps of visual and auditory space, and interact with one another in either a competitive or noncompetitive fashion. One of the important features of this model is that it incorporates a special tectopetal projection observed empirically arising from a region of association cortex in the cat, the anterior ectosylvian sulcus (AES). This cortical input plays a crucial role in mediating the development and expression of multisensory integration capabilities within the SC (Jiang et al., 2001, 2002; Alvarado et al., 2007, 2009; Yu et al., 2013, 2016; Rowland et al., 2014). In the basic model schematic, the sources of the competitive projections (representing “competitive regions” mostly outside of AES) are identified as C_v (visual) and C_a (auditory). The “noncompetitive regions” (exclusively inside the AES) that extend noncompetitive projections are identified as NC_v (visual) and NC_a (auditory). Competitive input regions project topographically into the SC and mutually suppress one another via inhibitory projections. Noncompetitive input regions project onto common compartments that bypass this competition, but are initially ineffective. Thus, in the naive state, the multisensory responses are dictated by a principle of competition, because the noncompetitive pathway is still immature. When cross-modal cues are experienced over many iterations of training in the model (simulating development), the noncompetitive pathway becomes shaped and strengthened, and comes to inhibit the competitive inputs. Thereafter, the outputs of SC neurons are driven by the noncompetitive inputs.

In the following equations (for descriptions of notation and parameters, see Table 1), Equation 1 describes the evolution of the output, z_i^h(t), of a unit i in region h with first order differential equation forced by its net input, u_i^h(t), which is transformed by a sigmoidal function, Φ(), in Equation 2.

Equation 3 describes how the net input, u_i^h(t), of a neuron at position i in the input region h is calculated by summing a random “noise” input [N(0,2.5)] with stimulation from external sources (I_i^h).

Each input region sends topographic projections into the SC that terminate on different compartments (3 compartments/unit). Each of the competitive input regions (C_a and C_v) target separate compartments, whose net input, I_i^s(t) (Eq. 4), is the sum of excitation from projections from the input regions and an inhibitory input (Y_i^h) from other regions in the model (expanded in Eq. 5).

The noncompetitive input regions target a common compartment (net input in Eq. 6).

The output of each compartment is calculated from these net inputs using Equations 1and 2. The net input to an SC unit is the weighted sum of outputs from these three compartments (Eq. 7), and an added random noise input [N(0,10)].

This net input is converted to output via Equations 1 and 2. During simulations of development (see below), the weights connecting the noncompetitive compartments to SC neurons are changed by a modified Hebbian learning rule that requires both inputs to exceed a threshold level (Eq. 8) before being strengthened. An adaptively scaled learning rate constrains this strengthening (Eq. 9).

Finally, the inhibitory effect of the noncompetitive regions over the other input sources is strengthened by a similar Hebbian learning rule (Eq. 10), with a learning rate, β, computed in Equation 11.

Unlike the model published by Cuppini et al. (2018), this model does not contain an inhibitory projection from the competitive inputs to the noncompetitive inputs. With this change the model can explain the present results without compromising its ability to explain those previously published.

Table 1.

Notation and parameters in equations and simulations

Parameter	Equation	Value	Identity
N	N/A	100	Number of units per simulated region
τ	1	3	Time constant (in time steps) for unit input/output transformation
θ	2	20	Midpoint of sigmoidal input transfer function
ρ	2	0.3	Scaling parameter of sigmoidal input transfer function
Itraining	3	30	Input magnitude used during training
I	3	17–22	Input magnitude used during testing
Lijh.m(t)	5	Varies	Trainable inhibitory weight of connection from unit at position j in region m to unit at position i in region h
Cv, Ca	4, 5, 11	N/A	Competitive input regions for visual (v) and auditory (a) modalities
Wc	4	42	Weight of inputs to competitive compartments.
NCv, NCa	6	N/A	Noncompetitive input regions for visual (v) and auditory (a) modalities
Wnc	6	21	Weight of inputs to noncompetitive compartments
N(0,10)	7	N/A	Indicates a random number selected from a normal distribution with mean 0, and SD 10
Wijsm.k(t)	7–9	Varies	Trainable excitatory weight from compartment k to the SC unit (i == j)
ϑC	8	0.7	Presynaptic threshold for excitatory weight modification
ϑN	8–10	0.4	Postsynaptic (SC) threshold for weight modification
ϑI	10	0.2	Presynaptic threshold for inhibitory weight modification
Wmax	9	25	Maximum trainable excitatory weight
Lmax	5–11	10	Maximum trainable inhibitory weight
α0	9	0.1	Baseline learning rate for excitatory weights
β0	11	0.001	Baseline learning rate for inhibitory weights

Exposure to a sensory cue was simulated by setting the external network input I to a value representing a point stimulus (I_training; Table 1), kept constant during a single training trial, updating the state of the network (Eqs. 1–7) until a steady state was reached, and applying the learning rules (Eqs. 8–11) to the plastic connections. Exposure to cross-modal cue combinations was simulated by setting input patterns for multiple subregions in the C and NC regions.

Simulations.

The model was simulated to determine its ability to quantitatively predict the multisensory responses of the naive and trained cohorts. To mimic the dark-rearing condition, the model in its default configuration (Fig. 1A), with ineffective cooperative pathways and mature competition only between non-AES subregions (see parameters in Table 1), was trained 5,000,000 trials containing only auditory stimuli. At the end of this training, predictions of the model were created by testing the architecture with simulated visual and auditory cues with intensities randomly-selected by a normal distribution (see parameters in Table 1), presented both alone and in spatiotemporal congruence (i.e., at matched locations in the input arrays). This procedure was repeated 10,000 times to generate 10,000 samples consisting of a prediction of the visual, auditory, and multisensory responses. The metrics of ME and UI were calculated for each sample and the relationship between ME and UI was evaluated using regression. To examine neuron-by-neuron predictability for each biological neuron, the database of 10,000 model samples was searched to find the simulation whose unisensory responses were most similar (minimizing Euclidian distance). The multisensory responses of the closest-matching model units were then compared to those of their biological counterparts.

The predictions of the model for the trained cohort were obtained in a different set of 10,000 simulations. The network, with the synaptic configuration obtained at the end of the previous training phase, was exposed to 5,000,000 spatiotemporally congruent pairs of visual-auditory stimuli. After each exposure, the network connectivity parameters were updated by a modified Hebbian learning rule (Eqs. 8–10). After this training, each network was retested with a pair of visual and auditory cues with randomly-selected intensities (see parameters in Table 1) presented alone and in spatiotemporal congruence. This yielded 10,000 samples of predicted unisensory and multisensory responses, as above. These predictions were then compared with the empirical data using the methods described above for the naive cohort.

Results

Unisensory and visual-auditory multisensory SC neurons were recorded from animals in three conditions: one in which animals were normally-reared (normal: n = 288), another in which animals were reared in the dark and served as the naive cohort with regard to visual-auditory experience (naïve: n = 256), and the last one, in which the naive population was then exposed to a visual-auditory training paradigm previously shown to instantiate multisensory enhancement capabilities in SC neurons (trained: n = 183; Yu et al., 2010). Only overt multisensory neurons, in which the visual and auditory stimuli elicited impulses when presented individually, were selected for analysis in this study. The incidence of these neurons was similar in the three cohorts: (normal: n = 102; naïve, n = 85, trained: n = 92). The incidence of neurons and their overall sensitivities reported here are consistent with other recordings from these and similar animals published previously (Yu et al., 2010).

The unisensory responses of normal and naive multisensory neurons

Mean visual response magnitudes in response to the standard visual and auditory stimuli used here were comparable across the normal and naive populations (normal: 5.57 ± 3.99 impulses/trial, naive: 5.22 ± 4.30 impulses/trial; p = 0.269). Auditory response magnitudes were slightly higher in the normal than the naive group (normal: 3.54 ± 2.96, naive: 2.15 ± 2.47; p = 1.51E−7). Nevertheless, in many cases in both groups auditory responses consisted of only a few impulses and their response magnitudes were not easily increased by varying stimulus intensity, and in others there was only a small range in which changes in auditory response magnitude were possible. Thus, as noted previously (Yu et al., 2010) it was more common for a multisensory neuron to have a higher visual than auditory response magnitude in these conditions, a preference that was somewhat more pronounced in the naive than in the normal population (normal: 72/102 = 71% of neurons, naive: 69/85 = 81%).

There were three additional differences of note regarding the unisensory responses in the normal and naive populations that impact the interpretation of the present findings. The first of these was that the UI scores were significantly lower (i.e., unisensory responses were more balanced) in the normal than in the naive population (mean UI: normal: 19.84 ± 32.24, naive: 50.14 ± 26.1; p = 9.36E−10). Underscoring this difference was the finding that the normal animals had only ∼1/5 of their neurons (20/102 = 19.6%) showing substantially “imbalanced” visual and auditory responses (UI > 50%); however, in the naive condition, imbalanced responses were apparent in nearly 50% of the neurons (42/85 = 49.4%). A second difference was the size of RFs. As noted previously (Yu et al., 2010) visual and auditory RFs were much larger in the naive (V mean diameter = 86.9 ± 32.6°, A mean diameter = 95.8 ± 36.4°) than in the normal (V = 51.36 ± 12.6°, A = 79.8 ± 24.4°) cohort. Last, the visual-auditory RF overlap was significantly lower in the naive (50 ± 23%) than in the normal (78 ± 28%) cohort.

Integration of spatiotemporally concordant visual-auditory cues

Of particular interest here were significant differences noted in the multisensory products elicited by spatiotemporally concordant visual-auditory cues in normal and naive animals. The first of these has been noted previously. Very few (6/85 = 7%) overt multisensory neurons in the naive animal showed significant enhancement in their responses to this combination of cross-modal cues, whereas such enhancement was characteristic of the normal population (95/102 = 93%). The multisensory products in the naive animals were much more likely to be classified as nonsignificant (49/85 = 58%), something that was seen only rarely in normal animals (7/102 = 6.9%). Typical exemplars illustrating this response difference in normal and naive multisensory neurons are provided in Figure 2. The large incidence of nonsignificant multisensory products in the neurons from naive animals is consistent with the standard interpretation that these neurons lack integration capabilities, and consequently disregard one of their inputs (Wallace et al., 2004; Yu et al., 2010, 2013; Xu et al., 2012, 2015, 2017). However, of special relevance here was the additional observation that a substantial portion of them also exhibited multisensory depression in response to spatiotemporally concordant cues (30/85 = 35%). This sort of interaction has rarely been observed previously (but see Alvarado et al., 2008) and suggested the presence of depressive multisensory interactions in naive neurons that prompted further examination.

Figure 2.

Typical unisensory and multisensory responses in naïve and normal SC neurons. Depicted for each of these two neurons are its responses to visual and auditory stimuli presented alone and together in spatiotemporal concordance. On the left are the impulse rasters for each response (ordered bottom-to-top), on the right are summary histograms for the average number of elicited impulses in each condition. Vertical lines through the bars represent the standard error of the mean SEM. A, The multisensory response in the naïve exemplar was not significantly greater than its largest unisensory response (here, V), and thus appeared to be insensitive to the auditory input in the VA condition. Response magnitudes (impulses/trial): V = 4.5, A = 3.65, VA = 5.1; UI = 10%, ME = 13% (multi vs uni, p = 0.307, Mann–Whitney U test). B, This result contrasts with the response pattern in the normal exemplar whose multisensory response was significantly enhanced by the auditory stimulus, becoming 117% greater than its strongest unisensory response (V). V = 6.17, A = 5.75, VA = 13.4; UI = 3%, ME=117%. **p = 7.50E−7.

That some naive neurons produced nonsignificant multisensory products and others produced significant multisensory depression in response to the same cross-modal stimulus configuration appeared to be dependent on the relative magnitude of the unisensory comparator responses (i.e., UI). In normal animals, unisensory imbalance decreases the magnitude of ME (Miller et al., 2015). Importantly, however, the multisensory product is nearly always enhanced regardless of the absolute level of that imbalance (Otto et al., 2013; Miller et al., 2015). This was obvious in the nature of the regression of ME versus UI (slope = −0.981, intercept = 125; Fig. 3A), which, although the slope was significantly negative (R² = 0.166; p = 1.24E-5), the regression remained >0 and significant enhancement was apparent in neurons even with highly imbalanced responses.

Figure 3.

Relationships between multisensory responses and unisensory imbalance in normal and naïve cohorts. A, Neurons from normally-reared animals produce their greatest response enhancements when the spatiotemporally concordant cues produced balanced unisensory responses. This is illustrated by the inverse relationship between ME and UI (dotted line). B, Naïve SC neurons showed a similar inverse relationship between ME and UI, but even balanced samples failed to produce significantly enhanced multisensory products, and imbalanced samples induced multisensory depression. C, Histograms summarizing the results. Vertical lines through the bars represent SEM. **p < 0.001, *p < 0.05.

But in the naive condition the result was very different. Although an inverse relationship between ME and UI was also evident, albeit less steep (slope = −0.44, intercept = 20.1, R² = 0.122; p = 5.9E−4; Fig. 3B), it began just >0 when imbalance was lowest, dipped <0 at ∼50% imbalance, and remained <0 at higher levels of imbalance. Furthermore, with rare exception, even the most balanced responses failed to yield significantly enhanced multisensory responses, and imbalanced responses yielded depressed multisensory responses that were weaker than one of the comparator unisensory responses (Fig. 3C). The coupling of depression with response imbalance helps explain why these interactions were not observed previously in neonatal (Wallace and Stein, 1997, 2000) and naive adult animals (Wallace et al., 2004; Yu et al., 2010, 2013; Xu et al., 2012, 2015, 2017). Previous studies commonly used stimuli that minimized unisensory responses to maximize their potential for multisensory enhancement (Meredith and Stein, 1986); these also elicited more “balanced” unisensory responses.

The transition from a nonsignificant multisensory product to multisensory depression that was exposed by increased unisensory imbalance is illustrated by the exemplar in Figure 4. This neuron was tested with visual and auditory stimuli of varying intensities. Those that produced two unisensory component responses of near equality, and were thus balanced, yielded a multisensory response roughly equivalent to the maximum unisensory response (i.e., a nonsignificant multisensory product). However, when stimuli produced imbalanced unisensory comparator responses, the multisensory product was less than the strongest of them alone (i.e., depression) and fell somewhere between the two. In short, when the unisensory inputs were balanced in naive neurons, the multisensory response was equivalent to the “strongest” unisensory response, and the null hypothesis that the neuron was responding to one input could not be rejected. This made it appear as if there had been no multisensory computation engaged. Only when there was substantial unisensory response imbalance was the competition revealed so that the null hypothesis could be rejected.

Figure 4.

Increasing unisensory imbalance revealed a naïve neuron's native state in which congruent cross-modal cues are treated as competitors and yield response depression. Shown are responses of a naïve neuron to visual and auditory stimuli of different intensities that produced three levels of response imbalance. Top row, The impulse rasters (left) and summary histograms (right) show that the unisensory response magnitudes differed very little. Combining the visual and auditory stimuli produced a multisensory response product that was not significantly different from the best unisensory comparator response (auditory). Response magnitudes (impulses/trial): V = 6.1, A = 7.55, VA = 8.33; UI = 11%, ME = 10% (Mann–Whitney U test, p = 0.287). Second row, The responses to the visual and auditory stimuli differed greatly, and their combination produced a response 15% below the best unisensory response. V = 2.25, A = 7.15, VA = 6.1; UI = 52%, ME = −15% (Mann–Whitney U test, p = 0.071). Bottom row, The visual and auditory response differences were greatest here, as was the level of depression produced by their combination (−25%). V = 0.6, A = 6.55, VA = 4.9; UI = 83%, ME = −25% (t test, p = 0.014). *p = 0.0198. Conventions are the same as in Figure 2.

Cross-modal sensory training produced a shift in the multisensory transform

Multisensory enhancement capabilities can be developed in naive adults when they are given repeated exposure to spatiotemporally congruent cross-modal stimuli, even while the animal is anesthetized (Yu et al., 2010, 2013; Xu et al., 2017). In the present study we compared the multisensory responses from the naive cohort of animals after they were provided with such cross-modal experience (i.e., the trained cohort) with those obtained from the normal and naive cohorts. With certain exceptions, the results show that neurons in the trained cohort of animals had become very much like those in normal animals.

Consistent with previous reports (Yu et al., 2010, 2013), the visual and auditory RFs of neurons in these animals had not contracted to normal dimensions. Nevertheless, they did show normal levels of multisensory overlap/alignment. Their visual response magnitudes were slightly (yet significantly) lower than those of the other two groups (visual = 4.5 ± 3.8 impulses/s; p = 0.037), but their auditory response magnitudes were comparable to those in normal animals (auditory = 3.5 ± 3.3 impulses/s, p = 0.99).

Interestingly, neurons in the trained animals also showed significantly more balanced responses than when in the naive condition (Fig. 5A); in fact, the incidence of neurons having balanced responses after cross-modal training (UI < 50% in 76/92 = 83%) was very close to that found in the normal condition (78%). Nevertheless, the mean UI score in the trained group (29.3%) was slightly lower than normal (p = 0.0355) to the same standard stimuli, suggesting that one of the consequences of cross-modal experience is to reduce the level of inequality between a neuron's unisensory sensitivities. However, the systematic manipulation of the stimuli in a large number of neurons that would be required to fully evaluate this hypothesis was not performed here.

Figure 5.

Sensory training develops normal multisensory integration capabilities in the naïve animal. A, After naïve animals experience the sensory training procedure, the mean level of unisensory imbalance in their multisensory responses decreased below that of even the normal animal. B, An inverse relationship between ME and UI identified in the normal and naïve cohorts was also identified in the trained cohort. The slopes (C) and the intercepts (D) of the regression lines fitting the relationship between ME and UI was quantitatively similar in the trained and normal conditions, but were significantly different from the line fit for the naïve condition. Nevertheless, some negative ME scores like those in the naïve condition remained in the trained animals. Plotted are the means and 95% confidence intervals for the slope and intercept parameters for the three groups. *p < 0.05.

Neurons in the trained group also showed the same inverse relationship between ME and UI as did those in normal animals (Fig. 5B; slope = −1.54, intercept = 115, R² = 0.424; p = 1.24E−12), and there were no significant differences in the slopes or intercepts of the best-fit regression line. However, neurons in both the trained and normal groups were significantly different from those in the naive condition (p < 0.001; Fig. 5C). It appeared as if the cross-modal exposure paradigm had rendered the naive animals normal in this respect.

The default multisensory computation is competitive

There are a number of underlying mechanisms that can yield the multisensory products observed in neurons of the naive animal, among them mutually inhibitory interactions between the different sensory modalities (Cuppini et al., 2018). According to this mechanism, unisensory signals in the naive condition are processed as if they refer to different events. This is similar to the computation used to process inputs from spatially noncongruent cross-modal cues in normal adults. Thus, it might be expected that spatially congruent and noncongruent cross-modal cues could elicit similar multisensory products in the naive condition.

To test this possibility, a sample of multisensory neurons in the normal (n = 90), naive (n = 62), and trained (n = 47) groups were tested with spatiotemporally congruent and spatially noncongruent visual-auditory cues. In noncongruent (i.e., spatially disparate) tests the visual stimulus was placed within its RF and the auditory stimulus was placed outside its RF (Fig. 6). In the normal animal, such disparate cues reliably evoked either nonsignificant (45/90 = 50%) or depressed (29/90 = 32%) multisensory products, and almost never yield enhanced responses (Fig. 6A). In contrast, in the naive animal these different spatial configurations did not reliably elicit different products. The percentage of nonsignificant to depressed products was 70%/24% when naive neurons were tested with spatially congruent cues and 74%/19% when tested with noncongruent cues. Even the mean magnitude of the depression in these two conditions were similar (congruent: −8.7 ± 29.6%, disparate: −2.0 ± 23.8%; paired t test, two-tailed, p = 0.17).

Figure 6.

Stimulus configuration does not affect the multisensory interaction in the naïve condition. Plotted is the magnitude of ME in individual neurons as a function of the cross-modal spatial configuration presented in each of the three different conditions examined. Each circle represents a single neuron, and the diagonal line is the line of equality between the results obtained in response to spatially congruent and noncongruent stimulus configurations. A, In the normal condition a neuron responded to visual and auditory stimuli that were within their overlapping RFs (“spatial congruence”) with enhanced multisensory responses. However, when one of those stimuli was presented outside its RF (spatial disparity or “spatial noncongruence”), the same neuron's response was usually depressed, revealing the competition. B, In the naïve condition, (i.e., no visual-auditory experience) however, there was no relationship between a neuron's multisensory response and the spatial configuration of the visual and auditory stimuli. Both congruent and noncongruent configurations resulted in similar degrees of competition as indicated by most circles clustering around the line of equality. C, Instantiation of the normal condition was achieved with the cross-modal training program.

A neuro-computational model of the default computation

The above results suggest that a competition exists between the two sensory channels in the naive state that is overcome by experience with spatiotemporally concordant cross-modal cues. This transition could be explained by an associative learning rule that strengthens convergent cross-modal tectopetal projections, thereby allowing them to override a default inhibitory network. To test the plausibility of this interpretation, a neuro-computational model containing these features was implemented and tested after it was given 5,000,000 exposures to auditory-alone stimuli (simulating dark-rearing, i.e., the naive condition), followed by 5,000,000 exposures to simulated congruent visual-auditory pairs.

To compare the present data with those predicted by this model, the relationship between ME and UI for the naive condition was presented alongside the model predictions in Figure 7A. The empirically-recorded visual and auditory responses were normalized by scaling between 0 and 1 by the maximum response for direct comparison to the model predictions (see Materials and Methods). Consistent with the empirical data, the competition implemented by the model in the naive state produced a nonsignificant interaction when the stimuli were balanced in effectiveness, but produced multisensory depression when they were imbalanced. Even with randomized input parameter values the model was able to accurately predict that neurons in the naive condition would exhibit a high incidence of “unbalanced” samples. And in fact, its predictions had excellent fidelity in their match to the individual neurons collected from the naive cohort (Fig. 7B,C). Unisensory responses were predicted with an R² = 0.993, multisensory responses with R² = 0.857. This comparison matched model units from 10,000 simulations whose unisensory responses closely resembled the (normalized) empirical unisensory response magnitudes. Note that overall of the matches are nearly perfect with the exception of a single point which, because of its extreme value, would have required a larger pool of random simulations to produce a closer match.

Figure 7.

The neuro-computational model predictions closely matched the empirical results from the naïve condition. A, The neuro-computational model predicted the inverse relationship between ME and UI that was observed in the physiological data. Results displayed are 500 U from the 10,000 model simulations whose unisensory response magnitudes most closely resembled those in the empirical sample. B, The model simulations whose unisensory response levels most closely matched those observed for each empirical sample were selected from a pool of 10,000 simulations. Plotted are the visual and auditory responses (normalized units, X and + symbols, respectively) of the empirical sample (x-axis) versus those of the closest-matched model unit (y-axis). Note that the symbols fall on the line of equality showing the close match. C, Shown is the fit between the multisensory responses of each empirically-recorded neuron (x-axis, normalized) and its prediction from the model (y-axis). Note the close model-empirical match, with symbols clustering around the line of equality.

As illustrated in Figure 8, the transition from multisensory competition to multisensory facilitation implemented by the experience-driven learning rule also accurately predicted the empirical data. In the model, multisensory experience causes noncompetitive and convergent cortico-collicular inputs to become strengthened by a modified Hebbian rule, thereby overcoming the default competition. Figure 8A illustrates the relationship between ME and UI predicted by the model (R² = 0.998), whereas Figure 8, B and C, illustrates the high neuron-by-neuron predictability of the model obtained from 10,000 simulations (R² = 0.631). The slight discrepancy in the model prediction (stronger multisensory responses at higher levels of unisensory efficacies) of the empirical data reflects the likelihood that the trained dark-reared cohort had not fully achieved the normal multisensory enhancement levels at this training duration (Yu et al., 2010).

Figure 8.

The neuro-computational model predictions closely matched the empirical results for the trained condition. Repeated exposure to concordant visual-auditory stimulus pairs trained convergence noncompetitive visual and auditory connections, leading to the development of multisensory enhancement capabilities in the neurons of these “VA trained” animals. A, The inverse trend in the VA Trained condition between ME and UI resembles that of the normal animal. B, C, There is also a good match between the model predictions of the (normalized) unisensory and multisensory response magnitudes of the empirical sample. Conventions are the same as in Figure 7.

Discussion

The ability of SC neurons to synthesize information across sensory modalities typically develops gradually during early postnatal life (e.g., postnatal weeks 4–12 in cat) as the animal acquires experience with congruent cross-modal stimuli (Wallace and Stein, 1997, 2000; Stein, 2005; Stein et al., 2014). If this experience is blocked by restricting the animal's experience with particular modality pairings (Wallace et al., 2004; Yu et al., 2010, 2013; Xu et al., 2014, 2015), or the covariance between these stimuli is disrupted (Xu et al., 2012), or the essential circuit for incorporating this information is compromised (Jiang et al., 2007; Rowland et al., 2014), this normal maturational transition is delayed until these conditions are resolved and appropriate experience is obtained (Yu et al., 2010; Xu et al., 2017). Traditionally this maturational progression in physiology, behavior and perception has been viewed as beginning from a default or naive state in which there is no multisensory integration capability, and transitioning to a mature state in which this capability has become evident (Bremner et al., 2012; Stein et al., 2014). In this sense, multisensory development has been conceived as similar to the development of other functional sensory capabilities, such as direction selectivity and velocity preference, which are initially absent and develop in an experience-dependent fashion (Aslin et al., 1981).

However, the present findings are not consistent with the idea that multisensory development proceeds in the same way as do the feature extraction capabilities of individual senses (i.e., from absent to present). Rather, they reveal a naive state of intersensory relations in which there exists a competition between the senses and that does not respect cross-modal spatial relationships. In this state all cross-modal stimulus configurations, both spatiotemporally disparate and congruent, can depress a neuron's response. The magnitude of this depression is most visible when the individual components of the cross-modal stimuli have a different impact on the target neuron (i.e., are imbalanced). In addition, SC neurons in naive animals were more likely than normal animals to have greater imbalance in their sensitivities to their visual and auditory inputs, amplifying the effects of this competition. These three features of naive multisensory neurons (lack of enhancement, presence of competition, greater imbalance) provide a more complete picture of the multisensory “impairment” in sensory restricted conditions than previously reported. Understanding this initial state of multisensory processes is crucial to understanding its normal trajectory as well as the etiology of anomalous development of multisensory information processing in autism spectrum disorder (Brandwein et al., 2013; Stevenson et al., 2014; Foxe et al., 2015), schizophrenia (Williams et al., 2010), and dyslexia (Hairston et al., 2005; van Laarhoven et al., 2016), as well as in patients in which early multisensory experience was compromised (Putzar et al., 2007, 2010). They require a very different perspective of multisensory development: that a preexisting state of competition must be overcome to yield functional enhancement.

Although this default computation may seem counterintuitive from previously-held perspectives, competition is a common epigenetic mechanism in the developing nervous system, and is crucial to the formation of stratified distributions of neuronal subtypes, sensory topographies, and cortical territories. As a default state, this competitive dynamic is consistent with the functional role of the SC in target selection and orientation behavior. The motor map of the SC, essential to accurate SC-mediated orientation behavior, is at least partially operational even before eye opening (Stein et al., 1980), and because orientation must select one target at a time, it is logical for the naive circuit to process any two signals as mutually exclusive targets “competitors” regardless of whether they are derived from the same, or different, senses (and despite the topographic overlap in the sensory maps and neuronal receptive fields). This functional organization is adaptive in the sense that it allows for efficient suppression of spurious noise and weak signals in favor of those that are of importance. However, it is not adaptive for real world conditions as it fails to use common information among the senses. Only when the circuit has learned that covarying signals from different senses likely derive from a common event is this initial competitive dynamic overridden and replaced by a cooperative one. Whether this transition is characteristic of multisensory neurons of other structures remains to be determined.

It is interesting to note that the presence of the competitive dynamic was not noted in earlier studies of SC multisensory maturation. This is likely due to the nature of the stimuli presented. Previous studies probing this maturation (in the neonate or sensory-restricted adult) used cues that were weakly effective in an effort to maximize the probability of observing multisensory enhancement as soon as it appeared (Wallace et al., 2004; Yu et al., 2010, 2013; Xu et al., 2012, 2015, 2017). This is because enhancement is maximized, and most apparent, when weakly effective modality-specific stimuli are combined (Meredith and Stein, 1986; Stein et al., 2009). Of course, this strategy also minimized the incidence of imbalanced unisensory responses, which is necessary to expose the competitive dynamic. This is important to keep in mind whenever exploring the consequences of any condition in which multisensory enhancement capabilities are believed to be compromised.

In addition to developing the cooperative dynamic, SC neurons must calibrate their sensitivities to the different sensory inputs before their normal multisensory integrative capabilities can be achieved. The presence of an imbalance in the impact of stimuli from different modalities is not uncommon, and has been detailed in normally-developed adult SC neurons (Miller et al., 2015), and in human perception (Otto et al., 2013). However, in the normal circumstance there is a reduction in the magnitude of enhancement as imbalance increases, not a progressive increase in the level of response depression. These two inverse trends relating ME to UI cross qualitatively different boundaries. Nonetheless, it is possible that similar logic underlies an inverse trend in both cases that, like the default competition, reflect the functional role of the SC in target selection, orientation, and attentional networks. If the efficacy of an input to a neuron reflects the likelihood of a potential target being at that neuron's location within the sensory map, imbalanced cross-modal inputs to a multisensory neuron represent conflicting signals regarding the preference of that location for an orientation response. This conflict makes the neuron's location less preferred over an alternative with balanced inputs.

These biological observations can be explained in more mechanistic terms by the neuro-computational model presented here that posits the existence of a default competitive circuit within the SC. In this model, topographically-organized inputs excite specific units and functionally suppress the sensory signals carried by all other inputs in the network. Crucially, this inhibition extends across sensory modalities. This inhibitory mechanism explains the inverse relationship between ME and UI. In the default state, cross-modal inputs, even when convergent onto a common target neuron, effectively compete with one another through mutual inhibition. This effect produces minimal multisensory inhibition in the activity elicited in the SC output unit when the inputs are balanced in efficacy because both modalities attenuate the excitation of the other one, but not completely, and the SC unit receives a reduced excitatory component from both modalities. The inhibition is maximal when the inputs are imbalanced, because the most effective modality is able to inhibit completely the excitatory input generated by the other one but at the same time is affected by the presence of the other modality, resulting in a reduced excitation reaching the SC unit.

In the model, an extra component develops to “bypass” this default competitive circuit via a special descending projection from AES (Jiang et al., 2001, 2002, 2006; Alvarado et al., 2009) that is strengthened via covariant multisensory experience and adapted Hebbian mechanisms. Unlike the inputs in the competitive circuit, these “cooperative” inputs do not extend mutual inhibition across sensory modalities (but do suppress the competitive inputs). Once strengthened, the influence of these AES-derived inputs comes to dominate SC responses and effectively serve as a binding signal across the SC sensory modalities. The model predicts that deactivation of the AES-SC projection would eliminate this cooperation, re-establishing the competition between the input modalities. Interestingly, this prediction already appears to have substantial empirical support from data obtained from SC neurons in normally reared animals (Alvarado et al., 2008).

These developmental events have been examined in animals deprived of cross-modal experience with the assumption that these events replicate those occurring in early life. Although previous studies demonstrate that many of the same requirements, time courses, and endpoints are very similar, the naive adult brain and the naive neonatal brain differ in many ways. Thus, there may be dissimilarities in how this maturational event progresses at different ages that are yet to be identified. Furthermore, the dark-reared animal is taken here as the model for naive-rearing. Other complementary manipulations that maintain animals in a state naive to particular types of multisensory experience (e.g., rearing in omnidirectional sound rooms, rearing without covariant cross-modal experience) would likely yield similar outcomes, but this has not yet been demonstrated.