Cortex Mediates Multisensory But Not Unisensory Integration in Superior Colliculus

Abstract

Converging cortical influences from the anterior ectosylvian sulcus and the rostral lateral suprasylvian sulcus were shown to have a multisensory-specific role in the integration of sensory information in superior colliculus (SC) neurons. These observations were based on changes induced by cryogenic deactivation of these cortico-SC projections. Thus, although the results indicated that they played a critical role in integrating SC responses to stimuli derived from different senses (i.e., visual–auditory), they played no role in synthesizing its responses to stimuli derived from within the same sense (visual–visual). This was evident even in the same multisensory neurons. The results suggest that very different neural circuits have evolved to code combinations of cross-modal and within-modal stimuli in the SC, and that the differences in multisensory and unisensory integration are likely caused by differences in the configuration of each neuron's functional inputs rather than to any inherent differences among the neurons themselves. The specificity of these descending influences was also apparent in the very different ways in which they affected responses to the component cross-modal stimuli and their actual integration. Furthermore, they appeared to target only multisensory neurons and not their unisensory neighbors.

Introduction

The superior colliculus (SC) uses information from various senses to effect orientation and localization behaviors (Stein and Meredith 1993; Jiang et al., 2002) (for discussion, see Calvert et al., 2004). Although purely unisensory SC neurons contribute to this function, the capacity of multisensory neurons to integrate the information arriving on modality-specific input channels makes them unique in this regard. The synthesis of independent forms of information derived from the same event is presumed to increase its salience, as evidenced by the fact that multisensory stimuli are more likely to be detected and elicit rapid orienting responses (Stein et al., 1988; Jiang et al., 2002). Likely underlying these behavioral effects at the single-neuron level is response enhancement, wherein a neuron's multisensory response significantly exceeds its response to either unisensory component stimulus and can, in some cases, even exceed their arithmetic sum (Meredith and Stein 1986a; Perrault et al., 2003; Stanford et al., 2005; Stanford and Stein 2007).

The unique nature of multisensory integration is underscored by a previous study demonstrating that SC multisensory neurons integrate within-modal cues quite differently from cross-modal cues (Alvarado et al., 2007). Thus, rather than additivity or superadditivity, simultaneous presentation of two excitatory stimuli from the same modality typically yields a response that is subadditive, being similar to, or less vigorous than, that to the more effective of the stimulus pair. From a functional perspective, it seems adaptive for the products of within-modal and cross-modal influences to be different (see Discussion). From a mechanistic perspective, a fundamental understanding requires identification of the neural architecture underlying the capacity of the same neuron to integrate excitatory influences differently depending on their source.

There is evidence to suggest that differences between unisensory and multisensory integration are the result of a special relationship that has evolved between neurons in specific regions of cortex and their SC target neurons, a relationship tailored specifically to support the process of multisensory integration. Accordingly, previous studies have shown that depriving the SC of unisensory inputs from subdivisions of the anterior ectosylvian sulcus (AES) abolishes multisensory enhancement without substantially altering modality-specific component responses (Wallace and Stein, 1994; Jiang et al., 2001). Thus, the descending projections would appear to play a very specific modulatory role, one necessary for promoting the additive and superadditive interactions typical of multisensory enhancement, but are not responsible for the primary modality-specific drive.

The present study was designed to determine whether, in fact, the modulatory influence of the AES cortico-SC projection is multisensory-specific and the principal substrate for differences between unisensory and multisensory integration in the SC. To do so, we examined the influence of temporary removal of cortical influence on unisensory and multisensory integration in the same multisensory neurons, as well on unisensory integration in neighboring unisensory neurons. The experiments represent a critical test of the notion that a unique anatomical substrate has evolved expressly for the purpose of combining information across multiple senses and, in doing so, endeavor to illuminate the essential distinctions between within-modal and cross-modal processing architectures.

Materials and Methods

All animal protocols used were in accordance with the Guide for the Care and Use of Laboratory Animals (National Institutes of Health Publication 86–23) and were approved by the Animal Care and Use Committee of the Wake Forest University School of Medicine, an Association for Assessment and Accreditation of Laboratory Animal Care accredited institution.

Surgical implantation

Experiments were performed in three adult cats after using the same surgical procedures as described previously (Jiang et al., 2001). Briefly, each animal was anesthetized via ketamine hydrochloride (30 mg/kg, i.m.) and acepromazine maleate (3–5 mg/kg, i.m.), intubated through the mouth, and then maintained with isoflurane (0.5–3%). Body temperature was maintained with a heating pad, and the head was held by a stereotaxic head holder. A craniotomy exposed the cortex overlying the SC and a hollow stainless steel cylinder was attached over it using surgical screws and dental cement. This provided access to the SC, and served as an atraumatic head holder during later recording experiments (McHaffie and Stein, 1983). AES and the adjacent rostral lateral suprasylvian sulcus (rLS) were exposed via a second craniotomy. The dura was opened and the sulcal walls of the AES and rLS were gently separated. Two cooling coils (7–11 × 3–7 mm), fashioned from loops of 19 gauge stainless-steel hypodermic tubing (inner diameter, 0.032 inches), and shaped appropriately for the sulci, were inserted. One coil was placed in the posterior part of the AES, where the auditory and visual modality-specific subregions are located. The other coil was placed in the rLS (Jiang et al., 2001). The area over the cooling coils was packed with gelfoam, the stems were fixed to the skull and the craniotomy was sealed. After the surgery, analgesics (butorphanol tartrate, 0.1–0.4 mg/kg/6 h) were administered as needed and antibiotics were given for 7–10 d (ceftriaxone, 20 mg/kg/bid). The initial recording session began 1–5 d after terminating antibiotic treatment.

Recording

Recording methods were also similar to those used previously (Alvarado et al., 2007). Anesthesia was induced with ketamine hydrochloride (30 mg/kg, i.m.) and acepromazine maleate (3–5 mg/kg, i.m.) followed by the paralytic pancuronium bromide (0.3 mg/kg). The animal was maintained with continuous ketamine hydrochloride (4–6 mg/kg), paralytic (0.1–0.2 mg/kg/h), and 5% dextrose Ringer solution (3–6 ml/h) infusion. It was artificially respired at a rate and volume necessary to keep end tidal CO₂ at ∼4.0%, and body temperature was maintained at 37–38°C using the heating pad. The values of CO₂ as well as blood pressure were monitored continuously using the Digital Vital Signs Monitor VetSpecs VSM7. The head was held via the atraumatic head-holder, the right pupil was dilated with ophthalmic atropine sulfate (1%), and a contact lens corrected refraction to focus the eye on a tangent screen. An opaque lens occluded the left eye. At the onset of a recording session, the recording chamber was cleaned and disinfected, and a glass-insulated tungsten electrode (tip diameter, 1–3 μm; impedance, 1–3 MΩ at 1 kHz) was lowered into the left SC and advanced in small steps by a hydraulic microdrive. Visual–auditory multisensory and unisensory visual neurons were sought using “search” stimuli as described below. Each neuron's responses were amplified, digitized via a window discriminator, displayed on an oscilloscope, and played through an audio monitor. Neuronal location was also recorded. At the end of the session, the electrode was removed, drug administration was discontinued, and the cylinder was disinfected and sealed. When stable respiration and locomotion were reinstated, the cat was returned to its home cage.

Neuronal search paradigm, receptive field mapping, and sensory tests

Visual search stimuli included moving and stationary flashing spots and bars of light projected onto a tangent screen located 45 cm in front of the animal.

Auditory search stimuli were broad-band (20–20,000 Hz) noise bursts, clicks, and snaps. Visual RFs were mapped with moving light bars. Auditory RFs were mapped with broad-band noise bursts 10 dB above threshold delivered by any of 16 hoop-mounted speakers separated from one another by 15° and from the animal by 15 cm. The speakers were mounted on a rotating hoop (radius, 30 cm), and the animal was positioned such its interaural axis was aligned with the axis of rotation of the hoop and perpendicular to the visual axis. Because the entire hoop could rotate around the interaural axis to achieve different azimuthal positions, the elevation could be examined without obstructing the animal's vision or blocking the visual stimulus (Meredith and Stein 1986a,b). RFs were recorded on standardized maps (Stein and Meredith 1993).

Visual test stimuli consisted of either one or two moving or stationary light bars placed within the best area of the RF (generally near the RF center). The bars (0.11–13.0 cd/m2 against a 0.10 cd/m2 background) were generated by a Silicon Graphics workstation and projected by a Barcodata projector onto a tangent screen that subtended ∼60° of visual angle in each hemifield. The stimuli could be moved in all directions across the RF at amplitudes of 1–110° and speeds of 1–400°/s.

Auditory test stimuli were computer generated white noise bursts within the best area of the neuron's auditory RF. The duration of the auditory stimuli ranged from 10 to 50 ms at intensities of 52–70 dB sound-pressure level (SPL) against a background SPL of 51.4–52.0 dB.

Cortical deactivation

Because inputs from the adjacent rLS play a similar role in some SC neurons (Jiang et al., 2001), both rLS and AES areas were deactivated simultaneously. Cryogenic deactivation was induced by circulating refrigerated water (0°C) through the cooling coils, an adaptation of a method developed by Lomber et al. (1999). Cryogenic deactivation has proved to be an effective, reversible method for studying cortico-SC influences on overt behavior (Jiang and Stein, 2003; Lomber et al., 2007; Malhotra and Lomber, 2007) and single-neuron responses (Ogasawara et al., 1984; Clemo and Stein, 1986; Meredith and Clemo, 1989; Wallace and Stein, 1994; Jiang et al., 2001, 2002). Each neuron's deactivation tests lasted 10–30 min, after which cortex was reactivated by circulating warm water (36–38°C) through the coils for at least 5 min

General testing paradigm

Each visual–auditory (VA) neuron was tested with three modality-specific stimuli: visual 1 alone (V₁), visual 2 alone (V₂), and auditory (A) alone; and three combined stimuli: visual 1 and visual 2 (V₁ -V₂), visual 1 and auditory (V₁-A), and visual 2 and auditory (V₂-A). To evaluate the neuronal responses three levels of stimulus effectiveness and a single stimulus duration for each modality-specific stimulus component were determined empirically for each neuron. These levels, which roughly spanned the neuron's dynamic range, were determined by manipulating the stimulus intensity of the modality-specific stimuli. To determine the lowest value, stimulus intensity was lowered and then raised until the minimum value that resulted in a reliable response was determined. To determine the upper level, stimulus intensity was raised progressively until an intensity level was found at which the response no longer increased. The intensity was then lowered and raised to confirm this intensity. The third value was the approximate midpoint between the upper and lower intensities. Similar procedures were used to find the three levels of visual effectiveness for unisensory neurons (Perrault et al., 2005; Stanford et al., 2005).

The same three levels of stimulus effectiveness were used in each of the experimental conditions (control, cortical deactivation and cortical reactivation), resulting in nine (3 × 3) combinations (blocks). In any given block, the same stimulus parameters (effectiveness and duration) defined individual modality-specific and combined stimuli. Eight repetitions of each stimulus were used and the stimulus conditions were interleaved in pseudo-random manner with 8–20 s between trials. Generally the paired within-modal and cross-modal stimuli were presented simultaneously. However, in several cases, a neuron's auditory and visual latencies were so different that a 50 ms delay (V before A) was required to ensure a robust multisensory interaction (Meredith et al., 1987). The resulting number of trials in each multisensory neuron was 432 (eight trials by six stimuli by nine blocks). A similar strategy was followed with unisensory visual neurons [V₁ alone; V₂ alone and V₁ -V₂ were interleaved in pseudorandom manner for a total of 216 trials (eight trials by three stimuli by nine blocks)].

Data analysis

The activity of individual neurons before and during cortical deactivation was quantified using three primary measures based on stimulus-driven firing rate. These included (1) the enhancement index, (2) a contrast index, and (3) the Z score as detailed below. Each neuron served as its own control during these comparisons, and to avoid misinterpretation, neurons that did not return to their predeactivation state after cortical reactivation were excluded from consideration.

Estimation of mean firing rate.

This was determined for each stimulus based on the mean number of impulses evoked during a common time window for that neuron. The time window bracketed the longest response train (beginning at stimulus onset). Spontaneous rates were measured for 1 s before the onset of the first stimulus during each set of trials and then normalized for the time window in which responses were counted (Jiang et al., 2001; Jiang and Stein, 2003). Although these rates were generally low, they were subtracted from stimulus-evoked responses.

Enhancement index.

Responses were analyzed to determine whether there was a statistically significant difference in the number of impulses evoked by the combined stimulus compared with the most effective modality-specific component stimulus. The relative magnitude of a significant difference was then evaluated using the enhancement index according to the formula (Meredith and Stein 1983) enhancement index = [(CM − SM_max)/(SM_max)] × 100, where CM equals the mean number of impulses/trial evoked by the combined stimuli (i.e., either visual–auditory or visual–visual) and SM_max equals the mean number of impulses per trial evoked by the most effective modality-specific stimulus.

Contrast index.

To facilitate comparison among neurons, the differences between responses to combined and modality-specific component stimuli were also quantified using a standard contrast index according to the formula (Motter, 1994) contrast index = (CM − SM_max)/(CM + SM_max).

This index distributes values in the range of −1 to 1, with zero corresponding to a combined stimulus response that is identical to the best modality-specific response. Positive values correspond to response enhancement and negative values indicate depression (i.e., combined stimulus response less than the best modality-specific response.

Z score.

Each combined response was evaluated with respect to the null hypothesis of simple summation of the modality-specific component responses according to the procedure described by Stanford et al. (2005). Briefly, for each stimulus combination (e.g., V-A), the mean number of impulses in the response was compared with a distribution of expected means based on simple summation of responses to its modality-specific component stimuli (e.g., V, A). To generate the predicted distribution, all possible sums (V₁ -V₂ for within-modal tests; V₁-A or V₂-A for the cross-modal tests) were computed. Thus, a sample distribution of 64 (8 × 8) possible sums was created from which eight trials were randomly selected (without replacement) and averaged to create a predicted mean combined response. This sampling and averaging operation was repeated 10,000 times to build a reference distribution of mean combined responses. The actual mean combined response was then compared directly to the distribution of means predicted from summation and this relationship expressed as a Z score. Using a criterion of p < 0.5, combined responses were thus classified as superadditive (Z > 1.96), additive (Z of −1.96 to +1.96), and subadditive (Z < −1.96).

Using Statistica for Windows, release 7.0 (StatSoft, Tulsa, OK), the differences among neuronal profiles were assessed with a variety of statistical treatments, depending on whether the sample distributions met the assumptions of normality based on the Kolmogorov–Smirnov test of normality (i.e., mean impulses, enhancement indices of the combined responses, Z scores of the combined responses and contrast indices of the combined responses). When this assumption was met, t tests and ANOVAs were used. When the assumption was not met, Mann–Whitney, Kruskal–Wallis ANOVA, Spearman's rank R, and χ² were used. All of the data were expressed as mean ± SE and the statistical significance was set at the p < 0.05 level.

Results

The impact of cortical deactivation on the integration of cross-modal and within-modal information was evaluated in 70 sensory-responsive SC neurons. This included 45 multisensory (visual–auditory) neurons and 25 unisensory (visual) neurons, all located in the deep layers of the structure.

The general response properties of these neurons, their physiologically defined modality convergence patterns, and integrative profiles proved to be similar to those described previously (Stein et al., 1976; Meredith and Stein 1983, 1996; Wallace et al., 1998; Alvarado et al., 2007). Thus, the majority of multisensory neurons (n = 40; 89%) responded with significantly more impulses to a cross-modal stimulus combination than to the most effective of their modality-specific component stimuli (King and Palmer, 1985; Meredith and Stein, 1986a; Peck, 1996; Kadunce et al., 1997; Stein, 1998; Wallace et al., 1998; Jiang et al., 2001; Perrault et al., 2003; Stanford et al., 2005; Alvarado et al., 2007). In contrast, and consistent with previous findings (Alvarado et al., 2007), few multisensory neurons (5 of 40, 12.5%) or neighboring unisensory neurons (3 of 25, 12%) showed response enhancement to within-modal stimulus combinations.

Effects of cortical deactivation on multisensory neurons

Responses to cross-modal stimuli

For most multisensory neurons (77.5%, 31 of 40), temporary removal of influences from AES and rLS all but abolished response enhancement to cross-modal stimuli without eliminating the responses to either modality-specific stimulus component. Figure 1 shows an example of one such neuron.

Figure 1.

Cortical deactivation affects multisensory integration in an SC neuron. At the top are the visual (dark ovoid) and the auditory (light ovoid) receptive fields of this multisensory neuron with icons showing the positions of the visual and auditory stimuli. The moving bar of light (arrow) is indicated by the ramp and the broad-band noise by the square wave above the rasters. The brain photograph shows cortical areas AES and rLS. A–C, Below the receptive fields are shown the neuron's responses to the visual, auditory, and combined visual-auditory stimuli in control (A), cortical deactivation (B), and reactivation conditions (C). Responses are displayed in rasters, histograms and summary bar graphs at three levels of visual stimulus effectiveness. A, Note that multisensory enhancement occurred at every stimulus effectiveness level, but was greatest at the lower end of the neuron's dynamic range. Z scores showed that at this level the enhancement was superadditive, transitioning to additivity at the highest level. B, Cortical deactivation (area in dotted lines) eliminated multisensory enhancement, rendering SC multisensory responses additive at all stimulus effectiveness levels. Modality-specific responses were not significantly affected at lowest level of stimulus effectiveness, but the impact was more evident as the responses increase in intensity. C, Multisensory enhancement was reestablished by cortical reactivation. Impulse counts were taken within a 1 s window. **p < 0.01.

In the control condition (Fig. 1A), the visual-auditory stimulus combination produced a response that was significantly enhanced compared with that of the visual stimulus alone (the more effective of the modality-specific stimuli) and did so at each level (1–3; top to bottom) of visual stimulus effectiveness. Consistent with the principle of “inverse effectiveness” (Meredith and Stein, 1986a; Stein and Meredith, 1993), proportionate enhancement decreased (from 193 to 44%) as the effectiveness of the visual stimulus increased, a trend common to the majority (31 of 40, 77.5%) of the neurons examined. Paralleling the reduction in enhancement was a monotonic decline in the Z score, signaling a change in the multisensory interaction from significantly superadditive (levels 1 and 2, Z >1.96) to one for which it is not possible to reject the null hypothesis of additivity (level 2, Z <1.96).

The importance of cortical influences for multisensory enhancement is illustrated in Figure 1 B, which shows the same SC neuron's unisensory and multisensory responses when AES and rLS were deactivated via cryogenic blockade. Although multisensory responses persisted at each level of stimulus effectiveness, they were substantially reduced in magnitude and no longer significantly greater than those produced by the visual stimulus alone (i.e., there was no longer any multisensory enhancement). Likewise, the superadditive interactions associated with large multisensory enhancements were no longer apparent, with Z scores in the additive range and nearly identical for each of the three stimulus effectiveness levels. As shown in Figure 1 C, reactivation (warming) of cortex restored both multisensory enhancement and superadditivity to near predeactivation levels. This example illustrates the strongly disproportionate effect that deactivation has on multisensory as compared with unisensory responses, a result emphasized in previous studies (Jiang et al., 2001). However, the effect of removing cortical influences was not exclusive to the multisensory response. For example, close inspection of Figure 1 B, suggests that the visual responses were somewhat reduced in response to cortical deactivation.

The relative impact of cortical deactivation over the unisensory responses was more evident when the modality-specific stimuli were considerably more effective. Although cortical deactivation abolished significant multisensory enhancement and superadditivity in all levels of stimulus effectiveness, the impact of cortical deactivation on the visual responses (the best modality-specific component) was more apparent at the highest level of visual stimulus effectiveness (Fig. 1, compare A, B). Indeed, across the sample, significant decrements in the responses to modality-specific component stimuli were routinely observed for all but the weakest stimuli.

These data indicate that the cortical influence is multifaceted, affecting both the vigor of modality-specific responses, but independently, the computation that governs their integration within the SC. Thus, the reduction in unisensory responsiveness does not explain the loss of multisensory enhancement and superadditivity. Indeed, given the seemingly inviolable principle of inverse effectiveness (Meredith and Stein, 1986a; Stein and Meredith, 1993), one would predict enhancement to be more robust and superadditivity more likely during deactivation because of the fact that multisensory combinations are now composed of less effective modality-specific component stimuli. That this is not the case is quite evident in Figure 1; in the control condition, combinations of the weakest stimuli yield superadditive interactions (levels 1 and 2), whereas, during deactivation, stimuli of comparable or lower effectiveness clearly fail to do so.

Interestingly, the magnitude of the effect of cortical deactivation on the individual responses and on the interaction itself tended to move in opposite directions. While the effect of cortical deactivation on modality-specific responses became more pronounced as stimulus effectiveness increased (Fig. 1), its influence on the interaction waned. Thus, as interactions deviated from superadditivity toward additivity, the impact of cortical deactivation declined, a trend that has implications for considering the cortical influence on unisensory integration described below.

Responses to within-modal stimuli

The influence of cortical deactivation on stimulus integration was highly selective for cross-modal stimulus combinations. Thus, despite having the above-described effect on the vigor of responses to modality-specific stimuli (e.g., visual), cortical deactivation had little or no effect on the integration of within-modal stimulus combinations. This is evident in Figure 2, which illustrates the effect of cortical deactivation on the integration of within-modal stimulus pairs for the same neuron depicted in Figure 1. In each case, the stimulus designated V₁ was identical to that in the corresponding cross-modal tests shown above, presented at the same intensity (levels 1–3) and in the same position within the receptive field (RF). Rather than an auditory stimulus, V₁ was paired with a physically identical stimulus (V₂), but in another position within the same RF. The two visual stimuli were presented together in the neuron's receptive field to approximate the spatial configuration of the corresponding cross-modal pair.

Figure 2.

Cortical deactivation fails to affect unisensory integration in a multisensory SC neuron. Within-modal visual tests (Visual 1 + Visual 2) were conducted in the same multisensory neuron depicted in Figure 1. A, In the control condition, the neuron showed no response enhancement and an underlying subadditive computation at all levels of stimulus effectiveness. B, Although cortical deactivation reduced the sensory responses (revealing that some of the sensory drive came from cortex), the underlying subadditive computation showed little change. C, Responses were re-established by cortical reactivation. Conventions are the same as in Figure 1.

It has been documented previously (Alvarado et al., 2007) that unisensory integration is distinct from multisensory integration. For the neuron shown here, the control condition showed that, unlike cross-modal stimuli, within-modal stimuli failed to elicit response enhancement at any level of stimulus effectiveness. Statistically, combined-stimulus responses were no more robust than those to the more effective of the two visual stimuli presented alone and the interaction was significantly subadditive (Z < −1.96) at each of the three levels of stimulus effectiveness. Thus, in contrast to cross-modal pairs, even relatively weakly effective visual stimuli failed to produce response enhancement when combined.

Although deactivation of AES and rLS reduced the vigor of the responses to the individual visual stimuli, the nature of their interaction was mostly unaltered, as evidenced by similar values for the enhancement index and Z score. Thus, responses to within-modal stimulus pairs that were uniformly subadditive before deactivation remained subadditive during the cortical deactivation. Considered with respect to the trend observed for the cross-modal cases, the data suggest that the cortical influence is specific to the response enhancement that was induced by integration, and to the extent that within-modal stimuli fail to produce enhancement, it appears to be a multisensory-specific phenomenon.

Effects of cortical deactivation on unisensory neurons

To determine whether the influences of AES and rLS cortex on unisensory processes (in both modality-specific and within-modal test conditions) were specific to multisensory SC neurons, within-modal tests were also conducted in neighboring unisensory neurons, an example of which is shown in Figure 3. As in multisensory neurons, there was essentially no response enhancement resulting from the addition of the second visual stimulus regardless of the relative level of stimulus effectiveness. Likewise, the within-modal interaction here was subadditive at every level of stimulus effectiveness (Fig. 3A). However, in contrast to multisensory neurons, cortical deactivation did not demonstrably affect the vigor of the unisensory neuron's response to either component of the within-modal stimulus complex. These differential effects of cortical deactivation were often seen on neighboring unisensory and multisensory neurons that were ∼100 μm apart. This finding suggests that AES and rLS target multisensory neurons and not their unisensory neighbors.

Figure 3.

Cortical deactivation fails to affect unisensory integration in a unisensory visual SC neuron. The within-modal visual responses of a unisensory neuron were similar to those of the multisensory neuron in Figure 2. A, In the control condition, the within-modal response was similar to that of the individual component stimuli and was characterized by subadditivity at all levels of stimulus effectiveness. B, C, Cortical deactivation (B) or reactivation (C) had little effect on the modality-specific component responses or the combined within-modal response, and did not alter the underlying subadditive computation. Conventions are the same as in previous figures.

Population profiles

Effect of cortical deactivation on enhancement and integrative mode

Before cortical deactivation, significant response enhancement was observed for 76%, (91 of 120) of all cross-modal stimulus combinations. With deactivating cortex, relatively few instances of significant multisensory response enhancement (17.5%; 21 of 120) were retained (χ² = 90.51; df = 1; p < 0.00001) and the mean value of the enhancement index across the population was reduced from 99.96 ± 8.64%% to 34.51 ± 6.26%%. Figure 4A plots the relationship between responses to the combined stimulus (ordinate) and those to the most effective modality-specific stimulus component (abscissa), both before (open symbols) and during (filled symbols) cortical deactivation. Before deactivation, the majority of points fall above the line of equality, consistent with multisensory enhancement. During deactivation, multisensory responses were reduced disproportionately and, although still above the line of equality in most cases, now more closely approximated the best unisensory response (in the majority of cases being statistically indistinguishable).

Figure 4.

Population responses reveal that the computations used by multisensory SC neurons to integrate cross-modal information differ depending on response vigor and that all depend on functional inputs from cortex. A, Each SC neuron's mean response to the cross-modal stimulus combination is plotted against its mean response to the most effective modality-specific component stimulus in both control (open circles) and cortical deactivation (closed circles) conditions. Lines of best fit describe these relationships, and their formulaic descriptions are shown at the bottom of the graph. Note that cortical deactivation shifted the line of best fit toward unity. B, Here, the mean enhancement indices obtained experimentally were parsed into six levels according to the total number of impulses in the multisensory response regardless of the test condition. They were plotted against the predicted mean responses obtained by averaging and summing each neuron's component responses. Note that during the control condition the enhancement index was inversely related to the number of impulses evoked. During cortical deactivation, however, enhancement was rendered minimal and there was no change in the index as a function of impulse number. C, D, These same relationships were noted when plotting combined versus predicted response (C) and mean Z score as a function of predicted sum (D). In the latter case, the superadditive computation obtained when multisensory responses contained low-impulse numbers were lost when responses contained higher impulse numbers, but no superadditivity and no corresponding changes in Z scores were evident during cortical deactivation. E, A cumulative distribution of Z scores shows that ∼60% of the multisensory computations were superadditive in control conditions, but <12% of them were retained during cortical deactivation. They shifted downward to additivity.

As suggested by the examples above, the magnitude of multisensory enhancement was inversely related to the best component stimulus effectiveness and, accordingly, the impact of cortical deactivation was maximal when the component stimuli were minimally effective. This is shown quite clearly in Figure 4 B, which plots the averages of the enhancement index as a function of the sum of the responses to the component stimuli (a composite index of effectiveness) for both the control (open symbols) and deactivation (filled symbols) conditions. In agreement with previous studies (Meredith and Stein, 1986a; Stein and Meredith, 1993; Wallace and Stein, 1997; Jiang et al., 2001; Kadunce et al., 2001; Perrault et al., 2003; Stanford et al., 2005; Alvarado et al., 2007), enhancement values declined steeply with increasing stimulus effectiveness in the control condition. During deactivation, mean enhancement was uniformly low across all response levels, with no inverse effectiveness in evidence (Mann–Whitney U = 3564; Z = −5.94; p < 0.0001) Thus, the effect of cortical deactivation declined in direct proportion to the decline in multisensory enhancement.

The same trends are apparent in Figure 4, C and D, which relate the multisensory response to the sum of the responses to the modality-specific component stimuli. Again, consistent with earlier studies (Stanford et al., 2005; Alvarado et al., 2007), the control condition (open symbols) shows that superadditivity (mean Z score >1.96) predominated at low stimulus efficacies only, giving way to additivity as the predicted sum increased (Fig. 4D). During deactivation (filled symbols), the mean Z score approximated zero (additivity) at all stimulus efficacies, once again illustrating that the effect of cortical deactivation was effectiveness dependent. Overall, the mean Z score was significantly reduced (from 2.59 ± 0.41 to 0.37 ± 0.21; Mann–Whitney U = 3984; Z = −5.10; p < 0.0001) reflecting a significant change (X² = 38.46; df = 2; p < 0.00001) in the relative proportion of superadditive (55.8 to 11.7%), additive (39.2 to 78.3%), and subadditive (5.0 to 10.0%) interactions, as shown by the cumulative distributions of Figure 4E.

As shown for the example neuron (Fig. 2), comparable effects were not obtained for unisensory integration (Fig. 5). Even before deactivation, the majority of within-modal pairs failed to yield enhancement, with combined responses approximating the best modality specific response in most cases (Fig. 5A, open symbols). Predeactivation enhancement index values were near zero (or slightly below) for all stimulus efficacies (Fig. 5B, open symbols) with an overall mean of −8.39 ± 4.29. It is readily apparent from Figure 5, A and B, that cortical deactivation had little effect on the enhancement index, which shifted from −8.39 ± 4.29 (control) to −6.55 ± 6.17 (deactivation). Likewise, there was little or no effect on the unisensory integrative mode. Responses were generally subadditive before and during cortical deactivation, and the slight shift in the mean Z score (−3.03 ± 0.23 to −2.49 ± 0.29) was not statistically significant. Accordingly, the relative proportions of superadditive, additive, and subadditive interactions were similar before and during cortical deactivation (control, 2.5, 29.2, 68.3%; deactivation, 2.5, 44.2, 53.3%) as shown in Figure 5E. Interestingly, Figure 5D shows a substantial trend toward increasing subadditivity as a function of increasing component stimulus effectiveness, an effect that may provide some insight into the underlying mechanism.

Figure 5.

Population responses reveal that cortex contributes marginally to the vigor with which multisensory SC neurons respond to within-modal stimuli but not at all to crafting the underlying computation. Data were taken from the same neurons used in Figure 4. Here, however, the comparisons are for responses to within-modal stimuli. A, C, Note that the within-modal responses were not significantly different from the most effective (best) unisensory responses (A), and were significantly lower than the predicted sum of the responses to the component stimuli (C). Although the within-modal responses were affected by cortical deactivation, the effect was minimal. B, D, E, Thus, the enhancement index was near or below 0, regardless of the number of impulses obtained for responses in both control and cortical deactivation conditions (B), and Z scores reveal that the underlying computations in both conditions were generally subadditive (D) and their distributions were also very similar (E).

On the basis of the unisensory integrative profile either before or during deactivation, unisensory neurons were virtually indistinguishable from their multisensory counterparts. As shown in Figure 6, unisensory neurons demonstrated no response enhancement and generally subadditive interactions to within-modal stimulus pairs in the control condition. These were unaffected by cortical deactivation, which yielded insignificant changes in the means of the enhancement index (5.83 ± 3.81 vs 2.03 ± 3.29), Z score (−2.36 ± 0.26 vs −2.64 ± 0.21), and relative proportion of superadditive (2.7 to 0%), additive (30.7 to 32.0%), and subadditive (66.7 to 68.0%) interactions. However, as noted above, and shown in more detail below, unisensory neurons differed from multisensory neurons in that cortical deactivation had seemingly no effect on the vigor of their modality-specific responsiveness either, suggesting that the AES and rLS projections do not target unisensory neurons.

Figure 6.

Neither the vigor nor the computations used by unisensory SC neurons to integrate within-modal stimuli involve cortex. The within-modal comparisons are identical to those conducted for neurons in Figure 5. A, As in Figure 5, the within-modal responses were not different from their responses to the best unisensory component responses. B, Thus, no enhancement was noted. C–E, The within-modal response was substantially lower than the predicted sum of unisensory component responses (C), and the underlying computation was generally subadditive. However, the absence of any demonstrable effect of cortical deactivation suggests that these neurons do not receive sensory inputs from these cortices.

Residual multisensory enhancement during deactivation

As described above, during deactivation of AES and rLS, only a small percentage (17.5%) of multisensory interactions showed enhancement. Although individually, the majority combined responses, during deactivation, failed to meet the statistical criterion for enhancement (i.e., they were not significantly greater than the best modality-specific component response), the mean value of the enhancement index was positive (34%) and consistently above zero for all stimulus efficacies (Fig. 4B), indicating that, as a population, combined responses tended to exceed the best modality-specific responses. This residual effect is most clear when contrasted with the near 0% average levels of enhancement obtained either before or during deactivation for unisensory integration (Fig. 5B). To quantify this further, Figure 7 plots distributions of a simple contrast index relating the combined response to the best modality-specific response (see Materials and Methods) with a value greater than zero indicating a combined response that is nominally greater than the best modality-specific component response. Note that, even during deactivation, the majority of multisensory interactions yielded positive contrast values (Fig. 7A, filled bars) with a mean of 0.09 ± 0.02. In contrast, values for unisensory interactions, either for multisensory neurons (Fig. 7B) or unisensory neurons (Fig. 7C) were evenly distributed about zero, indicating that, on average, combined responses did not differ from the best modality-specific component responses.

Figure 7.

The contrast index reveals inherent differences between multisensory and unisensory neurons. A–C, The graphs illustrate the contrast index for the integration of cross-modal (A) and within-modal (B) information in multisensory neurons and within-modal (C) for the unisensory neurons. A, Although cortical deactivation has a greater affect on the integration of cross-modal information, even during deactivation the majority of cross-modal interactions yielded positive contrast values (filled bars). In contrast, values for within-modal interactions, either for multisensory neurons (B) or unisensory neurons (C), were evenly distributed about zero, indicating that, on average, combined responses did not differ from the best modality-specific responses.

Effect of cortical deactivation on the magnitude of modality-specific responses

The examples shown above (Figs. 1, 2) indicate that cortical deactivation reduced the strength of the modality-specific responses. Although this has not been routinely observed in previous studies (Wallace et al., 1994; Jiang et al., 2001), such studies have examined the influence of cortical deactivation on a much more limited range of responses in emphasizing stimulus conditions that produce the largest multisensory enhancements and accordingly, the weakest modality-specific responses. As noted above, reductions in the vigor of modality-specific responses are less readily apparent when these responses are already very weak. Figure 8, A and B, illustrates the effect of cortical deactivation on the modality-specific responsiveness of multisensory neurons for the entire sample and covering a broad range of stimulus efficacies. Here, the vigor of responses to the auditory (black circle) and visual (V₁, white circle; V₂, gray circle) stimuli presented during deactivation are plotted against the control responses. Note that most points fall below the line of equality, as the majority of responses to V₁ (70 of 120; 58.33%), V₂ (72 of 120; 60%), and the auditory (73 of 120; 60.84%) stimuli were significantly reduced during deactivation. The magnitudes of response reduction were 40.09 ± 4.14% for V₁, 42.83 ± 3.77% for V_2, and 42.11 ± 5.31% for the auditory stimulus. As shown in Figure 8B, the reduction ranged from ∼30% to 60% and depended on stimulus effectiveness; the weakest responses corresponded to the lesser declines, perhaps indicative of a floor effect to the cortical influence.

Figure 8.

Modifications of the individual modality-specific responses during the cortical deactivation. The individual modality-specific responses during the cortical deactivation and in the control condition were plotted together across all levels of stimulus effectiveness. A, During the cross-modal tests, the majority of the modality-specific responses in visual 1, visual 2, and auditory fell below the line of unity during the cortical deactivation. B, There was a slight variation in the percentage of reduction with the stimulus effectiveness. When the actual number of impulses was low, the amount of reduction was smaller, whereas at higher number of impulses the reduction was bigger. Conversely, no apparent modifications in the modality-specific responses were observed in the unisensory neurons. C, D, The responses were clustered around the line of unity and their best fits were indistinguishable from it (C), and the magnitudes of the modifications found in the responses were low (D).

In marked contrast to multisensory neurons, Figure 8, C and D, confirms that deactivating cortex had no effect on the responsiveness of unisensory neurons. The responses were clustered around the line of unity, and the best fit lines for stimuli V₁ and V₂ each had near unity slope and intercepts near zero. Eighty-eight percent (66 of 75) of the response to V₁ and 86.67% (65 of 75) to V₂ did not change after the deactivation of the cortex and, accordingly, Figure 8D shows that the mean change during deactivation hovered near 0% for all stimulus efficacies for both V₁ (5.34 ± 4.55%) and V₂ (−5.68 ± 4.50%).

The effect of deactivation on the absolute magnitude of the combined responses (data not shown) was as expected, given the data already shown. Briefly, the average multisensory response was degraded by 68.25 ± 2.73% and was significantly below the control response levels (Mann–Whitney U = 2665; Z=-8.45; p < 0.00001), ranging from 81.2 ± 3.19% for combinations of the weakest modality-specific stimuli to 30.41 ± 7.32% for the most effective of them (r = −0.72; p < 0.0001). The decrease for within-modal combinations was far more modest at 45.20 ± 5.04%% (U = 4252.5; Z = 5.48; p < 0.00001) and reflected closely the decreases in responses to the individual stimulus components (Fig. 8A,B). Consistent with the data shown in Figures 8, C and D, there was no effect on the combined responses of unisensory neurons; a cortical deactivation mean decrease of 1.84 ± 4.36% was noted, but this was not statistically significant.

Discussion

The present study demonstrates that AES and rLS play a multisensory-specific role in determining the products of sensory integration in the SC. The specificity of this role is apparent in the striking contrast between their influence over the integration of cross-modal (visual–auditory) stimuli and the absence of such influence over the integration of within-modal (visual–visual) stimuli. The data also reveal that, although these cortices can influence responses to the individual cross-modal component stimuli, it is independent of that exerted over their actual integration. In addition, the magnitude of the cortical influence changes depending on the effectiveness of the stimuli being coded. Last, the data reveal that neighboring multisensory and unisensory SC neurons receive at least some of their modality-specific input from entirely different sources.

Cortico-SC influence on integration is multisensory-specific

The principal finding of this study is that deactivation of AES/rLS inputs alters multisensory but not unisensory integration, even in the same SC multisensory neurons. These data, along with those of previous studies, point toward a very specific corticotectal mechanism that modulates the magnitude of response enhancement resulting from combinations of excitatory stimuli. By using a wide range of modality-specific stimulus intensities, multisensory interactions were observed ranging from strongly superadditive (large enhancements) to subadditive (little or no enhancement). This made it possible to observe that the impact of deactivating AES/rLS inputs is in direct proportion to the magnitude of multisensory enhancement. It has no effect on responses to stimulus combinations that do not yield multisensory enhancement.

This failure to demonstrate effects of deactivating these cortices on unisensory integration does no real violence to current modes of thought. As shown in a previous study (Alvarado et al., 2007) and confirmed here, there is an essential distinction between multisensory and unisensory integration. When integrating the information contained in multiple cues from different senses, multisensory SC neurons show enhanced responses that are equal to or greater than the sum of the responses to the individual modality-specific component stimuli (Meredith and Stein, 1983; King and Palmer, 1985; Peck, 1987; Wallace et al., 1996; Kadunce et al., 1997; Jiang et al., 2001; Diederich and Colonius, 2004; Perrault et al., 2005; Stanford et al., 2005). In contrast, this was rarely observed when these same neurons integrated multiple inputs from the same sense. Instead, most responses were found to be subadditive, with responses to within-modal pairs substantially lower than the sum of their component responses.

By diminishing the magnitude of multisensory enhancement, cortical deactivation all but abolished the major distinction between multisensory and unisensory integration. Nevertheless, one important and unexpected distinction remained: multisensory integration was not completely eliminated. Although it was true that for any given neuron, statistically significant multisensory enhancement was rare after cortical deactivation, enhancement index values were often nominally greater than zero, and this was reflected in the population averages. Accordingly, this residual enhancement is reflected in the fact that, even during cortical deactivation, multisensory interactions tended toward additivity. In contrast, with or without cortical influence, unisensory integration illustrated no such enhancement, and interactions became progressively more subadditive as the effectiveness of the component stimuli increased.

The differences between unisensory and multisensory integration during cortical deactivation are likely to reflect some fundamental differences in the circuit elements subserving them and the functions they support. For example, Alvarado et al. (2007) provided evidence for suppressive interactions between within-modal pairs suggestive of competitive rather than reinforcing effects of excitatory pairs from the same modality. As they note, mutual antagonism, even for visual stimuli restricted to the same receptive field, could aid in selecting among potential goals for an orienting movement (Li and Basso, 2005). However, cues from different senses that occur in close spatial and temporal coincidence are likely derived from a singular event. In this case, their synergistic effects would enhance the likelihood of orienting to the initiating event, and this has been shown to be the case in behaving animals (Stein et al., 1988, 1989; Wilkinson et al., 1996; Stein, 1998; Jiang et al., 2002, 2007; Van Opstal and Munoz, 2004). Analogous multisensory benefits have also been repeatedly demonstrated in human subjects (Engelken and Stevens, 1989; Perrott et al., 1990; Hughes et al., 1994; Frens et al., 1995; Corneil and Munoz, 1996; Goldring et al., 1996; Harrington and Peck, 1998; Frassinetti et al., 2002; Lovelace et al., 2003; Bolognini et al., 2004; Laurienti et al., 2004).

Also contributing to the cross-modal/within-modal difference is the simple fact that cross-modal stimulus combinations activate mostly independent afferent populations, whereas nearby within-modal stimuli are likely to activate partially overlapping sets of afferents. The fact that unisensory integration became increasingly subadditive as the two within-modal stimulus components became more potent, and therefore presumably activated increasingly overlapping populations of visual afferents, is consistent with this interpretation.

Independent effects of cortex on multisensory integration and modality-specific responsiveness

Inputs to SC from AES (and presumably rLS) appear to derive solely from unisensory neurons (Wallace et al., 1993). Nevertheless, previous studies indicate that the depressive effect on SC multisensory responses produced by removing their collective or individual influences is not simply caused by reducing their modality-specific excitatory drive. Rather, these studies suggested that the effect is specific to the way in which the modality-specific inputs are integrated by an SC neuron, and were most evident on multisensory response enhancement (Wallace et al., 1994; Jiang et al., 2001). Specifically, cortical deactivation rendered the responses to cross-modal stimuli statistically indistinguishable from those evoked by the most effective modality-specific component stimulus, and did so without substantially affecting the modality-specific responses themselves.

The present results suggest that, in fact, AES and rLS inputs have independent effects on these two aspects of SC activity. One of the novel findings here is that removal of these cortical inputs does reduce responses to modality-specific stimuli, and is consistent with the fact that these areas are known to supply modality-specific inputs to SC neurons (Wallace et al., 1993). The apparent discrepancy between the current results and previous findings is most certainly caused by the present use of an extended range of stimulus conditions that includes far more effective modality-specific stimuli. As noted in the Introduction, previous studies have generally limited multisensory stimulus configurations to combinations of near-threshold modality-specific stimuli (Wallace et al., 1994; Jiang et al., 2001), thus decreasing the probability of observing significant response decrements resulting from a floor effect in the postsynaptic response.

Although different from previous results in this respect, the current results do not alter the fundamental conclusion of previous studies that the influence of the cortical projection on multisensory integration is independent of its effect on modality-specific drive. Thus, it is important to emphasize that the reduction in modality-specific drive is not causal to the loss of multisensory enhancement. Indeed, in and of itself, a simple reduction in modality-specific drive would be predicted to increase the incidence of multisensory enhancement according to the well documented principle of inverse effectiveness (Meredith and Stein, 1986a; Stein and Meredith, 1993; Stein et al., 2004). Evidence for integration-specific descending control has implications for behavior, underlying circuitry, and constitutes a primary constraint for biologically inspired computational models of the multisensory interaction (Anastasio et al., 2000; Rowland et al., 2007).

Distinct sources of modality-specific input for multisensory and unisensory neurons

Previous studies suggest that the SC is a primary site of multisensory interaction (Wilkinson et al., 1996; Jiang et al., 2001; Jiang and Stein 2003), with projections to SC multisensory neurons from the AES derived from unisensory neurons (e.g., auditory and visual). Presumably, the same pattern is derived from rLS, but this remains to be determined. Before the current study, there were no theoretical or empirical grounds to suggest that these modality-specific AES projections did not also target unisensory SC neurons. However, the data seem unequivocal in demonstrating that, unlike their multisensory counterparts, unisensory SC neurons are unaffected by deactivation of these cortices. Along with demonstrating differences in the afferent sources of input to unisensory and multisensory SC neurons, this observation reinforces the notion that these cortical regions are specialized to subserve multisensory integration.

Summary

The present study provides evidence that neurons in AES and rLS play a multisensory-specific role in modulating the responsiveness of SC multisensory neurons. In their absence, SC multisensory integration fails to yield its characteristic response enhancement. In contrast, these cortices play no apparent role in unisensory integration. We suggest that these descending influences likely comprise the anatomical substrate for the normal development, maintenance, and perhaps the context-dependent plasticity of multisensory integration that is necessary to orient to objects in the real world.