Visual- and saccade-related signals in the primate inferior colliculus

Abstract

The inferior colliculus (IC) is normally thought of as a predominantly auditory structure because of its early position in the ascending auditory pathway just before the auditory thalamus. Here, we show that a majority of IC neurons (64% of 180 neurons) in awake monkeys carry visual- and/or saccade-related signals in addition to their auditory responses (P < 0.05). The response patterns involve primarily excitatory visual responses, but also increased activity time-locked to the saccade, slow rises in activity time-locked to the onset of the visual stimulus, and inhibitory responses. The presence of these visual-related signals suggests that the IC plays a role in integrating visual and auditory information. More broadly, our results show that interactions between sensory pathways can occur at very early points in sensory processing streams, which implies that multisensory integration may be a low-level rather than an exclusively high-level process.

Recent investigations into multisensory processing have advanced our understanding of the extent of perceptual interactions between sensory stimuli (e.g., refs. 1–5) and have begun to suggest that interactions between the sensory modalities recruit brain regions historically considered to be unimodal (for review, see ref. 6). If perception is often a multimodal process, then it makes sense for supposedly unimodal brain regions to be influenced by other modalities. However, this emerging paradigm shift has largely focused on cortical stages of processing (for review, see ref. 6). An important test of this view is to investigate multisensory influences in earlier stages of processing. If interactions between sensory modalities are frequently useful or even necessary for forming an understanding of the sensory environment, then the important neural task of integrating across sensory systems may not be deferred to late stages of processing but may get underway quickly, recruiting precortical “unimodal” brain regions as well as later stages.

The inferior colliculus (IC) is a logical place to test this view. The IC's importance to hearing is not in doubt, given its early and prominent position in the ascending auditory pathway. Recent work has suggested that the IC may also bear a responsibility for mediating visual influences over hearing. In barn owls exposed to a visual scene that is displaced by prisms, IC neurons modify their connections to the superior colliculus (SC) (7–13), and they are capable of responding to visual stimuli but only when inhibitory influences within the SC are blocked pharmacologically (14). Circumstantial evidence that the mammalian IC may also play a role in visual–auditory interactions has come from a number of sources. We and others have found that the primate IC contains neurons with activity that depends on eye position (15–17), a signal that is necessary for compensating for the dissociations between visual and auditory spatial locations that occur naturally when the eyes move. There is also anatomical evidence for projections to the IC from the retina (18–20), visual cortex (21), and the SC (22, 23). However, a previous physiological study found evidence for only sparse visual responsiveness in the IC of anesthetized cats (<10% of neurons), and such visual responses were only reported among neurons that lacked auditory responses (ref. 24; see also ref. 25). The “gating” of visual responses in the IC by SC disinhibition demonstrated by Gutfreund et al. (14) in anesthetized owls suggests that visual inputs to the IC are prevalent, but normally silent.

It seems plausible that the use of anesthesia in these previous physiological studies may have led to an underestimation of the importance of visual signals in the IC. Such visual responses might occur normally in awake animals, especially when the visual stimulus is to be the target of a saccadic eye movement. Accordingly, we assessed the activity of IC neurons in monkeys performing a task involving the presentation of a visual stimulus, a saccade to that stimulus, followed by the presentation of a sound (Fig. 1).^‡

Fig. 1.

Behavioral task. The monkey was required to make a saccade to the visual stimulus and fixate for 600–900 ms before the sound was turned on. Visual activity was assessed by counting the number of spikes occurring during the periods 50–250 and 100–500 ms after the onset of the visual stimulus and/or 50–250 ms after the saccade to that visual stimulus. (This latter time period was used because the eye movement served to bring the visual stimulus to a new location on the retina.) Saccade-related activity was assessed by counting spikes during a 100-ms period centered on the entry of the eyes into the acceptance window surrounding the visual stimulus. Final classification of neurons into response categories was done based on the responses during these time windows in conjunction with visual inspection of PETHs. Auditory activity was assessed during the sound presentation.

Results

We found that 63.9% of neurons (115 of 180) carried statistically significant changes in activity associated with either the onset of the visual stimulus, the saccade to that visual stimulus, or both (P < 0.05 after Bonferroni's correction for multiple tests) (Table 1).^§ Fig. 2 shows an example neuron. The rasters and perievent time histogram (PETH) in Fig. 2 a and d are time-locked to the onset of the visual stimulus [light-emitting diode (LED)], sorted by the reaction time of the saccade to that visual stimulus (red line on the rasters). This neuron showed a visual response time-locked to the onset of the visual stimulus, with a latency of ≈70 ms [see also supporting information (SI) Fig. 6]. When the rasters and PETH are realigned such that they are synchronized to the onset of the saccade to the LED (Fig. 2 b and e), an additional brief burst in activity at the time of the saccade is evident. This neuron was classified as having both excitatory visual activity and saccade-related activity. An estimate of the receptive field of the neuron is shown in Fig. 2g. The receptive field included the fovea; consequently, the activity of the neuron remained elevated above baseline throughout the trial while the monkey fixated the visual stimulus. The neuron also had a clear, vigorous, short-latency auditory response, as shown by the rasters and PETH in Fig. 2 c and f.

Table 1.

Results of statistical analysis and classification of neurons

Test	Total		Monkey M		Monkey X		Monkey C
	N	%	N	%	N	%	N	%
Visual or saccade-related signals	115	63.9	52	62.7	31	50.0	23	65.7
Primarily visual, excitatory	63	35.0	29	34.9	13	21.0	21	60.0
Primarily motor, excitatory	8	4.4	1	1.2	7	11.3	0	0.0
Visual-motor	27	15.0	17	20.5	10	16.1	0	0.0
Nonspecific	2	1.1	0	0	1	1.6	1	2.9
Visual, slow rise*	9	5.0	9	10.8	0	0.0	0	0.0
Visual, inhibitory	10	5.6	4	4.8	5	8.1	1	2.9

Neurons were tested statistically as described in Materials and Methods. Neurons were classified as having visual or saccade-related signals (first row) if they showed a significant (P < 0.0127) change in activity 50–250 ms after the onset of the LED, 50–250 ms after the saccade to the LED, 50 ms before to 50 ms after saccade onset, or 100–500 ms after LED onset (for the slow-rise response patterns). Neurons were then subcategorized into the categories listed on the second through seventh rows by inspection of the rasters and PETHs sorted by saccade reaction time. Classification as having a visual component of the response required that at least part of the change in activity either appear time-locked to the visual stimulus or be time-locked but after the saccade. Classification as having a motor-related component required that the elevation in activity occur during or preceding the saccade.

*Includes four neurons that were also classified as having short-latency visual responses.

Fig. 2.

A neuron with both visual and saccade-related activity. (a–c) The rasters show the activity on individual trials, synchronized on the onset of the LED, the onset of the saccade to the LED, or the onset of the sound. All rasters are sorted according to the latency of the saccade to the LED. (d–f) The PETHs are binned in 10-ms bins and are not smoothed. (g) The response of the neuron during the 50–250 ms after LED onset as a function of the location of the LED with respect to the eyes at the time of LED onset (smoothed with a 5° bin stepping in 1° increments). The color of the background of the plot reflects the average activity before LED onset across all eye positions, which is also indicated numerically by the white line on the color scale. Retinal locations that were tested for visual responses stand out from this background by their pixilated appearance on the plot, and the relative activity level can be compared with baseline by comparing the color of the pixilated regions with the overall background color.

Fig. 3 shows an additional 12 example neurons with vigorous increases in activity time-locked to the onset of the fixation light. As in Fig. 2 a and d, the rasters and PETHs are aligned on the onset of the fixation light, and the rasters are sorted by the reaction time of the saccade to the fixation light. All of the neurons in Fig. 3 were classified as having either visual-related activity (Fig. 3 a–d, f, and i–l) or visual- and saccade-related activity (Fig. 3 e, g, and h) because of a second peak in activity time-locked to the saccade. Because of space limitations, the saccade-locked rasters and PETH are not shown.

Fig. 3.

Twelve additional example neurons with visually evoked (a–d, f, and i–l) or visual- and saccade-related (e, g, and h) activity. Rasters and PETHs are aligned on fixation light onset, and the rasters are sorted by saccade reaction time (the conventions are the same as in Fig. 2).

Three additional example neurons illustrating other patterns of visual or saccade-related activity are shown in Fig. 4. The neuron illustrated in Fig. 4a appeared to have a mainly saccade-related pattern of activity. This saccade-related discharge was more vigorous than the auditory evoked activity of this neuron (Fig. 4a Right). The neuron in Fig. 4b had both a saccade-related burst and a slow-rising increase in activity beginning ≈200 ms after the saccade to the LED. The slow-rising activity was still rising several hundred milliseconds later in the trial, when the sound came on, as can be seen in Fig. 4 Right for rasters and PETH during the period before sound onset.^¶ An example of a neuron with a powerful and sustained inhibitory visual response is shown in Fig. 4c. Most neurons with inhibitory responses showed more transient and weaker inhibition than this example.

Fig. 4.

Three example neurons showing other patterns of visually evoked activity. (a) Mainly saccade-related pattern of activity. (b) Saccade-related burst and slow-rising increase in activity. (c) Powerful and sustained inhibitory response. All three neurons also exhibited auditory responses. The conventions are the same as for the rasters and PETHs in Fig. 2.

The receptive fields of visually responsive neurons exhibited a variety of patterns, as illustrated in Fig. 5. Some neurons had small, fairly focal receptive fields, as shown in Fig. 5a. Other receptive fields were larger, but still contiguous and bounded (Fig. 5b). Some neurons responded to all visual stimulus locations, either with excitation (Fig. 5c, same neuron as Fig. 3c) or inhibition (Fig. 5d, same neuron as Fig. 4c). Others responded preferentially to visual stimuli at eccentric retinal positions, regardless of radial direction (Fig. 5e). Overall, the receptive fields of 53.0% of the visually responsive neurons (61 of 115) could be successfully fit with either a 2D Gaussian or a planar function (individual P < 0.05).

Fig. 5.

The receptive fields of five example neurons. (a) Small, fairly focal receptive field. (b) Large receptive field spanning both contralateral and ipsilateral locations. (c) Large receptive field with best area in the ipsilateral hemifield. (d) Inhibition at nearly all tested locations. (e) “Annular” receptive field, with responses to visual stimuli at eccentric positions. The conventions are the same as in Fig. 2g.

The visual responses reported here had several features that distinguished them from the eye position sensitivity we and others have described in the IC (15–17). First, most of the visual responses were time-locked to the onset of the fixation light (i.e., a period in which eye position was random). Second, the visual responses were generally excitatory, whereas eye position sensitivity could take the form of either increases or decreases in activity as the eyes moved either left or right. Third, many of the visually responsive neurons were sensitive to the location of the LED (the planar or Gaussian function fitting). Because the baseline activity was subtracted for this statistical test, any effects solely caused by eye position were eliminated. Fourth, we have previously reported eye position sensitivity in the absence of a visual stimulus (17). Thus, visual stimuli and eye position are two distinct factors influencing IC neural activity.

The presence of both visual and eye position signals in the IC suggests that the IC may be playing a role in mediating visual influences over hearing. If so, then the visual and eye position signals might co-occur in the same neurons, and they did: neurons that exhibited visual responses also tended to show eye position sensitivity (SI Table 2; P < 0.001, χ² test). The frame of reference of the visual receptive fields, and the relationship between the visual, auditory, and eye position spatial sensitivity is uncertain at this point and will be worthy of further investigation.

We also checked whether the presence of visual sensitivity was correlated with any aspects of auditory sensitivity but found no relationship between the presence of auditory responsiveness, frequency tuning, or the temporal profile of the auditory response (SI Table 2). There was also no difference in the latency of auditory responses in neurons with and without visual responses (P > 0.05, t test; the median latency was 14 ms).

The visual-evoked activity also differed in its properties from the pattern expected of a reward-related or attentional signal. We reasoned that if the signals were really related to attention or reward, they should be substantially diminished or eliminated when the animal was insufficiently attentive or motivated to perform the task correctly. Among neurons that responded to the onset of the fixation light, the responses occurred regardless of whether the monkey successfully achieved and maintained fixation throughout the trial: we found that 48 of 67 or 71.6% of neurons classified as “visual” or “visual-motor” had statistically significant responses (P < 0.05) to the fixation light even on unsuccessful, and unrewarded, trials.

Discussion

The results from this study show that visual-related and saccade-related signals are prevalent in the IC, coexisting with vigorous auditory signals. These signals are present under normal circumstances (i.e., in an awake animal making saccades to visual stimuli, a situation that is common in a natural setting). All three monkeys showed a similar prevalence of visual or saccade-related signals (Table 1), but there were some differences in the nature of these signals. Specifically, monkey C did not show any activity that could be classified as containing a saccade-related component. The visual responses of this monkey also seemed to be more restricted to foveal locations. This could account for the lack of saccade-related activity: when a visual stimulus is on the fovea, no additional saccade is required to look to that stimulus. The visual responses of monkey C tended to occur after the saccade to the fixation light, which brought that stimulus onto the fovea. At present, it is unclear why the receptive fields in this monkey were more likely to contain the fovea than was the case in the other two animals. One possible explanation is that the IC contains a retinotopic map and that our recordings sampled this map unevenly in the three animals. Future experiments will be needed to investigate this question.

The presence of a saccade-related component to the discharge patterns nicely parallels previous findings in barn owls (14), and supports the view that the SC is at least one of the physiologically relevant sources of input to the IC. Responses that are primarily visual might arise either from the SC or the projections from retina or visual cortex. Some of the receptive field attributes seen in IC neurons (e.g., the large size, inclusion of the ipsilateral field, and unusual shapes) are not typical of the receptive fields found in any of the potential sources of input and suggest that there must be a substantial amount of convergence of projections from individual neurons in these visual input structures onto IC neurons, either directly or via interneurons.

Throughout this study, we have used the terms visual, visual-related, or saccade-related to describe the activity patterns. Could the activity really be attention- or reward-related? Several factors argue against this interpretation. First, the response patterns share the temporal and spatial characteristics of responses in classical visual and saccade-related areas, namely the time-locking of the responses to visual stimulus or saccade onset, and the presence of receptive fields. Second, the activity occurred on trials that were not initiated and/or completed successfully (and no reward was delivered), as well as on those that were successfully performed. Third, visual responses have been reported in the IC of anesthetized preparations in which no behavioral task was involved (14, 24). Fourth, the response properties differed from the reward-related influences that we have previously explored: this reward-related activity was most pronounced later in the trial when the reward delivery was imminent (26). The subset of neurons showing a slowly rising response is the best candidate for a population having visually triggered reward signals, but these neurons were a small fraction of our data set (2.8%). Finally, it is plausible for the IC to contain visual responses as the IC receives anatomical inputs from areas known to have visual responses [the retina, visual cortex, and the SC (18–23)].

More broadly, these findings contribute to the emerging paradigm shift suggesting that the predominantly unimodal areas of the brain may actually be subject to multisensory influences. It has been known since the 1960s that the primary visual cortex contains some auditory responses (for review, see ref. 27), and more recently responses in human auditory cortex have been shown by imaging studies to be modulated by visual stimuli in both hearing (28–30) and deaf subjects (31, 32). Furthermore, audiovisual convergence has been demonstrated in the auditory cortex of monkeys (33–35). The IC is located two steps earlier in the auditory pathway than auditory cortex and may serve as a source of visual influences on auditory cortical activity. At present, overt unimodal visual responses in auditory cortex do not appear to be as extensive as those in the IC (34, 35), but visual modulation of auditory responses is quite prevalent and is especially evident in the local field potential (35). In auditory cortex, visual influences depend on the behavioral task (34) and the nature of the visual stimuli (35). The same may ultimately prove true of the IC's visual signals. In addition, the advent of studies in which eye position (and thus retinal stimulus location) is monitored and controlled will facilitate investigations of visual responses in these areas.

That the brain gets started on the task of merging sources of information so early in sensory processing streams may reflect the profound importance of multisensory integration to perception. It remains to be seen exactly how early visual responses might be present: the IC projects to still earlier positions such as the superior olivary complex, and, via the olivocochlear bundle, to the outer hair cells in the cochlea (36). Thus, visual influences may be transmitted to even earlier stages of processing, an intriguing possibility indeed.

That said, the presence of visual influences in the auditory pathway does not necessarily mean that the auditory pathway participates in visual perception, but more likely reflects a mechanism by which visual stimuli can influence auditory perception. Indeed, a recent study in a human patient with a unilateral lesion to the IC showed a reduction in the bimodal interactions of the McGurk effect when the stimuli were presented in the contralesional hemifield (37). Additional studies comparing the IC responses elicited by visual stimuli that are naturally associated with sounds with those elicited by visual stimuli that are normally silent may help elucidate the purpose of visual signals in the IC.

Materials and Methods

Animals and Animal Care.

All procedures were conducted in accordance with the principles of laboratory animal care of the National Institutes of Health (publication 86-23, revised 1985) and were approved by the Institutional Animal Care and Use Committee at Dartmouth College. Three monkeys (M, X, and C) served as subjects for these experiments.

Surgical and Recording Procedures.

Surgical procedures were performed by using isoflurane anesthesia and postoperative analgesia. The monkeys underwent an initial surgery to implant a head post for restraining the head and a scleral eye coil for monitoring eye position (38, 39). After the animals were trained to fixate visual stimuli, a recording cylinder was implanted over a craniotomy. The cylinder was positioned to allow electrodes to approach the IC on the left side of the brain at an angle of 33° from vertical in the coronal plane (40). The entry of the electrode into the IC was readily apparent based on the emergence of auditory responsiveness. The action potentials of individual IC neurons were isolated from those of surrounding neurons by using a window discriminator (Bak Electronics, Germantown, MD) and their time of occurrence was stored for off-line analysis. All neurons included in this study either had statistically significant auditory responses (n = 159; two-tailed paired t test of the discharge rate during sound presentation compared with the fixation period before sound presentation, P < 0.05) or were located in the immediate vicinity of auditory responsive neurons (i.e., the multiunit activity at the recording site was auditory, but the particular neuron under study did not have a statistically significant response to the sounds presented; n = 21).

Experimental Setup and Behavioral Task.

All experimental and behavioral training sessions were conducted in complete darkness in a single-walled sound-attenuation chamber (Industrial Acoustics, Bronx, NY) lined with sound-absorbing foam (3-inch painted SonexOne; Sonex, Minneapolis, MN). We monitored each monkey's eye position by using the scleral eye-coil technique (38, 39) (sampling rate, 500 Hz). Each trial began with the onset of a visual stimulus consisting of an LED (luminance, ≈26.4 candelas/m²) (Fig. 1). The location of the LED in space could be at one of three, eight, or nine positions (vertical position, 0°; horizontal positions: −12, 0, 12; n = 12 neurons; −24 to 24 by 6; n = 30 neurons; or −15 to 20 by 5; n = 138 neurons). Eye position was monitored but not controlled at the time of LED onset, that is, the monkey was free to look anywhere during the intertrial interval, and thus the initial retinal location of the LED varied from trial to trial. The natural variation in eye position at the beginning of the trial together with the range of LED locations permitted an estimate of the location and structure of visual receptive fields in retinal coordinates, but not full disambiguation of the reference frame used by visually responsive neurons.

The monkey was required to respond to the LED by making a saccade to its location within 3 s. After fixating the LED for 600–900 ms, a 500-ms sound (200 ms for seven cells) was played from one of up to nine speaker locations on the horizontal meridian in the central 40–50° range (model TWO25V2; Audax). Auditory stimuli were bandpass white noise bursts (rise time, 10 ms; 500 Hz to 18 kHz; 51 ± 2 dB sound pressure level;“A” weighting; Bruel Kjaer, Norcross, GA; model 2237 integrating sounds level meter with model 4137 condenser microphone). The animal maintained fixation of the LED during sound presentation and for 300–600 ms afterward to receive a liquid reward. In monkey C, additional trials were conducted in which the fixation light went off after the sound had been playing for 500 ms, and the monkey was required to make a saccade to the location of the sound. These data are reported in a separate study.^‖ Trials in which the animal failed to maintain fixation were terminated and discarded from the analysis except as noted for the analysis of visual responsiveness on aborted trials.

Frequency response functions were assessed for 109 of the 180 experiments, when time and unit isolation permitted. Tones at a fixed intensity (51 ± 2 dB sound pressure level) and covering a range of frequencies (usually 400 Hz to 12 kHz in approximately one-third octave increments) were used for this. The animals did not perform a behavioral task for this phase of the experiments, although eye position was monitored.

Statistical Analyses.

Visual responses were assessed statistically by comparing the spike rate during either the 50–250 ms after the onset of the LED or the 50- to 250-ms period after the saccade to the LED to a matching 200-ms time period before LED onset by using a two-tailed paired t test. (The visual response window after the saccade was used because the eye movement served to bring the visual stimulus to a fresh location on the retina.) Saccade-related activity was also assessed by using a spike-counting window from 50 ms before to 50 ms after the time at which the eyes entered the acceptance window surrounding the fixation target. Neurons that passed one or more of these statistical tests were then classified as having visual, saccade-related, or combined visual- and saccade-related activity by inspection of PETHs synchronized on visual stimulus onset vs. the saccade. Additionally, some neurons had very long latency slow rising visual response patterns evident in the PETHs. These were assessed statistically by using a time window 100–500 ms after LED onset.

The individual criterion P value for these tests was <0.0127, providing an overall Bonferroni-corrected α level of 0.05. SI Fig. 7 shows the P values of the neurons classified as having significant visual- or motor-related responses of these different types, as well as neurons with values of P < 0.05, which were not counted toward the total. The distributions are skewed toward values of P < 0.0001. Thus, changes in the criterion P values would have changed the results only minimally; inclusion of neurons with values of P < 0.05 would have brought the total proportion of neurons showing visual-related increases in activity to 71%.

We tested for sensitivity to the retinal location of the LED by using 2D Gaussian and planar fits (P < 0.05). These tests involved relating the activity of the neuron during the 50- to 250-ms window after LED onset to the location of the LED on the retina. Retinal location varied because of variations in the animal's eye position at the time the LED was illuminated (the animal was freely viewing at this point in time) and because of the location of the LED in space. Baseline activity was subtracted, so that this test reported the effects of the retinal location of the stimulus above any influence of eye position on the spontaneous activity.

We conducted an analysis of aborted trials to help determine whether any task-related factors such as arousal, attention, or motivation to achieve a reward could account for our findings. This analysis involved trials in which the monkey either never achieved fixation or achieved but then aborted fixation early. We conducted this test only in neurons that were classified as having visual or visual-motor activity on correctly performed trials. We used only the first response window (50–250 ms after LED onset) and not the saccade-triggered windows because a saccade to the LED did not occur in most trials. Because only one statistical test was used, the P value was 0.05.

As mentioned earlier, auditory responses were assessed by comparing the number of spikes during 0–500 ms after sound onset (0–200 ms for the cells tested with 200-ms sounds) to the corresponding period just before sound onset (P < 0.05, t test).

Recording Locations.

The locations of our recording penetrations were identified by using MRI at the Dartmouth Brain Imaging Center [1.5 T scanner (GE, Waukesha, WI); 3D T1 weighted gradient echo pulse sequence; 5-inch receive-only surface coil] (SI Fig. 8). One or more tungsten electrodes were inserted into the brain for the scan; these electrodes were readily visible in the images and served as reference points for the reconstruction of other recording locations (e.g., ref. 15). In monkey X, the recording site locations were confirmed histologically as well as by MRI (SI Fig. 9). Standard histological techniques were used as follows: the brain was fixed with formalin and sliced in 50-μm sections that were stained with cresyl violet. This work was performed by David T. Larue and Jeffery A. Winer (University of California, Berkeley, CA), and a figure has been published in ref. 17. Because of the long period over which the recordings took place, we did not attempt to assign specific individual sites to the IC subdivisions that have been identified in other species [e.g., cat (41–44)].

The physiological properties of the neurons in our recordings were consistent with the IC in terms of showing that vigorous auditory discharges, frequency sensitivity, and response latencies were comparable with previous reports. Also, we note that the auditory responses differed from those previously observed in the neighboring SC, which does not typically show auditory responsiveness unless monkeys are performing saccades to auditory stimuli (45); our monkeys did not make auditory-guided saccades in this study.

Abbreviations

IC

inferior colliculus

SC

superior colliculus

PETH

perievent time histogram

LED

light-emitting diode.