Cross-modal responses in the primary visual cortex encode complex objects and correlate with tactile discrimination

Abstract

Cortical areas that directly receive sensory inputs from the thalamus were long thought to be exclusively dedicated to a single modality, originating separate labeled lines. In the past decade, however, several independent lines of research have demonstrated cross-modal responses in primary sensory areas. To investigate whether these responses represent behaviorally relevant information, we carried out neuronal recordings in the primary somatosensory cortex (S1) and primary visual cortex (V1) of rats as they performed whisker-based tasks in the dark. During the free exploration of novel objects, V1 and S1 responses carried comparable amounts of information about object identity. During execution of an aperture tactile discrimination task, tactile recruitment was slower and less robust in V1 than in S1. However, V1 tactile responses correlated significantly with performance across sessions. Altogether, the results support the notion that primary sensory areas have a preference for a given modality but can engage in meaningful cross-modal processing depending on task demand.

The anatomical segregation of the ascending sensory pathways, as well as the observed specificity of electrophysiological responses, led to the once-prevalent notion that each primary cortical area represents the highly specialized functional target of feed-forward labeled lines that are supposed to define a given sensory system (1–3). According to this view, primary sensory areas would be exclusively dedicated to the processing of a single sensory modality, whereas multisensory integration would take place only in higher-level associative areas.

Notwithstanding its textbook penetration (4), the “labeled lines” view of cortical function was challenged early on by Karl Lashley (5) and Donald Hebb (6), who argued in favor of widely distributed cortical processing. Evidence of distributed processing has mounted since then (7, 8). Although the sensory areas of the mammalian cortex are separated according to dominant modalities, anatomical projections link primary sensory areas to multisensory regions (9–12). For instance, rats are capable of cross-modal object recognition when asked to visually recognize an object with which they had previous tactile experience (13), and stimulation of the rat's whiskers triggers cortical activation that spreads over time far beyond the primary somatosensory cortex (S1) (14–17). In primates, multisensory stimulation enhances responses in the primary sensory cortex, in comparison with unisensory stimulation of the dominant modality (18–22). In humans, brain imaging studies have demonstrated that the primary visual cortex (V1) is engaged by tactile processing in blind subjects (23). Such cross-modality seems useful for task execution because transcranial magnetic stimulation of V1 disrupts tactile discrimination (24). Furthermore, occipital activation in blind subjects is positively correlated with performance in a nonvisual memory task, suggesting that visual circuits are functionally recruited for nonvisual processing on demand (25). In agreement with these findings, imaging studies of nonblind subjects showed that V1 becomes responsive to tactile inputs after brief darkness adaptation (26, 27). Altogether, these results indicate that primary sensory areas have a preference for a dominant modality but are capable of cross-modal processing (28–30).

Despite the increasing acceptance of cross-modal processing in primary sensory areas, the phenomenon remains controversial. The passive presentation of simple stimuli to behaving primates resulted in cross-modal phase resetting of local field potentials (LFPs) (21, 22) but failed to reveal correspondent changes at the spike level (21, 22, 31). However, cross-modal processing at the level of both LFPs and spikes was observed with naturalistic stimuli (32). In line with these results, V1 neurons in rats display robust rate changes during novel-object exploration in the dark (33). It remains unknown whether such responses carry any useful information. Therefore, here we asked whether V1 cross-modal spike responses represent behaviorally relevant information that is positively correlated with perceptual discrimination during natural or operant behavior.

Results

We performed extracellular recordings of spikes (SI Appendix, Fig. S1) and LFPs in adult rats (SI Appendix, Supporting Methods 1). Each animal was chronically implanted with two multielectrode arrays centered on the binocular region of V1 and on the barrel field of S1 (SI Appendix, Fig. S2). Electrodes were implanted in cortical layer V. The placement of individual electrodes with respect to layer and area boundaries was confirmed by the anatomical reconstruction of electrode tracks within regions rich in cytochrome oxidase (SI Appendix, Fig. S2 and Supporting Methods 2), a mitochondrial enzyme most abundant in primary sensory areas (34).

Cross-Modal Changes in Firing Rates.

To investigate cross-modal processing in primary sensory cortices during spontaneous behavior, rats (n = 13) were recorded in the dark (0 lx) as they used their facial whiskers to freely explore four novel objects of different shapes and textures (Fig. 1A and SI Appendix, Supporting Methods 3). A total of 411 neurons were recorded from V1, and 188 neurons were recorded from S1. The data revealed significant firing-rate changes during object exploration in the dark (Fig. 1B). On average, 35% of V1 neurons and 55% of S1 neurons changed their rates during blind novel-object exposure, in comparison with preexploration periods (post hoc with corrected α across animals ≤0.007). Most V1 neuronal responses consisted of excitation (81%), with 19% showing inhibition. Similar numbers were found in S1: 84% of the significantly modulated neurons showed excitation, and 16% showed inhibition (SI Appendix, Supporting Methods 4).

Fig. 1.

Firing-rate changes in V1 during object exploration. (A) Animals explored novel objects in the dark. (B) Rate modulation in V1 and S1 neuronal ensembles before, during, and after exploration. Red vertical lines indicate the beginning and end time points of a core 10-min epoch within the total 20-min exploration period. Color scale represents normalized firing rates from the minimum to the maximum of each neuron.

To estimate the strength of V1 cross-modal responses during blind object exploration, we investigated V1 rate changes during presentation of a novel movie composed of natural images (n = 5 animals, 194 neurons). Movie presentation at 190 lx resulted in significant rate change for 82% of V1 neurons, with 81% of excitation and 19% of inhibition (post hoc with corrected α across animals ≤0.0016). When V1 neurons that showed significant rate increase in response to both object exploration and movie presentation were analyzed, we found no significant rate difference between the two stimulation conditions in comparison with the baseline (median firing rates: baseline, 3.1 Hz; objects, 5.4 Hz; movie, 5.8 Hz; Wilcoxon test, P > 0.999).

Cross-Modal Spike Responses Encode Complex Objects.

Previously, we interpreted electrophysiological and molecular changes in V1 during object exploration in the dark as a nonspecific effect of arousal (33). If this were the case, however, V1 responses during blind exploration should represent a general alert signal, carrying no information whatsoever about object identity. To test this hypothesis, we first compared the mean activation levels of individual V1 and S1 neurons during object exploration (n = 6 animals, 142 and 125 neurons from S1 and V1, respectively). No significant difference among objects was detected for 95% of the V1 neurons and 98% of the S1 neurons (paired two-tailed Mann–Whitney test of interspike intervals for three object pairs, P < 0.02 corrected for the number of neurons compared). This result indicates that mean firing rates do not encode object identity for most single neurons.

Next, we asked whether object identity could be encoded by S1 or V1 neuronal ensembles during novel-object exploration in the dark. Spike records were used to train binary classifier models (SI Appendix, Supporting Methods 5) to build a map of neuronal responses evoked by each isolated object (n = 4 animals, 101 and 108 neurons from S1 and V1, respectively), such as in recent studies of neurons from the macaque posterior parietal cortex (35) and inferior temporal cortex (36). The area under a receiver operating characteristic curve (AUROC) (37) was used as an estimate of classification accuracy for each specific object (Fig. 2). High AUROC values significantly above chance (medians >0.8) were reached by both S1 and V1 ensembles for three objects: “ball,” “brush,” and “urchin” (paired two-tailed Mann–Whitney test of medians with P ≤ 0.03). In contrast, the object “food” was poorly discriminated by both S1 and V1 ensembles (median <0.6; paired two-tailed Mann–Whitney test of medians with P ≥ 0.35), possibly reflecting a switch from exploratory to consumption behavior. For all classifier models, data from V1 ensembles did not differ significantly from values calculated for S1 neuronal ensembles (SI Appendix, Fig. S3). In contrast, surrogated datasets did not produce significant object classification above chance level (SI Appendix, Fig. S4 and Supporting Methods 6).

Fig. 2.

Complex objects explored in the dark were similarly encoded by S1 and V1 neuronal ensembles. Shown are mean AUROC values for object classification with a naive Bayes binary classifier fed with neuronal ensemble data from S1 and V1 (n = 4 animals, 10 samples per animal, median ± quartiles). S1 and V1 values were not significantly different (paired two-tailed Mann–Whitney test, P = 0.69 for ball, P = 0.89 for brush, P = 0.49 for urchin, and P = 0.34 for food, with α = 0.0125 corrected for four comparisons).

Given the more direct recruitment of S1 neurons than V1 neurons for tactile processing, it is possible that tactile processing in V1 depends on a few specialized cross-modal neurons, whereas S1 would display a more balanced distribution of processing across its neuronal population. To investigate the coding scheme of cross-modal representations in V1, we performed a neuron-dropping analysis of the classifier results; this bootstrap method reveals how much information is lost, on average, as neuronal ensembles decrease their size from n to 1 neuron (38). Substantial shape differences in neuron-dropping curves would indicate that the nonvisual processing is different between S1 and V1 neuronal ensembles. To our surprise, very similar neuron-dropping curves were observed for both areas (Fig. 3A). For brush, urchin, and ball, AUROC values > 0.7 were similarly achieved with S1 or V1 ensembles as small as five neurons. In both areas, maximum AUROC values between 0.75 and 0.95 were achieved by ensembles of ∼10 neurons (Fig. 3A).

Fig. 3.

Neuron-dropping curves for object coding were equivalent for V1 and S1 neuronal ensembles. (A) Neuron-dropping curves for object classification in S1 and V1 using a naive Bayes binary classifier. Symbols on the right represent different animals: circle (Rat 4), square (Rat 5), triangle (Rat 9), and rhombus (Rat 12). (B) Parameters b and d of the double exponential fit of the neuron-dropping curves do not segregate according to cortical area. No statistically significant differences were found between V1 and S1 for both parameters (two-tailed Wilcoxon signed-rank test; ball: P_b = 0.6857 and P_d = 0.6857, brush: P_b =1.0 and P_d = 0.0571, urchin: P_b = 1.0 and P_d = 0.2, and food: P_b = 0.4857 and P_d = 0.6857 with α = 0.00625 corrected for eight comparisons). Same symbols are used as described in A.

To quantitatively compare the neuron-dropping curves, they were modeled by using the exponential fit AUROC(n) = a * exp(b * n) + c * exp(d * n), in which parameters b and d represent the growth factors with respect to the number of neurons n in each ensemble recorded (Fig. 3B). This analysis confirmed that there was no significant difference in the shapes of the neuron-dropping curves (paired two-tailed Mann–Whitney test of a comparison between S1 and V1 for parameters b and d, respectively, P = 0.69 and 0.69 for ball, P = 1.0 and 0.06 for brush, P = 1.0 and 0.2 for urchin, and P = 0.49 and 0.69 for food, with α = 0.0125 corrected for four comparisons). Thus, we did not observe that a few special V1 neurons have privileged access to cross-modal inputs. Instead, V1 neuronal ensembles seem to encode object identity with the same ensemble statistics of S1 neurons.

Altogether, the results suggest that object-specific information is similarly encoded by S1 and V1 neuronal ensembles during tactile object exploration in the dark. Alternatively, could rats actually see under our recordings conditions? To rule out this possibility, we investigated object recognition under four different visuo-tactile conditions (n = 8 per group; SI Appendix, Supporting Methods 7): (i) objects with different shapes and visual appearances, explored with lights on; (ii) objects with different shapes and visual appearances, explored with lights off; (iii) objects with same shapes but different visual appearances, explored with lights on; and (iv) objects with same shapes but different visual appearances, explored with lights off. When objects with different shapes and visual appearances were used, discrimination was excellent with lights either on or off (Fig. 4, first and second diagrams; Wilcoxon test, P = 0.00047 and P = 0.00078, respectively). For objects with identical shapes but different visual appearance, discrimination with lights on was reduced but still significant (Fig. 4, third diagram; P = 0.037), whereas discrimination with lights off was null (Fig. 4, fourth diagram; P = 0.779).

Fig. 4.

Influence of visual and tactile inputs on the object recognition task. The recognition of objects with different shapes and visual appearances was substantial irrespective of illumination. The recognition of objects with identical shapes and different visual appearances was less prominent but still significant with lights on, but it was abolished with lights off. The rats discriminated objects with different shapes in the dark better than they discriminated objects with different visual appearances but identical shapes (compare second and third diagrams).

Cross-Modal Spike Responses Correlate with Perceptual Discrimination.

To further investigate the relationship between the behavioral need for tactile processing and V1 cross-modal recruitment, rats were recorded in the dark (n = 3, 166 and 99 neurons recorded from V1 and S1, respectively) while performing a whisker-based tactile discrimination task (39, 40) (Fig. 5A and SI Appendix, Supporting Methods 8). To solve this task, rats use their facial whiskers to establish bilateral contact with metal bars that set either narrow or wide apertures (39). This contact occurs around nose poke (NP) and conveys task-relevant information regarding aperture width to various somatosensory relays, including S1 (40). Significant spike responses were conspicuous in V1 (Fig. 5B). A rate-change analysis (post hoc with corrected α across animals ≤0.004) showed that 32% of the V1 neurons displayed significant modulation around NP, with 17% of excitation and 83% of inhibition. For comparison, 53% of the S1 neurons had their firing rates modulated by the tactile stimulus, with 16% of excitation and 84% of inhibition (post hoc with corrected α across animals ≤0.004). Using cumulative sums to detect sharp rate modulations instead of bulk mean changes (SI Appendix, Supporting Methods 4), we found that 38% of the V1 neurons were responsive, with 52% showing excitation and 48% showing inhibition. For comparison, 41% of the S1 neurons showed significant responses, comprising excitation (61%) and inhibition (39%). Most of the S1 responses occurred at −100 or +100 ms from NP (SI Appendix, Fig. S5; −40 ± 33 ms, mean latency ± SEM). Significant V1 responses showed a much wider distribution of latencies (SI Appendix, Fig. S5; +110 ± 32 ms, mean latency ± SEM) and, overall, significantly longer responses (V1 > S1, Wilcoxon test, P = 0.025). However, both areas contained neurons that responded very early in the task (SI Appendix, Fig. S5, black arrow at 150 ms before NP).

Fig. 5.

Tactile inputs trigger V1 spike responses during whisker-based aperture discrimination. (A) Animals were trained to associate narrow or wide apertures to liquid reward dispensed on left or right. Apertures (lateral bars) sampled with whiskers around NP. (B) Representative peristimulus time histograms (Lower) and raster plots (Upper) of cross-modal responses in V1 during task execution. Time 0 = NP.

To search for task-related changes in neuronal synchrony, we calculated neuronal cross-correlations (41) between all pairs of recorded neurons (SI Appendix, Supporting Methods 9). Several S1 × V1 neuronal pairs showed increased correlated discharge (Fig. 6A). As expected for tactile stimulation (42), synchrony was overall significantly greater in S1 neuronal pairs than in V1 or S1 × V1 pairs (Fig. 6A and SI Appendix, Fig. S6). We also investigated cross-correlations across the different trial epochs (Fig. 6B). We detected increased S1 cross-correlations immediately after the central door opening (CD), soon followed by increased S1 × V1 cross-correlations at ∼300 ms after CD and finally by increased V1 synchrony around when whisker contact with the aperture is initiated. These results suggest a late engagement of V1 neuronal ensembles in tactile processing.

Fig. 6.

Cross-correlations across areas and trial epochs. (A) Significant normalized peak cross-correlations around NP for representative S1, V1, and S1 × V1 neuronal pairs. Notice that two S1 neurons with high synchrony between them (1a and 11a) displayed very different cross-correlations with respect to the same V1 neuron (31d). Red lines represent shuffled data. (B) Group data for cross-correlations calculated around the CD, NP, and reward (RW) (SI Appendix, Supporting Methods 8). Notice that V1 neuronal pairs shows increased synchronization around stimulus sampling (NP) up to the time of rewarded choice (RW).

To estimate stimulus-related tactile information in cortical spiking activity as the trial progressed, we performed pattern classification with a learning vector quantization (LVQ) classifier. Both V1 and S1 neuronal populations contained substantial amounts of information during execution of the tactile discrimination task in the dark (Fig. 7A). Most interestingly, we found strong linear correlations between animal performance in the task and the LVQ prediction of stimulus type based solely on the activity of S1 or V1 neuronal ensembles (Fig. 7B; R > 0.8 and P < 0.02 for both areas). Altogether, the results suggest that cross-modal spike responses in V1 are used for tactile coding during natural as well as operant behavior.

Fig. 7.

V1 spike responses encode tactile information during whisker-based aperture discrimination. (A) Representative example of the evolution of tactile information over trial time using a LVQ classifier (mean ± SEM). During task execution, S1 and V1 neuronal ensembles reached significant amounts of tactile information, with faster time course and higher magnitude in S1. The gray area represents the estimated period of contact between whiskers and stimulus. (B) Linear correlations between animal performance in the tactile task and peak neuronal ensemble prediction of stimulus type based on the LVQ classifier. Data are from 11 recording sessions.

Discussion

Here we report cross-modal responses of V1 neurons when rats use their whiskers to explore objects in the absence of visible light. The attempt to decode object identity based solely on spike recordings showed that relatively small V1 or S1 ensembles (∼10 neurons) hold specific object-related information. During execution of a whisker-based aperture-discrimination task in the dark, V1 neurons showed marked rate modulation. Furthermore, stimulus-related information in V1 neuronal ensembles was proportional to performance in the tactile task, in line with human data showing a positive correlation between V1 engagement and performance in a nonvisual task (25).

Interpretation of the results depends largely on the assumption that our recording conditions represent total darkness. For the tactile discrimination task, it has been previously determined that performance drops to chance when the whiskers are cut (39). Because rats have neither red nor infrared vision (43, 44), they are effectively blind under the infrared illumination used in our recordings. The investigation of object recognition under different sensory conditions confirmed that no visual cues were available for object recognition when visible lights were off (Fig. 4). Thus, our electrophysiological data constitute evidence that V1 neuronal ensembles are recruited for tactile processing in the absence of visible light.

During object exploration, ∼81% of the responses in V1 were excitatory. For the tactile discrimination task, the result was reversed: ∼83% responses were inhibitory. This difference is probably related to the contrast of novelty versus familiarity that characterizes the tasks. Animals freely explore novel objects in a nonstereotyped manner, whereas the tactile discrimination task involves repetitive, stereotyped whisker contact with familiar stimuli. Our results differ from those obtained with passive flutter stimulation of awake monkeys (31), which failed to elicit cross-modal changes in spike rates in primary auditory and tactile cortical areas. Two protocol differences may explain this discrepancy. We used active stimulation, which depends on the animal's initiative to move and explore. During active whisker contact with the objects, proprioceptive and motor feedback may both contribute to the recruitment of V1 for the computation of object identity in the dark. In contrast, Lemus et al. (31) used passive stimuli, which may greatly reduce the need for cross-modal processing in primary sensory areas. Another difference is that we investigated visual and tactile stimuli, a combination that may be more prone to cross-modal processing than the tactile and auditory stimulation used by Lemus et al. (31).

The present neurophysiological and behavioral data add compelling evidence against the notion that sensory processing in the cortex is implemented by segregated labeled lines for unimodal signals that can only mix in associative areas (1–3, 31). Instead, our results support the growing body of evidence that points to a distributed organization of the sensory neocortex (14–30). This organization may be a mere corollary of the fact that the brain has a small-world organization able to ensure fast access from everywhere (45–47). Neurons with cross-modal responses in primary sensory areas probably act as information hubs that regulate multisensory cortical recruitment under various regimes of sensory stimulation. Because V1 latencies are significantly larger than S1 latencies, corticocortical horizontal propagation likely contributes to V1 tactile responses (16, 17, 48, 49). The horizontal propagation of activity in V1 is inversely correlated with the contrast of visual stimulation (50), which may explain the tactile recruitment of V1 in the dark. However, the very early latency peak in both S1 and V1 implies direct thalamocortical somatic sensory influence on selected V1 neurons, probably involving tactile inputs via the superior colliculus (51–55).

We do not propose a functional equivalence between S1 and V1. The predominance of S1 over V1 among tactile-responsive neurons (∼55–35%), the longer latencies of most V1 responses in the tactile discrimination task (SI Appendix, Fig. S5), the higher S1 neuronal synchrony in comparison with V1 only or S1 × V1 neuronal pairs (SI Appendix, Fig. S6), the significant increase in V1 synchrony as the trial progresses from stimulus sampling to rewarded choice (Fig. 6B), and the increase of stimulus-related information first in S1 and then in V1 (Fig. 7A) all indicate that tactile recruitment is slower and less robust in V1 than in S1. However, V1 and S1 yield comparable amounts of information in tactile tasks when visible light is not present to visually engage V1 (Figs. 2, 3, and 7). This result indicates that the contributions of S1 and V1 for tactile coding are quantitatively different but qualitatively similar, by analogy with similar poll numbers for two electorates of different sizes. In Lashleyan terms, the tactile contributions of S1 and V1 in the dark seem “equipotent but with different masses” (5).

Altogether, our results support the notion that the neocortex, rather than constituting a mosaic of areas operating independently from one another, processes information according to the perceptual task at hand. Increased task sharing minimizes the number of idle processors, a feature exhibited by cooperative computer grids (56) and competitive “mixture of experts” computer models (57). Future experiments shall elucidate the roles of cooperation and competition for task sharing by primary neocortical areas.

Methods

Experimental Animals.

A total of 49 adult male Long–Evans rats (300–350 g) were used for electrophysiological (n = 17) and behavioral (n = 32) experiments. Housing as well as surgical and recording procedures were in accordance with the Edmond and Lily Safra International Institute of Neuroscience of Natal Committee for Ethics in Animal Experimentation, the National Institutes of Health guidelines, and the Duke University Institutional Animal Care and Use Committee.

Electrophysiological Recordings.

As previously described (33), multielectrode 4 × 4 arrays with 250-μm spacing (Teflon-coated tungsten wires, 35- or 50-μm diameter, >500 Ohm at 1 KHz) were surgically implanted in S1 and V1 under anesthesia, according to coordinates shown in SI Appendix, Fig. S2A. Some animals (n = 4) came from a previous study (33) and also received hippocampal implants. Spikes and LFPs were simultaneously recorded with a 96-channel Multichannel Acquisition Processor (MAP; Plexon Inc.) (SI Appendix, Supporting Methods 1).

Binary Classifier Models.

Five classifier models were used to analyze data recorded during the free exploration of novel objects: multilayer perceptron (58), radial basis functions (58), support vector machines (58), decision tree (59–61), and naive Bayes classifier (61, 62). Data input is detailed in SI Appendix, Supporting Methods 5. The LVQ model (58) was used to implement decision predictors (40, 63, 64) for the tactile discrimination experiment (Fig. 7 A and B).