Coding of stimulus sequences by population responses in visual cortex

Abstract

Neuronal populations in sensory cortex represent the time-changing sensory input through a spatiotemporal code. What are the rules that govern this code? We measured membrane potentials and spikes from neuronal populations in cat visual cortex (V1), through voltage-sensitive dyes and electrode arrays. We first characterized the population response to a single orientation. As response amplitude grew, population tuning width remained constant for membrane potential responses and became progressively sharper for spike responses. We then asked how these single-orientation responses combine to code for successive orientations. We found that they combine through simple linear summation. Linearity, however, is violated after stimulus offset, when responses exhibit an unexplained persistence. Thanks to linearity, the interactions between responses to successive stimuli are minimal. We demonstrate that higher cortical areas may reconstruct the stimulus sequence from V1 population responses through a simple instantaneous decoder. In area V1, therefore, spatial and temporal coding operate largely independently.

The computations performed by sensory cortex involve large neuronal populations whose activity evolves over time to code an ever changing sensory input^{1, 2}. We wish to understand the rules that govern this dynamical code. Such rules would describe the computations performed by the underlying circuits. We also wish to understand the strategies required to decode these responses, as these strategies must be followed by downstream neurons that interpret V1 population activity.

In some neural systems, the spatial and temporal aspects of the neural code are inextricably linked³. For example, in the insect olfactory system the population activity that codes for a sequence of two odors differs not only from the representation of either odor alone but also from a mixture of the two⁴. Similarly, in the mammalian motor cortex, the temporal evolution of the population activity that codes for a movement describes a complex trajectory in state space⁵. An analogous situation may occur in rodent somato-sensory cortex, where responses to sequences of whiskers differ substantially from responses to individual whiskers⁶.

We asked if such a linkage of spatial and temporal coding is present also in visual cortex, and specifically in the representation of stimulus orientation by the primary visual area (V1). Orientation selectivity in V1 is a paradigmatic example of cortical elaboration of thalamic sensory afferents, providing a testbed for theories of cortical function^{7, 8}.

Studies of the dynamics of V1 population activity concentrated on membrane potential responses to individual stimuli, and found a dissociation of spatial and temporal coding. Using voltage-sensitive dye imaging, these studies found that the orientation selectivity of the population remained constant over the course of the response^{9, 10}. This invariance is consistent with tuning width measurements made from the membrane potential of single neurons: this tuning width typically remains constant during the course of a response¹¹.

Do the spike responses to individual stimuli show a similar dissociation of spatial and temporal coding? The dynamics of V1 spike responses to oriented stimuli have been studied extensively in single neurons. Over the course of the response, the orientation bandwidth of neurons has been found to sharpen^12–16, to broaden¹⁷ or to remain largely constant^{8, 18}. This variety may stem in part from differences in experimental procedures or in data analysis. Given the variety of results observed even within individual studies, however, it remains unknown whether the spike response to a single orientation of V1 populations shows any dynamics such as sharpening of selectivity over time.

More generally, it is not known if the population response to individual orientations can predict population responses to stimulus sequences. The interactions between responses to subsequent orientations could be complex. Indeed, the spike responses of individual V1 neurons to pairs of orientations display effects ranging from suppression to a repulsion of tuning curves^19–22. We don’t know to what degree these interactions may affect the overall population responses.

Similarly, it is not known how population activity evolves once the stimulus drive has been removed. Some V1 neurons give prolonged responses at stimulus offset²³, perhaps providing support to perceptual phenomena of visual persistence²⁴. These offset responses may reflect intracortical interactions that reverberate activity, possibly causing attractors in the dynamics of cortical activation^{25, 26}. Attractors would explain why spontaneous activity in V1 favors patterns that resemble those elicited by oriented stimuli^{25, 27}. Perhaps, once the responses attain one of these patterns, they linger there even after the initial drive is removed.

Finally, we don't know how stimulus sequences coded into population responses should be decoded by later stages of the visual system. Simple rules have been proposed for the decoding of stimulus orientation from V1 populations^{1, 2, 28–31}. These rules, however, have been rarely tested on actual population responses, and in particular they have never been applied to dynamically changing population responses. Can the time-evolving population activity of area V1 be decoded using simple rules?

Results

We monitored the dynamics of population activity in visual cortex, measuring both spikes and membrane potentials in identical stimulus conditions. To measure a population’s spikes we recorded from 10×10 electrode arrays³². To measure a population’s membrane potentials we used voltage-sensitive dye (VSD) imaging³³. VSD imaging reveals the membrane potential responses in superficial layers³⁴ with <10 ms temporal resolution^{9, 10, 33}. In V1, its signals reflect largely the responses of complex cells¹⁰.

Seeking to activate multiple subpopulations of neurons in rapid sequence, we used an “orientation noise” stimulus. In this stimulus, gratings of random orientation (each flashed for 32 ms in random phase) are presented in sequence^14,8. To study how cortical responses transition between resting and stimulated states and vice versa, we interleaved at random times a fraction (~30%) of blank frames, uniform gray screens that had the same mean luminance as the gratings. This orientation noise stimulus elicited lively spike responses³⁵ and strong VSD signals (Supplementary Fig. 1).

Population responses to an oriented stimulus

The orientation noise stimuli yielded detailed maps of orientation preference (Fig. 1a,b). We measured the average activation profile of the membrane potentials following a given orientation (Fig. 1a). Activation generally emerged ~45 ms after stimulus onset and assumed the characteristic patchy structure of orientation maps³⁶. As expected³⁷, stimuli with lower contrast elicited slower responses (Supplementary Fig. 2). Responses to orthogonal stimuli tended to be complementary, so the single-orientation maps could be combined in a map of orientation preference³⁶, where hue labels the preferred orientation and intensity reflects the selectivity of the response (Fig. 1b).

Fig. 1.

Population responses to an oriented stimulus. (a) Average maps of VSD fluorescence triggered on the appearance of a 90° grating, at various delays from grating onset. Scale bar indicates 1 mm. (b) Combining maps obtained with multiple stimulus orientations yields a map of orientation preference. Hue indicates the preferred orientation and brightness indicates tuning strength. (c) Summary of the membrane potential responses of the population. Responses of pixels with similar orientation preference were averaged and the result plotted as a function of preferred orientation (relative to stimulus orientation). The mean across orientations was removed. Black curves indicate best fitting Gaussian functions, error bars indicate ±1 s.e. (across 6 stimulus orientations). (d) Two superimposed population responses separated by 20 ms interval. (e) Summary of the spike responses of the population. The same methods as those in a–c were applied to the firing rate measured with the electrode array. (f) Two superimposed population responses separated by 20 ms interval. Panels a–d are from experiment 56-4-1, panels e,f from experiment 75-5-16.

To investigate the selectivity of these responses we expressed them as a function of preferred orientation (Fig. 1c,d). Having assigned to each pixel a preferred orientation (Fig. 1b), we summarized the activity of the population (Fig. 1a) by averaging together the responses of pixels with similar orientation preference⁹ (Fig. 1c). For this analysis, further, we averaged responses to stimuli of different orientations, expressed relative to the difference between stimulus orientation and preferred orientation. The shape of these responses remained constant: once responses at different times are normalized by their amplitude, their profiles become very similar (Fig. 1d).

We applied a similar analysis to the spike responses and obtained fairly different results (Fig. 1e,f). We identified the preferred orientation of each site in the electrode array, and averaged together responses of sites with similar preference (Fig. 1e). Unlike the membrane potential responses, the spike responses did show a dependence of tuning width on time, with population activity being more narrowly distributed at the peak of the response than at earlier or later times (Fig. 1f).

To compare the population responses measured from spikes and membrane potentials, we fitted them with circular Gaussian functions³⁸ (Fig. 2a–b). These provided excellent fits (Fig. 1c,e) and yield two parameters: amplitude and tuning width. For membrane potential the amplitude peaked 74±2 ms (n=8) after stimulus onset (Fig. 2a, blue). For spikes the amplitude peaked earlier, at 50±2 ms (n=6) (Fig. 2a, red). Tuning width (half-width at half-height) for membrane potential remained fairly constant during the course of the responses, and averaged 31.2°±0.3° (Fig. 2b, blue). For spikes, instead, tuning width decreased by 27%, from close to 30° to almost 20°, before broadening again (Fig. 2b, red).

Fig. 2.

Properties of single-orientation responses. (a) Time course of response amplitude (from Gaussian fits in Fig. 1c,e) for membrane potential (dark gray) and spikes (light gray). Shaded regions indicate ±1 s.e. (n=8 hemispheres for membrane potential, 6 hemispheres for spikes). Black dots indicate data points above background level. (b) Same, for the tuning width (half-width at half height). Only values corresponding to amplitudes above background level (black dots in a) are shown. (c) Time course of response amplitude for isolated single units (n=31). (d) Time course of tuning width for isolated single units for time points above background level (black dots in panel c). (e) Population responses for membrane potential (e) and spikes (f) were decomposed into two additive terms: a tuned component (which varies both in orientation and in time, shown on the bottom) and a baseline component (which varies only in time, shown by the black curve on top). To account for variability, responses are expressed as z-scores (mean across repeats divided by standard deviation across repeats). (f) Tuned component of the spike responses with the unturned component shown on top as well. Panel e is from experiment 56-4-1, panel f is from experiment 75-5-16.

The sharpening observed in population responses measured from spikes is seen also in individual neurons (Fig. 2c–d). We studied the spike responses of 32 well-isolated single units within the population. Similar to the multiunit spike activity, these single-unit responses peaked 50±2 ms after stimulus onset (Fig. 2c) and decreased in tuning width from close to 30° to almost 20° before broadening again (Fig. 2d). This clear sequence of sharpening and broadening during the course of the response was seen in all neurons, with no exceptions.

The difference in tuning width between membrane potential and spikes agrees with intracellular measures in single neurons, and is largely due to the spike threshold³⁸. The difference in timing is less expected. It is likely due to two factors. First, neurons tend to fire during the rising phase of the underlying membrane potential³⁹. Second, an early untuned depolarization^{8, 9, 11} may help the potential reach spike threshold in the initial portion of the responses. The properties of this depolarization are described below.

The population response to a single oriented stimulus could be decomposed into the sum two components, one untuned and one tuned^{8, 9, 11} (Fig. 2e,f). The untuned component is the mean activity across preferred orientations; it varies only in time (Fig. 2e,f, curves). The residual tuned component varies both in time and in preferred orientation (Fig. 2e,f colored panels). Consistent with earlier reports^{8, 9, 11}, the untuned component led in time the tuned component.

Responses to a sequence of orientations

Having characterized the population response to single oriented stimuli, we asked whether a sequential application of this elemental population response can predict the responses to the full stimulus sequence.

We first considered the membrane potential responses, and found that we could predict them based on simple summation (Fig. 3). We expressed the stimulus as a function of orientation and time (Fig. 3a), and predicted responses by repeatedly summing the elemental population response (Fig. 3b) appropriately shifted in time and orientation (a two-dimensional convolution). The result is a linear prediction of the responses (Fig. 3c). Aside from occasional discrepancies, seen especially in overall amplitude, the predicted responses resembled the actual responses (Fig. 3d). Both predicted and actual responses showed peaks of activity that shifted to coincide with the appropriate stimulus orientation. Distributions of predicted vs. actual responses cluster along a curve that deviates from linearity only for large responses (Fig. 3e). We could indeed improve on the predictions of the model by fitting a mildly compressive nonlinear function to these distributions and applying it after the summation⁴⁰. The resulting model explained 98±1% of the variance of the responses (s.d., n=8). The correlation between predicted and actual responses was high, at 0.69±0.06 (s.d., p<0.005., n=8).

Fig. 3.

Predicting the membrane potential responses of the population to the full stimulus sequence. (a) The stimulus expressed as an image, with ones (dots) indicating the sequence of stimulus orientations as function of time, and zeros elsewhere (gray). For graphical purposes, the orientation axis was duplicated to cover the full 360° range. Thus each grating appears as two dots separated by 180°. Only 2 s of stimulation are shown here (typical stimuli lasted 30 s). Dots have been shifted in time by 74 ms to compensate for the delay of the membrane potential signal. (b) Elemental population response to a single orientation (from Fig. 2e0). The asterisk denotes convolution. (c) Population responses predicted by convolving the stimulus with the elemental response. For graphical purposes we advanced these responses in time by 58 ms to appear roughly contemporaneous with the stimulus. (d) The measured population responses, similarly shifted in time. (e) Relationship between model predictions and measured responses. Dots are mean values taken across n=8 hemispheres. Gray regions indicate ±1 s.d. Panels a–d are from experiment 56-4-1.

We obtained similar results for the spike responses (Fig. 4). The summation model (Fig. 4a,b) captured the transient nature of these responses and the sequence of peak activations (Fig. 4c,d). Just as for membrane potential, the major deviations occurred for high and low values of the response (Fig. 4e). Thus, the model performance was improved by including a compressive nonlinearity, yielding a correlation between predicted and actual responses of 0.58±0.06 (s.d., p<0.005., n = 6, Fig. 4e).

Fig. 4.

Predicting the spike responses of the population to the full stimulus sequence. Format as in Fig. 3. Panels a–d are from experiment 75-5-16.

These results indicate that population responses to the stimulus sequence are simply the sum of consecutive responses to the individual elements in the sequence. On the other hand, there is a noticeable scatter in the relation between predicted and actual population responses, both for membrane potential (Fig. 3e) and for spikes (Fig. 4e). Are the deviations systematic? To address this question we focused on two key stimulus conditions: the transitions from orientation to orientation and the transitions from stimulus to blank.

Effect of recent history of stimulation

We asked if the summation model could explain how the population’s activity depends on the orientations seen in the immediate past (Fig. 5). We focused on four “orientation jumps”, in which consecutive gratings differed in orientation by: 0° (the same orientation), 90°, +45°, and −45°.

Fig. 5.

Predicting the interactions between population responses to successive orientations. (a) Average membrane potential responses of the population to two successive stimuli of the same (top panel) or different orientations. The schematic at left describes the stimulus. From top to bottom, successive stimuli differing in orientation by 0°, 90°, −45°, and +45°. (b) Predictions of the summation model for the data in a. (c–d) Same as a–b, for the spike responses of the population. (e) Attraction of population response profiles during ±45° jumps in orientation. Profiles are measured at the peak time of the second response (60.5±1.7 ms from the onset of the second stimulus). Curves are model predictions. Average of 5 hemispheres. (f) Same as in e, for the spike responses of the population. Panels a–b are from experiment 69-1-5, panels c–d are from experiment 75-5-16.

The membrane potential responses of the population represented faithfully the end points of the orientation jumps, but tended to interpolate through neurons with intermediate orientation preferences during the transitions (Fig. 5a, e). For 0° changes, the membrane potential response showed a single prolonged activation (Fig. 5a, row 1): there was no clear demarcation between the responses to the two stimuli even though in 75% of the cases the transition between stimuli involved a change in spatial phase. For a 90° jump, conversely, the population response showed a distinct transition (Fig. 5a, row 2), involving in sequence two separate subpopulations with orthogonal orientation preferences. The responses to ±45° jumps, finally, showed a mixture of these behaviors (Fig. 5a, rows 3,4); responses show two distinct peaks, but these peaks are joined by a response passing through the intermediate orientations. The profiles of the membrane potential responses during the transition showed a marked attraction between response to first and second stimulus: the peak of the population response shows an average shift of 20.7±2.6° (s.e., n=6) toward the preceding stimulus orientation (Fig. 5e). Similar results were obtained when examining smaller orientation jumps (23° or 30°, not shown).

Similar though less marked effects were seen in the spike responses of the population (Fig. 5c, f), which followed more faithfully the transitions between orientations. Though the responses to 0° changes showed a single prolonged activation similar to that seen in membrane potential (Fig. 5c, row 1), responses to jumps by 90°, 45° and −45° showed clearly distinct peaks, with little response in neurons with intermediate orientation preferences (Fig. 5c, rows 2–4). Just as for membrane potential, however, the population responses exhibited a marked attraction towards the orientation of the previous stimulus, with a shift of 6.3±2.8° (s.e., n=5) toward the preceding stimulus orientation (Fig. 5f).

All these effects were explained accurately by the simple summation model. From the population responses predicted by the model we computed average responses to orientation jumps (Fig. 5b,d). They closely resemble the data, explaining 91±3% (s.d., n = 8) of the variance for membrane potential (Fig. 5b), and 94.3±2% (s.d., n = 6) of the variance for spikes (Fig. 5d). Specifically, the model accurately predicts the attraction in population responses that follows a ±45° jump, both in membrane potentials and in spikes (Fig. 5e,f, dotted lines). Therefore, even when the population responses appear to be traveling waves that involve neurons with orientation preference intermediate between two successive orientations, these responses are more usefully described as the sum of two elemental waves – the successive responses to the individual orientations.

Persistence of population responses

We then asked how the population response evolves once the stimulus drive has been removed. We measured the population activity that follows stimulus offset and tried to explain it with the simple summation model (Fig. 6).

Fig. 6.

Unexplained persistence of population responses after stimulus offset. (a) When the visual stimulus is removed (stimulus-to-blank conditions, as shown on top by the contrast change), membrane potential activity in the population persists for well over 50 ms. Gray symbols indicate that before and after the stimulus-blank sequence, the stimulus could have any orientation or be blank (b) Same as in a, for spike responses. (c–d) The prediction of the summation model is substantially shorter for both spike responses and membrane potential. (e) The persistence is tuned for orientation: a separable model obtained through singular-value decomposition of the response shown in a yields a very similar profile. (f) Same as in e, for spike responses. (g–h) Temporal dynamics of responses (red) and predictions (black). Shaded areas indicate ±1 s.d. (n = 6 conditions). (i) Difference between the predicted and actual time courses averaged over hemispheres. Shaded areas indicate ±1 s.d. (n=8 hemispheres). (j) Same, for the firing rate of the population responses (n=6 hemispheres). Panels a,c,e from experiment 68-3-5, panels b,d,f from experiment 75-5-16.

The membrane potential responses showed a persistence that could not be explained by the summation model. We averaged all cases when a stimulus was followed by a blank (Fig. 6a), and compared them to the model prediction (Fig. 6c). The measured responses clearly outlasted the prediction (Fig. 6g, red vs. black). The population responses persisted 48±3 ms (n=8) longer than the predicted responses, with the peak difference occurring 72±24 ms after stimulus offset (Fig. 6i). Membrane potential responses, therefore, persisted well beyond what would be expected from the elemental response to a given orientation.

The inadequacy of the model in predicting the offset responses is not due to erroneous choice of model parameters. The model involves convolution with a filter, the average population response to a single orientation. This filter is biased toward stimulus-to-stimulus transitions (70% of the data). To assess the impact of this bias we computed the filter using only stimulus-to-blank transitions. By definition, the resulting model did a perfect job at predicting the offset responses. However, it did significantly worse in describing the average orientation-to-orientation transitions: explained variance dropped from 91% to 75%. Therefore, the summation model could not explain population responses to both kinds of transition: from stimulus to stimulus, and from stimulus to blank.

We observed a similar persistence in the population responses measured from spikes. The spike responses to a stimulus followed by a blank (Fig. 6b) had a longer tail than the prediction of the summation model (Fig. 6d). The responses during the blank stimulus persisted 30±4 ms (n=6) longer than predicted (Fig. 6h, red vs. black), and the peak difference occurred 64±31 ms after stimulus offset (Fig. 6j).

The failure of the summation model in predicting persistence is not simply due to an inadequacy in dealing with blank stimuli: the model encountered no difficulties in predicting the responses to stimuli following a blank. We saw no consistent difference between model predictions and actual responses in onset latency, peak amplitude, and width of the response. Thus, cortical patterns of activity produced by a stimulus are similar whether they follow another response or a period of spontaneous activity.

Two observations suggest that the origin of response persistence is cortical. First, the persistence is tuned for orientation: it involves the same subpopulation of neurons that were initially activated by the stimulus (Fig. 6e,f). Indeed, the averaged response to a stimulus followed by a blank was separable, both for membrane potential responses (Fig. 6, compare e to a) and for spike responses (Fig. 6, compare f to b). Second, the persistence is most pronounced in the superficial layers of cortex. In a control experiment using multiprobe electrodes spanning the depth of cortex, we computed field potentials and the associated profile of current source density. We compared this profile with the predictions of the summation model, and found persistence in the superficial layers but not in deeper layers (Supplementary Fig. 3). Together, these results suggest that response persistence is not inherited from the thalamus. Its causes may lie in the circuitry of cortex.

Decoding population responses

Finally, we asked how the spike responses of the population should be decoded to reconstruct the stimulus sequence. We have seen that at any instant, population responses depend not only on the present stimulus, but also on the recent history of stimulation. This dependence was explained by the linear summation of elemental responses to individual orientations. In principle, therefore, to estimate the present stimulus orientation, a decoder should know the rules of summation, the shape of the elemental response, and the recent history of stimulation. Could a decoder neglect these aspects and still interpret the population responses instantaneously and accurately?

We implemented a simple Bayesian decoder^{1, 28, 31} that operates on instantaneous population firing rates, with no knowledge of previous stimuli and responses or of the summation rule (Fig. 7a,b). This decoder quantizes time in 8 ms intervals. For each interval, it measures the likelihood that the present stimulus has orientation θ given a population response. This likelihood is the product of the probabilities of observing each of the responses in the bins of orientation preference, conditioned on θ. We obtained these probabilities from the averaged population response to a single orientation (Fig. 7a), by measuring the mean and variance across stimulus presentations (Fig. 7b).

Fig. 7.

Decoding stimulus orientation from the population responses. (a) The average response to an individual stimulus orientation. The dotted lines indicate the interval when this response is significantly higher than baseline. (b) Probability that a spike response is evoked by a specific orientation (computed within the time interval indicated in a). (c) Firing rates of the population in response to a stimulus sequence, averaged over 10 trials. (d) Firing rates for an individual trial. (e) Stimulus likelihood estimated by the instantaneous Bayesian decoder from the single-trial responses. (f) The stimulus sequence (shifted by 50 ms to compensate for response delay). (g) Confusion matrix comparing the estimated orientation to the actual stimulus orientation. (h) The average stimulus likelihood when a stimulus is followed by a blank. Panels a–h are from experiment 79-12-16.

Such a simple decoder predicted accurately the sequence of stimulus orientations (Fig. 7c–g). To be realistic, the decoder must operate on population responses measured in single-trials (Fig. 7d), which can be considerably noisier than responses averaged over trials (Fig. 7c). The decoder predicts a distribution of stimulus likelihoods (Fig. 7e). The maxima of this distribution are the decoded orientations. They matched very closely the actual stimulus orientations (Fig. 7f), with a circular correlation of r = 0.68 in the example data set (Fig. 7g) and of r = 0.67 ± 0.03 (s.e. p < 10⁻⁶) across n = 5 experiments.

Because of response persistence, during blank intervals the decoder continued to predict the presence of the previous orientation (Fig. 7h). To study this “perceived persistence”, we averaged all conditions when a stimulus was followed by a blank. Consistent with the persistence of population responses at stimulus offset (Fig. 6f), the distribution of stimulus likelihoods during a blank interval was not flat, but rather resembled the distribution obtained during the preceding stimulus (Fig. 7h). In other words, during the blank intervals, the decoder indicates that the stimulus has the orientation of the preceding stimulus. These perceived orientations simply reflect the fact that the population responses during a blank interval maintain a trace of the previous stimulus.

We performed this formal analysis on the spike responses as these constitute the output of area V1 and are read by subsequent visual areas. An informal analysis indicates that it would be equally easy to decode the population responses measured from membrane potential (Supplementary Fig. 1 and Supplementary Movie).

We conclude that it is not necessary for a decoder of population activity in area V1 to know the relationship between temporal and spatial encoding. We have seen that temporal and spatial encoding in V1 are linked to each other in two ways. First, the elemental population responses to a single orientation measured from spikes show sharpening over time. Second, the population responses retain a trace of previous stimuli, interacting with each other through summation. These properties of the coding mechanisms, however, can be safely ignored by a subsequent decoder. Indeed, a decoder with no knowledge of any linkage between temporal and spatial encoding could successfully predict the sequence of input stimuli.

Discussion

Using electrophysiology and imaging, we have studied how large populations of neurons in area V1 code for stimulus orientation. We described how population activity changes over time to reflect a rapidly changing sensory input, and we showed that the dynamics of this activity are summarized by a set of simple rules.

We first investigated the population responses to a single orientation. When measuring membrane potential, we found that the width of the population profile remained constant during the course of the responses. This result is consistent with earlier measurements obtained in populations with periodic stimuli^{9, 10}, and in single neurons with random stimuli¹¹. When measuring spike responses, however, we found that activity exhibited a pronounced sharpening during the course of the response. This sharpening was present not only in the overall population profile, but also in all individual neurons in the sample, with no exceptions.

The uniformity of sharpening across neurons is surprising given the variety of observations reported in previous single-neuron studies^{12–17,8, 18}. This variety may be explained by multiple factors. First, by differences in stimulus attributes such as size: large stimuli like ours lead to clearer sharpening⁴¹. Second, by differences in cell sample, including layers and location in the orientation preference map⁴². Third, by differences in data analysis, e.g. the way responses are normalized¹⁶, and whether one’s measure of orientation selectivity discounts an untuned response^{8, 9, 11, 14}.

We then investigated the interactions between oriented stimuli presented in succession, and asked if they could be predicted from the elemental responses to single orientations. We found that simple summation was surprisingly successful.

Our data reveal that strong interactions between successive orientations are prevalent in the population responses, both in membrane potential and in spike activity. Population activity is attracted towards the orientation of the preceding stimulus, and such attraction corresponds to repulsion of tuning curves of individual neurons. These results agree with studies in single neurons, where suppression near the conditioning stimulus¹⁹ can accompany an enhancement at other orientations, resulting in the repulsion of tuning curves away from the conditioning stimulus^20–22.

With the exception of one study²¹, previous reports of interactions between subsequent orientations in single neurons did not ask whether such interactions would be explained by simple summation. Our data support this interpretation. Specifically, summation could predict with high accuracy the population responses to changes in orientation. These responses can appear as traveling waves, peaking at a range of intermediate orientations between the stimulus orientations. However, the summation model reveals that they really are the sum of two elemental waves.

The fundamental linearity that we observed in population responses may be surprising given the nonlinearities seen in individual V1 neurons. The summation model is linear: population responses are a weighted sum of past stimulus orientations. The static nonlinearity at the output of the sum does not alter this fundamental linearity⁴⁰. By comparison, complex cells, which constitute the bulk of the signal in our imaging experiments¹⁰, are markedly nonlinear in their integration of spatiotemporal inputs^{43, 44}. Even simple cells, whose spatial summation properties are more linear⁴³ exhibit strong temporal nonlinearities⁴⁵. Perhaps the orientation noise stimulus places the cortex in a regime where it operates more linearly (possibly because contrast is mostly constant and fairly high). Indeed, as a first approximation the spike responses of individual neurons to this stimulus can be predicted by the summation model³⁵.

Yet, linear summation cannot be a complete description of V1 population responses. For instance, linear summation would not explain population responses to stimuli of different contrast. Linearity would predict that responses scale proportionally with contrast, while V1 responses vary nonlinearly with contrast, both in amplitude and in time course^{37, 43} (Supplementary Fig. 2). Accounting for these nonlinearities would likely require extending the model to include gain control mechanisms present at all stages of the early visual system⁴³.

Indeed, we found a novel deviation from linearity: activity persists once the stimulus drive is removed (Fig. 6). This prolonged activation could be seen as a prolongation of the response to the stimulus (“persistence”) or a response to the disappearance of the stimulus (“off-response”). We favor the first description for two reasons. First, the lag between an off-response and the preceding on-response would resemble the stimulus duration (32 ms), whereas the prolonged activation peaked at a later time. Second, an “off-response” is de facto already included in the average stimulus-triggered population responses: the stimulus disappears (on average half of the pixels change gray level) every time there is a change in orientation.

We argued that this neural persistence is likely to be due to cortical mechanisms, possibly implicating the strong recurrent excitation that characterizes cortical circuits⁴⁶. Recurrent circuits in area V1 may favor states that signal a single orientation^{7, 8, 25–27, 47}. Such states might act as attractors^{25, 26}. The stimulus before the blank might drive the network in one of these attractors; when the drive is removed, activity persists because it is transiently trapped in the attractor. Attractors are probably weak compared to the feedforward drive, or they would lead to striking hallucinations⁴⁷. However, their influence might be strong at low contrast or during a blank stimulus, when the feedforward drive is weaker or absent.

Perceptually, the neural persistence that we have observed may be a correlate of phenomena of visual persistence, which is in turn related to iconic memory²⁴. Such correlates had been previously found in higher visual areas^48–50. Smaller but similar effects have been seen in area V1²³. The dynamics of these responses, however, had been studied only in few neurons and had been characterized qualitatively. Our results indicate that persistence is robustly present in the population responses of area V1, and indicate that this persistence is not due to the off-responses that are generally expected from a receptive field.

Finally, we found that the time-evolving spike responses of a V1 population can be read-out through a simple instantaneous decoder. Multiple methods have been proposed for decoding stimulus orientation from V1 populations^{1, 2, 28–31}. These methods had been rarely tested on actual population responses, and in particular they had not been applied to reconstruct the attributes of stimulus sequences from dynamically changing population responses. We implemented a simple decoder that operates instantaneously, and found it to do an excellent job in predicting the stimulus orientation.

The good performance of the decoder is surprising for a number of reasons. First, it is typically assumed that a decoder needs to take into account the large amount of variability that is shared among neurons²; our decoder ignores this covariance and yet it performs very well. Second, our decoder ignores a number of factors that would appear to be crucial: the rules of interaction between subsequent elemental responses, the shape and duration of the elemental response, and the recent history of stimulation. We suggest that the linearity of summation that we have demonstrated reduces the importance of these factors. Indeed, these factors are essential to decode the population activity in systems where the spatial and temporal aspects of the neural code are inextricably linked³.

In conclusion, we have provided a set of rules that govern the dynamical code by which populations of V1 neurons represent the attributes of a rapidly changing stimulus. Such rules ultimately describe the computations performed by the underlying circuits. We have also provided a simple example of the strategy that downstream neurons could follow to decode the population responses. By elucidating both aspects – coding and decoding – these findings may ultimately help relate neural activity to perception.

Methods

Physiology

We report results measured in 7 cats (8 hemispheres) for the VSD experiments and in additional 4 cats (6 hemispheres) for the electrical recordings. Young adult cats (2–4 Kg) were anesthetized first with Ketamine (22 mg/kg i/m) and Xylazine (1.1 mg/kg i/m) and then with Sodium Penthotal (0.5–2 mg/kg/hr i/v) and Fentanyl (typically 10 µg/kg/hr i/v), supplemented with inhalation of N₂O (typically 70/30 with O₂). A 1 cm craniotomy was performed over area V1 (usually area 18, occasionally area 17). The eyes were treated with topical atropine and phenylephrine, and protected with contact lenses. A neuromuscular blocker was given to prevent eye movements (pancuronium bromide, 0.15 mg/kg/hr, i.v.). The animal was artificially respirated, and received periodic doses of an antibiotic (Cephazolin, 20 mg/kg IM, twice daily), an anti-edematic steroid (Dexamethasone, 0.4 mg/kg daily), and an anticholinergic agent (atropine sulfate, 0.05 mg/kg, i/m, daily). Fluid balance was maintained by intravenous infusion. The level of anesthesia was monitored through the EEG. Additional physiological parameters that were monitored include temperature, heart rate, end-tidal CO₂, and lung pressure. Experiments typically lasted 48–72 hours. Procedures were approved by the Institutional Animal Care and Use Committee.

Stimuli

The stimulus consisted of full-field stationary gratings flashed in random sequence for 40 ms each¹⁴. The gratings had one of 4 spatial phases and one of 4–8 orientations. Their contrast was typically 50%, and the spatial frequency was the optimal one as assessed by a preliminary experiment (typically, 0.2 cpd). Sequences were broken into 4–8 segments lasting 6 s each. Randomly interleaved with the gratings were blank frames (also lasting 40 ms), which occurred with a probability of about 30%. An additional 6 s control segment consisted entirely of blanks. Segments were presented in random order and each block of segments was generally presented 10 times. Stimuli were viewed monocularly with the eye contralateral to the hemisphere being imaged.

Imaging

Methods for VSD imaging were developed by Grinvald and collaborators^{9, 33} and described in our previous work¹⁰. We stained the cortex with the VSD RH-1692 and imaged its fluorescence in 15–30 mm² of V1. The dye was circulated in a chamber over the cortex for 3 hours, and washed out with saline. We acquired images with a CMOS digital camera (1M60 Dalsa, Waterloo, Ontario), as part of the Imager 3001 setup (Optical Imaging Inc, Rehovot, Israel). Images were acquired at a frame rate of 110 Hz, with spatial resolution of 0.028 mm per pixel. Additional spatial filtering was performed offline (bandpass, 0.2–2.2 cycles/mm), except when measuring overall activation (Fig. 2a). Frame acquisition was synchronized with the respirator. Illumination from a 100 W halogen light was delivered through two optic fibers. Excitation and emission filters were bandpass, at 630±10 nm and 665±10 nm.

Population responses were measured by grouping pixels in 24 bins according to their preferred orientation and averaging their responses to obtain a single data point for each preferred orientation bin.

To analyze the VSD responses to orientation noise, we subtracted the response to the blank 6-s segment from the response to each 6-s segment; this subtraction removed artifacts due to respiration, which was synchronized with sequence onset. The resulting responses were then bandpass filtered between 1 and 25 Hz.

Z-scores were computed by averaging the responses across the 10 repeats, and dividing responses by the s.d. computed across these repeats.

Array recordings

We implanted a 10 × 10 electrode array (0.4 mm separation and 1.5 mm electrode length) in the same patch of cortex. To avoid excessive cortical damage, the arrays were inserted at high speeds (around 8 m/s) using a pneumatic insertion device. Insertion depths were about 0.8–1 mm. Due to the curvature of the cortex the depth of penetration varied across sites. The array and surrounding tissue was covered in 1.5% agar to improve stability. Signals that crossed a specified threshold in each electrode were saved to disk for spike sorting at a later stage. These threshold crossing events are defined as multi-unit activity. Well tuned multi-unit activity was typically recorded from most of the 96 electrodes. Many electrodes also contributed well isolated single unit recordings. Traces were acquired at 12 kHz, and firing rates were obtained by low-pass filtering the spike trains with a cutoff at 25 Hz.

Untuned response component

The untuned component of the membrane potential and spike responses is the mean activity across preferred orientations. In VSD imaging, signals of such low spatial frequency are dominated by noise. We removed this component from population responses because it reflects the activation of the entire region regardless of preferred orientation.

Event-related analysis

To compute the average response to a generic oriented stimulus, we used event related analysis and averaged the responses in a 200 ms window around the time of occurrence of a given oriented grating. This procedure was applied each time a grating was presented and the data sorted and averaged according to the stimulus orientation.

Conditional average responses (second-order interactions) were computed with the same algorithm. Event related analysis was performed on those conditions in which a stimulus was preceded by (1) an identical stimulus (2) a stimulus with 45° orientation difference (3) 90° orientation difference, or (4) −45° orientation difference.

Predicted population responses

As illustrated in Fig. 3a,b, the linear predicted responses L_θ(t) are computed by convolving the average response to a stimulus F_θ(t) with the sequence of stimuli S(φ,t):

$$L_{\theta }(t)={\displaystyle {\int }_0^{\pi }d\phi }{\displaystyle {\int }_0^{\infty }S(\phi ,t)F_{\theta -\phi }(t-T)dT.}$$

Here, the stimulus S(φ,t) is 1 if the orientation of the stimulus presented at time t was equal to φ, and 0 otherwise, and θ indicates the preferred orientation of a subpopulation.

Finally, the predicted responses R_θ(t) are computed by passing the linear prediction through a static nonlinearity f to account for the discrepancy between linear model and prediction at high and low values of the responses: R_θ(t)=f(L_θ(t)). The static nonlinearity is fitted to the data so as to be optimal⁴⁰.

Percentage of variance

The percentage of variance for orientation jumps was computed as

$$<[1-\frac{{\displaystyle \sum _{t,\theta }{(R_{t,\theta }-P_{t,\theta })}^2}}{{\displaystyle \sum _{t,\theta }{(R_{t,\theta }-{\langle R_{t,\theta }\rangle }_{t,\theta })}^2}}]{>}_{S1\to S2},$$

where R and P are responses and prediction for a given orientation jump. The sums, ${\displaystyle \sum _{t,\theta }}$, and the average, 〈 〉_t,θ, are computed for time t between 50 ms and 150 ms after stimulus onset, an interval during which the response amplitude is significantly above the baseline noise. The average 〈 〉_s1→s2 is done over all stimulus transitions in 8 hemispheres.

Bayesian decoder

To decode population responses we computed firing rate over 8 ms time bins, and for each of these bins we estimated the probability that the stimulus had a certain orientation φ. Given a population response R = (r₁,r₂,…,r_n), and a set of orientations {φ₁,φ₂,…,φ_n} with equal probability of presentation p(φ), the probability of stimulus φ_j is given by

$$p({\phi }_j|r_1,r_2,\dots ,r_n)=p(r_1|{\phi }_j)p(r_2|{\phi }_j)\dots p(r_n|{\phi }_j)p(s)/{\displaystyle \sum _{i=1}^{n}p(r_1|{\phi }_j)p(r_2|{\phi }_j)\dots p(r_n|{\phi }_i)}$$

Here we make the simplifying assumption that p(r_i|φ) and p(r_j|φ) are independent. In reality they are not, but in separate analyses (not shown) we have also computed the covariance matrix and included it in the model, and found that doing so did not improve the quality of the decoding.

For each response r_i (i = 1,…n; n = 24) we computed the distribution p(r_i|φ_j) as:

$$p(r_i|{\phi }_j)=\frac{1}{\sigma \sqrt{2\pi }}{e}^{\frac{{(\overline{r}({\phi }_j)-r_i)}^2}{2{\sigma }^2}}$$

where r̄(φ_f) is the mean across stimulus presentations of the response to the stimulus φ_j, and σ is the associated standard deviation. We obtained the n = 24 tuning curves used to derive the distributions p(r_i|φ_j) (Fig. 7a,b), by doing 24 circular shifts of the average elemental response along the orientation axis. Only the time average of the elemental response (within a time interval centered on the peak time, Fig. 7a, dotted lines) is used to construct the tuning curves.