Novel stimuli evoke excess activity in the mouse primary visual cortex

Significance

Rapid detection and processing of stimulus novelty are key elements of adaptive behavior. Predictive coding theories postulate that novel stimuli should be encoded differently from familiar stimuli. Here, we show that the majority of neurons in layer 2/3 of the mouse primary visual cortex exhibit a significant excess response to novel visual stimuli. The distinction between novel and familiar images developed rapidly, requiring only a few repeated presentations. We show that this phenomenon can be described by a model of cascading adaptation. This ubiquitous mechanism makes it likely that similar computations could be carried out in many brain areas.

Abstract

To explore how neural circuits represent novel versus familiar inputs, we presented mice with repeated sets of images with novel images sparsely substituted. Using two-photon calcium imaging to record from layer 2/3 neurons in the mouse primary visual cortex, we found that novel images evoked excess activity in the majority of neurons. This novelty response rapidly emerged, arising with a time constant of 2.6 ± 0.9 s. When a new image set was repeatedly presented, a majority of neurons had similarly elevated activity for the first few presentations, which decayed to steady state with a time constant of 1.4 ± 0.4 s. When we increased the number of images in the set, the novelty response’s amplitude decreased, defining a capacity to store ∼15 familiar images under our conditions. These results could be explained quantitatively using an adaptive subunit model in which presynaptic neurons have individual tuning and gain control. This result shows that local neural circuits can create different representations for novel versus familiar inputs using generic, widely available mechanisms.

Because the behavioral consequences of a sensory stimulus can depend on whether that stimulus is novel or familiar, sensory systems can benefit from employing different representations of novel versus familiar stimuli. At the level of human psychophysics, stimulus novelty can enhance salience and capture attention (1–3), while familiarity can speed visual search (4). Novelty also affects aversive conditioning (5–7) and fear conditioning (8, 9). In human brain imaging, novel stimuli have been shown to generate the mismatch negativity (MMN) (10, 11) while repeated stimuli lead to repetition suppression (12). Explicit representation of novelty has been shown at higher stages of the sensory hierarchy, such as in the hippocampus (13) and inferotemporal cortex (14–16), and has been interpreted as a possible substrate of recognition memory (17). Lower in sensory hierarchies, the representation of novelty can be enhanced by stimulus-specific adaptation (SSA) (18–21) as well as by gain control (22, 23). Novelty signals are also prominently present in midbrain dopamine neurons (24).

Explicit representation of stimulus novelty is also related to theories of predictive coding, in which neural circuits carry out computations that emphasize novel or surprising information. Theories of predictive coding have had a long history, starting with ideas about how the receptive field structure of retinal ganglion cells more efficiently encodes natural visual scenes by removing redundant data (25–28) and including the idea that active adaptation may aid in this process (18). Theories of predictive coding in the neocortex have typically focused on the idea that feedback from higher cortical areas encodes a prediction about lower-level sensory data (29) that is subtracted from the lower-level representation, so that the signals traveling up the cortical hierarchy represent surprise or novelty (30, 31). However, a recent study failed to find these signatures of predictive coding (32).

Here, we investigate novelty processing in the mouse primary visual cortex. We repeatedly presented a set of images, each composed of a random superposition of Gabor functions, and then occasionally presented novel images drawn from the same ensemble. Using two-photon imaging of the Ca²⁺ sensor GCaMP6f to measure neural activity in layer 2/3 of awake, head-fixed mice (33), we found that the majority of neurons exhibited excess activity in response to a novel image. This distinction between novel versus familiar images was quickly reached, emerging with a time constant of 2.6 ± 0.9 s. Similarly, when we began presenting a new set of images, a majority of the neurons exhibited elevated firing that relaxed to a steady state with a time constant of 1.4 ± 0.4 s. When we presented novel images within larger image sets, the amplitude of novelty response decreased, defining a capacity of the system to encode ∼15 familiar images. All of these findings could be explained qualitatively using an adaptive subunit model in which neurons presynaptic to a recorded neuron have both individual tuning to visual stimuli and adaptive gain control.

Results

In order to explore how the primary visual cortex encodes novelty, we used two-photon Ca²⁺ fluorescence imaging in mice that were awake and head-fixed but free to move on a styrofoam ball placed below them (Fig. 1A) (34, 35). Animals were transgenics from a Thy1 line (GP5.3; Janelia) that expressed the protein GCaMP6f (36) in excitatory neurons. In order to locate the primary visual cortex, we first carried out large-scale brain mapping with a one-photon macroscope using drifting bars (Fig. 1B and SI Appendix, Supplementary Methods). We then selected a field of view in V1 and imaged at cellular resolution in layer 2/3. We manually identified regions of interest (ROIs) corresponding to cell bodies having a “halo” pattern of fluorescence indicating expression mostly in the cytoplasm (Fig. 1C). After averaging across pixels in an ROI to obtain the average fluorescence F, we constructed the time course of the fractional change in fluorescence (ΔF/F). The activity in single ROIs typically exhibited sparse events on a background (Fig. 1E).

Fig. 1.

Measuring a novelty response. (A) Awake mice were head-fixed and placed on an air-suspended styrofoam ball. Visual stimuli were projected on a toroidal screen surrounding the animal. Neural activity was recorded with a two-photon microscope. (B) Wide-field image of visual areas, as determined by one-photon fluorescence measurements (see Materials and Methods). The black square within area V1 shows the size of the field of view for the two-photon microscope. (C) Portion of a field of view taken by the two-photon microscope with ROIs shown in red. (Scale bar, 50 µm.) (D) Stimulus design. Step 1: Images were constructed from a superposition of randomly chosen Gabor functions. Step 2: A set of different images was formed and presented repeatedly in the same order; image sets are represented by plotting the image index versus time. Step 3: Occasionally, an image was substituted by unique novel images drawn from the same image ensemble. (E) Example activity traces with the times of novel image presentations shown in red; all novel images were unique. (F, Top) Matrix of trial-by-trial responses of an example cell to novel images. (F, Middle) Activity averaged across trials. (F, Bottom) Repeated sequence with the time of novel images shown in red.

In this two-photon setup a toroidal projection screen was placed around the mouse, which allowed us to display visual stimuli. We wanted to create an ensemble of diverse images that shared the same low-level statistics but were different in detail. We also wanted those images to drive V1 strongly. To this end, we chose images that consisted of a random superposition of 100 Gabor functions on a gray background. Those Gabor functions were drawn from a distribution that matched the range of receptive field parameters found in the literature for mice (SI Appendix, Supplementary Methods) (Fig. 1D) (37, 38). Images constructed in this way were statistically similar in global light level, contrast, and spatial scale. We then formed sets of three or four of those images, each displayed for 250 or 300 ms, depending on the specific experiment. In order to create a familiar stimulus, we repeated this image set many times without blanks in between. Occasionally, we substituted one image with a novel image drawn from the same distribution (Fig. 1D).

The Novelty Response.

In the novelty experiment we repeated a set of four images, each with a duration of 250 ms. A single image set was repeated for ∼10 min while novel images were randomly substituted every ∼6 s (SI Appendix, Supplementary Methods). All novel images were unique images drawn from the same ensemble of random Gabor images that the repeated image set was drawn from. Neurons generally showed a noisy, weakly modulated response to the repeated image set. However, when a novel image was substituted, neurons showed a large, brief response (Fig. 1 E and F). Some neurons responded to a majority of the novel images (example cell in Fig. 1F), indicating that their novelty response was somewhat unspecific to the particular locations and orientations of the Gabor patches overlapping the cell’s receptive field.

To quantify this response across the neural population, we subtracted the activity triggered by the repeated image set from that triggered by the presentation of a novel image (Fig. 2A, black line). We found that a large majority of the 1,134 neural responses in our sample showed, on average, excess activity to a novel image (Fig. 2B) (P < 0.05 for 878/1,134 = 77%; SI Appendix, Supplementary Methods). The amplitude of the excess activity varied across the population ranging from slightly negative for some neurons up to values greater than ΔF/F ∼ 0.2 (Fig. 2C). We formed averages over groups of neurons sorted by their rank (Fig. 2D); this analysis showed that the temporal dynamics of the novelty response were the same, regardless of response amplitude.

Fig. 2.

Population summary for the novelty response. (A, Top) Repeated image set containing novel images shown in orange. (A, Middle) Activity of one example neuron, averaged across trials with and without novel images (red vs. blue). (A, Bottom) Excess activity due to the occurrence of novel images (black). (B) Excess activity for 1,134 trial-averaged neural responses (rows) plotted versus time relative to the occurrence of novel images and sorted by response amplitude (color scale = z-score). (C) Histogram of amplitudes of the excess activity for all cells. (D) Excess neural activity (normalized) for different groups of neurons sorted by response rank (colors). (E) GCaMP6f response to a single spike, taken from ref. 36. (F) Population-averaged excess activity (black line; ΔF/F) with a curve fit capturing the response dynamics (dotted black line) and a model of the spiking rate (blue line).

In order to better quantify the response amplitude, we first averaged the raw responses across all the neurons (Fig. 2F, black line). Then, we fit a simple curve to capture the dynamics of the novelty response (Fig. 2F, dotted line). Finally, we quantified the calcium dynamics by analyzing published data (36) (Fig. 2E) and deconvolved to estimate the spiking rate of the population as a function of time (Fig. 2F, blue line). Even though the averaged excess signal was relatively small in peak amplitude (∼0.04 ΔF/F), deconvolution revealed a significant increase in spike rate from a baseline of ∼1.2 Hz per cell up to a peak of ∼4 Hz per cell. The integrated area of this excess response corresponded to about 0.5 spikes per cell per trial. Multiplying by the number of neurons, we estimate that a novel image elicits ∼150,000 excess spikes in all of V1 (SI Appendix, Supplementary Methods). Therefore, the fact that the novelty response was present in such a large fraction of neurons led to a substantial increase in population activity.

Neuropil Subtraction.

Because the novelty response was so widely distributed among neurons, we were concerned that signals from the neuropil might be making a significant contribution. To this end, we estimated the local neuropil activity around each neuron by constructing an annulus around the soma and averaging over all of the pixels in this annulus. We first identified fast events in the neuropil signal, which are likely to be action potentials from other neurons. If these signals were to “bleed over” into the ROI we were observing, then we would see a corresponding event, albeit at a small amplitude, in the ROI for every such fast signal. We found many examples of fast signals in the neuropil that were not reflected in the ROI (SI Appendix, Fig. S1A), implying that contamination of the ROI signal by the neuropil was not significant. However, it is still possible that this contamination was present but noisy enough that it was hard to measure on a single trial.

To this end, we constructed the average activity in the ROI triggered on the peak of fast events in the neuropil, with both signals normalized so that the peak of the neuropil signal had an amplitude of unity (SI Appendix, Fig. S1B). The peak signal in the ROIs was 0.27, which implied that neuropil may have contaminated the ROI signal by as much as 27%. However, it is also possible that fast events in the neuropil were distinct from fast events in the ROI but were correlated in their occurrence. To further investigate, we examined the trial-by-trial amplitude of fast neuropil events versus the corresponding amplitude of ROI signals both as a histogram of the response ratio (SI Appendix, Fig. S1C) and as a scatter plot (SI Appendix, Fig. S1D). These data have considerable scatter, but they do not rule out the possibility of direct contamination.

Therefore, we examined the novelty response after subtracting a maximal amount of contamination from the raw fluorescence trace, F_cell = F_ROI – 0.27 F_neuropil. This neuropil subtraction had a negligible effect on the novelty response (SI Appendix, Fig. S1 E and F). Exploring the neuropil subtraction process in more detail, we found that the population-averaged fluorescence in the neuropil and ROIs were qualitatively the same, so that subtraction only produced a small baseline offset (SI Appendix, Fig. S1G). At the single-cell level, subtraction reduced the fraction of neurons with a significant novelty response from 0.89 to 0.79 in this examined dataset (SI Appendix, Fig. S1 H and I). Overall, we concluded that the novelty response is clearly present in individual neurons and not merely an artifact of neuropil contamination. Because our data are also consistent with the neuropil containing signals that are distinct but correlated with those in the ROI, we subsequently report uncorrected fluorescence data.

At the same time, we found that novelty responses were strongly reflected in the average neuropil signal. When we plotted the mean neuropil signal following a novel image, we found clear novelty responses in all of the annuli studied (SI Appendix, Fig. S2 A and D). This signal was similar to that found in ROIs for both its amplitude distribution (SI Appendix, Fig. S2B) and population average (SI Appendix, Fig. S2C). Overall, the novelty response in the neuropil was similar to that found in ROIs, but with somewhat greater extent. These results are all consistent with the finding that most neurons had a novelty response, as processes from nearby neurons comprise much of the neuropil.

Is the Novelty Response Caused by Surprise?

Sudden, unexpected visual stimuli, like a dark looming disk from above (39) or the onset of a bright light (40), can cause strong behavioral responses in mice, like flight or freezing. Thus, the novel image might trigger changes in locomotion, which could then influence neural activity. To address this possibility, we tested for changes in running speed at the onset of a novel image. Even though our mice showed strong alternations between restfulness and bouts of running (Fig. 3A), we found that there was no change in running speed that was triggered by a novel image (Fig. 3B). From these control analyses, we conclude that the novelty response is not the result of the animal’s change in locomotion.

Fig. 3.

The novelty response is not caused by running, pupil size, or eye movements. (A) Traces of behavioral variables during the course of a 1-h experiment. (Top) Pupil displacement from resting position. (Middle) Pupil diameter. (Bottom) Running speed. (B) Behavioral variables triggered on the occurrence of novel images (red dotted line). Shaded areas are uncertainty estimates. (Top) Pupil displacement (blue). (Upper Middle) Pupil diameter (green). (Lower Middle) Running speed (red). (Bottom) Trial-averaged response of the neural population (black).

Could the novelty response be the result of other forms of behavioral surprise? Surprise can also lead to activation of the sympathetic nervous system, which in turn can increase the evoked firing rates of neurons in V1 through release of norepinephrine and acetylcholine in V1 (41–44). Changes in pupil diameter can be used as a proxy for changes in norepinephrine activity (45). Thus, we tracked the animal’s eye position and measured their pupil diameter. This allowed us to test whether the presentation of a novel image changed the pupil diameter of the mouse. While pupil diameter did fluctuate strongly during a recording session (Fig. 3A), this fluctuation was not correlated with the onset of a novel image (Fig. 3B).

Finally, behavioral surprise could cause eye movements, which could then lead to increased neural activity. Thus, we tested whether the novel image triggered saccadic eye movements. We found no correlation between the onset of a novel image and eye velocity (Fig. 3B). Together, these analyses strongly suggest that the novelty response is not caused by a generalized startle or behavioral surprise response. While this result might seem counterintuitive, it is consistent with the fact that the presentation of a novel image under these conditions has very low salience for human observers. Readers are encouraged to view video clips of the visual stimulus (Movies S1 and S2).

Influence of Locomotion and Eye Movements.

It is known that Ca²⁺ signals in V1 can be modulated by the animal’s locomotion (46–48). Perhaps the novelty response is greatly enhanced by locomotion? To test this possibility, we averaged the Ca²⁺ response to a novel image over periods when the animal was either running or still. We found an enhancement of the novelty response during running (SI Appendix, Fig. S3A), similar to previous reports. Combined across our entire recording, we found a mild positive correlation between running speed and neural activity (SI Appendix, Fig. S3B).

Pupil dilation can also modulate neural activity in the primary visual cortex. To test whether pupil diameter enhanced the novelty response, we similarly averaged neural activity on trials with dilated versus constricted pupils. Pupil dilation led to a substantial increase in baseline activity but little change in the amplitude of the novelty response relative to baseline (SI Appendix, Fig. S3C). At the same time, the correlation between pupil diameter and neural activity was stronger than for running (SI Appendix, Fig. S3D).

Finally, we also tested whether saccadic eye movements enhanced the novelty response. Similar to the influence of pupil diameter, we found that on trials with a saccadic eye movement neural activity had a higher baseline than for trials with no eye movements (SI Appendix, Fig. S3E). However, the amplitude of the novelty response relative to baseline was unchanged. We found fairly little correlation between saccades and neural activity (SI Appendix, Fig. S3F), suggesting that the increase baseline activity may have resulted from correlation between eye movements and the animal’s modulatory state.

Influence of Image Order.

Is the novelty response generated by a violation in the expected temporal order of images or by the spatial novelty of the image alone? To this end, we designed an order violation experiment by occasionally switching the order of familiar images rather than inserting novel images (SI Appendix, Supplementary Methods). We found very little excess activity in this experiment (Fig. 4), suggesting that the novelty response does not signal a violation of the temporal order per se, but rather the spatial novelty of an image. We nevertheless retained a stimulus design that presents familiar images in a consistent order. This temporally regular presentation of familiar images within a set avoids the trial-to-trial variability that would be otherwise be caused by a randomized presentation order.

Fig. 4.

The novelty response is not caused by changes of image order. (A) Population-averaged excess activity triggered by novel images. (B) Population-averaged excess activity triggered by an image order violation. In A and B, error bands were computed by first taking the average of all cell responses for each mouse and then computing the SEM across those five traces (gray). Error bands are therefore indicative of mouse-to-mouse variability.

Emergence of the Novelty Response.

Conceptually, a neuron cannot exhibit a differential response to a novel image until the original set of images is repeated at least once. We wanted to see how many repetitions of a sequence were necessary so that neurons would show an elevated response to a novel image. To this end, we designed a variable repetition experiment in which we displayed sets with three images of 300-ms duration. We presented a given image set in a block with either L = 1, 3, 8, 18, or 38 repetitions of the image set before showing a novel image (Fig. 5A and SI Appendix, Supplementary Methods). Each block contained a unique image set and a unique choice of repetitions, L. Blocks followed each other seamlessly with no intertrial period or blank frame.

Fig. 5.

Novelty responses emerge quickly within a repeated sequence. (A) In the variable repetition experiment, we formed blocks containing a new, randomly chosen image set (shown in blue; previous block shown in green). This unique image set was presented once and then immediately followed by L repetitions. Next, an image set was presented with a unique novel image substituted (shown by red bars). Finally, two more image sets were presented before a new block began (shown in purple). (Bottom) Neural activity averaged over the entire population for different choices of L (curves offset for clarity). (Top) Amplitude of the novelty response versus the number of repetitions, L. Dotted line: Exponential curve fit; error bars are SEM over n = 5 mice. (B, Top) Design of the repeated image set experiment. A given image set was repeated in a block until adaptation reached steady-state (blue). The same image set was presented again after a variable delay during which other image sets were presented (green). (B, Bottom) Transient response amplitude versus delay with an exponential curve fit (red); error bars are SEM over n = 5 mice.

We found that the novelty response emerged rapidly. Significant excess activity was observed in the population after as few as L = 1 repeats of a new image set (Fig. 5A, Bottom). The effect increased with more repetitions and saturated at L ∼ 20 (Fig. 5 A, Top). Fitting an exponential curve to the effect amplitude versus number of repetitions revealed a time constant of τ_emerge = 2.6 ± 0.9 s, or alternatively 2.9 ± 1.0 repeats, for the emergence of the novelty response.

The Transient Response.

We also noticed that neurons exhibited elevated activity when we began presenting a new image set (Fig. 5A, elevated activity near t = 0). In this experiment, each adaptation block was always preceded seamlessly by a block that used a different image set with no intertrial period or blank frame in between. Therefore, this initial elevated activity is conceptually very similar to the novelty response. This transient response adapted strongly as a given image set was repeated, quickly reaching a steady state. This adaptation process had a time constant of τ_adapt = 1.4 ± 0.4 s in the variable repetition experiment.

A transient response is also evident after the occurrence of a novel image, because then a new image set was presented (Fig. 5A, elevated activity near t = 40 s). Similar to the case of the novelty response, a given image set must be repeated before a response can emerge from the transition to a novel sequence. We measured the amplitude of the transient response elicited by the transition to a novel sequence as a function of the number of repeats of the preceding image set, L, and found that the emergence of the transient response closely paralleled that of the novelty response (Fig. 5 A, Top).

How long does adaptation to familiar images persist? All sequences used in the variable repetition experiment were different. To answer this question, we designed a repeated image set experiment that adapted neurons to a given image set and then, after various lags, presented the same sequence again (Fig. 5B and SI Appendix, Supplementary Methods). After a sufficiently long recovery time, we expect the transient response to recover to its maximum amplitude, similar to what we found for the emergence of the novelty response. To make sure we did not create transition effects due to the lack of neural stimulation during the recovery period, we filled this period with other image sets.

We measured the amplitude of the transient response as the difference between the response to the last few repeats of the previous sequence and the peak calcium fluorescence of the response elicited by the new sequence. The previous sequence was repeated sufficiently often so neurons had reached steady-state activity. When we plotted the amplitude of the transient response as a function of the time since the same image set was presented, we found an exponential rise with a time constant of τ_recovery = 33 ± 11 s (Fig. 5B). The amplitude at a time interval of zero was set to zero, because in this case there was no pause between the adaptor stimulus and the probe, and thus no transition that could elicit an excess response.

The Novelty Response as a Probe of Capacity.

In order for a neural circuit to distinguish novel from familiar images, it must maintain some representation of familiar images. To explore the limitations of this representation, we designed a variable image number experiment, in which we varied the number of images in the set. Similar to the variable repetition experiment, we formed adaptation blocks. In each block, i, we randomly chose the number of images to be S_i = 3, 6, 9, or 12 and then randomly generated a new image set of this size (Fig. 6A and SI Appendix, Supplementary Methods). In all blocks, the given image set was presented for 17 times before a novel image was introduced in the 18th trial. This allowed us to measure the amplitude of the novelty response after adapting to image sets of different size.

Fig. 6.

Novelty response from within larger image sets. (A) Image sets with either S = 3, 6, 9, or 12 images were presented 17 times before one of the images was substituted by a unique novel image. All images were presented for 300 ms each. (B) Trial-averaged response of the neural population to novel images during image sets of different size, S (shown in color). (C) Amplitude of the novelty response versus image set size, S. Dotted line is an exponential curve fit; error bars are SEM for n = 5 mice. (D) Population-averaged steady-state neural activity versus image set size, S; error bars are SEM for n = 5 mice.

We found that the novelty response systematically decreased as the number of images, S, increased (Fig. 6 B and C). This result is consistent with the intuition that as the neural circuit encodes more images, then yet another image will seem less novel. Although the novelty response decreased with larger image sets, it was still present for sets of S = 12 images. This indicates that under our experimental conditions the capacity was at least as large as 12 images. Another way of characterizing the capacity is to examine the trend of novelty response amplitude versus size of the image set, S. Here, we found an exponential decrease with a decay constant of τ_encode = 15 images (Fig. 6C). This decay constant can be seen as a measure of the capacity to store familiar images.

One way that the neural circuit could generate a smaller novelty response could be by having a higher steady-state response for larger image sets but always the same response level to novel images. In this case, there would be less dynamic range to generate a large novelty response. However, we found that this was not the case (Fig. 6D). Because the adaptation process reached the same steady-state activity as a function of the image set size, S, we interpret the decrease in the novelty response amplitude as arising from limitations in the representation of familiar images. The constancy of steady-state neuron activity across image sets of different sizes is consistent with previous observations that cortical adaptation helps to achieve a form of population homeostasis (49).

Adaptation versus Steady-State Response.

In order to study the connection between this novelty response and known adaptation phenomena in V1, we examined how the amplitude of the transient response depended on the steady-state activity. In classical contrast adaptation in V1, neurons show a transiently elevated firing in response to higher contrast stimuli that adapts down to a lower steady-state level. Then, when contrast is reduced, firing rates start lower and recover back to a higher steady-state level (Fig. 7 A, Left) (50, 51). Our results were qualitatively different, in that a transition to any new set of images always caused a transiently elevated activity that decayed downward to a new baseline that could be higher or lower than the previous steady-state activity (Fig. 7 A, Right). Although our stimuli do not differ globally in contrast, the contrast in a local region the size of a cell’s receptive field does differ across images. In addition, the local orientation can change across images, which can drive orientation-selective neurons at different strengths.

Fig. 7.

Classical contrast adaptation versus the novelty response. (A, Left) Schematic of classical contrast adaptation. The transition from high to low contrast typically causes a reduction in firing followed by a slow recovery to higher steady-state firing. (A, Right) Schematic of the novelty response. All transitions to a new image set cause a transient response, regardless of the steady-state activity. (B) Example trace of population neural activity for transitions to image sets that drive stronger (Top) versus weaker (Bottom) steady-state activity. (C) Amplitude of the transient response versus the ratio of steady-state response to the current versus preceding image set; dots are for each of 36 rank-ordered groups of neurons (see main text). (D) Transient activity versus steady-state activity; dots are averages over rank-ordered groups of 60 neurons. Black dotted line: unity; red line: linear curve fit. (Upper Inset) Normalized activity of neurons with large steady-state responses (from blue oval). (Lower Inset) Normalized activity of neurons with weak steady-state responses (from magenta oval).

In order to explore those different types of stimulus transitions further, we separated out individual stimuli according to how much steady-state activity they evoked in individual neurons. For this, we used the repeated image set experiment and calculated the event-triggered activity (ETA) of each neuron during each unique image set. Because of the great heterogeneity among neurons, we divided the ETAs from all of our measured conditions into six groups according to the rank of their average steady-state activity. We further divided each of these into six more groups according to the average steady-state activity during the preceding sequence. This resulted in a 6-by-6 matrix of activity traces, where rows show neural responses with increasing steady-state activity to the current image set and columns show increasing steady-state activity to the preceding image set (SI Appendix, Fig. S4). In this manner, we could systematically investigate how the transient response depended on the steady-state response of the current and preceding image sets.

We found a transient response with excess activity following nearly all transitions between different levels of steady-state activity. Of note, we show an example group with a transition to much larger steady-state activity along with the opposite transition (Fig. 7B). In both cases, a transient response was clearly seen. We summarized our results by plotting the amplitude of the excess transient activity versus the ratio of steady-state activity in current versus preceding image sets (Fig. 7C). We found robust excess transient activity across these conditions.

A more detailed analysis of the transient response showed a mix of additive and multiplicative component. To demonstrate this, we plotted the activity during the first presentation of a new image set (transient activity) versus the activity during the last presentation (steady-state activity; Fig. 7D). If the transient activity was a multiple of the steady-state activity, then these data would lie along a line with slope greater than 1. If instead the transient response was independent of the steady-state activity, then the transient activity would have an additive offset.

We found a combination of both effects. Neurons with larger steady-state activity had transient activity scaled by a factor ∼1.36, while neurons with smaller steady-state activity had an additive offset of ∼0.016 ΔF/F. Thus, in weakly responding cells, the additive component contributed the most, whereas in strongly responding cells the multiplicative component dominated. Because the population-averaged novelty response has an amplitude of ∼0.04 ΔF/F (Fig. 4A), the additive component contributes ∼0.016/0.04 = 40% and the multiplicative component contributes ∼60%.

In addition, we found that the dynamics of adaptation depended on the steady-state activity. Neurons with larger steady-state activity tended to have a slower decay of activity (τ_slow = 4.0 ± 0.3 s), reminiscent of contrast adaptation (Fig. 7 D, Upper Inset). Conversely, neurons with smaller steady-state activity had a faster decay of activity (τ_fast = 0.93 ± 0.06 s) (Fig. 7 D, Lower Inset).

A principal component analysis of the temporal profiles of activity following the transition to a new image set resulted in two principal components capturing 87% of the variance (Fig. 8 A and B). While the second principal component showed fast, transient activity, the first component showed a faster initial decay on top of a slower decay of activity. However, linear combinations of the two first principal components closely matched both the fast, transient response observed in weakly active cells and the slowly decaying response observed in strongly active cells (Fig. 8 C and D).

Fig. 8.

Two principal component analysis (PCA) components capture most of the response variance when transitioning from one set of images to the next. (A) The first five PCA components, scaled by their component weights, from responses transitioning from one set of images to the next set of images. The duration of one repeat of an image sequence was 900 ms. (B) Variance explained by each of the first 20 principal components. The two first principal components captured 87% of the variance. (C) The decay in response amplitude of the strongly active cells was well captured by a linear combination of the first two principal components. (D) Similarly, the decay in response amplitude of the weakly active cells was also well captured by another linear combination of the first two principal components.

We can understand these results by assuming that neurons exhibit a combination of novelty response and classical contrast adaptation. Because the novelty response tended to have a modest amplitude, this activity would barely be visible in a neuron with large steady-state activity. Since these neurons would typically have lower steady-state activity in the preceding sequence, we can think of the neuron as experiencing higher effective contrast, which would trigger contrast adaptation. On the other hand, neurons with weak steady-state activity would be experiencing a lower effective contrast or would simply not have enough steady-state activity to exhibit classic contrast adaption. Thus, for these neurons, most of their activity would arise from the novelty response. Together, these results argue that the novelty response may be distinct from contrast adaptation in V1 neurons.

Adaptive Subunit Model.

In order to explain and unify our observations, we formulated an adaptive subunit model. The idea is that a given neuron that we recorded from in layer 2/3 received inputs from many other cortical neurons (subunits) that need not have been observed and that each of these presynaptic neurons had its own individual tuning to visual stimuli along with a cell-intrinsic mechanism of adaptation (Fig. 9A). Because each presynaptic neuron had different tuning to stimuli, there was always a subset of those subunits that had high sensitivity to any novel image. As that image was repeated, those subunits adapted, giving rise to lower activity in the recorded neuron. At the same time, subunits that were poorly stimulated by a given set of images will recover their full sensitivity, so that excess activity was observed in response to a different image.

Fig. 9.

Adaptive subunit model. (A) Schematic of the model: Images, i, provide input to M subunits, j, via synapses, p_ij, and subunits provide input to N neurons, k, via synapses, q_jk. (B) Illustration of subunit dynamics: A sequence of images (Bottom) drives sparse responses in a subunit (Top) that decrease over time due to a reduction in gain (Middle) during each image presentation. (C) Population-averaged firing rate (Bottom) in neurons during presentation of three different image sets (Top). (D) Population-averaged firing rate during adaptation blocks with different numbers of repetitions of the same image set before the presentation of a novel image, L; dotted line is an exponential curve fit. (E) Amplitude of the novelty response, N, versus number of repetitions, L; dotted line is an exponential curve fit. (F) Population-averaged firing rate following presentation of a novel image during adaptation blocks with different sizes of image set, S. (G) Amplitude of the novelty response, N, versus size of the image set, S; dotted line is an exponential curve fit. (H) Amplitude of the transient response, T, plotted as a function of the time interval between image set presentations; dotted line is an exponential curve fit that defines the recovery time, τ_recovery. All error bars are SEM calculated over n = 4 instantiations.

In this model (SI Appendix, Supplementary Methods), subunits have a cellular time constant, τ_cell, chosen to be 20 ms, along with a time scale of adaptation, τ_slow, chosen to be 30 s to match the measured recovery time constant (Fig. 5B). The most likely mechanism underlying this gain control is an adaptive ion channel (52), such as the Na⁺-dependent K⁺ current found in V1 neurons, which has a time constant of 10 to 30 s (51, 53). Other mechanisms of spike frequency adaptation can have similar time scales and effects (54).

This process of adaptation causes the subunit’s gain and hence its response to repeated presentations of the same image to decrease (Fig. 9B). Notice that the induction of adaptation depends on τ_slow as well as both strength parameters, p₀ and q₀. However, because of the sparseness of subunit activation, recovery is determined primarily by τ_slow. Recorded cells, k, integrate over all of their subunits with randomly chosen synaptic weights, q_jk, and with the same cellular time constant, τ_cell. The model contains many recorded cells and averages over their activity to compare with population-averaged experimental data.

Population activity in the model starts out high during the presentation of a new image set and then adapts down to a lower steady state (Fig. 9C). When a novel image is presented, there is excess activity (Fig. 9C). This excess activity decays to a steady-state baseline with a time constant, τ_adapt = 2.3 s (Fig. 9D), that agrees with experimental data (Fig. 5). All new image sets and novel images evoke similar levels of increased activity.

When we increase the number of image-set repetitions, L, before the presentation of a novel image, the amplitude of the novelty response, N, increases (Fig. 9D) with a roughly exponential dependence on L (Fig. 9E). This trend is dominated by the fact that the transient response had not fully adapted away for small L. In addition, there is a modest recovery of sensitivity of the subunits most strongly driven by the novel stimulus that leads to a small increase in the absolute amplitude of the novelty response. This recovery occurs because most of the subunits driven by the novel image were not activated by the image set, allowing their gain to recover.

When we increase the size of the image set, S, the amplitude of the novelty response, N, decreases (Fig. 9F) with a roughly exponential dependence (Fig. 9G). This phenomenon results from cross-adaptation. Subunits tended to have a strong response to a small fraction of all the images and weak responses to the other images. With a larger image set, the weak responses to these other images induces increasingly more adaptation. Because of this accumulated weak adaptation, subunits had a smaller gain when the novel image is presented.

Finally, the transient response for a given image set, T, recovers with a time constant of τ_fit = 30.2 s, consistent with our measurements (Fig. 5B). Taken together, the adaptive subunit model provides a qualitative match for all of our experimental observations with 6 free parameters (SI Appendix, Supplementary Methods). Only three of these parameters were fully adjusted to match experimental data (p₀, q₀, and τ_slow); the others were constrained to reasonable values.

Discussion

In this study we found that the excitatory neurons in layer 2/3 of the mouse primary visual cortex exhibited a distinctive pattern of excess activity when a novel image was presented among a set of familiar images. This excess activity consisted of moderate spiking, estimated to be ∼0.5 spike per neuron, which was present in a large fraction of all neurons (Fig. 2B). Because of this widely distributed activity, the population spiking rate increased by a factor of approximately fourfold. This excess activity did not result from changes in running speed, pupil diameter, or eye movements (Fig. 3 and SI Appendix, Fig. S3). When we switched from one familiar set of images to a new set, there was a similar distributed pattern of transient excess neural activity.

The process that determined the novelty of an image was rapid, as the differential response between novel and familiar images could be observed after only one repetition of an image set (Fig. 5A). The characteristic time scale over which the novelty response emerged was approximately three repeats (Fig. 5 A, Inset). Similarly, the excess activity evoked by the switching to a new image set decayed to steady state after approximately two repeats of that image set. The familiarity of images must be maintained by some form of memory. By presenting novel images in the context of image sets of different durations, S, we found that the novelty response decayed for larger S (Fig. 6). This defined a capacity for the system to encode ∼15 familiar images, under our experimental conditions. This representation of familiarity decayed relatively slowly, with a recovery from adaptation on a time scale of ∼30 s (Fig. 5B).

Mechanisms of the Novelty Response.

In order to gain insight into the circuit mechanisms responsible for novelty responses, we formulated an adaptive subunit model. In this model, each observed neuron received rectified input from many subunits that each had different tuning for stimuli—a form similar to Hubel and Wiesel’s model of the complex cell (55) as well as models of cascaded adaptation (56–58). In addition, each subunit had its own gain control. This model could account for all of the main qualitative features of our data, including: 1) excess activity evoked by all novel images, 2) rapid emergence of the novelty response among a new set of familiar images, 3) rapid decay of excess activity as novel images were repeated, 4) decay of the novelty response among larger sets of familiar images, and 5) recovery of full activity with a longer time scale.

While mechanisms of short-term synaptic plasticity, like depression, appear to be nearly ubiquitous in early sensory pathways (59), we did not explicitly include such terms. This was for two reasons: 1) The dominant time scale of synaptic depression in layer 2/3 of V1 is ∼400 ms (60), which is too fast to account for our observations, and 2) the gain control in each subunit adjusts the strength of presynaptic inputs in a manner similar to depression, obviating the need for another term. In our model, the gain control variable in subunits accounted for all adaptation in subcortical pathways as well as adaptation in layer 4 of V1. As such, we used a stronger gain control parameter (p₀ > q₀). Breaking up all of these processes into multiple biophysical mechanisms would presumably have allowed for a closer quantitative match to our observations but would have been achieved at the cost of a more complicated model. One strength of our resulting model is that it captures such a wide range of phenomena with a relatively simple structure.

Contrast adaptation also leads to excess activity when the stimulus changes to an image that strongly activates a given neuron (61). However, in classical contrast adaptation, the transition back to an image that more weakly activates that neuron evokes reduced activity that rises to its steady state (50, 51). Our data instead showed excess activity for all transitions between image sets, including those that led to lower steady-state activity (Fig. 6). Contrast adaptation in V1 is pattern-specific (62–64). This property could lead to excess activity for all stimulus transitions, as for each image there could be unadapted receptive field subunits that had high sensitivity (65). In fact, this structure is captured by our adaptive subunit model. However, some properties of the novelty response differ from pattern-specific contrast adaptation: 1) Compared to its steady state, the excess activity had not just a multiplicative component characteristic of contrast adaptation, but also an additive component (Fig. 7D) and 2) the dynamics of excess activity exhibited two principle components, with a slow component matching contrast adaptation and an additional fast component (Fig. 8). Furthermore, most studies of cortical adaptation have focused on steady-state activity and not on transient dynamics.

Another closely related phenomenon is SSA, first reported in the primary auditory cortex (19). Construing this phenomenon broadly, our findings are certainly an example of adaptation that is stimulus-specific. While our specific experimental design differs from previous experiments, our results share many features of SSA reported in A1 (66–68). Construed more narrowly, SSA, defined as a reduced response to familiar stimuli, has been distinguished from deviance detection per se, a phenomenon linked to the MMN (69). One paradigm to define deviance detection is to compare the response of a neuron to an equally rare stimulus when the background is a single common stimulus versus many different stimuli (the many standards control) (70).

Neural deviance responses have been observed in V1 (21) in awake mice and A1 first in membrane potentials of pyramidal cells and interneurons as a delayed signal (71) and also as spikes in awake mice (72). Deviance detection in the auditory system builds along the hierarchy from inferior colliculus to the auditory belt (72) and is found prominently in the prefrontal cortex (73). It is also noteworthy that auditory deviance detection was statistically significant in awake mice but not anesthetized rats. Deviance detection in the visual system was decreased when prefrontal cortex input to V1 was inhibited (74). This suggests an involvement of the prefrontal cortex specifically in the generation of a deviance response. This could potentially explain the absence of a statistically significant deviance response in A1 of anesthetized rats.

In the context of these previous results, we believe that the novelty response most likely has components due both to stimulus-specific adaptation and deviance detection. However, we cannot make exactly the same decomposition, because we used different experimental design. It is also worth noting that the distinction between deviance detection and stimulus-specific adaptation does not cleanly map onto different circuit mechanisms. In particular, the adaptive subunit model, which might appear to be a model of stimulus specific adaptation alone, actually can exhibit deviance detection. This occurs when the subunits in the model respond to more than one stimulus, resulting in a greater level of “cross-adaptation” in the many standards control than in the oddball experiment. Given the diversity of neural tuning in the cortex—for even as simple a property as orientation tuning (75, 76)—the assumption of cross-talk in the adaptive subunit model seems reasonable. Because inhibition of the prefrontal cortex reduces the amplitude of deviance detection in V1 (74), deviance detection may have properties that are not captured by the adaptive subunit model. Further investigation will be needed to resolve the full set of mechanisms that give rise to the novelty response.

Interpretation of the Novelty Response.

The existence of a novelty response implies that the system has some representation or “memory” of familiar images. We want to be clear, however, that this form of memory may not have any relationship to the memories encoded by the hippocampus. In fact, the structure of the adaptive subunit model suggests that this memory could be primarily encoded by synaptic and cellular variables of neurons within V1. At the same time, recent results have shown that responses in V1 can be influenced by activity in the hippocampus (77, 78), so it would be premature to rule out any such involvement.

This form of memory for familiar images might not allow explicit retrieval of the remembered information, but it can help serve a “passive” role in determining whether an image has been seen before or not—a phenomenon known as recognition memory (79). While the perirhinal cortex is thought to play a central role in recognition memory (17), the novelty response in V1 may contribute to this ability as well.

The novelty response can also be thought of as a form of predictive coding, which emphasizes the representation of novel or surprising information. While many theories of predictive coding in the neocortex focus on the role of descending feedback (80, 81), mechanisms of adaptation can also contribute (82).

The fact that a simple and biophysically common mechanism can give rise to the novelty response suggests that similar processing may be present at many stages of the sensory processing hierarchy. If this operation were repeated at multiple stages, then adaptation would cascade and potentially get stronger, faster, or longer-lasting. In fact, more than half of the neurons in the human hippocampus had responses to a novel image that disappeared after a single presentation (13). In perirhinal and inferotemporal cortex, more than half of recorded neurons show an image-specific response reduction after a single presentation (14, 16, 83, 84). Furthermore, these novelty responses lasted for ∼24 h, as compared with the ∼60-s duration we measured in V1 (Fig. 4B). If a mechanism like the adaptive subunit model existed at multiple cortical stages, then novelty responses could be present for increasingly complex stimulus features at higher stages of the hierarchy, without needing to invoke further changes, like adaptive synapses (56, 85). Thus, this simple mechanism could lead to powerful adaptive processing across the cortical hierarchy.

Materials and Methods

A summary of our methods is provided here, with additional details contained in SI Appendix, Supplementary Methods. All experiments were performed according to the Guide for the Care and Use of Laboratory Animals (86), and procedures were approved by Princeton University’s Animal Care and Use Committee. In short, a cranial window was implanted over V1 in five transgenic mice expressing GCaMP6f under the thy1 promoter (36, 87). After a recovery period, activity was recorded from layer 2/3 neurons, while mice passively viewed repeated images containing random Gabor functions. For surgical details, details about the imaging procedure and our analysis pipeline SI Appendix, Supplementary Methods. Repeated and unique images were presented in various sequences, either for 250 ms or 300 ms per image. Experiments lasted between 10 min and 1 h. See SI Appendix, Supplementary Methods for a detailed description of the stimuli. We built a computer model that mimics our experimental results. This model contained two layers of units, representing layer 2/3 neurons and their upstream inputs. Those input neurons were differentially driven by the stimulus. Both sets of neurons dynamically adapted to their input. The model contained eight parameters that were either picked based on plausible assumptions or estimated form a fit to the data. Parameters and equations are described in SI Appendix, Supplementary Methods.

Data Availability

Raw data used for analysis are available upon request. All other data are included in the main text and/or SI Appendix.