Skip to main content
NCBI home page
Elsevier Sponsored Documents logoLink to Elsevier Sponsored Documents
. 2023 Feb 22;42(3):112096. doi: 10.1016/j.celrep.2023.112096

Locomotion-induced gain of visual responses cannot explain visuomotor mismatch responses in layer 2/3 of primary visual cortex

Anna Vasilevskaya 1,2, Felix C Widmer 1,2, Georg B Keller 1,2, Rebecca Jordan 1,3,4,∗
PMCID: PMC9945359  PMID: 36821437

Summary

The aim of this work is to provide a comment on a recent paper by Muzzu and Saleem (2021), which claims that visuomotor mismatch responses in mouse visual cortex can be explained by a locomotion-induced gain of visual halt responses. Our primary concern is that without directly comparing these responses with mismatch responses, the claim that one response can explain the other appears difficult to uphold, more so because previous work finds that a uniform locomotion-induced gain cannot explain mismatch responses. To support these arguments, we analyze layer 2/3 calcium imaging datasets and show that coupling between visual flow and locomotion greatly enhances mismatch responses in an experience-dependent manner compared with halts in non-coupled visual flow. This is consistent with mismatch responses representing visuomotor prediction errors. Thus, we conclude that while feature selectivity might contribute to mismatch responses in mouse visual cortex, it cannot explain these responses.

Keywords: neocortex, primary visual cortex, predictive processing, prediction error, locomotion, vision

Graphical abstract

graphic file with name fx1.jpg

Open in a new tab

Highlights

  • •

    Evidence for visuomotor mismatch responses signaling prediction errors is reviewed

  • •

    Locomotion-related gain of visual halt responses depends on visuomotor coupling

  • •

    Ongoing experience of uncoupled visual flow reduces locomotion-related gain

  • •

    Uniform locomotion-related gain cannot account for mismatch responses

In a recent issue of Cell Reports, Muzzu and Saleem (2021) propose that visual halt responses amplified by locomotion can explain visuomotor mismatch responses in V1. Vasilevskaya et al. argue that this explanation is insufficient because locomotion-induced gain is not uniform across stimuli, visuomotor coupling conditions, or experience.

Introduction

Predictive processing is a theoretical framework for a computational description of brain function. It postulates that an internal model of the environment is learned and used to predict sensory inputs.1,2,3,4,5,6 A key neural component of these models is prediction error neurons that compute and report discrepancies between actual sensory input and the prediction of sensory input. Prediction error signals, in turn, are used to drive corrective learning in the internal model. Predictive processing has long been hypothesized as a framework capable of describing cortical function,6 but physiological evidence for this has only started to accumulate in the last two decades across a variety of species and cortical areas.7,8,9,10,11,12 In the primary visual cortex (V1), much of this evidence has come from head-fixed mice locomoting in a virtual reality environment with visual feedback coupled to locomotion speed. Pyramidal neurons in layer 2/3 of V1 respond strongly when mismatches between locomotion speed and visual flow speed are presented, which consist of a brief halt in otherwise coupled visual flow during locomotion.9 One parsimonious explanation for these responses is that individual neurons compute differences between an inhibitory visual flow-driven input and a top-down excitatory prediction of visual flow based on locomotion.13,14,15 This would constitute a negative prediction error computation, driving spiking responses when there is less visual flow than expected based on locomotion. Several key pieces of evidence point toward mismatch responses resulting from this computation, which are summarized in Figure 1.

Figure 1.

Figure 1

Open in a new tab

Graphical summary of the key evidence that visuomotor mismatch responses in layer 2/3 of V1 arise from prediction error computation

Prediction error computation consists of a subtractive (or divisive, see Spratling16) comparison between sensory input and a prediction of that sensory input, for instance based on movements like locomotion. This can be achieved using a balance of excitation and inhibition from sensory and prediction sources. The key pieces of evidence supporting the idea that visuomotor mismatch responses in layer 2/3 of V1 represent prediction error signals are as follows:

(A) Mismatch responses cannot be replicated with the visual flow halts alone, uncoupled from locomotion.9,13,17 Here and in the subsequent panels of the figure, the black lines illustrate the mean population response of pyramidal cells in layer 2/3 of V1.

(B) The responses cannot be explained by a uniform gain increase of visual responses during locomotion (explained in detail in the main text).17 The lines on the right illustrate the population responses to the different stimuli (orange: mismatch, gray: playback halt during stationary periods, purple: visual flow during locomotion, green: visual flow during stationary periods).

(C) Responses likely cannot be explained by a surprise induced neuromodulatory signal, since calcium responses to local mismatches in small parts of the visual field evoke no pupil dilation (which is a proxy for neuromodulatory activity), but are equivalent in size to responses to full field mismatches (which do evoke pupil dilation).17

(D) Responses scale linearly with the degree of error between locomotion speed and visual flow speed.9,17

(E) Prediction errors should be computed from the difference between visual flow and locomotion speed, and visual flow and locomotion have opposing signs of influence on the membrane potential of layer 2/3 neurons, consistent with a subtractive comparison between the two sources of information.14 Somatostatin-positive interneurons are consistent with providing visually driven inhibition onto mismatch-responsive neurons.13

(F) Mismatch responses depend on the visuomotor coupling experience of the animal: mismatch responses are indistinguishable from passive visual flow halts if the animal is raised with no coupling between visual flow and locomotion.13

The main claim of the recent paper Muzzu and Saleem 18 is that responses to visual flow halts, enhanced by locomotion, could instead account for the visuomotor mismatch responses described in the literature. Please note, we are not questioning the validity of the results presented in the paper, and testing and falsifying models proposed by previous work is essential. However, we would contend that the main conclusion—that visuomotor mismatch responses can be explained by visual flow offset responses modulated by locomotion—is poorly backed up by the data. This is primarily because the coupling between visual flow and locomotion is never introduced, and therefore the mismatch responses, which the work claims to be able to explain, are not measured.

It is important to note that “visuomotor mismatch” describes a stimulus comprising a perturbation (a sudden and brief reduction of visual flow speed to zero) in a condition of closed-loop coupling between locomotion and visual flow feedback (see Figure 1A). The authors argue that responses to such mismatches can be explained without precise coupling between visual flow and locomotion speeds. Instead, they argue that mismatch responses could result from sensory responses to visual flow halts that undergo a general gain increase during locomotion. The three pieces of evidence used to back up this claim are as follows:

  • (1) Passively presented (open-loop) visual flow halts can evoke increased firing in cells that are tuned to low temporal frequencies of drifting visual gratings.

  • (2) These responses can be amplified by locomotion.

  • (3) These responses are not biased to the forward-backward direction across the population.

Points 1 and 2 have been previously addressed by the visuomotor mismatch literature. More importantly, the experiments in Muzzu and Saleem18 do not show that visuomotor mismatch responses are quantifiably explained by these phenomena. Regarding point 1, it has been demonstrated previously that visual flow halts (termed “playback halts” in the literature) evoke responses in V1 neurons, similar to the stimuli presented in the current paper.9,13,17 However, across the population of layer 2/3 neurons, playback halt responses correlate poorly with mismatch responses.13 Thus, it is likely different populations of neurons that respond to visuomotor mismatch and playback halts. In addition, the authors demonstrate that the neurons most responsive to passively viewed visual flow halts are neurons with a temporal frequency tuning to low visual flow speeds. The population response of mismatch neurons indeed exhibits a near-linear decrease in activity with increasing visual flow speeds that could result in tuning for low spatial frequencies (in the range 0–6 Hz; compare Figure S3B from Attinger et al. 13 with Figure 3A in Muzzu and Saleem.18 Thus, the fact that neurons that respond to visual flow halt also have low temporal frequency tuning is not inconsistent with the predictive processing model.

Regarding point 2, it has been shown that locomotion increases the gain of visual responses, on average 2-fold,17,19,20,21,22 by mechanisms that are context and cell-type specific.23 However, this locomotion-induced gain is on average 5-fold smaller than the gain calculated between responses to visuomotor mismatches and playback halts.17 That is, on average, calcium responses to visuomotor mismatch are ten times larger than responses to the identical visual flow stimulus played back to a stationary mouse (playback halts) (Figure 1B). This is a key piece of evidence that argues against the idea that locomotion-based amplification of visual responses explains visuomotor mismatch responses. The experiments in Muzzu and Saleem18 do not address this, given that there is no quantification of responses to mismatches in yoked visual feedback during locomotion. While the authors discuss the possibility that the coupled visual flow condition could reveal true mismatch responses, we are of the opinion that this should be experimentally addressed via quantification of mismatch responses if the conclusion of the paper is that “feature selectivity can explain mismatch signals in mouse visual cortex.”

Point 3 is based on the argument that evidence of a V1 population bias for the horizontal direction of visual flow halts (naso-temporal) is required for visuomotor mismatch signaling, as this is the direction of visual flow self-generated by forward locomotion. The authors do not find such a bias in their recorded population in awake mice. Interestingly, a population bias for naso-temporal visual flow selectivity has been found in the retina and in layer 2/3 of V1 in anesthetized mice, suggesting it exists in the feedforward signal.24 While it makes sense to match the dynamic range of your sensors to the statistical properties of the input that are ethologically relevant, the bias toward horizontal visual flow is not what one would expect to find based on a predictive processing model of V1. In a freely moving animal, visual flow can occur in any direction during movements of the head, during turns and rears, or during eye movements. It would therefore make sense for V1 to have neurons representing visuomotor mismatches with direction tuning to all directions the animals can self-generate. Head fixing an animal and restricting the possible directions of locomotion to only forward-backward is unlikely to result in an immediate re-organization of the circuit to overrepresent visual flow or halts along this single axis of motion. In addition, given that in the experiments of Muzzu and Saleem, the mice never experience head-fixed closed-loop visual flow and that the stimuli presented are not visuomotor mismatches, we do not know whether the orientation/direction tuning profile they find would generalize to mismatch responses. We therefore would argue that the lack of population direction tuning bias for halts in naso-temporal visual flow, especially in the context of the experimental conditions used by Muzzu and Saleem, is not evidence against the predictive processing model of V1.

Finally, mismatch responses are thought to be computed in layer 2/3 neurons, while layer 5 neurons exhibit visuomotor integration characteristics that are not consistent with a visuomotor error computation.14 Analyzing a published two-photon imaging dataset, Muzzu and Saleem find that tuning for low temporal frequencies is overrepresented in superficial layers but not in deep layers. This is consistent with the finding that the neurons that compute mismatch responses are primarily found in layer 2/3. Given that different cortical layers very likely have different computational functions, it is essential that experiments are performed in such a way that layer identity of recorded neurons can be reconstructed.

Overall, none of the three arguments made in Muzzu and Saleem (1–3) directly refute predictive processing as an explanation for visuomotor mismatch responses, nor do they demonstrate that flow halt responses can explain mismatch responses.

At this point, it is important to note that predictive processing is just one of many possible models that would explain mismatch responses. Muzzu and Saleem’s 18 claim is that a visual halt response combined with a locomotion-induced gain of visual responses would also explain mismatch responses. As was shown in Zmarz and Keller,17 where the locomotion-induced gain of flow halts was directly compared with the locomotion-induced gain of flow onsets, the effect of locomotion on visual stimuli is not uniform. The apparent gain is substantially larger for flow halts than it is for flow onsets (i.e., classical visual stimuli). Thus, an explanation for mismatch responses based on locomotion-related gain would require a visual response to the flow halt combined with a locomotion gain that is stimulus specific.

Results

To directly test whether a stimulus specific locomotion gain could explain layer 2/3 mismatch responses, we compared the locomotion-induced gain of visual flow halt responses in closed- and open-loop conditions (Figure 2A). Assuming that a stimulus-specific locomotion gain is sufficient to explain mismatch responses, then precise coupling between visual flow and locomotion should have little effect on the size of mismatch responses. In other words, responses evoked by mismatch in the closed-loop condition should be very similar to responses evoked by passive visual flow halts during locomotion in the open-loop condition. To directly assess this, we analyzed three previously acquired two-photon calcium imaging datasets, recorded from layer 2/3 of V1, totaling 7,094 neurons in 45 mice (see STAR Methods). Two of these have been previously published and are publicly available,13,25 while one is not yet published. Mice were first exposed to the closed-loop condition where visual flow was coupled to locomotion speed, and sudden 1 s halts in visual flow during locomotion were used to evoke mismatch responses. This was followed by an open-loop condition where the self-generated visual flow from the preceding closed-loop session was replayed to the mouse uncoupled from locomotion. The 1 s halts in visual flow (“playback halts”) could therefore occur either during stationary periods or during locomotion. Consistent with locomotion increasing the gain of visual responses, in the open-loop session, we found that playback halts during locomotion evoked a larger population response compared with playback halts during stationary periods (playback halt response, mean ± standard deviation [SD] ΔF/F: stationary: 0.3% ± 5.8%, locomoting: 1.1% ± 8.7%, p < 10−10, paired t test). However, mismatches in otherwise coupled visual flow evoked significantly larger responses, on average double the size of those evoked by playback halts during locomotion (mismatch response, mean ± SD ΔF/F: 2.2% ± 9.7%; mismatch vs. playback halt during locomotion, p < 10−10, paired t test) (Figures 2B and 2E). These differences cannot be accounted for by differences in locomotion speed, as average locomotion speeds in both conditions were similar (Figure 2C). By experimental design, the only difference between the two conditions was the coupling between visual flow and locomotion preceding the visual flow halt (Figure 2D).

Figure 2.

Figure 2

Open in a new tab

Visuomotor coupling enhances mismatch responses in an experience-dependent manner

(A) Schematics to show the three conditions in which responses to visual flow halts were assessed. The green line illustrates visual flow speed, and the purple line illustrates locomotion speed. Left: mice are stationary while observing a 1 s visual flow halt (playback halt, gray shading). Middle: mice are locomoting and observe a 1 s visual flow halt, but locomotion and visual flow are not coupled. Right: mice observe a 1 s visual flow halt while locomoting in a closed-loop condition. These events we refer to as visuomotor mismatches (orange shading).

(B) Heatmaps of the average responses to visual flow halts corresponding to the three conditions in (A), sorted across 7,094 neurons. Top row: heatmaps were sorted independently for each condition. Bottom row: the same data but with heatmaps that were sorted according to the responses to playback (PB) halt, stationary (leftmost heatmap). Note, the correlations between responses in the different conditions were relatively low.

(C) Average locomotion speed during the visual flow halt stimuli in each condition.

(D) The correlation between locomotion speed and visual flow speed across the whole session was close to 0 in the open-loop condition and close to 1 in the closed-loop condition. Note, the only reason closed-loop correlation is less than 1 is due to the presentation of mismatches.

(E) Population average responses to visual flow halt stimuli in the three conditions for mice raised with (left, coupled trained) and without (right, non-coupled trained) visuomotor coupling experience. Shading shows standard error of the mean. Lines below the plots indicate statistical differences between responses as color coded to the left (gray: not significant; black: p < 0.01; paired t test). The comparison being made for each line is indicated by the combination of colors on the left.

(F) Mean responses to mismatch and PB halt during locomoting in coupled trained animals, in the early and late blocks (see STAR Methods) of closed-loop and open-loop sessions, respectively (where at least 7,000 neurons met the criterion of having at least three trials in a given condition). In the experimental paradigm, open-loop sessions always were always presented after closed-loop sessions. Thus, the longer the animal was exposed to an open-loop condition, the smaller the PB halt locomoting responses became. Error bars show standard error of the mean. n.s. p > 0.05, ∗∗∗p < 0.001, paired t test.

(G) Population average responses to mismatch (orange) and PB halts during locomotion (purple) in coupled trained animals, in the early and late blocks (see STAR Methods) of closed-loop and open-loop sessions, respectively (where at least 7,000 neurons met the criterion of having at least three trials in a given condition). Shading shows standard error of the mean. Lines below the plots indicate statistical differences between responses as color coded to the left (gray: not significant; black: p < 0.01; paired t test). The comparison being made for each line is indicated by the combination of colors on the left.

This result is consistent with the idea that in the closed-loop condition, strong predictions of visual flow during locomotion drive prediction error responses when the visual flow is lower than expected and directly opposes the idea that a uniform locomotion-induced gain of visual flow halt responses, whether stimulus specific or not, can explain mismatch responses. Note that predictions likely still play a reduced role in the enhancement of playback halt responses during locomotion in the open-loop condition given that mice have a lifetime of experience with coupled locomotion and visual flow. Thus, the effect of visuomotor coupling on mismatch responses shown here is potentially an underestimate. To assess locomotion-related gain in mice raised without coupling between forward locomotion and backward visual flow (non-coupled trained), we assessed visual flow halt responses in mice raised in an open-loop virtual environment (2,072 neurons from 9 mice13). In these mice, locomotion increased playback halt responses (playback halt response, mean ± SD ΔF/F: stationary: 0.4% ± 5%, locomoting: 1.2% ± 6.4%, p < 10−7, paired t test), but the enhancing effect of visuomotor coupling on mismatch responses was entirely absent (mismatch response, mean ± SD ΔF/F: 0.9% ± 6.7%; mismatch vs. playback halt during locomotion, p < 0.05, paired t test) (Figure 2E).

An interesting question the results of Muzzu and Saleem raise is on what timescale does the experience of coupling, or the absence of coupling, shape visuomotor prediction error responses? Exposed to an open-loop environment, the system likely gradually reduces predictions of visual flow given movement, thereby decreasing prediction error responses. If mice are raised without visuomotor coupling, mismatch responses are not increased by visuomotor coupling, consistent with a lack of visuomotor prediction error responses (Figure 2E). When transitioning from a closed-loop virtual environment to an open-loop virtual environment, locomotion onset-related mismatch responses decay with a half-life of a few minutes in normally raised animals (see Figure 3B in Keller et al.9). To test whether playback halt responses during locomotion undergo a similar reduction in amplitude as a function of time since the start of the open-loop condition, we compared these responses between early (0–250 s) and late (500–750 s) segments of the open-loop condition in mice raised with visuomotor coupling. All experiments started with a closed-loop condition that was followed by an open-loop condition. In mice raised with visuomotor coupling, the size of playback halt responses during locomotion declined between the early and late periods in the open-loop session toward a value close to the average playback halt response during stationary periods (playback halt response during locomotion, mean ± SD ΔF/F: early: 1.4% ± 11.3%, late: 0.7% ± 10.7%, p < 10−10, paired t test) (Figures 2F and 2G). This reduction in response could not be explained by a general time-dependent change since mismatch responses showed no sign of reduction between early and late epochs of the closed-loop session (mismatch response, mean ± SD ΔF/F: early: 1.8% ± 11.1%, late: 1.9% ± 11.9%, p = 0.90, paired t test). Neither could the reduction in playback halt response be explained by a reduction in locomotion speed (p = 0.74, paired t test between locomotion speeds in early and late epochs). These results are consistent with locomotion-induced gain being strongly dependent on the recent experience of visuomotor coupling.

Discussion

Based on our results, visuomotor mismatch responses can be described either as a stimulus-specific and experience-dependent locomotion gain or as a visuomotor prediction error. Both are technically valid explanations of mismatch responses. However, we would argue that a stimulus-specific and experience-dependent locomotion gain is not a useful model given that it does not parametrize the stimulus specificity and hence does not generalize or make testable predictions. It is simply a description of the observations that does not address the question of why it would make sense to compute such a signal. Most importantly, such a model is not normative and fails to predict the results summarized in Figure 1. By contrast, a predictive processing model that interprets mismatch responses as prediction errors explains all of these results and has direct computational application.26,27,28

An interesting future endeavor that could help advance our understanding and move this discussion forward would be to characterize feature selectivity in the context of prediction error computations. Movement is only one source of predictions that contributes to shaping responses in V1. Spatial location in a known environment also results in input to V1 consistent with top-down predictions of visual input,29,30 and experience with associations between sounds and visual stimuli results in audiovisual predictions that are sent from auditory cortex to V1.31 Visual input from areas surrounding the classical receptive field likely provides visual surround predictions that can be used to compute responses that underlie end stopping, surround suppression,6 or inverse receptive fields.32 Similarly, flow halt responses can be understood as deviations from predictions based on recent stimulus history. Interestingly, in mice raised without visuomotor coupling, these flow halt responses are larger than in normal mice.13 This would indicate that different possible sources of predictions are weighted as a function of how reliable they have been over the course of the experience of the mouse. Normally, movement is the major predictor of visual feedback. All eye, head, or translational movements result in highly predictable visual flow feedback throughout the life of the mouse. In the experimentally induced absence of some of this coupling, other sources of predictions, like stimulus history, could become more important. Thus, prediction error responses need to be understood in a context of a diverse set of sources of different predictions. How different neurons in V1 are influenced by these different sources of predictions is still largely unknown.

In summary, our primary concern with the alternative explanation for visuomotor mismatch responses in V1 provided by Muzzu and Saleem18 is that visual flow is never coupled to locomotion, and therefore mismatch responses are not quantified. We argue that without such quantification, it is difficult to provide evidence for a useful alternative explanation for mismatch responses or any other neural response. The comparison between responses to passive visual flow halts and mismatches we present here (Figure 2) provides this quantification and, alongside previous findings (Figure 1), demonstrates that locomotion-induced gain cannot account for mismatch responses in layer 2/3 of V1. Overall, we consider the results presented in Muzzu and Saleem18 both valid and valuable for the field; however, we find the interpretation and the title are not well supported due to the concerns we outlined here.

Limitations of the study

The analyses to determine the impact of visuomotor coupling on playback halt responses were somewhat limited by the structure of the experiments. First, while there were no notable time-dependent changes in mismatch responses in the closed-loop condition (Figures 2F and 2G), to completely rule out other time-dependent effects on the reduction in playback halt responses, ideally one would reinitiate a second session of the closed-loop condition and observe a recovery of mismatch response amplitude. Second, to assess whether individual neurons respond significantly more to mismatch compared with playback halt responses, we would need to perform a statistical test for each neuron for the difference between responses to playback halts during locomotion and responses to mismatch. However, due to variation in locomotion behavior and a relatively low number of trials per stimulus (we included data from mice that had at least 5 playback halt and 5 mismatch stimuli), the comparison of playback halt and mismatch responses for individual neurons is underpowered. Finally, it is not possible to determine from our calcium imaging data whether mismatch neurons receive a visual excitation selective for low visual speeds, as suggested by Muzzu and Saleem,18 or whether they receive visually driven inhibition, as postulated by predictive processing.4

STAR★Methods

Key resources table

REAGENT or RESOURCE SOURCE IDENTIFIER
Deposited data
2-photon imaging dataset Attinger et al.13 https://data.fmi.ch/PublicationSupplementRepo/
2-photon imaging dataset Widmer et al.25 https://data.fmi.ch/PublicationSupplementRepo/
2-photon imaging dataset This manuscript https://data.fmi.ch/PublicationSupplementRepo/
Software and algorithms
MATLAB, 2021b https://www.mathworks.com/products/matlab.html RRID: SCR_001622
Analysis code This manuscript https://data.fmi.ch/PublicationSupplementRepo/
Open in a new tab

Resource availability

Lead contact

Further information and requests for resources and reagents should be directed to and will be fulfilled by the lead contact, Rebecca Jordan (rjordan3@ed.ac.uk).

Materials availability

This study did not generate any new materials or reagents.

Experimental model and subject details

Details of the experimental subjects (Mus musculus, C57BL/6J) have been described previously.13,25

Method details

Description of the dataset

All details of data acquisition were described previously.13,25 Raw data consisted of two-photon calcium imaging recordings from V1 layer 2/3 pyramidal neurons, totaling 10 005 neurons recorded in 66 mice with visuomotor experience, and 2561 neurons recorded in 9 mice raised with only open loop visual experience. Neurons undergoing analyses were selected based on criteria below, described under ‘Calculation of responses and exclusion criteria’). Data were recorded at 10 Hz or 15 Hz. 15 Hz data were down sampled to 10 Hz for analysis. All data comprised the following stimulus presentation structure. First, a session with 500 s or 750 s closed loop virtual reality experience, in which visual flow was yoked with locomotion speed. In this condition, brief 1 s visual flow halts were presented to the mice during locomotion, which we refer to as mismatch stimuli. Next, two or three open loop sessions were presented with the same duration as the closed loop session, in which visual flow recorded from the initial closed loop sessions was replayed to the mouse, and thus not yoked to locomotion. In this condition, the visual flow halts were referred to as playback halts and could occur both during locomotion and stationary behavior.

Quantification and statistical analysis

Definition of locomotion and stationary trials

Locomotion speed for each trial was assessed by averaging the locomotion speed in a time window of −1.5 to +1.5 s from stimulus onset. Due to differences in the measurement of locomotion speed across the datasets, the threshold for locomotion was determined as a velocity above 12% of the 95th percentile of the speed distribution within a dataset, while the threshold for stationary periods was defined as a velocity below the 2% of the 95th percentile.

Calculation of responses and exclusion criteria

For each neuron, the ΔF/F response in each trial was first baseline subtracted using the mean ΔF/F in the 1 s preceding the stimulus, before being averaged across trials. By design, mismatch triggers and playback halts while locomoting occur during locomotion. Hence the probability of locomotion at the time of the trigger is 100% but reduces with increasing time from the trigger. As a result of this, neurons whose activity correlates positively with locomotion will exhibit increases of activity around time of the trigger. To correct for these non-specific increases in ΔF/F during mismatch and playback halts while locomoting we calculated sham responses and subtracted these from mismatch responses and playback halt responses during locomotion. Sham responses were calculated based on 1000 random triggers, selected based on the same locomotion criteria used to calculate actual responses. For Figures 2B and 2E, only neurons with at least 5 trials in all three stimulus conditions (mismatch, playback halt during locomotion and during stationary periods) were included in the analyses (coupled trained, n = 7094; non-coupled trained, n = 2072). To calculate average responses for each neuron (e.g., numbers reported in the text, and in Figure 2F), we took the mean of the average response in a window 500 ms–1500 ms after stimulus onset. In all statistical tests, responses were compared with a paired t test.

Early vs late response comparisons

To compare responses in early and late parts of the session (Figures 2F and 2G), we took responses during the first 250 s of the first open loop or closed loop sessions to calculate early responses. To calculate late responses, we used responses in the last 250 s of the closed loop condition to calculate mismatches (250 s–500 s for 3625 neurons with 500 s duration closed loop sessions, or 500 s–750 s for 5539 neurons with 750 s duration closed loop sessions), while we took the third block of 250 s in the open loop condition (500 s–750 s) to calculate playback halt responses. To calculate the average responses, only neurons with at least three trials in a given condition (mismatch, playback halt during locomotion, early and late) were included (more than 7000 neurons in each condition).

Acknowledgments

We thank the members of the Keller lab for discussion and support. This work has received funding from the Swiss National Science Foundation, the Novartis Research Foundation, the Human Frontier Science Program (LT000077/2019-L) to R.J., and the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement no. 865617).

Author contributions

R.J. designed and performed all analyses. F.C.W. performed two-photon imaging experiments. A.V., G.B.K., and R.J. wrote the manuscript.

Declaration of interests

The authors declare no competing interests.

Published: February 22, 2023

Data and code availability

All 2-photon imaging datasets have been deposited at https://data.fmi.ch/PublicationSupplementRepo and are publicly available as of the date of publication. Links to each dataset are listed in the key resources table. All original code has been deposited at https://data.fmi.ch/PublicationSupplementRepo and is publicly available as of the date of publication. The link is listed in the key resources table. Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

References

  • 1.Bastos A.M., Usrey W.M., Adams R.A., Mangun G.R., Fries P., Friston K.J. Canonical microcircuits for predictive coding. Neuron. 2012;76:695–711. doi: 10.1016/j.neuron.2012.10.038. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Clark A. Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behav. Brain Sci. 2013;36:181–204. doi: 10.1017/S0140525X12000477. [DOI] [PubMed] [Google Scholar]
  • 3.Jordan M.I., Rumelhart D.E. Forward models: supervised learning with a distal teacher. Cognit. Sci. 1992;16:307–354. doi: 10.1207/s15516709cog1603_1. [DOI] [Google Scholar]
  • 4.Keller G.B., Mrsic-Flogel T.D. Predictive processing: a canonical cortical computation. Neuron. 2018;100:424–435. doi: 10.1016/j.neuron.2018.10.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Koster-Hale J., Saxe R. Theory of mind: a neural prediction problem. Neuron. 2013;79:836–848. doi: 10.1016/j.neuron.2013.08.020. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Rao R.P., Ballard D.H. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 1999;2:79–87. doi: 10.1038/4580. [DOI] [PubMed] [Google Scholar]
  • 7.Eliades S.J., Wang X. Neural substrates of vocalization feedback monitoring in primate auditory cortex. Nature. 2008;453:1102–1106. doi: 10.1038/nature06910. [DOI] [PubMed] [Google Scholar]
  • 8.Heindorf M., Arber S., Keller G.B. Mouse motor cortex coordinates the behavioral response to unpredicted sensory feedback. Neuron. 2018;99:1040–1054.e5. doi: 10.1016/j.neuron.2018.07.046. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Keller G.B., Bonhoeffer T., Hübener M. Sensorimotor mismatch signals in primary visual cortex of the behaving mouse. Neuron. 2012;74:809–815. doi: 10.1016/j.neuron.2012.03.040. [DOI] [PubMed] [Google Scholar]
  • 10.Roth M.M., Dahmen J.C., Muir D.R., Imhof F., Martini F.J., Hofer S.B. Thalamic nuclei convey diverse contextual information to layer 1 of visual cortex. Nat. Neurosci. 2015;19:299–307. doi: 10.1038/nn.4197. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Schneider D.M., Sundararajan J., Mooney R. A cortical filter that learns to suppress the acoustic consequences of movement. Nature. 2018;561:391–395. doi: 10.1038/s41586-018-0520-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Stanley J., Miall R.C. Functional activation in parieto-premotor and visual areas dependent on congruency between hand movement and visual stimuli during motor-visual priming. Neuroimage. 2007;34:290–299. doi: 10.1016/j.neuroimage.2006.08.043. [DOI] [PubMed] [Google Scholar]
  • 13.Attinger A., Wang B., Keller G.B. Visuomotor coupling shapes the functional development of mouse visual cortex. Cell. 2017;169:1291–1302.e14. doi: 10.1016/j.cell.2017.05.023. [DOI] [PubMed] [Google Scholar]
  • 14.Jordan R., Keller G.B. Opposing influence of top-down and bottom-up input on excitatory layer 2/3 neurons in mouse primary visual cortex. Neuron. 2020;108:1194–1206.e5. doi: 10.1016/j.neuron.2020.09.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Leinweber M., Ward D.R., Sobczak J.M., Attinger A., Keller G.B. A sensorimotor circuit in mouse cortex for visual flow predictions. Neuron. 2017;95:1420–1432.e5. doi: 10.1016/j.neuron.2017.08.036. [DOI] [PubMed] [Google Scholar]
  • 16.Spratling M.W. Fitting predictive coding to the neurophysiological data. Brain Res. 2019;1720:146313. doi: 10.1016/j.brainres.2019.146313. [DOI] [PubMed] [Google Scholar]
  • 17.Zmarz P., Keller G.B. Mismatch receptive fields in mouse visual cortex. Neuron. 2016;92:766–772. doi: 10.1016/j.neuron.2016.09.057. [DOI] [PubMed] [Google Scholar]
  • 18.Muzzu T., Saleem A.B. Feature selectivity can explain mismatch signals in mouse visual cortex. Cell Rep. 2021;37:109772. doi: 10.1016/j.celrep.2021.109772. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Bennett C., Arroyo S., Hestrin S. Subthreshold mechanisms underlying state-dependent modulation of visual responses. Neuron. 2013;80:350–357. doi: 10.1016/j.neuron.2013.08.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Erisken S., Vaiceliunaite A., Jurjut O., Fiorini M., Katzner S., Busse L. Effects of locomotion extend throughout the mouse early visual system. Curr. Biol. 2014;24:2899–2907. doi: 10.1016/j.cub.2014.10.045. [DOI] [PubMed] [Google Scholar]
  • 21.Niell C.M., Stryker M.P. Highly selective receptive fields in mouse visual cortex. J. Neurosci. 2008;28:7520–7536. doi: 10.1523/JNEUROSCI.0623-08.2008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Polack P.-O., Friedman J., Golshani P. Cellular mechanisms of brain-state-dependent gain modulation in visual cortex. Nat. Neurosci. 2013;16:1331–1339. doi: 10.1038/nn.3464. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Pakan J.M., Lowe S.C., Dylda E., Keemink S.W., Currie S.P., Coutts C.A., Rochefort N.L. Behavioral-state modulation of inhibition is context-dependent and cell type specific in mouse visual cortex. Elife. 2016;5:e14985. doi: 10.7554/eLife.14985. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Hillier D., Fiscella M., Drinnenberg A., Trenholm S., Rompani S.B., Raics Z., Katona G., Juettner J., Hierlemann A., Rozsa B., Roska B. Causal evidence for retina dependent and independent visual motion computations in mouse cortex. Nat. Neurosci. 2017;20:960–968. doi: 10.1038/nn.4566. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Widmer F.C., O’Toole S.M., Keller G.B. NMDA receptors in visual cortex are necessary for normal visuomotor integration and skill learning. Elife. 2022;11:e71476. doi: 10.7554/eLife.71476. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Lotter W., Kreiman G., Cox D. 2016. Deep predictive coding networks for video prediction and unsupervised learning. (5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings). [Google Scholar]
  • 27.Straka Z., Svoboda T., Hoffmann M. PreCNet: next frame video prediction based on predictive coding. arXiv. 2020 doi: 10.48550/arXiv.2004.14878. Preprint at. [DOI] [PubMed] [Google Scholar]
  • 28.Whittington J.C.R., Bogacz R. MIT Press Journals; 2017. An Approximation of the Error Backpropagation Algorithm in a Predictive Coding Network with Local Hebbian Synaptic Plasticity, Neural Computation. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Fiser A., Mahringer D., Oyibo H.K., Petersen A.V., Leinweber M., Keller G.B. Experience-dependent spatial expectations in mouse visual cortex. Nat. Neurosci. 2016;19:1658–1664. doi: 10.1038/nn.4385. [DOI] [PubMed] [Google Scholar]
  • 30.Pakan J.M.P., Currie S.P., Fischer L., Rochefort N.L. The impact of visual cues, reward, and motor feedback on the representation of behaviorally relevant spatial locations in primary visual cortex. Cell Rep. 2018;24:2521–2528. doi: 10.1016/j.celrep.2018.08.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Garner A.R., Keller G.B. A cortical circuit for audio-visual predictions. Nat. Neurosci. 2022;25:98–105. doi: 10.1038/s41593-021-00974-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Keller A.J., Roth M.M., Scanziani M. Feedback generates a second receptive field in neurons of the visual cortex. Nature. 2020;582:545–549. doi: 10.1038/s41586-020-2319-4. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

All 2-photon imaging datasets have been deposited at https://data.fmi.ch/PublicationSupplementRepo and are publicly available as of the date of publication. Links to each dataset are listed in the key resources table. All original code has been deposited at https://data.fmi.ch/PublicationSupplementRepo and is publicly available as of the date of publication. The link is listed in the key resources table. Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

RESOURCES