Limited Evidence for Sensory Prediction Error Responses in Visual Cortex of Macaques and Humans

Abstract

A recent formulation of predictive coding theory proposes that a subset of neurons in each cortical area encodes sensory prediction errors, the difference between predictions relayed from higher cortex and the sensory input. Here, we test for evidence of prediction error responses in spiking responses and local field potentials (LFP) recorded in primary visual cortex and area V4 of macaque monkeys, and in complementary electroencephalographic (EEG) scalp recordings in human participants. We presented a fixed sequence of visual stimuli on most trials, and violated the expected ordering on a small subset of trials. Under predictive coding theory, pattern-violating stimuli should trigger robust prediction errors, but we found that spiking, LFP and EEG responses to expected and pattern-violating stimuli were nearly identical. Our results challenge the assertion that a fundamental computational motif in sensory cortex is to signal prediction errors, at least those based on predictions derived from temporal patterns of visual stimulation.

Introduction

The physical environment is structured. Sensory systems exploit this structure to represent and process information about the environment efficiently. Predictive coding—a broad concept encompassing several distinct algorithms (Spratling 2017)—is an information processing strategy in which structure is used to make predictions that “explain away” expected signals.

There are many well-established examples of predictive coding (Keller and Mrsic-Flogel 2018). For instance, the receptive field structure of retinal ganglion cells has an antagonistic center-surround organization which allows these cells to explain away luminance within the receptive field when it is matched by signals in the surround (Barlow 1961; Srinivasan et al. 1982). Cortical neurons also have suppressive surrounds which can be viewed as instantiating a form of predictive coding (Rao and Ballard 1999; Spratling 2010; Lochmann et al. 2012). Simple mechanisms of adaptation, such as the neuronal fatigue caused by recent sensory drive, can be viewed as a way in which recent experience explains away responses to persistent or recurring stimuli (Hosoya et al. 2005; Lochmann et al. 2012; Solomon and Kohn 2014).

One specific, recent formulation of predictive coding has received a great deal of attention. It proposes a general theory of cortical architecture and function, in which each level of cortical processing includes two sets of neurons: one that predicts the sensory input and relays this information to lower areas via feedback connections; and another set that computes the difference between this prediction and the observed sensory input—the prediction error—and relays this difference to higher cortex via feedforward connections (Friston 2005; Bastos et al. 2012; Clark 2013). The predictions relayed to lower cortex are changed from moment-to-moment based on recent sensory experience, reflecting a constantly updating internal model of the sensory environment. Hereafter, we refer to this proposal as “predictive coding theory” though predictive coding is a broader concept that extends beyond this specific formulation.

Experimental support for predictive coding theory has relied primarily on demonstrations of expectation or repetition suppression (Clark 2013; Summerfield and de Lange 2014). In expectation suppression, a cue or general knowledge of stimulus statistics generates an expectation of which sensory stimulus will appear. Weaker responses are often observed to expected stimuli. Under predictive coding theory, weaker responses arise because expected stimuli produce a smaller prediction error. In repetition suppression, a repeated stimulus evokes a weaker response. This is also attributed to a smaller prediction error, arising from the expectation that once a stimulus appears it will likely persist or reappear.

Repetition and expectation suppression are most commonly measured with functional magnetic resonance imaging (fMRI) (e.g., Summerfield et al. 2008; Egner et al. 2010; Kok et al. 2012) or EEG (e.g., Melloni et al. 2011; Wacongne et al. 2011). However, these signals cannot be directly linked to the prediction error signals relayed between brain areas through spiking responses. For instance, weaker fMRI responses to an expected stimulus may arise from sharper neuronal tuning (Kok et al. 2012; Alink et al. 2018), so that fewer neurons are recruited, rather than from weaker responses in neurons encoding prediction errors.

Neuronal studies of predictive coding theory have focused primarily on simple repetition suppression or “oddball” experiments, in which the response to a frequently presented standard stimulus is compared with that evoked by a rarely presented deviant. Weaker responses are usually observed to the standard, which is consistent with a smaller prediction error. However, weaker responses may instead reflect simple adaptation (Ulanovsky et al. 2003; May and Tiitinen 2010; Kaliukhovich and Vogels 2010; Vogels 2016), arising from neuronal or synaptic fatigue (Kohn 2007; Solomon and Kohn 2014), rather than an explicit comparison of predictions from higher cortex with sensory input.

Because prediction error coding and simple adaptation are conflated in repetition suppression measurements, a more discerning test of predictive coding theory is to compare responses evoked by a standard pattern of stimuli, which establish an expectation, to those evoked by stimuli that violate that pattern, which should result in prediction errors. In EEG and fMRI studies, this approach has revealed prediction error responses to pattern violations under some conditions (e.g., Bekinschtein et al. 2009; Wacongne et al. 2011; Stefanics et al. 2014; Symonds et al. 2017). Pattern-violation responses have also been observed in neuronal spiking activity in retinal (Schwartz et al. 2007) and some rodent cortical studies (Yaron et al. 2012; Latimer et al. 2019; Homann et al. 2019). But whether such responses are a robust feature of sensory cortical encoding is unclear. Further, neuronal pattern-violation responses have not been reported in primate sensory cortex where the processing hierarchy, which lies at the heart of predictive coding theory, is most clearly established (Felleman and Van Essen 1991; but see Chao et al. 2018 for electrocorticogram responses in some cortical areas of nonhuman primates).

Here, we report recordings of neuronal responses in early (V1) and mid-level (V4) visual cortex of awake, fixating macaque monkeys, evoked by a diverse set of standard patterns and their violations. Patterns evoked robust repetition suppression in both areas, but violations of the expected pattern led to minimal modulation of neuronal responses. That is, there was little evidence of responses that might reflect prediction errors. To better understand the relationship between these measurements and those available from EEG experiments, we measured evoked potentials in human participants using the same stimulus protocol. These responses revealed no evidence for pattern-violation responses, unless participants were explicitly instructed to identify violations. Our results show sensory cortical responses that are inconsistent with basic tenets of predictive coding theory.

Materials and Methods

Headpost and Array Implant Surgery

Monkeys were first implanted with a headpost to allow head fixation. Animals were premedicated with glycopyrrolate (0.01 mg/kg) and diazepam (1.5 mg/kg) before inducing anesthesia with ketamine (10 mg/kg). They were then intubated and anesthesia was maintained with isofluorane (1-2%). An intravenous catheter was inserted in a hindlimb to administer fluids continuously (0.9% NaCl or Normosol). Monkeys were then transferred to a stereotaxic frame for the rest of the surgery, which was performed under strictly sterile conditions. Throughout the surgery, vital signs (ECG, SpO₂, CO₂, temperature) were monitored and maintained at physiologically normal values. The headpost was attached to the skull with titanium screws. During recovery, animals were provided with antibiotics (Ceftiflex) and analgesics (buprenex). All procedures were approved by the Institutional Animal Care and Use Committee of the Albert Einstein College of Medicine and were in compliance with the guidelines set forth in the National Institutes of Health’s Guide for the Care and Use of Laboratory Animals.

We then trained monkeys to visually fixate for a liquid reward. Once they performed this task adequately (performing 1000 trials per session, within the fixation window detailed below), we performed surgery to implant microelectrode arrays. General procedures were identical to those described above. To implant the array, a craniotomy was made over visual areas V1 and V4, followed by resection of the dura. One 96-channel and one 48-channel microelectrode array (1 mm electrode length, 0.4 mm spacing, Blackrock Microsystems) were implanted in V1, and another 48-channel microelectrode array (0.4 mm spacing) was implanted in V4, on the prelunate gyrus. We do not know the cortical depth at which our recordings were made, but the electrode length (1 mm) makes it probable that recordings were strongly biased to the superficial or middle layers.

Visual Stimuli

Visual stimuli were generated using custom software (EXPO) and displayed on a calibrated cathode ray tube monitor (1024 × 768 pixel resolution, 100 Hz refresh rate) viewed at a distance of 64 cm. Monkeys fixated a small white square (0.2 × 0.2°) at the center of the monitor while a grating sequence was presented. Gratings were presented within a circular aperture surrounded by a gray field of average luminance (~40 cd/m²).

In each session, we presented a “standard” sequence of gratings (80% of trials), and two “deviant” sequences (each 10% of trials). Each sequence comprised two orthogonal gratings (1.5 cpd, 5 Hz, 6° diameter, full contrast, unless otherwise noted), referred to as gratings X and O. The standard sequence of gratings was XXXOX, and deviants were either XXOXX (termed early deviant) or XXXXO (late deviant). Each grating in the sequences was presented for 0.1 s followed by 0.1 s of a gray screen. Each session was complemented with a second session in which the X and O stimuli were swapped. Recordings from paired sessions were made either on the same day or 1–2 days apart. Monkeys were given a liquid reward after each sequence if they maintained fixation within a 1.6 (Monkey M) or 1.3° diameter (Monkey C) window. The three sequences were rewarded equally.

We presented several additional variations of this paradigm. All variants used the same parameter as in the main experiment described above except for the following variations: (1) To manipulate stimulus size, gratings were presented in a smaller aperture (1.75° diameter). (2) To manipulate stimulus contrast, the X and O gratings had the same orientation but different Michelson contrast values (0.25 and 1). (3) To manipulate stimulus duration, we presented each grating for 0.2 s followed by a 0.2 s interstimulus interval (consisting of a gray screen). Sequences consisted of 3 gratings: the standard sequence was XOX, and the early and late deviant sequences were OXX and XXO, respectively. (4) To heighten the salience of the deviant sequences, we provided twice the volume of liquid reward on those trials, relative to standard trials. We also presented an omission stimulus, in which we presented two sequences that comprised a single grating: in these experiments, a standard sequence (XXXXX) was presented 90% of the time, and in 10% of the time one grating was omitted (XXX_X). All other parameters in these sessions were as in the main experiment.

In one animal (monkey C), V1 and V4 spatial receptive fields were retinotopically aligned, at an eccentricity of ~ 2–4°. We therefore centered stimuli on the aggregate receptive field of the V1 and V4 neurons. In the second animal (monkey M), the V1 receptive fields were located at eccentricities at ~ 1–2° and the V4 receptive fields at ~5–10°. For this animal, we placed the gratings between the V1 and V4 aggregate receptive fields (~3°), so that the grating covered a portion of the V1 and V4 neurons’ receptive fields.

Monkey Neurophysiology

Signals from each electrode were amplified and band-pass filtered (0.3–7.5 kHz) using commercial acquisition systems (Cerebus, Blackrock Microsystems and Grapevine, Ripple). Spiking data were acquired by digitizing waveform segments that exceeded a threshold at 30 kHz. Waveform segments were manually sorted offline using commercial software (Plexon Offline Sorter), to isolate single unit activity and small multiunit clusters. Local field potentials (LFPs) were acquired by low-pass filtering (0.3–250 Hz) the broadband signal and digitizing at 1 kHz.

In one monkey (Monkey C), we observed cross-talk between channels of the array, either due to a manufacturing defect or to compromise of the insulation isolating signals from different channels during or following implantation. Cross-talk was evident as the presence of the same action potential waveform at precisely the same time (within 1 ms) on different electrodes. To remove these cases, we calculated the percentage of spikes that occurred within 1 ms of each other, across all pairings of electrodes. If that value exceeded 10%, we removed 1 of the units in the pair. Roughly 2–3% of pairs showed evidence of crosstalk by this criterion.

We measured responses during a 200 ms window, beginning at stimulus onset and encompassing the interstimulus interval (100 ms) after stimulus offset. We used this window because the responses of most neurons persisted into this epoch. For LFPs, responses were defined as the root-mean-square (rms) of the trial-averaged signal during this epoch (as in Dubey and Ray 2016). For each condition and electrode, we subtracted the value of the trial-averaged LFP in a baseline period, measured 0–25 ms after stimulus onset.

For analyses of responses to the O stimulus, we limited our measurements to neurons with an evoked response of at least 1 sp/s to each O stimulus in the standard sequence. In addition, we required that that response was larger than the mean spontaneous firing rate (measured in a window from −20 to +30 ms relative to the establishment of fixation) plus three times the standard error of that mean. We confirmed that neurons that did not respond to the O stimulus in the standard sequence also did not respond to that stimulus in the deviant sequences (i.e., that our selection did not bias toward cells that fired solely to deviants). When analyzing responses to the deviant X stimulus, we required a response of at least 1 sp/s to each X stimulus in the sequence. For LFP analyses, we excluded cases that showed an rms value less than 10 μV.

We detected microsaccades with an approach used in our previous work (Jasper et al. 2019), a method based closely on that of Horwitz and Albright (2003). Briefly, we smoothed the eye position time series with a Gaussian kernel (6 ms standard deviation), and then computed its derivative. A microsaccade was defined as an event with velocity ≥10°/s lasting for at least 8 ms. We removed any response (defined as the activity 0–200 ms after stimulus onset) during which a microsaccade occurred. Microsaccades occurred during 31% of standard O presentations and 23% of deviant O presentations, a rate expected given that microsaccades are typically made 1–2 times per second.

Human Neurophysiology

The study involved 25 healthy adult participants. Data from three participants were excluded due to poor task performance (d-prime values of 1 or lower), leaving 22 participants (16 females; age 27 ± 4 SD years old). Prior to testing, all participants provided written informed consent after they were told about the experiment, in accordance with the Declaration of Helsinki. The protocol was approved by the Internal Review Board of the Albert Einstein College of Medicine. All participants had normal or corrected-to-normal vision.

As in animal experiments, visual stimuli were presented using EXPO and a calibrated monitor (at 800 x 600 pixel resolution, 60 Hz refresh rate). Participants viewed the monitor from a distance of 57 cm, maintained using a chin rest. Stimuli were sinusoidal gratings (6° diameter, 2 Hz drift rate, 1 cpd) presented at full contrast. Participants were instructed to maintain fixation on a small cue (0.1°); gratings were centered at this location. We presented the same standard (80% of cases) and deviant sequences used in animal experiments, where X and O stimuli were horizontal and vertical gratings (or vice versa). We also presented sequences in which X and O differed by 10°; these revealed nearly identical effects and will be presented elsewhere. Each stimulus was presented for 200 ms, and followed by a 500 ms interstimulus interval during which a gray screen was presented.

Stimuli were presented in blocks of roughly 3 min duration, consisting of 60 presentations of each deviant and 480 presentations of standards, randomly interleaved. The five-stimulus sequences were presented as a continuous sequence; that is, after the interstimulus interval, the next sequence began immediately. Participants completed 10 blocks (except one subject, who did 9). Each session lasted for approximately 1.5 h.

While viewing sequences, participants performed one of two tasks. In the first, participants were instructed to count the number of individual gratings presented in each block. The mean error rate (the total number of stimuli divided by the count number) for this task was 3% (SD 3), ranging from 0 to 8% across individuals. In the second task, participants were informed about the standard sequence (3 gratings of the same orientation followed by a different orientation) and asked to press a button whenever they detected a violation of this expected pattern. Participants were given brief practice on this task, before data collection commenced. The mean hit rate for early and late deviations was 87% (SD 9) and 71% (SD 16), respectively.

EEG was conducted using a 32-channel electrode cap placed according to the modified International 10–20 System, with additional electrodes placed on the left and right mastoids (LM and RM, respectively). Horizontal eye movements were measured by recording the horizontal electro-oculogram (EOG) in a bipolar configuration between F7 and F8 electrodes. Vertical EOG was monitored using the FP1 electrode in a bipolar configuration with an external electrode placed below the left eye. The reference electrode was placed at tip of the nose. F3 was used as the ground electrode. Impedance was below 5 kΩ across all electrodes. The EEG and EOG were digitized (Neuroscan Synamps amplifier, Compumedics Corp.) at 500 Hz (0.05–100 Hz bandpass). The EEG was then filtered offline (0.1–30 Hz) using a finite impulse response filter with zero phase shift and a roll-off slope of 24 dB/octave using Neuroscan SCAN software 4.3 for PC.

Further analysis was performed in Matlab. Responses to the relevant X and O stimuli were extracted and re-zeroed using the response measured 0–25 ms after stimulus onset. Epochs containing activity exceeding ±75 μV on any electrode were excluded from further analysis to eliminate artifact activity stemming from eye movements. Artifact-free epochs were then averaged together by stimulus type separately in each condition and used in the analyses.

In preliminary analysis, we performed a repeated-measure ANOVA to test for differences between O stimuli in the standard and two deviant sequences, using responses measured at the OZ, PO9, PO10, O1, and O2 electrode. We also used this approach to test for an interaction between variations in responses to the O stimuli and these recording electrodes. We found no interaction and so averaged responses across these electrodes in further analysis. We focused on these electrodes because they provided the most well-defined visually evoked signals and were thus most comparable with the neuronal responses in early cortex. In additional analyses (Fig. 8), we explored signals from all recorded electrodes.

Figure 8 .

Scalp current source density analysis of human EEG responses. (A) Difference between average EEG response to the early deviant and standard (top, blue), and late deviant and standard (bottom, yellow), for responses recorded during the counting task. Time axis markers indicate onset of each stimulus in the sequence. Responses from four electrodes are shown. Insets show scalp topography of current source density analysis of the difference waveforms. Topography is calculated at times of 1577, 1682, and 2043 ms (from left to right) for the top trace; and for 2235, 2379, and 2673 ms for the bottom. (B) Difference traces for the active task. Responses from 5 electrodes are shown. Topography is calculated at times of 1556, 1688, and 2052 ms (from left to right) for the top trace; and for 2235, 2379, and 2673 ms for the bottom. Gray bars indicate epochs during which stimuli were presented.

All indications of variance are standard errors of the mean, unless otherwise indicated. Statistical tests on ratios were performed on log-transformed values.

Results

To create an expectation for an upcoming stimulus, we presented a standard temporal sequence comprised of two gratings (termed, X and O). This standard sequence (XXXOX) was presented on 80% of trials. We violated the expected ordering of the gratings in two deviant sequences, each shown on 10% of trials. In one deviant, the O stimulus was shown earlier in the sequence (XXOXX, termed early deviant); in the other, the O stimulus was presented at the end of the sequence (XXXXO, termed late deviant; Fig. 1A; see Methods for details).

Figure 1 .

Stimuli and responses expected from predictive coding. (A) The standard sequence XXXOX was presented on 80% of trials. Early (XXOXX) and late (XXXXO) deviant sequences were presented on the remaining trials. We considered each stimulus as having either probability 1 or 0 (second row) of appearing in each epoch (gray vertical bands). Solid lines show the probability for the X stimulus; dotted lines for O. (B) The logic relating repetition suppression experiments (top row) to predictive coding. When a stimulus is repeated, the response it evokes weakens with each successive presentation (bottom). This is attributed to a growing expectation or prediction (middle row, black) for the stimulus to reappear. (C) Expected outcome if predictions (top black) are the most frequently presented sequence (the standard). The responses for each sequence are shown below (green, standard; blue, early deviant; yellow, late deviant). (D) Expected outcome if predictions reflect the combined probabilities for each sequence. (E) Expected outcome based on the same predictions as (D), but predictions are updated after each stimulus epoch. (F) Expected outcome if predictions are that a recently encountered stimulus will reappear. Predictions grow in magnitude with each successive presentation.

To understand the responses that our stimuli might be expected to produce according to predictive coding theory, it is helpful to consider first a simple repetition suppression experiment. When a stimulus is repeated (Fig. 1B, top, light gray), it evokes progressively weaker responses in driven neurons (bottom, dark gray). Predictive coding theory posits that this weakening reflects a growing expectation of later presentations (black), resulting in a smaller predictive coding error or response. Note that in this scenario, the prediction is in lock step with the stimulus. We return to this point in Discussion.

We next consider two scenarios for responses to our sequences, which can be viewed as the sum of two stimuli. At each time, each stimulus has a probability of 0 or 1 of being present; the two stimuli do not appear at the same time (Fig. 1A; bottom, X in solid lines, O in dotted). For simplicity, we assume equal neuronal responses to the X and O stimulus. If neurons responded more to one of the stimuli, this would not affect comparisons of responses to a given stimulus in different contexts—for example, an O stimulus that occurred at the expected compared with unexpected time.

In our first scenario, the visual system learns the ordering of stimuli in the sequence—or, the “global” pattern (Bekinschtein et al. 2009). In this case, higher cortical areas might predict that the standard sequence, the most probable, will be presented on each trial (Fig. 2C, black). If the prediction is perfect, there should be no response (no prediction error) to any stimulus in the standard sequence (left, green) because each stimulus in the sequence is exactly the one expected. Both the early (center, blue) and late (right, yellow) deviant sequences should elicit a strong response to the O stimulus, which occurs at an unexpected time. One X stimulus in each deviant sequence should also elicit a strong response because it occurs when an O stimulus would have appeared in the standard sequence. We note that predictions with no uncertainty are likely contrived: imperfect predictions would yield some response, proportional to the mismatch between prediction and sensory drive. Critically, as long as some prediction is provided, unexpected O and X stimuli (deviant sequences) should generate stronger responses than expected ones (standard sequence).

Figure 2 .

Comparison of spiking responses to the O stimulus, in standard and deviant sequences. (A,B) Response of a sample V1 (A) and V4 (B) unit to the early deviant (blue), standard (green) and late deviant (yellow) sequence. Gray shading indicates the response to the O stimulus. (C,D) Average peristimulus time histograms for V1 (C) and V4 (D) units, for the O stimulus in each of the 3 sequences. Color conventions as in (A,B). Line thickness indicates SEM. (E,F) Comparison of the response to the O stimulus in the standard sequence (abscissa) with the response to the O stimulus in the early (blue) or late (yellow) deviant O, in V1 (E) and V4 (F). Each dot corresponds to a single unit or sorted multiunit cluster. Open symbols from monkey C; filled from monkey M.

Predictions based on the global pattern might instead be a weighted mixture of the different sequences, rather than reflecting solely the most common sequence. This prediction is equivalent to the uncertainty that an X or O stimulus will be presented in each epoch of the sequence (Fig. 1D, black). In this case, the first two stimuli in the sequence should evoke no response, because they involve no uncertainty. The final three stimuli of the standard sequence, however, should evoke some response because the prediction is biased toward but not matched to these stimuli (green). The unexpected stimuli in the deviant sequences would evoke strong responses (blue, yellow). Finally, predictions derived from learning global statistics might evolve during the trial, taking into account the stimuli observed in that trial thus far (i.e., conditional probabilities; Fig. 1E, black). For instance, the early appearance of the O stimulus in the early deviant sequence (blue) is unexpected and generates a strong response. But once the early O is observed, there is no ambiguity about the remaining stimuli in the sequence so these should evoke no response.

A second scenario is that predictions might be based on the “local” rather than the global pattern. When either an X or O stimulus is presented, the expectation that same stimulus will be evident in the next epoch increases (as in basic repetition suppression; Fig. 1F, black); when the stimulus is not shown, the expectation that it will appear in the next epoch decreases (to a minimum of zero). If we assume the expectation for the X and O stimuli are updated independently, with the prediction error reflecting their combined influence, the responses should resemble Figure 1F. Responses to the X stimulus should decrease as the sequences progress. In addition, the response to the O stimulus should be strongest in the late deviant sequence and weakest in the early deviant. This is because the expectation of an X stimulus grows with repetition; when an O is instead presented, the prediction error reflects both the unexpected appearance of the O stimulus and the violation of the expectation that an X will appear.

In summary, our sequences could generate a range of neuronal responses, depending on precisely what is predicted. Importantly, prediction errors can be assessed by comparing responses to the same stimulus in different contexts (the O stimulus, which can appear at an expected or unexpected time) or an unexpected repetition of a just-encountered stimulus (as in the fourth X of the late deviant sequence). Neither case relies on comparing responses to two different stimuli as in traditional “oddball” protocols, a comparison for which it is difficult to rule out the possibility that any observed differences arise from stimulus-specific adaptation.

Spiking Responses to Deviant O Stimuli Show no Evidence of a Pattern-Violation Response

Both the local and global scenarios suggest that the response to the O stimulus should differ between standard and deviant sequences, if neuronal responses reflect prediction errors. To test this, we recorded responses to these sequences from two alert nonhuman primates performing a fixation task. The X and O stimuli were large, high contrast gratings, whose orientation differed by 90°. We obtained responses simultaneously from units in V1 (mean population of 47.8 ± 1.2 visually responsive units per session) and V4 (26.0 ± 1.1 units). Animals performed an average of 345 ± 22.7 trials per session, and 1.8 sessions per day.

Figure 2A shows responses of an example V1 neuron to the standard sequence (green) and to each of the two deviants (blue and yellow). This neuron responded more strongly to the O than X stimulus, due to its orientation preference. More importantly, the responses to the O gratings were similar whether that stimulus occurred at the expected time in the sequence (standard, green) or at unexpected time (deviants, blue and yellow). Responses of a sample V4 unit recorded in the same session showed a similar behavior (Fig. 2B).

To determine whether the responses to expected and unexpected O stimuli differed, we first inspected the PSTHs for each O stimulus, averaged over all responsive cells (Fig. 2C,D; 16 sessions in 2 animals). These did not reveal notable differences in a particular epoch, so we measured each neuron’s response as the average firing rate in a 200 ms epoch beginning at stimulus onset. In both V1 and V4, the responses to the standard O and the early deviant O were similar (Fig. 2E,F, top; for V1, 15.6 ± 0.6 vs. 14.7 ± 0.5 sp/s; for V4, 19.7 ± 0.8 vs. 18.2 ± 0.8 sp/s), as were responses to the standard O and the late deviant O (Fig. 2E,F; bottom; for V1, 15.2 ± 0.5 sp/s; for V4, 18.7 ± 0.8).

We summarized the evidence for pattern-violation responses by computing the geometric mean ratio, across cells, of the response to the O stimulus in the deviant compared with the standard sequences (Fig. 3A). For the early deviant, the mean ratio was 1.05 in V1 (P < 0.001 for difference from a ratio of 1) and 1.08 in V4 (P < 0.001). For the late deviant, the ratios were, 1.00 in V1 (P = 0.58) and 1.02 in V4 (P = 0.05). Thus, on average, across the population, we found at most an 8% average modulation of neuronal responses to a stimulus which violated the expected pattern.

Figure 3 .

Stimulus variants. (A) Histogram with blue bars shows the ratio of the response to the O stimulus in the early deviant sequence compared with the O stimulus in the standard sequence. Each case is a single- or multi-unit cluster. Histogram with yellow bars shows the corresponding values for the O stimulus in the late deviant sequence. V1 data plotted on left; V4 data on right. X and O stimuli are large sinusoidal gratings differing in orientation by 90 deg. (B) Comparison of responses to deviant and standard O stimuli, when X and O stimuli were small sinusoidal gratings. (C) Comparison of responses to deviant and standard O stimuli, when X and O stimuli differed in contrast not orientation. (D) Comparison of responses to deviant and standard O stimuli, when X and O stimuli were large gratings presented for 200 ms. (E) Comparison of responses to deviant and standard O stimuli, when only standard sequences were presented for the first 200 behavioral trials, to ensure the establishment of an expected sequence. (F) Comparison of responses to deviant and standard O stimuli, when trials with deviant sequences provided twice the reward volume as trials with the standard sequence. In all histograms, ratios less than 0.33 or greater than 3 are placed in the first or final bin, respectively, for display purposes only.

In our preceding analysis, we assumed that each of the recorded neurons was an independent sample. Because neurons were recorded from chronically implanted arrays, it is likely that some units were recorded across multiple sessions. To provide a more conservative test of response modulation by pattern violation—one in which statistical power is not inflated by the assumption of independence—we determined the percentage of cells which showed statistically distinguishable responses for deviants relative to the standard (bootstrap, P < 0.05). We required that both deviant responses were larger than the standard response, as expected if neurons encoded prediction errors defined by the “global” predictions of Figure 1C–E. Only a small percentage of neurons showed statistically significant modulation (in V1, 0.8% of cases; in V4, 1.6% of cases, with a false positive rate of 0.25%). Similarly, we tested whether cells conformed to the “local” prediction of Figure 1F (testing whether responses to early O stimulus were significantly weaker than to the standard whereas responses to that late O deviant were significantly greater). Fewer than 1% of V1 and V4 cells had responses consistent with this prediction. Thus, few neurons showed evidence of encoding pattern violations.

We considered that our results might be affected by fixational eye movements, which can substantially modulate visual cortical responses (Bair and O’Keefe 1998; Martinez-Conde et al. 2009). To test this possibility, we identified all epochs during which a microsaccade was made during the presentation of an O stimulus (see Methods). We then compared responses measured on trials without microsaccades, with those during which a microsaccade occurred. In every case—comparing the early and late deviant O with the standard O, in either V1 or V4—the response ratio was near 1 (range of 0.98–1.07). There was no significant difference in response ratio for the two subsets of trials (P > 0.05, Bonferroni correction for 4 comparisons). Thus, the absence of V1 or V4 response modulation by pattern violation cannot be attributed to fixational eye movements.

We conclude that there is little modulation of V1 and V4 neuronal responsivity by whether a stimulus appears at the expected or unexpected time during a stimulus sequence.

Stimulus Variants

We have thus far only considered responses to sequences of large high-contrast gratings. We tested several variants of this stimulus, in search of stronger pattern-violation responses.

Stimulus size: We recorded in additional sessions using smaller grating stimuli—1.75° in diameter (6 sessions in 2 animals). Large stimuli recruit surround suppression, which can be viewed as a form of hierarchical, predictive coding (Rao and Ballard 1999). Thus, we thought prediction-error responses induced by sequences of small gratings might somehow differ from those induced by sequences of large gratings. However, we found that neuronal responses to the deviant O stimuli were nearly indistinguishable from the responses to the standard O stimulus. In V1, the average response ratio (deviant/standard) was 1.04 for the early deviant and 0.98 for the late deviant (P = 0.02 and P = 0.28, respectively; Fig. 3B, left). In V4, the corresponding average ratios were 1.06 (P < 0.001) and 1.09 (P < 0.001). On average, 1.8% of V1 and V4 units showed significant modulation by pattern violations, defined by either global or local predictions.

Contrast: We recorded responses to sequences in which X and O stimuli had the same orientation but differed in contrast (0.25 for one stimulus, and full contrast for the other; 6 sessions in 2 animals). Previous fMRI work has reported that V4 responds robustly to increments or decrements of contrast (Gardner et al. 2005), indicating an encoding of contrast change (perhaps a type of prediction error; see also Boehnke et al. 2011 for related work in the colliculus). Thus, we thought pattern-violation responses might be apparent for sequences of stimuli with different contrasts, at least in V4. However, neuronal responses to the deviant O stimuli were similar to those for the standard O (Fig. 3C). In V1, the response ratio for the early deviant O was 1.03 (P = 0.02) and for the late deviant 1.03 (P = 0.02). In V4, the respective response ratios were 1.07 (P < 0.001) and 1.03 (P = 0.08). On average, 3.8% of V1 and V4 units showed significant modulation by pattern violations, defined by either global or local predictions.

Presentation duration: We used sequences in which each X and O was presented for a longer duration (200 rather than 100 ms presentations; 2 sessions in 1 animal). Previous studies have found that expectation violation signals are more evident in later response epochs (Todoric and de Lange 2012; Chen et al. 2015; Schwiedrzik and Freiwald 2017). By prolonging the presentation of each sequence element—to a duration similar to a visual fixation—and measuring responses in a later epoch (150–300 ms after stimulus onset), we thought we might find pattern-violation responses, but we did not (Fig. 3D). In V1, the mean ratio was 1.01 for the early deviant (P = 0.7) and 0.95 for the late deviant (P = 0.001). In V4, the mean ratio was 1.04 for the early deviant (P = 0.3) and 0.92 for the late deviant (P = 0.002). On average, 0.6% of V1 and V4 units showed significant modulation by pattern violations, defined by either global or local predictions.

Establishing a stronger expectation for the standard: We considered that perhaps deviant sequences failed to elicit pattern-violation responses because they were randomly interleaved with the standard sequence. Perhaps the occasional presentation of a deviant sequence interfered with the establishment of a strong expectation for the standard sequence. We therefore conducted additional sessions in which we presented the standard sequence for the first 200 trials, before randomly interleaving standard (80% of trials) and deviant (10% for each version) sequences (2 sessions in 1 animal). We only analyzed responses after the initial 200 trials, used to set expectation. Neuronal responses to the early and late deviant O were nearly identical to the standard O (Fig. 3E; average ratio for early deviant: 1.03, P = 0.08 in V1; 1.06, P = 0.01 in V4; for late deviant: 1.01, P = 0.56 in V1; 1.1, P < 0.001 in V4). On average, 2.4% of V1 and V4 units showed significant modulation by pattern violations, defined by either global or local predictions.

Enhancing the salience of deviant sequences: We sought to heighten the behavioral salience of the deviant sequences (2 sessions in 1 animal), thinking greater behavioral relevance might reveal pattern-violation responses. We manipulated salience by doubling the reward provided on deviant trials. Still, we found no consistent effect on response ratios in V1 (for early deviant: 1.1, P < 0.001; for late deviant: 0.99, P = 0.66), and only a slight elevation of response ratios in V4 (1.09 for early, P < 0.001; 1.06 for late, P < 0.001; Fig. 3F). On average, 2.0% of V1 and V4 units showed significant modulation by pattern violations, defined by either global or local predictions.

Omission responses: We tested whether the omission of an expected stimulus would trigger a measurable response, as previously reported in the retina (Schwartz et al. 2007) and several mouse V1 studies (Latimer et al. 2019; Homann et al. 2019). We presented a sequence of 5 identical gratings on 90% of trials; on remaining trials, we removed the fourth grating from the sequence (4 sessions in 1 animal). There was no response to the omitted stimulus evident in the population PSTHs in V1 or V4 (Fig. 4A,B). In V1, the response during the omitted stimulus was indistinguishable from the spontaneous firing rate measured before sequence onset (Fig. 4C, P = 0.13, paired t-test), and much weaker than the response to the grating presented during that epoch in the standard sequence (Fig. 4D; P < 0.001). In V4, the response to the omitted stimulus was weaker than the spontaneous firing rate before sequence onset (Fig. 4E, P < 0.001), but similar to the spontaneous firing rate in the interstimulus interval between the preceding stimuli (Fig. 4B). Responses to the omitted stimulus were also weaker than the response to the corresponding grating in the standard sequence (Fig. 4F; P < 0.001).

Figure 4 .

Omission stimulus. (A,B) Population V1 (A) and V4 (B) PSTHs for response to a standard sequence of 5 gratings (green) and to infrequent (10% of trials) presentations of the same sequence but with the fourth grating omitted (gray). Dotted lines show the average spontaneous firing rate. (C) Comparison of the spontaneous firing rate (abscissa) and the response to the omitted grating (ordinate), for V1 units. (D) Comparison of the response to the fourth grating in the sequence (abscissa) and the response measured when that grating was omitted (ordinate). (E,F) As in (C,D) but for V4 units. Responses less than 0.1 sp/s are displayed along the axes.

In V4, we did observe that the response onset latency for the final stimulus in the omission sequence occurred earlier (gray, Fig. 4B) than the response to that same stimulus embedded in the standard sequence (green). This difference in onset latency was not evident in V1 (Fig. 4A). One possibility is that the change in V4 onset latency is a form of prediction error response, with responses to an expected stimulus (the final stimulus of the standard sequence) being delayed relative to those that are less expected (the final stimulus after an omission). An alternative—and in our view, more likely—possibility is that the delayed onset in the standard sequence is a consequence of sensory adaptation, as described in previous work (Saul 1995; see also Fig. 5B for shifts in onset latency during the preceding stimuli). In the omission sequence, the absence of a stimulus before the final stimulus presumably allows for some recovery from adaptation, so response onset is more rapid.

Figure 5 .

Response to X stimuli in the late deviant sequence. (A) Example PSTHs to the late deviant sequence in V1 (left) and V4 (right). Colored bars under the PSTHs indicate when each of the four X stimuli were presented; thin gray line indicates when the O stimulus was presented. (B) Population PSTHs for responses to X₁–X₄, for V1 (left) and V4 (right). (C) Ratio of the response to stimulus X_i, compared with X₁, for V1 (left) and V4 (right). Error bars, within each symbol, indicate SEM.

In summary, we tested several stimulus variants: manipulating grating size and contrast, lengthening the presentation duration, providing extended exposure to standard sequences, enhancing the reward provided on trials with deviant sequences, and measuring responses to an expected but omitted stimulus. None of these variants revealed compelling evidence for pattern-violation responses. Only a small percentage of cells showed significant modulation of responsivity by pattern-violations.

X Repetition Induces Robust Suppression but not Pattern-Violation Responses

The weak evidence of pattern-violation responses to O stimuli in deviant sequences might simply reflect that the sequences created no expectation. To test this possibility, we focused on the late deviant sequence—XXXXO—in our standard paradigm (Fig. 2). We tested whether responses to X stimuli were weakened with each successive presentation. That is, we tested for repetition suppression, the phenomenon interpreted as foundational evidence for predictive coding theory (Clark 2013). In addition, we compared the response to the fourth X in the sequence to those to preceding X stimuli, a pattern violation which involved no change in stimulus properties relative to those preceding stimuli.

Spiking responses became weaker with each successive presentation of the X stimulus, including the fourth “unexpected” repetition, as shown for example V1 and V4 units in Figure 5A. In V1, this response reduction was driven by a reduction in the amplitude of the onset transient, as evident in the population PSTH (Fig. 5B, left; see also Patterson et al. 2013). In V4, responses to later stimuli in the sequence were weaker because response onset was delayed (Fig. 5B, right).

To quantify the degree to which responses weakened across stimulus presentations for each unit, we calculated the ratio of the response to each X, relative to the first presentation. In both V1 and V4, this ratio decreased monotonically with repetition number (Fig. 5C). The responses to the second X, for instance, were significantly weaker than to the first presentation (P < 0.001 in both V1 and V4), indicating robust repetition suppression. This suppression indicates that our sequences induced some “expectation” in the sense that weaker responses have been interpreted as evidence for smaller prediction errors.

The observation that responses decreased with each successive presentation of the X stimulus also applied to the fourth X in the sequence, which was an unexpected stimulus based on the standard pattern. The response to that stimulus was significantly smaller than the response to preceding X stimulus in both V1 (0.86 vs. 0.92; P < 0.001; paired t-test) and V4 (0.79 vs. 0.87; P < 0.001). Thus, an unexpected repetition of a stimulus produces no pattern-violation response in either V1 or V4.

Notably, the repetition suppression evident in the responses to the fourth X stimulus, when it was an unexpected stimulus, was stronger than the suppression observed to the fourth X in the omission experiment, when that stimulus was expected. In the long deviant sequence, the response ratio for the fourth X compared with the third X stimulus was 0.93 in V1 and 0.91 in V4; in the omission sequence, the corresponding ratios were 1.05 and 1.01. This is entirely contrary to the expected outcome for pattern-violation responses.

LFP Responses do not Signal Pattern Violations

We next analyzed the LFPs recorded in V1 and V4 simultaneously with the neuronal spiking responses. The LFP is an aggregate electrical signal that reflects both synaptic events and active neuronal conductances, and can provide a sensitive measure of local neural activity (Einevoll et al. 2013). In the current context, we thought the LFP might capture weak but broadly shared pattern-violation signals (Polterovich et al. 2018; Vinken et al. 2018), which might be undetectable with spiking activity.

We focused our analyses on LFPs for O stimuli in the sequences of large, high contrast gratings (sessions of Fig. 2). LFPs evoked by O stimuli in the standard and deviant sequences were visually indistinguishable (Fig. 6A, for population average V1 LFPs in one session; Fig. 6B for overlay of responses to the different O stimuli). To quantify LFP responses, we computed the rms amplitude of the trial-averaged response for each electrode in each session. This metric is equally sensitive to positive and negative components of the signal, and thus does not require assumptions about which components might differ across conditions. We then compared the rms amplitude for responses to the O stimuli in each of the deviant sequences, relative to the O stimulus in the standard sequence (Fig. 6C). The ratio of these responses was between 0.97 and 1.03 (i.e., a 3% modulation), except for V4 responses to the early deviant in one animal (15% smaller than the standard; blue trace, Fig. 6B, bottom).

Figure 6 .

Comparison of LFP responses to the O stimulus, in standard and deviant sequences. (A) Trial-average LFP response from one electrode in V1, for standard (green), early (blue) and late (yellow) sequences. (B) Average LFP responses in V1 (top) and V4 (bottom) for each of the 3 sequences. Color convention as in (A). (C) Comparison of LFP response to the O stimulus in the standard sequence (abscissa) with the response to the O stimulus in the early (left, blue) and late (right, yellow) deviant, in V1 (top) and V4 (bottom). Each symbol represents data from one electrode in one recording session.

Thus, LFPs in V1 and V4 show little evidence of pattern-violation responses.

Responses to Unexpected Stimuli in Human Visual Cortex Depend on Task Engagement

Much of the work on repetition suppression and predictive coding has been performed in human participants, using either EEG or fMRI. We wondered whether our stimulus protocol would induce different effects in macroscopic measurements of human brain activity. We therefore conducted complementary human EEG recordings, using the same stimulus design as in monkey experiments (but with slightly different temporal and spatial parameters, see Methods). These additional experiments also allowed us to assess the effect of task engagement on evoked responses by altering verbal instructions given to participants.

In the first experiment, participants were instructed to maintain fixation and count the number of gratings presented. The task thus required participants to be alert and attend the stimuli, but it did not require awareness or detection of patterns or pattern violations. Each grating of the sequence evoked a clear scalp potential at each of the occipital electrodes (O1, O2, OZ, PO9, PO10, where visual-evoked responses have greatest signal-to-noise ratio), and in their average (Fig. 7A). The average response to the O stimulus in the standard sequences (Fig. 7A, bottom left, green) and in the deviant sequences (early, blue; late, yellow) were indistinguishable (repeated measures ANOVA on average EEG amplitude from 0 to 700 ms after stimulus onset, F(2,18) < 0.01; P > 0.5), indicating no signaling of the pattern violations. We also found no evidence of pattern-violation responses evoked by the unexpected fourth X stimulus in the late deviant sequences, which were indistinguishable from the preceding X stimuli (Fig. 7A; bottom, right, repeated measures ANOVA, F(3,27) = 1.3, P = 0.29). The absence of repetition suppression in these responses likely reflects the relatively brief stimulus presentation (200 ms) and substantial interstimulus interval (500 ms).

Figure 7 .

Human EEG responses to standard and deviant sequences. (A) Top: Average response to the standard (green), early deviant (blue) and late deviant (yellow) sequences, recorded while subjects performed a counting task. Responses shown are the average across the occipital electrodes (OZ, PO9, PO10, O1, and O2). Gray bars indicate epochs during which stimuli were presented. Line thickness indicates SEM. Bottom: Overlay of the responses to the O stimulus in each of the three sequences (left), and the responses to each X stimulus in the late deviant sequence (right). (B) As in (A) but recorded while subjects performed a task requiring them to detect violations of the pattern in the standard sequence.

We then collected additional sessions in which participants were informed about the patterned structure of the stimulus sequences and instructed to indicate with a button press each time the standard pattern was violated. Thus, participants pressed the response button to the unexpected O stimulus in early deviant sequences and to the unexpected fourth X in the late deviant sequence. In this task, the evoked responses to deviant sequences differed markedly from responses to standard sequences (Fig. 7B). These differences were evident both to the O stimuli in deviant sequences (Fig. 7B, bottom left) and to the unexpected X stimulus in the late deviant (bottom right). For the O stimuli, the differences were evident 200–250 ms after stimulus onset, determined by repeated-measures ANOVA conducted on successive 50 ms epochs of the response (P < 0.05, Bonferroni corrected). For the X stimulus, the difference was first evident 250–300 ms after stimulus onset.

To determine whether these pattern-violation responses reflected processes early in the visual stream or were more closely tied to attentional processes associated with target detection (i.e., a P3b response; Luck 2014), we first subtracted the average response to the standard sequence from the average responses to each of the deviant sequences. This subtraction eliminated the sensory ERP components elicited by the presentation of each stimulus (e.g., P1). The resultant difference signal was minimal, for responses recorded while participants performed the counting task (shown for Pz, Oz, P09, and P010 electrodes in Fig. 8A). For responses obtained when the participants performed the pattern violation task, both deviants produced a robust difference signal (Fig. 8B).

We then applied current source density analysis to these difference signals to sharpen the spatial representation of the underlying current dipoles and reveal the corresponding scalp topography (Fig. 8A,B; insets). Pattern deviants triggered a clear P3b component, which was strongest at the Pz electrode (thick solid line), as expected. In addition, deviants evoked robust signals preceding the P3b, which were strongest at electrodes over occipital cortex. Thus, it appears that task engagement can recruit some type of pattern-violation response within visual cortex when detecting the pattern violation is an explicit task goal.

Discussion

To test whether visual cortical responses are consistent with predictive coding theory, we presented a fixed sequence of grating stimuli on most trials, and infrequently presented deviant sequences in which the stimulus ordering was altered. We measured cortical responses to these sequences at three spatial scales: spiking activity and LFPs, in V1 and V4 of macaque monkeys; and EEGs, in human observers. We found little evidence for pattern-violation signals, except in EEG signals when human participants were explicitly told to report violations of the expected sequence. Our results suggest that the vast majority of neurons in early and midlevel cortex do not encode sensory prediction errors, at least those which might be generated from established temporal patterns of visual input.

Evidence Against Prediction Error Responses

Our stimulus design allows us to distinguish between effects that might be attributed to simple adaptation—as in a basic repetition suppression or “oddball” paradigms—and effects which can only be attributed to a mismatch (or prediction error) between an expected stimulus and the one experienced. Importantly, the expectation in our design was based on a passively experienced, regular sequence of stimuli. This expectation does not require explicit cueing, which may recruit internal cognitive processes not directly tied to sensory processing. Importantly, predictive coding theory is a general theory of sensory representation, rather than one predicated on specific task goals (Aitchison and Lengyel 2017).

One might argue that defining expectation based on a passive viewing of sequences of grating stimuli is a narrow and contrived test of predictive coding theory. Perhaps the theory is correct, but our stimulus protocol does not generate sufficiently strong expectations to reveal its relevance. However, we found that the successive presentation of X stimuli in our sequences resulted in progressively weaker responses (Fig. 5). This weakening—repetition suppression—is frequently cited as evidence for predictive coding. But this weakening can only support the theory if sequences result in predictions that strengthen with each exposure (and thus generate smaller prediction errors, or responses). If predictions grow in strength during a sequence then violations of those predictions should trigger stronger responses, which we did not observe.

We measured responses in V1 and V4 because encoding is relatively well understood in these areas. Our stimuli involved manipulations of grating contrast, size, and orientation, which robustly modulate neuronal responsivity in both V1 and V4 (Lennie and Movshon 2005; Roe et al. 2012). Thus, the weak evidence for pattern-violation responses in these areas cannot be attributed to a mismatch between the stimuli used and the area targeted. An additional benefit of targeting V1 and V4 is that these areas occupy distinct levels of the visual hierarchy (Felleman and Van Essen 1991). By recording responses to a common set of stimuli in both areas (in the same animals, at the same time), we were able to compare signals that might be sent from V1 to V4 (presumed to relay prediction errors) and those that might be relayed in the opposite direction (from V4 to V1, presumably providing predictions to V1). We observed similar repetition suppression and near absence of pattern-violation signals in both areas, and no evidence for a hierarchical organization of prediction errors as proposed by predictive coding theory (see Vinken et al. 2017 for a contrary outcome in rodent cortex).

We also found no evidence, in either V1 or V4, for the existence of two functional classes of neurons: those which might provide predictions to lower cortex, and another which might encode the mismatch between the prediction and the encountered sensory input. Only these latter neurons should exhibit repetition suppression (if this suppression reflects prediction errors), whereas our data and previous studies indicate that nearly all neurons show weaker responses to a repeated stimulus. Further, there was no evidence of bimodality in the response distributions of Figure 4 (Hartigan’s dip test, P > 0.4 in all cases), as one might expect there to be if a substantial subset of neurons encoded sensory prediction errors. However, because our recordings involved planar microelectrode arrays, we cannot exclude the possibility that two functional classes exist, if these were separated across cortical layers.

Our simple conceptual model (Fig. 1) points out an overlooked assumption of attempts to relate neuronal responses to predictive coding theory, raising an additional challenge to the theory. Specifically, the weaker responses in repetition suppression are evident at stimulus onset, which requires the prediction to appear precisely when the stimulus does (as in Fig. 1B; see also Wacongne et al. 2012 for this behavior in a predictive coding model). This, in turn, requires knowledge of when a stimulus will reappear, or the maintenance of a prediction signal during the interstimulus interval. The first possibility is illogical since repetition suppression does not require learning an interstimulus interval; it is evident after a single stimulus presentation and for a wide range of interstimulus intervals (Priebe et al. 2002; Patterson et al. 2014). The proposal of a maintained prediction signal between stimulus presentations is also incompatible with cortical physiology. When a stimulus ends, the response it evokes rapidly decays (e.g., Figs 4 and 5). If predictions carry forward in time to produce repetition suppression, then they should cause prediction errors (or perhaps, more accurately, a “surprise” response or unsigned prediction error; Lieder et al. 2013) during the interstimulus interval rather than a return to the prestimulus spontaneous firing rate. Experimental tests of predictive coding theory often treat sensory input as discrete isolated events, neglecting to account for how cortex responds in the interstimulus interval.

We note that our conceptual model makes a number of simplifying assumptions. For instance, predictions involve an expectation that a particular stimulus will appear. Predictions might instead consist of multiple factors: one being that a stimulus will appear, another being what its features will be, and so on. In this case, the responses produced by prediction error coding of our sequences might differ from the scenarios we consider. We note that if predictions simply involved the appearance of a stimulus, regardless of its identity, than stimulus omissions should have triggered a prediction error response, which we did not observe (Fig. 4). Clearer specification of the nature of top–down predictions (e.g., their tuning) would be needed to provide experimentally testable predictions of more subtle and sophisticated schemes.

Comparison with Previous Work

Our experiments involve relatively brief experience with grating sequences (hundreds of trials in each session). This is different from studies of “statistical learning,” in which two stimuli are presented in a fixed order, over hundreds or thousands of repetitions distributed across multiple days or weeks. This form of longer term learning establishes an expectation of transition probabilities between images (Meyer and Olson 2011; Kaposvari et al. 2016), resulting in weaker responses to the expected, trailing image in area V2 and IT cortex in monkeys (Ramachandran et al. 2017; Schwiedrzik and Freiwald 2017; Huang et al. 2018), a potential correlate of predictive coding. Similarly, responses to naturalistic images (whose properties are presumably learned through extended experience) and scrambled controls differ in a way which appears consistent with some aspects of predictive coding theory (Issa et al. 2018). Our results do not conflict with these observations since they involve vastly different time scales. Importantly, predictive coding theory is proposed as a general theory of cortical function. Although it requires training—in the sense that predictions are based on prior knowledge or experience—the theory is meant to be broadly relevant to our visual experience, not only stereotyped patterns of input that are experienced repeatedly over days.

A perhaps surprising discrepancy between our study and previous work is the absence of omission responses (Fig. 4). Omission responses have been reported in a number of human EEG studies (e.g., Nordby et al. 1994; Wacongne et al. 2011). They have also been reported in some neuronal studies of early sensory processing. For instance, Schwartz et al. (2007) reported robust omission responses in the retina though these responses were only observed for sequences with repetition rates above 6 Hz, and thus might not reflect a general prediction error coding scheme. A few studies have also reported omission responses in V1. In some studies (Gavornik and Bear 2014; Fiser et al. 2016) these responses were observed after extensive exposure to stimulus sequences. But in other studies omission responses were evident after a within-session exposure (Homann et al. 2019; Latimer et al. 2019), similar to ours. The discrepancy between those studies and ours might reflect differences in the stimuli or species used. Consistent with the latter possibility, we note that in primate cortex “statistical learning” results in weaker responses to the expected stimulus (Meyer and Olson 2011; Kaposvari et al. 2016; Ramachandran et al. 2017), whereas in mouse V1 it leads to stronger responses (Gavornik and Bear 2014). In any case, our results show that omission responses are not a universal feature of sensory encoding.

Many previous studies that sought to test predictive coding theory have relied on measuring the mismatch negativity (MMN), a scalp potential evoked when an unexpected stimulus is presented (Näätänen et al. 2001; Sussman et al. 2014; Stefanics et al. 2014). In the simplest stimulus paradigm, the MMN is evoked by a rarely presented stimulus. However, this form of the MMN can be readily explained by neural fatigue or adaptation. More sophisticated paradigms—for instance, using sequences of stimuli—have provided robust evidence for prediction violation responses (Sussman et al. 1998). For instance, using stimulus sequences similar to ours, Bekinschtein et al. (2009) found that the responses to rarely presented stimuli triggered an MMN in early sensory cortex. But signals encoding a violation of an expected sequence were only evident in fronto-parietal networks, and required task engagement. Similarly, Wacongne et al. (2011) found that stimulus changes within a pattern (X vs. O) could elicit an MMN in auditory cortex, but that responses to violations of an expected pattern were more evident in fronto-parietal areas. These pattern-violation responses required subjects to be attentive but did not require engagement in a pattern-violation detection task. Finally, Chao et al. (2018) studied auditory pattern-violations responses in macaque monkeys using electrocorticography, during passive listening. They found pattern-violation responses were not evident in early auditory cortex under these conditions, but were observed in anterior temporal cortex. Our EEG findings are broadly consistent with these studies, in that we found pattern-violation responses were not evident in early sensory cortex under attentive viewing. However, in our study, task engagement did reveal evidence for prediction violation responses in early sensory cortex, not only in fronto-parietal networks as in these previous EEG studies.

We note that in our experimental design the enhanced sensory EEG response during the performance of the task could be either due to the detection of the target (i.e., the task goal) or to the target being unexpected. In previous work in the auditory system (Sussman et al. 2002; Max et al. 2015; Symonds et al. 2017), we have found that under some stimulus conditions (slow stimulus presentation rates) the enhanced responses required both pattern-task engagement (a “top–down” influence) and a violation of the expected sequence; neither influence was sufficient to elicit an enhanced response. However, when stimuli are presented rapidly, the expectation of the auditory pattern is established even without task engagement (Sussman and Gumenyuk 2005; Symonds et al. 2017), unlike the findings presented here. Thus, changing the temporal parameters of patterns presented to the visual system might yield different results.

Predictive Coding and Cortical Function

Our data do not disprove the existence of prediction error coding in sensory cortex. Across neurons, there was a small (up to 8%) modulation of responsivity for expected compared with unexpected stimuli, though this modulation was not always consistent for violations associated with early compared with late deviants. A small percentage of neurons (~1%) did show consistent modulation of the type expected for prediction error coding. One possibility is that prediction error coding is performed by these neurons. Another possibility is that our recording strategy failed to sample the relevant population, though this explanation makes it nearly impossible to test any theory—proponents can always seek recourse in an undetected subpopulation of neurons. A more straightforward interpretation of our data is that the theory has limited merit, at least as a description of the function of most sensory cortical neurons.

We also emphasize our data do not argue against the general concept of predictive coding. Well-established cortical mechanisms of visual processing, such as receptive field surrounds (Rao and Ballard 1999; Spratling 2010; Lochmann et al. 2012) and simple sensory adaptation (Hosoya et al. 2005; Lochmann et al. 2012; Solomon and Kohn 2014), can be viewed as instantiations of predictive coding. Nor do our results argue against modulation of sensory responses by expectation. It is well established that sensory cortex is affected by internal cognitive processes like expectation and attention (Carrasco 2011; Cohen and Maunsell 2014), and these likely involve “top–down” signaling. It is also clear that sensory systems must use prior knowledge to properly infer the state of the world (i.e., the causes of the sensory signals they receive).

The critical distinctions between these more general concepts and those of predictive coding theory explored here is the explicit comparison of a top–down prediction and bottom-up sensory evidence, and a representation of their difference—the prediction error—in sensory cortex. Our data provide little evidence in support of this view as a central motif of cortical visual processing, when predictions are based on the temporal pattern of recently encountered stimuli.

Author Contributions

SSS, HT, ES, AK designed the research; SSS, HT collected the data; SSS, HT, ES, AK analyzed the data; AK drafted the manuscript; SSS, HT, ES, AK revised the manuscript.

Funding

This work was supported by National Institute of Health (EY024858 and EY028626).

Notes

We thank Jean Demarco and members of the Kohn lab for assistance with data collection. We thank Anthony Norcia for helpful discussion of the EEG responses. The authors declare no competing financial interests. Conflict of Interest: None declared.