Abstract
A novel neural signature of active visual processing has recently been described in the form of the “perceptual echo”, in which the cross-correlation between a sequence of randomly fluctuating luminance values and occipital electrophysiological signals exhibits a long-lasting periodic (∼100 ms cycle) reverberation of the input stimulus (VanRullen and Macdonald, 2012). As yet, however, the mechanisms underlying the perceptual echo and its function remain unknown. Reasoning that natural visual signals often contain temporally predictable, though nonperiodic features, we hypothesized that the perceptual echo may reflect a periodic process associated with regularity learning. To test this hypothesis, we presented subjects with successive repetitions of a rapid nonperiodic luminance sequence, and examined the effects on the perceptual echo, finding that echo amplitude linearly increased with the number of presentations of a given luminance sequence. These data suggest that the perceptual echo reflects a neural signature of regularity learning.
Furthermore, when a set of repeated sequences was followed by a sequence with inverted luminance polarities, the echo amplitude decreased to the same level evoked by a novel stimulus sequence. Crucially, when the original stimulus sequence was re-presented, the echo amplitude returned to a level consistent with the number of presentations of this sequence, indicating that the visual system retained sequence-specific information, for many seconds, even in the presence of intervening visual input. Altogether, our results reveal a previously undiscovered regularity learning mechanism within the human visual system, reflected by the perceptual echo.
SIGNIFICANCE STATEMENT How the brain encodes and learns fast-changing but nonperiodic visual input remains unknown, even though such visual input characterizes natural scenes. We investigated whether the phenomenon of “perceptual echo” might index such learning. The perceptual echo is a long-lasting reverberation between a rapidly changing visual input and evoked neural activity, apparent in cross-correlations between occipital EEG and stimulus sequences, peaking in the alpha (∼10 Hz) range. We indeed found that perceptual echo is enhanced by repeatedly presenting the same visual sequence, indicating that the human visual system can rapidly and automatically learn regularities embedded within fast-changing dynamic sequences. These results point to a previously undiscovered regularity learning mechanism, operating at a rate defined by the alpha frequency.
Keywords: alpha, cross-correlation, EEG, perceptual echo, regularity learning, temporal sequence learning
Introduction
There is accumulating evidence that complex spatial-temporal dynamics of visual stimuli are encoded early in the visual processing stream (Chubykin et al., 2013; Gavornik and Bear, 2014; Cooke and Bear, 2015; Zold and Shuler, 2015). However, relatively little is known about how the human visual system tracks, or learns about, such rapidly changing stimulus sequences. One recent finding relevant to this question is the “perceptual echo” (VanRullen and MacDonald, 2012). This is long-lasting reverberation between a rapidly changing visual input and evoked neural activity, apparent in cross-correlations between occipital EEG and the stimulus itself. Specifically, when visually presenting a nonperiodic dynamic sequence, whose luminance randomly fluctuated at a rate of 160 Hz, VanRullen and MacDonald (2012) found that the occipital EEG response displayed a periodic reverberation or “echo” of the input sequence, which persisted for at least 1 s, and was found specifically in the alpha (∼10 Hz) frequency range of the cross-correlation function, primarily over occipital electrodes and was observable at the group level. Importantly, the absence of such a reverberation when the luminance sequence of each trial was cross-correlated with EEG recorded on a different trial underlines that the perceptual echo is a true oscillatory response to the (nonperiodic) visual stimulation sequence, and not a general property of ongoing EEG signals in response to this kind of stimulation. Furthermore, the seconds-long duration of the perceptual echo suggests a long-lasting representation of fast-changing sensory information over time. As yet, however, the functional relevance and the underlying mechanism of the perceptual echo have remained unclear.
Here, we examine the idea that perceptual echo in visual cortex is a neural signature of learning of rapid temporal regularities in ongoing sensory stimulation (following previous studies, Gavornik and Bear, 2014), we use the term learning to refer specifically to evidence of brain signals distinguishing previously presented stimuli from novel stimuli).This idea is motivated by the suggestion that perceptual echo in visual cortex repeatedly “replays” neural activation associated with afferent visual information. Supporting this view, studies in awake monkeys (Eagleman and Dragoi, 2012), mice, and rats (Xu et al., 2012; Gavornik and Bear, 2014) found that after learning an association between a cue and a sequence stimulus, presenting the cue alone can elicit the same (temporally extended) neural activation pattern as evoked by the actual stimulus. This replay activation pattern has been interpreted as a mechanism that may facilitate learning and memory consolidation (Skaggs and McNaughton, 1996; Euston et al., 2007).
We reasoned that, if the perceptual echo is indeed associated with visual regularity learning, the perceptual echo response should change across repeated presentations of the same luminance sequence, as the visual cortex encodes regularities across sequence repetitions. In two experiments we tested this prediction by using random dynamic luminance sequences that were predictable across repeated presentations. In Experiment 1, we presented random dynamic luminance sequences that were each repeated four times and we calculated the perceptual echo for each presentation. In Experiment 2, we added an “inverse” luminance sequence (which has the inverse luminance polarity of the original sequence) following the presentation of the fourth repetition of a sequence, which was followed by an additional presentation of the original sequence, to test whether changes in perceptual echo were specific to the (nonperiodic) temporal and luminance information within a given sequence.
Anticipating our results, Experiment 1 showed that the amplitude of the perceptual echo increased with the number of sequence presentations of a luminance sequence, in line with the idea that the perceptual echo reflects a neural signature of regularity learning. Experiment 2 further supported this interpretation by showing the increased perceptual echo response accompanying repeated sequence presentations was abolished by the presentation of an inverse luminance sequence. In addition, the echo response recovered when the original sequence was presented again, indicating that the visual system retained information about the sequence even when challenged by a novel input sequence.
Altogether, our data provide first evidence for a mechanism by which the human visual system rapidly learns regularities in fast nonperiodic stimulus sequences, and in doing so provides a functional account of the alpha-band perceptual echo response.
Materials and Methods
In both experiments participants were seated in a dimly-lit electromagnetically shielded room and their heads were stabilized in a head-and-chin rest to maintain centrality 50 cm away from a LaCie Electron blue IV 22” CRT monitor, which had been manually gamma corrected. Luminance-sequence stimuli were generated and presented using the Psychophysics toolbox (Brainard, 1997). These were constructed based on VanRullen and Macdonald (2012) as follows. Each sequence consisted of a disc stimulus subtending a visual angle of 7° and placed at 7.5° above a fixation point (a dim gray circle with a 0.2° visual angle). The luminance of the disc altered randomly at a rate of 160 Hz, so that each sequence consisted of 500 luminance frames (Fig. 1). Thus, each sequence comprised a rapid nonperiodic sequence of luminance changes. To ensure equal power of all temporal frequencies within each sequence, all sequences were processed by a whitening procedure. Fourier components of each sequence were obtained by a fast Fourier transform. Power at all frequencies of each random sequence were equalized by normalizing the amplitudes of its Fourier components. An inverse Fourier transform was then applied to reconstruct the sequence. Thus, sequences were not distinguishable by power characteristics of their temporal frequencies.
Figure 1.
Example of trial structure in Experiment 1. For each trial a specific random luminance sequence (e.g., sequence A) was presented four times. Each luminance sequence lasted 3.125 s and consisted of a disc with randomly changing luminance at 160 Hz. Sequences were separated by an ISI of 3 s. Response sequences, which contained an embedded square image (1 s duration, random onset time within sequence), were distributed at random points in-between trials throughout the experiment. Participants were requested to press a key after each sequence if they noticed a square.
EEG data in both experiments were recorded using a 64 channel ANT Neuro amplifier at a sampling rate of 2048 Hz. A 64 channel Waveguard EEG cap (ANT Neuro) using standard Ag/AgCl electrodes placed according to the 10–20 system, using an average reference. Horizontal and vertical eye movements were recorded using two independent electrode pairs. Impedances of recording electrodes were maintained <10 kΩ. No analog filter was applied during on-line recording. For both experiments preprocessing and data analyses were performed using the EEGLAB toolbox (Delorme and Makeig, 2004) under MATLAB (MathWorks), and custom MATLAB scripts. The acquired EEG data were downsampled to 160 Hz and filtered using a 2–80 Hz bandpass finite impulse response filter. Independent component analysis was used to identify and remove ocular artifacts. Data in each trial were then epoched from 0 to 3.125 s time-locked to the stimulus onset (start of the luminance sequence).
Experiment 1
Participants.
Twenty-one healthy University of Sussex students (11 male) with normal or corrected-to-normal vision completed the experiment (11 male, 18–36 years; mean age 24.9 years). Participants provided informed consent before the taking part and received £10 or course credits as compensation for their time. The experiment was approved by the University of Sussex ethics committee.
Experimental design.
Each experimental session comprised of 60 trials using 60 unique random nonperiodic luminance sequences. Each trial consisted of four presentations of the same sequence. Each presentation lasted for 3.125 s and successive presentations were separated by an intersequence interval (ISI) of 3 s.
To ensure participants maintained attention, 60 additional “response sequences” were distributed within the 60 (experimental) trials. Each response sequence comprised a unique nonperiodic luminance sequence, which also contained an embedded square target (3.75°) distinguished by having a nonchanging luminance for 1 s. The target appeared with onset time selected from a uniformly distributed random time during the sequence presentation. Participants were informed that the experiment was a visual detection task in which they were required to press the spacebar on a standard keyboard at the end of a sequence whenever they detected the target. Each response sequence was randomly assigned to a position in-between experimental trials. This was done individually for every response sequence, resulting in the possibility of there being one, more than one, or zero, response sequence between any two experimental trials. This design made it unlikely that participants could predict the onset of an experimental trial based on the occurrence of a response sequence.
Participants were not informed that there would be any repetition of the luminance sequences. The entire experimental session consisted of 300 (240 standard and 60 response) sequences and took ∼1 h to complete. In a post-experiment interview participants were asked if they noticed any repetition of luminance sequences in the experiment. The following questions were asked:
Did you notice any experimental manipulation during this experiment?
Did you notice any relationship between any two sequences?
Did you notice any repetition of any of the sequences?
Analysis.
To identify the perceptual echo, we calculated cross-correlations between (preprocessed) EEG time series and luminance time series for all experimental sequences and sensors using the same approach as VanRullen and Macdonald (2012). The cross-correlation procedure calculated the “impulse response function” (IRF) of the EEG as follows:
![]() |
where stim and eeg denote the standardized stimulus sequence and the corresponding standardized EEG response, respectively.
The amplitude spectra of the cross-correlation functions were calculated on lags between −1.5 and 1.5 s (Fig. 2A). For each participant we then averaged the cross-correlations across each sequence for all sensors, which revealed a long-lasting post-onset oscillation in the alpha range (Fig. 2B). Phase and power were computed by means of a continuous wavelet transform of single-trial data for the frequency range from 3 to 80 Hz. Grand average intertrial coherence (ITC) was calculated across all 14 participants and sequence presentations (4) for data from Experiment 1 (Fig. 2C). Calculations for the spectral estimate of the normalized power spectrum were run on a −1 to 1.5 s epoch using 400 ms Hanning-windowed sinusoidal wavelet. ITC values were calculated for 80 EEG frequency bands between 2 and 80 Hz. This revealed that the long-lasting post-onset oscillation in the alpha range was due to the strong phase coherence (across trials) of the cross-correlation time series within the alpha frequency range (∼10 Hz; Fig. 2C). These findings replicate the perceptual echo effect observed by VanRullen and Macdonald (2012). To quantify the amplitude of the echo response, we applied a fast Fourier transform on the average cross-correlation between 0 and 1 s for each participant and sensor to extract the alpha range (8–12 Hz) power. We emphasize that this is the power of the (average) cross-correlation function, not of neural oscillations themselves.
Figure 2.
Recording and computation of perceptual echo for a representative subject. A, For each luminance sequence presentation and EEG sensor, we computed the cross-correlation (−1.5:1.5 s) between the luminance value series and the EEG time series. This provides an ensemble of cross-correlation time series for each luminance sequence, indexed by sensor and presentation number. B, As by VanRullen and Macdonald (2012), averaging the cross-correlations for each luminance sequence revealed a long lasting oscillation (∼1.5 s) in the alpha range: the perceptual echo (blue line). The red line shows the same analysis with shuffled data (see Materials and Methods), in which no perceptual echo is observed. C, Grand average ITC of the cross-correlation time series, averaged across all participants and sequence presentations (4) from Experiment 1. The group level analysis reveals that the perceptual echo is due to the strong ITC within the alpha frequency range (∼10 Hz). Phase and power were computed by means of a continuous wavelet transform of single-trial data for the frequency range from 2 to 80 Hz. Calculations for the spectral estimate of the normalized power spectrum were run on −1 to 1.5 s epoch using 400 ms Hanning-windowed sinusoidal wavelet. ITC values were calculated for 80 EEG frequency bands between 2 and 80 Hz.
To compare the echo response across successive presentations of a luminance sequence, the cross-correlations were averaged by the order of presentation for each sensor, across trials (e.g., all cross-correlations for the first presentation of a given random luminance sequence were averaged across trials, and the same for all second, third, and fourth presentations). We then computed the amplitude of the echo response for each presentation and participant. To verify that the perceptual echo was driven by the EEG response to a specific luminance sequence and not by variations in ongoing alpha activity in the raw EEG signal, we created a “shuffled” set of cross-correlations by randomly rearranging the EEG time series with stimulus sequences from different trials from Experiment 1. As can be seen from Figure 2B (red line), this procedure leads to a complete absence of echo response, confirming that the echo response is not driven by the ongoing alpha-band EEG response.
We choose to analyze the cross-correlation between luminance values of stimulus sequences and event-related EEG signals rather than a more classical visual-evoked response to isolate the IRF of the visual system. An impulse response is the reaction of any dynamic system in response to some external change. Classical stimulus-evoked responses [such as visual-evoked responses, (VEPs)] can be viewed as a special case of impulse response function where the sensory input event is a single isolated event (Lalor et al., 2006). From this perspective, the perceptual echo can be considered as the superposition of visually evoked responses to each stimulus frame, rather than a (standard) VEP evoked by the sequence onset. This perspective has some ecological validity because sensory afferents are rarely single discrete events that would evoke isolated VEPs. Because natural visual input occurs in a rapid and continuous fashion, impulse response functions are well suited to characterize the response of the visual systems in these circumstances. Overall, because the central aim of the current study and of previous research investigating the perceptual echo (VanRullen and Macdonald, 2012) was to examine exactly how the visual system processes these types of dynamic rapid sequential inputs, we therefore used a type of analysis best suited for this purpose.
Statistical analysis.
We compared the echo response for each presentation of a luminance sequence using a polynomial trend analysis which is commonly applied to evaluate the relationship between levels of variables in fixed-effect models. In the present study, the main independent variable, the number of sequence presentations, consists of four fixed levels (i.e., Presentations 1, 2, 3, 4). The polynomial trend analysis is the most applicable method for assessing the linear, quadratic, and, cubic trends with the added benefit of avoiding the issue of multiple comparisons.
Experiment 2
Participants.
Eighteen participants took part in Experiment 2 (12 females, 20–36 years; mean age 25.2 years, none of whom took part in Experiment 1). Participants provided informed consent before the taking part and received £10 or course credits as compensation for their time. The experiment was approved by the University of Sussex ethics committee.
Experimental design.
In Experiment 2, four repeated presentations of a specific luminance sequence were followed, first by an inverse version of that luminance sequence, and second by another instance of the original (noninverse) sequence. Each inverse sequence was created by inverting the luminance polarity, of the original sequence, i.e., reversing black and white relative to the middle gray level (Fig. 3). This preserved the relative luminance values of the sequence while also maintaining a flat luminance power spectrum. These manipulations were designed to test the specificity of changes in echo responses (across presentations) to regularities embedded in the luminance sequences.
Figure 3.
Experiment 2 trial structure. In a control trial, six presentations of the same sequence were shown. In an experimental trial, after four presentations of a given sequence, the inverse-polarity luminance sequence was presented, followed by a final replay of the original (noninverse) sequence. Each sequence lasted 3.125 s, separated by an ISI of 3 s.
There were a total of 120 trials in Experiment 2, divided into “control” and “experimental” conditions. In the control condition (60 trials), a specific 3.125 s sequence was presented six times with a 3 s ISI. In the experimental condition (60 trials), a sequence was repeated four times. Then, on the fifth presentation, the inverse luminance sequence was presented. Following the inverse luminance sequence, the original sequence was presented again, making a total of six presentations of a sequence for each trial (Fig. 3).
Experiment 2 did not use the response sequences of Experiment 1. This was to avoid a highly unlikely confound arising from the distribution of response sequences falling by chance into a regular pattern between trials in Experiment 1. Potentially, this may have led participants to develop strategies to deploy more attention toward the end of a trial in anticipation of a possible response sequence. Instead, in Experiment 2, participants were asked to indicate whether the average luminance level of each sequence was brighter or darker than the luminance of the fixation circle, by pressing the left arrow for darker and right arrow for lighter, after each sequence presentation. This modification meant that all sequences were “response” sequences, ensuring that participants had to maintain equal attention to all sequences. The average luminance of a sequence was closely comparable across all sequence presentations (the SD of the average luminance of all sequences relative to the entire luminance dynamic range of all sequences was 0.018% in Experiment 1 and 0.019% in Experiment 2). Therefore, participants could not use the average luminance of a sequence as a potential cue to encode a sequence. The luminance of the fixation circle was constant, meaning that the participant's judgements about the average luminance after each sequence were always based on similar information. Across all response sequences, participants rated 63.5% of sequences as being brighter than the fixation. One participant indicated for all sequences that the average luminance was brighter than the fixation; this may have been due to a perceptual bias when rating similar information across sequences, the data from this participant still showed a robust echo response and survived the exclusion criteria.
Statistical analysis.
The same procedure used in Experiment 1 was used to identify the perceptual echo for all six presentations of a sequence (including the inverse sequence) for each trial in Experiment 2.
Traditional frequentist hypothesis testing (e.g., t test) does not provide a quantitative measure about how strongly the data supports the null hypothesis. In Experiment 2, for cases in which permutation t tests did not display significant differences in echo amplitude between presentation number, we therefore used Bayesian analysis to further evaluate to what extent the echo amplitude supported the null hypothesis (i.e., a conclusion of no difference in echo amplitude between presentations) or the alternative hypothesis (i.e., echo amplitude was different between presentations). We quantified how close to the null or alternative hypothesis each result was by Bayes factor (BF) analyses of paired sample t tests using JASP (JASP Team, 2016) with a default Cauchy prior of 0.707 half-width at half-maximum (Rouder et al., 2009).
Exclusion criteria.
In both experiments, we found that the magnitude of the echo response varied widely across individuals. Participants with no significant perceptual echo were excluded from the data analysis, as follows. First, cross-correlation time series were averaged across all trials, then the distribution of absolute values across lags were derived from the averaged cross-correlation for real and shuffled data separately. The nonparametric Kolmogorov–Smirnov test was then performed to examine the similarity of the two distributions. We excluded those participants with p > 0.001 (i.e., those participants for whom echo responses were not significantly different between the shuffled and nonshuffled data). Seven of 21 participants were excluded using this procedure in Experiment 1. Four participants were excluded using this procedure in Experiment 2, meaning that 14 participants' data were retained for further analyses.
Results
Experiment 1
Figure 4 shows the average alpha (8–12 Hz) power of the cross-correlation from Experiment 1, i.e., the amplitude of the perceptual echo, across all participants and sequence presentations. We refer to this amplitude simply as “echo response” from here on. We found a maximal echo response over occipital sensors, centered over POz, consistent with previous findings (VanRullen and Macdonald, 2012). To maximize the sensitivity to any potential effect of stimulus sequence repetition on echo response, the following analyses were therefore conducted with data from POz only.
Figure 4.
The topography of the echo response averaged across all sequence presentations and participants. The topographic plot displays the 8–12 Hz envelope of the echo response and reveals a maximum at POz with a gradual decrease from posterior to anterior electrodes. Arbitrary units, a.u.
To test whether repetitions of a luminance sequence modulated echo response, we compared the echo responses for each sequence presentation in Experiment 1 using polynomial trend contrasts (Fig. 5A). Supporting our hypothesis that successive presentations would increase echo response, we found a strong linear trend from Presentation 1 to Presentation 4 (F(1,13) = 7.32, p = 0.018). Post hoc paired t tests revealed a larger echo response for Presentation 4 compared with Presentation 1 (bootstrapping test with 10,000 resamples, p < 0.01; Fig. 5A; Howell, 2012).
Figure 5.
A, Experiment 1. Perceptual echo amplitude (echo response) for each sequence presentation, averaged across all participants, for electrode POz. The echo response displayed a linear increase across successive presentations. B, t Value map of echo response comparing Presentation 4 to Presentation 1 across all participants. The maximum difference in echo response was located over occipital electrodes (× indicates areas of significant difference between presentations, multiple t test between all electrodes, Bonferroni corrected). C, Average of all participants' power from 8 to 12 Hz (μV2) for each sequence presentation for all trials for electrode POz. Average alpha power of the EEG time series was not affected by the repetition of a luminance sequence. Average power is measured in microvolts squared. D, Time-frequency analysis of the effect of sequence repetition on perceptual echo power. The plot shows the difference in time-frequency analysis of the perceptual echo (cross-correlation function) between Presentations 4 and 1, averaged across luminance sequences and subjects, for POz. The echo differs significantly between Presentations 4 and 1 within the alpha frequency range from ∼375 to 505 ms (8–12 Hz). The outlined area highlighted with a red arrow indicates significant differences using multiple t tests, p < 0.05, FDR corrected. Average echo amplitude is measured in arbitrary units (a.u.). To effectively represent the variance within subjects, error bars show SE of the difference between the first presentation and all subsequent presentations. **p < 0.01.
One might wonder whether the increase described above could be attributed to a general effect of the repetition of luminance sequences on the spectral power of the occipital EEG. To test this possibility, we computed the EEG amplitude spectra of the 3.125 epochs for every luminance sequence presentation and then averaged the amplitude spectra across each sequence presentation. We then compared the average EEG amplitude spectra for each sequence presentation using polynomial contrasts, which did not reveal a significant linear relationship between Presentations 1 and 4 (F(1,13) = 0.65, p = 0.44). Further post hoc paired t tests between the four presentations confirmed this result (p > 0.28 for all presentations). We next compared the linear trend for the perceptual echo and raw EEG alpha amplitudes. Paired t tests revealed a significantly stronger linear trend for echo amplitude compared with raw EEG alpha amplitude, t(13) = 2.56, p = 0.019 as a result of sequence repetition. To further examine the relationship between the perceptual echo and EEG alpha amplitudes, we performed correlation analyses and computed correlation coefficients using the data from each individual participant. We then tested whether the correlation coefficients were significantly above the chance level (0) at the group level. We found no significant correlation between raw EEG alpha and the perceptual echo amplitude across presentations using both Pearson correlation (average correlation coefficient across subjects = −0.22, t(13) = −1.28, p = 0.22) and Spearman correlation tests (average correlation coefficient = −0.12, t(13) = −0.93, p = 0.37), indicating that there is no significant linear relationship between the amplitude of the perceptual echo and ongoing EEG alpha activity. Together, the increase in echo response with successive sequence presentations observed in Experiment 1 cannot be attributed to a general effect of EEG alpha response to these stimuli (Fig. 5C).
To examine the topography of the change in echo response from the first to the last presentation, we performed t tests on echo amplitude between Presentations 1 and 4 across subjects and electrodes and plotted the t values. The largest difference in echo response was observed over occipital electrodes (Fig. 5B).
To examine the time course of the repetition enhancement of the echo response, we next computed the difference in echo response between Presentations 1 and 4 in time-frequency space using FFTs (Hanning window tapering from −100 to 1000 ms poststimulus onset and a frequency range of 2–80 Hz). This confirmed the repetition enhancement of the echo response, by showing that the echo response for Presentation 4 was significantly larger than for Presentation 1, with this effect concentrated within the period of the cross-correlation between 375 and 505 ms after sequence onset (p values <0.05, FDR corrected; Fig. 5D).
In the post-test questionnaire, all participants responded “no” to all of the questions, ruling out explicit learning of a sequence as driving the increase in echo response with repeated sequence presentations.
Summarizing, the results from Experiment 1 demonstrate that the amplitude of the perceptual echo response increases in a linear fashion with successive repetitions of a specific dynamic luminance sequence, even though participants were not aware of these repetitions. This finding shows that the visual system can encode regularities defining repetitions of a specific luminance sequence. We next set out to determine the robustness and stability of this finding.
Experiment 2
Experiment 2 was designed to investigate the robustness of the increase in echo response with repeated sequence presentations, across time and intervening sensory input. We reasoned that, if the echo response reflects regularity learning, the dissimilar luminance polarities of an inverse sequence should abolish any signature of such learning in the echo response, because the luminance contingencies of the sequence as a whole would be completely different whereas all other temporal and visual characteristics are preserved. We further reasoned that, if the learning process is robust across time and to intervening sensory input, the echo response amplitude should recover or further increase when the original sequence reappeared following the inverse luminance sequence.
To assess whether the main effect of sequence repetition on echo response was also present in Experiment 2, we compared the echo response from Presentation 1 to Presentation 4 (data were pooled from experimental and control conditions) using a polynomial contrast analysis to test the linear dependence between Presentations 1 and 4. Although showing an unexpectedly high average echo response for the third presentation, the results remain supportive of a linear trend from Presentation 1 to Presentation 4 (F(1,13) = 6.35, p = 0.026; Fig. 6A) Similar to Experiment 1 post hoc paired t tests revealed a larger echo response for Presentation 4 compared with Presentation 1 (bootstrapping test with 10,000 resamples, p < 0.01).
Figure 6.
A, Echo response as a function of repetition of luminance sequences. For the first four presentations of a given sequence, we observed a linear increase in echo response with successive presentations, similar to Experiment 1 (data pooled across experimental and control trials). B, Critically, a subsequent inverse sequence (experimental trials) showed a reduction in echo response to a level similar to initial presentation of a luminance sequence, compared with a fifth presentation of a noninverse sequence. C, Re-presenting the original sequence following the inverse sequence (Replay, experimental trials), showed a recovery of echo response indistinguishable from a sixth successive presentation. Echo response is measured in arbitrary units (a.u.). To effectively represent the variance within subjects, error bars show SE of the difference between the first presentation and all subsequent presentations. **p < 0.01. For clarity the colors of each bar plot are consistent with Figure 3.
We next examined whether presentation of an inverse luminance sequence would abolish the increase in echo response seen after four successive presentations. When considering which sequence presentation in the control condition to compare with the inverse sequence, we took into account that other factors such as (for example) neural adaptation and changes of vigilance may occur across a series of successive stimulus presentations as used in the current study. We reasoned that these factors would only depend on the presentation number, but not on stimulus type (e.g., inverted vs noninverted). To match the overall presentation number we therefore compared the echo response elicited by the fifth presentation (control condition) with the inverse sequence (i.e., the fifth presentation in the experimental condition), because in this comparison the number of stimulus presentations is equated, and only the stimulus type (or the immediately preceding stimulus history) differs. We found that the echo response elicited by the inverse sequence was significantly lower than elicited by a fifth (noninverse) presentation (two-tailed paired t test, t(13) = 4.36, p < 0.002, bootstrap, 10,000 resamples; Fig. 6B).
We also compared the echo response elicited by the inverse sequence and Presentation 1 in experimental trials and found no significant differences between the two presentations (two-tailed paired t test, t(13) = 0.02, p = 0.71, bootstrap, 10,000 resamples). We found BF = 0.29 (<0.33), which strongly supports the null (i.e., no difference) over the alternative hypothesis (Dienes, 2011).
Together these results indicate that an inverse luminance sequence, carefully controlled for a range of perceptual properties, was processed in a similar manner as a new luminance sequence, in terms of echo response. This confirms that the visual system is encoding precise sequence information, rather than only general properties of luminance sequences (e.g., time-frequency dynamic, auto-correlation) or its visual characteristics (e.g., luminance range and variance), because these more general features are shared with the inverse luminance sequence.
We next compared the topography of the echo response for the fifth sequence presentations between experimental and control trials. We performed t tests on echo response between these presentations across subjects and electrodes and plotted the resulting t values (Fig. 7A). Similar to Experiment 1, the largest difference in the echo response was found over occipital electrodes, with a maximum over POz.
Figure 7.
A, t Value map of echo response difference between Presentation 5 (control condition) and the inverse luminance sequence (experimental condition) for all subjects, in Experiment 2. A significant difference was centered over occipital electrode POz (outlined, FDR corrected). B, t Value map of echo response difference between Presentation 6 (control condition) and the replay sequence (experimental condition) for all subjects, in Experiment 2. There were no significant differences in echo response between the two presentations. No other regional differences, positive or negative, reached statistical significance.
We then examined whether sequence information encoded across Presentations 1–4 could persist even after the presentation of an inverse luminance sequence. We reasoned that if such information does persist, re-presenting the original sequence after the inverse luminance sequence should restore the echo response to a level comparable to six sequential presentations of a particular sequence. Using the same logic as above, we compared the echo response elicited by Presentation 6 (control condition) with replay (i.e., the sixth presentation in the experimental condition), as the number of stimulus presentations is directly comparable, and only the stimulus type (or the immediately preceding stimulus history) differs. No significant difference was observed (Fig. 6C); indeed, echo amplitudes for these two presentations were almost identical (t = 0.01, p = 0.99). We computed BF to evaluate whether the echo response elicited by the replay sequence was supportive of the null hypothesis (i.e., no difference in echo amplitude between the replay and Presentation 6) or the alternative hypothesis (i.e., echo amplitude is different between replay and Presentation 6). A BF = 0.27 provides strong evidence (<0.33) for accepting the null over the alternative hypothesis, indicating that sequence-specific information about a particular luminance sequence persists, for >9 s, even in the presence of intervening visual input.
We also examined the topography of the difference in echo response between Presentation 6 (control) and replay (experimental) sequences. We performed t tests on echo response between these presentations across participants and electrodes and plotted the resulting t values (Fig. 7B). There were no significant differences at any sensor, indicating that the echo response to the replay luminance sequence displayed a similar topography and magnitude as Presentation 6.
Finally, to further validate the linear trend of echo response for each presentation of a luminance sequence found in both experiments, we generated 2000 surrogate shuffled datasets in which each EEG time series recorded for a specific trial number (matching the trial number in the main analysis) was cross-correlated with a luminance sequence presented in a different (shuffled) trial, but with the same presentation number. We reasoned that if the observed linear trend of the echo response was caused by an EEG-induced effect, and was not specific to the input stimulus sequence, we should observe a similar linear trend in the shuffled datasets to that observed in the real data (Figs. 5, 6).
We performed permutation tests in which the linear trend for the real data were compared with the linear trends of the 2000 shuffled datasets separately for each experiment. For each surrogate cross-correlation dataset, we performed a linear trend test, producing an F value for each of the 2000 shuffled datasets for each experiment. We then computed p values by examining the rank of F values from the linear trend reported in the paper for both experiments to the F-value distribution for the 2000 shuffled datasets. The results of this permutation test, showed p = 0.012 for Experiment 1 and p = 0.014 for Experiment 2, with both distributions of t values being centered around 0, strongly supporting our conclusion that the linear trend in perceptual echo for successive presentations of the same luminance sequence is stimulus-dependent, and cannot be attributed due to broader EEG-induced effects.
Discussion
In two experiments, we investigated the functional relevance of the previously observed alpha-band perceptual echo, testing the hypothesis that it reflects a regularity learning mechanism that can encode dynamic visual sequences within the visual cortex. Supporting this hypothesis, Experiment 1 showed that the amplitude of the perceptual echo (the echo response) is enhanced by repetitions of an identical rapid nonperiodic luminance sequence. Control analyses using shuffled data excluded the possibility that increases in echo response could be attributed to general changes in induced alpha-band EEG responses resulting from sequence repetitions, suggesting instead that these increases reflected a neural signature of regularity learning. Experiment 2 established that the repetition enhancement of perceptual response reflects specific sequence information. Following four presentations of a specific sequence we presented an inverse luminance sequence, which preserved all nonsequential spectral and temporal properties of the original sequence. Strikingly, the echo response for the inverse sequence returned to a level comparable with presentation of a novel sequence (Fig. 6B). When the original stimulus sequence was presented again (following the inverse sequence), the echo response recovered to a level consistent with the number of presentations of this sequence, demonstrating that information about an encoded sequence persisted for >9 s and was robust to intervening visual input.
Previous behavioral studies have demonstrated that humans are able to learn temporal sequences presented at different rates (1.7–8.3 Hz) consisting of spatiotemporal information (Song et al., 2008), combinations of visual features (color and spatial; Gheysen et al., 2011), and object orientations (Luft et al., 2015). Our results extend these findings by showing that the visual system is capable of sequence learning even with stimuli presented orders of magnitude faster (160 Hz) than previously used. This sensitivity to rapidly changing input is in line with known properties of the human auditory system, where auditory sequence learning has been described behaviorally for random and meaningless input signals (Gaussian random noise; Agus et al., 2010). Notably, as in the present study, behavioral signatures of sequence-specific learning were found to persist for seconds and be robust to intervening auditory inputs.
Considering the neural substrates of sequence learning, both human and animal studies have implicated brain regions outside primary sensory areas. For example, in monkeys, information about spatiotemporal sequences is encoded in inferior temporal cortex and also in V4 (Li and DiCarlo, 2008; Meyer and Olson, 2011). However, Gavornik and Bear (2014) found that, in mice, it was possible to detect the encoding of spatiotemporal sequence information early in the visual stream, within primary visual cortex. These findings are compatible with the surface localization of the perceptual echo, and its increase across repeated sequence presentations, to (human) visual regions.
What mechanisms could be responsible for the increase in echo response observed with successive presentations of luminance sequences? One possibility is that this increase may be a consequence of the increasing similarity, across sequence repetitions, between luminance changes in the stimulus sequence, and changes in the evoked EEG response. This would result in stronger cross-correlations between the EEG and the luminance sequence, with successive presentations, and hence an increase in echo amplitude. This could be thought of as “sharpening” of the neuronal assemblies representing visual sequences, reflecting increasingly precise representations of repeated sequences. Supporting this interpretation, previous studies on perceptual learning have shown that repeated exposure to the same visual input sharpens the responsiveness of neuronal assemblies representing these inputs (Seitz and Dinse, 2007). For example, monkeys trained on an orientation discrimination task show a post-training decrease in response variability and an increase in the slope of orientation-tuning curves in V4 for the trained orientations (Yang and Maunsell, 2004). Similarly, the phenomenon of repetition suppression may be caused by a sharpening or tuning mechanism, which occurs when repeated exposure to a stimulus leads to a more precise and more efficient neural representation of that stimulus (Desimone, 1996; Kok et al., 2012). At a mechanistic level, neuronal sharpening is thought to depend on Hebbian processes, among which spike-timing-dependent plasticity (STDP) is particularly significant when dealing with temporal relationships (Bi and Poo, 1998). Relevant to the present study, STDP has been shown to make neurons extremely sensitive to repeating spatiotemporal patterns (Guyonneau et al., 2005; Masquelier et al., 2008, 2009) and displays the highest speed and efficacy when spike timing is reliable (Markram et al., 1997). Therefore, the repeated presentation of a luminance sequence may have increased both the sensitivity and reliability of neural responses facilitating the rapid encoding of sequence information reflected in the changes in perceptual echo we describe.
Could the results of the present study be accounted for by stimulus related changes to ongoing alpha activity or phase modulation of alpha oscillations? It is possible that the amplitude of specific evoked EEG responses to the luminance sequence could increase across successive repetitions. This would lead to an increased signal-to-noise ratio and thus also to stronger cross-correlations between the EEG time series and luminance sequences. Although the opposite phenomenon is commonly observed (repetition suppression), a wide range of studies investigating perceptual learning have indeed found that the neural response to a stimulus can be enhanced by repeated exposure to identical sensory input (Karni and Sagi, 1991; Vogels, 2010). Indeed, the original investigation of the perceptual echo by VanRullen and Macdonald (2012) found that the frequency peak and power of the spontaneous alpha rhythm correlated with the frequency peak and amplitude of the perceptual echo across subjects. This finding suggests that the perceptual echo is intimately linked to occipital alpha rhythms. Additionally, the perceptual echo can only be observed after averaging the cross-correlation time series across trials, which suggests that phase modulation time-locked to each stimulus frame, is a key feature of the perceptual echo. Indeed, we show that there is a strong phase coherence between cross-correlation time series centered within the alpha frequency range (Fig. 2C). Therefore, both the ongoing alpha rhythm and phase modulation of this rhythm are critical in the production of the perceptual echo.
However, our data does not support the conclusion that these factors can account for the modulation of the perceptual echo in response to repeated presentations of rapid luminance sequences. First, the power of the raw EEG alpha did not display the same linear trend as found for the perceptual echo across stimulus presentation, which would be expected if the neural response to sequence repetition were driven by modulations in the amplitude of the alpha-band-evoked EEG signal (Fig. 5C). Second, both the current study and previous research by VanRullen and Macdonald (2012) show that when the cross-correlation is performed on EEG time series and stimulus sequences from different trials, the ∼10 Hz power in cross-correlation is abolished (Fig. 2B). Together these findings strongly suggest that the perceptual echo is not driven by EEG responses to rapid luminance sequences and that it is indeed stimulus-dependent.
In summary, the enhancement of the echo response with successive presentations of a sequence is most plausibly due to an increase in similarity between luminance changes in the stimulus sequence and changes in the evoked EEG response. A candidate mechanism for this process is STDP, through sharpening the population response for each specific sequence, leading to an enhanced echo response with successive presentations of a sequence. The echo response is linked to occipital alpha rhythms a phase-modulation within this frequency range; however, it is stimulus-dependent and independent of general features of EEG.
A striking, indeed defining, feature of the perceptual echo is its periodicity. One interpretation of this feature is that the visual system is actively engaging in repeated patterns of activity in response to ongoing sensory stimulation. In this view, the periodicity of the perceptual echo may reflect a process that updates and replays temporal representations at a rate defined by occipital alpha. This interpretation is in line with “predictive processing” accounts of perception and brain function, in which the brain continuously generates and updates predictions about incoming sensory signals (Friston, 2005; Clark, 2012; Hohwy, 2013; Seth, 2014). Extended formulations of this account suggest that hierarchical predictive generative models update prior knowledge by extracting and encoding hidden spatial and temporal regularities in the environment (Tenenbaum et al., 2011). In our study, increases in echo response with successive repetitions of a sequence demonstrate that participants were (implicitly) learning information about each sequence, which may reflect perceptual predictions (Bayesian priors) being updated with each sequence. In this view the perceptual echo may reflect an iterative process that updates priors about the luminance dynamics of a sequence, communicating perceptual predictions at rate defined by the alpha frequency. Further research will be needed to examine connections between perceptual echo and predictive processing, for example by manipulating expectations about temporal regularities, within or across sequence presentations.
Conclusion
The perceptual echo is a long-lasting periodic reverberation in the EEG response to dynamic visual stimulation, reflected by an alpha-band peak in the cross-correlation function between a rapidly changing random dynamic luminance sequence and the raw occipital EEG response. We investigated the functional properties of the perceptual echo, finding that it can be enhanced by repeatedly presenting the same visual sequence, indicating that the human visual system can rapidly and automatically learn regularities embedded within such fast-changing dynamic stimulus sequences. By comparing echo responses for inverse and noninverse luminance sequences, we further showed that the increase in echo response was sensitive to specific sequence information. Finally, we show that the encoded sequence information can persist over many seconds even in the presence of additional intervening sensory input. Together, our results provide evidence that the perceptual echo reflects the existence of a previously undiscovered regularity learning mechanism, which operates at a rate defined by the alpha frequency band.
Footnotes
This work was supported by a graduate teaching assistantship from the School of Engineering and Informatics at the University of Sussex, a scholarship provided by Sackler Centre for Consciousness Science, and the ministry of education Taiwan to A.Y.-C.C.; the Dr. Mortimer and Theresa Sackler Foundation to A.K.S. and D.J.S.; an ERC Consolidator Grant P-CYCLES 614244 to R.V.; and a PRESTO Grant from Japan Science and Technology Agency to R.K.
The authors declare no competing financial interests.
References
- Agus TR, Thorpe SJ, Pressnitzer D (2010) Rapid formation of robust auditory memories: insights from noise. Neuron 66:610–618. 10.1016/j.neuron.2010.04.014 [DOI] [PubMed] [Google Scholar]
- Bi GQ, Poo MM (1998) Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J Neurosci 18:10464–10472. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Brainard DH. (1997) The psychophysics toolbox. Spat Vis 10:433–436. 10.1163/156856897X00357 [DOI] [PubMed] [Google Scholar]
- Chubykin AA, Roach EB, Bear MF, Shuler MG (2013) A cholinergic mechanism for reward timing within primary visual cortex. Neuron 77:723–735. 10.1016/j.neuron.2012.12.039 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Clark A. (2012) Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behav Brain Sci 36:181–204. [DOI] [PubMed] [Google Scholar]
- Cooke SF, Bear MF (2015) Visual recognition memory: a view from V1. Curr Opin Neurobiol 35:57–65. 10.1016/j.conb.2015.06.008 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Delorme A, Makeig S (2004) EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neruosci Methods 134:9–21. 10.1016/j.jneumeth.2003.10.009 [DOI] [PubMed] [Google Scholar]
- Desimone R. (1996) Neural mechanisms for visual memory and their role in attention. Proc Natl Acad Sci U S A 93:13494–13499. 10.1073/pnas.93.24.13494 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Dienes Z. (2011) Bayesian versus orthodox statistics: which side are you on? Perspect Psychol Sci 6:274–290. 10.1177/1745691611406920 [DOI] [PubMed] [Google Scholar]
- Eagleman SL, Dragoi V (2012) Image sequence reactivation in awake V4 networks. Proc Natl Acad Sci U S A 109:19450–19455. 10.1073/pnas.1212059109 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Euston DR, Tatsuno M, McNaughton BL (2007) Fast-forward playback of recent memory sequences in prefrontal cortex during sleep. Science 318:1147–1150. 10.1126/science.1148979 [DOI] [PubMed] [Google Scholar]
- Friston K. (2005) A theory of cortical responses. Philos Trans R Soc Lond B Biol Sci 360:815–836. 10.1098/rstb.2005.1622 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Gavornik JP, Bear MF (2014) Higher brain functions served by the lowly rodent primary visual cortex. Learn Mem 21:527–533. 10.1101/lm.034355.114 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Gheysen F, Van Opstal F, Roggeman C, Van Waelvelde H, Fias W (2011) The neural basis of implicit perceptual sequence learning. Front Hum Neurosci 5:137. 10.3389/fnhum.2011.00137 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Guyonneau R, VanRullen R, Thorpe SJ (2005) Neurons tune to the earliest spikes through STDP. Neural Comput 17:859–879. 10.1162/0899766053429390 [DOI] [PubMed] [Google Scholar]
- Hohwy J. (2013) The predictive mind. Oxford UP.
- Howell DC. (2012) Statistical methods for psychology, Ed 8 Belmont, CA: Wadsworth Publishing. [Google Scholar]
- JASP Team (2016) JASP (version 0.7.5.5) [Computer Software]. Available at https://jasp-stats.org/.
- Karni A, Sagi D (1991) Where practice makes perfect in texture discrimination: evidence for primary visual cortex plasticity. Proc Natl Acad Sci U S A 88:4966–4970. 10.1073/pnas.88.11.4966 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kok P, Jehee JF, de Lange FP (2012) Less is more: expectation sharpens representations in the primary visual cortex. Neuron 75:265–270. 10.1016/j.neuron.2012.04.034 [DOI] [PubMed] [Google Scholar]
- Lalor EC, Pearlmutter BA, Reilly RB, McDarby G, Foxe JJ (2006) The VESPA: a method for the rapid estimation of a visual evoked potential. Neuroimage 32:1549–1561. 10.1016/j.neuroimage.2006.05.054 [DOI] [PubMed] [Google Scholar]
- Li N, DiCarlo JJ (2008) Unsupervised natural experience rapidly alters invariant object representation in visual cortex. Science 321:1502–1507. 10.1126/science.1160028 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Luft CD, Meeson A, Welchman AE, Kourtzi Z (2015) Decoding the future from past experience: learning shapes predictions in early visual cortex. J Neurophysiol 113:3159–3171. 10.1152/jn.00753.2014 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Markram H, Lübke J, Frotscher M, Sakmann B (1997) Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science 275:213–215. 10.1126/science.275.5297.213 [DOI] [PubMed] [Google Scholar]
- Masquelier T, Guyonneau R, Thorpe SJ (2008) Spike timing dependent plasticity finds the start of repeating patterns in continuous spike trains. PLOS ONE 3:e1377. 10.1371/journal.pone.0001377 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Masquelier T, Guyonneau R, Thorpe SJ (2009) Competitive STDP-based spike pattern learning. Neural Comput 21:1259–1276. 10.1162/neco.2008.06-08-804 [DOI] [PubMed] [Google Scholar]
- Meyer T, Olson CR (2011) Statistical learning of visual transitions in monkey inferotemporal cortex. Proc Natl Acad Sci U S A 108:19401–19406. 10.1073/pnas.1112895108 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Rouder JN, Speckman PL, Sun D, Morey RD, Iverson G (2009) Bayesian t tests for accepting and rejecting the null hypothesis. Psychon Bull Rev 16:225–237. 10.3758/PBR.16.2.225 [DOI] [PubMed] [Google Scholar]
- Seitz AR, Dinse HR (2007) A common framework for perceptual learning. Curr Opin Neurobiol 17:148–153. 10.1016/j.conb.2007.02.004 [DOI] [PubMed] [Google Scholar]
- Seth AK. (2014) A predictive processing theory of sensorimotor contingencies: explaining the puzzle of perceptual presence and its absence in synesthesia. Cogn Neurosci 5:97–118. 10.1080/17588928.2013.877880 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Skaggs WE, McNaughton BL (1996) Replay of neuronal firing sequences in rat hippocampus during sleep following spatial experience. Science 271:1870–1873. 10.1126/science.271.5257.1870 [DOI] [PubMed] [Google Scholar]
- Song S, Howard JH Jr, Howard DV (2008) Perceptual sequence learning in a serial reaction time task. Exp Brain Res 189:145–158. 10.1007/s00221-008-1411-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tenenbaum JB, Kemp C, Griffiths TL, Goodman ND (2011) How to grow a mind: statistics, structure, and abstraction. Science 331:1279–1285. 10.1126/science.1192788 [DOI] [PubMed] [Google Scholar]
- VanRullen R, Macdonald JS (2012) Perceptual echoes at 10 Hz in the human brain. Curr Biol 22:995–999. 10.1016/j.cub.2012.03.050 [DOI] [PubMed] [Google Scholar]
- Vogels R. (2010) Mechanisms of visual perceptual learning in macaque visual cortex. Top Cogn Sci 2:239–250. 10.1111/j.1756-8765.2009.01051.x [DOI] [PubMed] [Google Scholar]
- Xu S, Jiang W, Poo MM, Dan Y (2012) Activity recall in a visual cortical ensemble. Nat Neurosci 15:449–455. 10.1038/nn.3036 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Yang T, Maunsell JH (2004) The effect of perceptual learning on neuronal responses in monkey visual area V4. J Neurosci 24:1617–1626. 10.1523/JNEUROSCI.4442-03.2004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zold CL, Hussain Shuler MG (2015) Theta oscillations in visual cortex emerge with experience to convey expected reward time and experienced reward rate. J Neurosci 35:9603–9614. 10.1523/JNEUROSCI.0296-15.2015 [DOI] [PMC free article] [PubMed] [Google Scholar]