Reactivation of visual-evoked activity in human cortical networks

Abstract

In the absence of sensory input, neuronal networks are far from being silent. Whether spontaneous changes in ongoing activity reflect previous sensory experience or stochastic fluctuations in brain activity is not well understood. Here we demonstrate reactivation of stimulus-evoked activity that is distributed across large areas in the human brain. We performed simultaneous electrocorticography recordings from occipital, parietal, temporal, and frontal areas in awake humans in the presence and absence of sensory stimulation. We found that, in the absence of visual input, repeated exposure to brief natural movies induces robust stimulus-specific reactivation at individual recording sites. The reactivation sites were characterized by greater global connectivity compared with those sites that did not exhibit reactivation. Our results indicate a surprising degree of short-term plasticity across multiple networks in the human brain as a result of repeated exposure to unattended information.

one important issue in neuroscience is whether the encoding of sensory inputs is maintained across neuronal networks after sensory stimulation has disappeared. We addressed this issue by examining whether brain areas in human neocortex are able to spontaneously evoke previous visual-induced patterns of activity in the absence of sensory visual input. A related phenomenon, known as “replay,” has been reported in rat hippocampus, where specific place cells coactive during a running task were more likely to be reactivated during subsequent slow-wave sleep (Foster and Wilson 2006; Gupta et al. 2010; Skaggs and McNaughton 1996; Wilson and McNaughton 1994). Subsequent studies not only have supported the fact that task-coactivated hippocampal neurons are reactivated during posttask slow-wave (O'Neill et al. 2008) and REM (Louie and Wilson 2001) sleep but also have shown that the temporal firing pattern of responses reoccurs in the same order as during the task (Lee and Wilson 2002; Skaggs and McNaughton 1996). Although reactivation was originally reported in the hippocampus as a mechanism of memory consolidation (Buzsaki 1998; Marr 1971; McClelland et al. 1995), it may constitute a fundamental property of neural ensembles in many brain areas. Indeed, electrophysiological studies in cat visual cortex have shown that ongoing activity resembles previously evoked responses to natural movies (Han et al. 2008; Yao et al. 2007). More recently, Tambini and Davachi (2013) used functional MRI to investigate reactivation in the human brain and found that the hippocampus is involved in stimulus coding during rest. However, whether and how reactivation occurs across distributed cortical networks in the awake human brain has rarely been addressed (but see Deuker et al. 2013; Fuentemilla et al. 2010; Jafarpour et al. 2014; Kuhl et al. 2012), and hence it is an issue that continues to be poorly understood.

We examined here the capacity of human brain networks to exhibit reactivation using subdural electrocorticography (ECoG) recording, a technique that allows us to capture real-time signals from multiple brain areas simultaneously. Response reactivation was investigated by using a presentation of movie strips reminiscent of stimuli encountered during natural viewing. We found that repeated, brief stimulation with natural movies causes a significant “memory trace” in a subsequent blank trial and an increased similarity between the stimulus-evoked response and the ongoing response pattern.

MATERIALS AND METHODS

Subjects.

Four adult patients with medically refractory epilepsy underwent ECoG to localize seizures and participated in the study after providing informed consent approved by the University of Texas, Medical School at Houston committee for the protection of human subjects (Table 1).

Table 1.

Subjects

ID No.	Age, yr	Sex	Handed	Dis Hemis
TS018	21	M	Right	R
TA441	28	F	Left	R
TA442	47	M	Left	L
TA460	55	M	Right	R

Dis Hemis, hemisphere identified as containing the seizure-onset zone.

Behavioral task.

The experiment consisted of the following stimulus sequence: 25 Prestimulus trials, 25 interleaved Movie/Blank trials, and 25 Poststimulus trials (presented in this order; Fig. 1A). Stimuli were presented binocularly on a computer screen of uniform gray background (35 cd/m²) at a viewing distance of 57 cm. Stimuli were edited in MATLAB (MathWorks, Natick, MA) and ImageJ (National Institutes of Health, Bethesda, MD) and run in MATLAB with Psychophysics Toolbox (Brainard 1997). Each trial lasted for 5 s [with an intertrial interval (ITI) of 10 s], and subjects were required to maintain fixation in the center of the screen by solving a letter task in which a letter (in the middle of the screen within a 2° × 2° square) was presented every 250 ms (letters could be repeated within the sequence). Seven different letters were presented throughout the entire experiment (a, b, m, x, y, u, v), and the order of the letter presentation was randomized across trials and subjects (the letters were lowercase and printed in black). At the beginning of the session subjects were told that one specific letter would need to be identified (e.g., the letter “a”). Half of the trials contained the cued letter, whereas the other half of the trials did not contain the cued letter. Subjects were required to press a computer button (in each Pre, Post, and Movie/Blank trial) to indicate, in a two-alternative forced-choice task at the end of the trial, whether the cued letter was present in the sequence.

Fig. 1.

Reactivation experiment in the human brain. A: experimental paradigm: each session consisted of 25 prestimulus (Pre) trials, 25 interleaved Movie/Blank trials, and 25 poststimulus (Post) trials. During the Movie trials, a 5-s cartoon was presented on the screen, and during the Pre, Blank, and Post trials, subjects were exposed to a gray blank screen. For all trials, a letter was presented in the middle of the screen and at the end of the trial the subject had to indicate whether the cued letter was present during the trial by pressing a key. B: band-filtered and Hilbert amplitude (HA) ECoG responses in specific frequency bands. Reactivation was examined across frequency bands (θ: 4–8 Hz; α: 8–16; β: 16–32, γ: 32–128 and hγ: 128–200 Hz) by filtering the ECoG signal with zero-phase, band-pass, recursive digital filters having a stopband attenuation of 80 dB. We computed the HA (red line) as the absolute value of the analytical signal obtained after applying the Hilbert transform to filtered ECoG signal (purple line). HA and the correlation distance were subsequently used to identify the reactivation electrodes. The correlation distance between HA of 2 signals is less sensitive to the phase difference between signals than the correlation distance between the signals themselves.

The experiment started with a block of 25 Prestimulus trials consisting of a changing letter in the center of the screen (uniform gray background). The next 50 trials consisted of alternating Movie/Blank trials (25 each) repeating the same movie within each session. This design was expected to build up a propensity to reactivation with subsequent repetitions of a movie clip. In the movie trials the letters were presented in the middle of the screen while the background consisted of a 5-s cartoon movie (occupying the full screen). In the Blank trials subjects were exposed to the letter task on a gray screen, and these trials were identical to the Prestimulus trials. The cartoon movie was presented without the sound since we were interested in visual processing only and not auditory or cross-modal effects. The choice of our stimulus (cartoon movies) was motivated by the fact that cartoons are very colorful and contain rapid animation and this type of stimulus has been found to drive visually responsive neurons most effectively compared with static natural scenes. At the end of the Movie/Blank trials we presented 25 Poststimulus trials identical to the Prestimulus trials (letter task on a gray background). To verify whether our reactivation effects are stimulus specific, in some sessions we presented two different cartoon movies. The two movies were presented in a block design, i.e., Pre¹-Movie¹/Blank¹-Post¹-Pre²-Movie²/Blank²-Post². Behavioral performance in the letter task in each block of 25 trials was at least 85% (we did not observe significant differences in task performance across blocks of trials). This indicates that significant potential differences in the degree of task engagement or overall alertness in these trials were unlikely to influence our neurophysiological recordings.

Patient electrophysiology.

Electrophysiological methods and electrode localization were similar to those described previously (Conner et al. 2011). In brief, subdural circular platinum-iridium electrodes with a top hat design (4.5-mm overall diameter, 3-mm cortical contact, 10-mm interelectrode distance) were implanted and placed solely on the basis of clinical considerations with standard techniques (Tandon 2008). Electrode localization was verified by coregistering a postoperative CT imaging with a preoperative MRI structural image. Lobar and gyral labels were assigned by an expert in human neuroanatomy (N. Tandon). ECoG signals were sampled at 1,000 Hz with Nihon Kohden NeuroFax (Japan) and a recording bandwidth from 0.15 to 300 Hz. Signals were referenced to a common average consisting of all nonictal electrodes over lateral frontal and lateral temporal areas to minimize the effect of the referencing scheme on synchronization measures (Nunez and Srinivasan 2006). Recordings were then imported into MATLAB for postprocessing.

Anatomical imaging.

Anatomical imaging data were acquired with a 3-T whole-body MR scanner (Philips Medical Systems, Bothell, WA) equipped with a 16-channel SENSE head coil prior to surgery. A magnetization-prepared 180° radio-frequency pulses and rapid gradient-echo (MP-RAGE) sequence with 1-mm-thick sagittal slices and an in-plane resolution of 0.938 × 0.938 mm and functional MRI volumes (33 axial slices, 3-mm slice thickness, 2.75 in-plane resolution, 30-ms TE, 2,015-ms TR, 90° flip angle) were collected. For each subject, a three-dimensional reconstruction of the pial surface was generated with FreeSurfer v4.5 (Dale et al. 1999). Subdural electrodes were localized on the surface with CT scans taken after implantation and intraoperative photographs at the time of grid placement and resection (Pieters et al. 2013). For representation in a common coordinate space, the subdural electrodes were displayed on the MNI-N27 surface with a 12-parameter affine transformation.

Epileptic spiking detection.

The criterion for inclusion in the analysis was that each electrode had a minimum of 10 trials in each condition (Pre, Movie, Blank, and Post) with no epileptic spiking activity. For each condition, we computed the mean and standard deviation (std) of the absolute values of the raw ECoG signal (by concatenating all the trials) in order to compute a threshold (TH) = mean + 3 × std. Subsequently, for each trial, we computed the time interval for which the absolute raw signal was >TH. “Spiking trials,” which were defined as those trials for which the absolute raw signal was >TH for at least 250 ms, were removed from the analysis. The entire electrode record was excluded from further data analysis if the number of trials with normal (nonepileptic) activity was <10 (in any stimulus condition).

Spectrogram computation.

Spectrograms of the ECoG signal in the three conditions (Pre, Movie, Blank) were computed with the multitaper power spectral density estimator pmtm from MATLAB. We used moving time windows of 500-ms length, with 100-ms overlap. The ECoG raw signals were resampled by decimation at 500 Hz. After the spectrograms were computed, we limited the spectra to 128 Hz, and by using interpolation we distributed the spectral information in frequency bands having an equal number of frequency points (Fig. 2). To quantify the difference between the spectrograms we used the RMS log spectral distance (SD) computed in frequency bands:

$$\text{SD}\left(P_{xx},P_{yy}\right)=\sqrt{\frac{1}{N}{\displaystyle {\sum }_{i=1}^N{\left\{\ln \left[P_{xx}\left(f_i\right)\right]-\ln \left[P_{yy}\left(f_i\right)\right]\right\}}^2}}$$ (1)

Fig. 2.

Example spectrograms revealing stimulus reactivation. A–C: spectrograms for subject P1, electrode 34. E–G: spectrograms for subject P2, electrode 18. Power spectrum is shown in decibels in all the panels. A and E: spectrograms of the Movie condition. B and F: the difference in decibels between the spectrograms of Movie and Pre conditions. C and G: the difference between the spectrograms of Movie and Blank conditions. D and H: the RMS spectral distance (SD; see materials and methods) between Movie and Pre conditions; SD(Movie,Pre), magenta line in D and H, is larger than the spectral distance between movie and blank, SD(Movie,Blank), red line in D and H, in all frequency bands, for both examples.

where P_xx, P_yy were spectral densities for two time signals x(t) and y(t) and f_i (i = 1,2 . . . N) were the frequencies included in the frequency band. For each spectrum in the spectrogram, corresponding to a certain time, we computed the SDs of the frequency bands with Eq. 1. Then, for all the spectra in the spectrogram, we averaged the SDs of the frequency bands (Fig. 2, D and H).

Digital filtering in specific frequency bands.

To examine reactivation across EEG frequency bands (θ: 4–8 Hz; α: 8–16; β: 16–32, γ: 32–128 Hz; high γ: 128–200 Hz), we filtered the recorded ECoG signal with digital nonrecursive band-pass filters. To increase filtering efficiency, we resampled the ECoG signal (recorded with 1 kHz sampling frequency) at 250 Hz for low-band frequency bands (θ to β) and at 500 Hz for the γ bands. The resampling allowed us to design efficient equiripple band-pass filters with low transition width between the pass and stop bands (1 Hz for low-frequency bands and 2 Hz for the γ bands), low ripple in the band-pass band (1 dB), and high attenuation in the stop bands (40 dB) (Oppenheim and Schafer 1989). All signals were filtered in both the forward and reverse directions. As a result of this bidirectional filtering, we were able to double the stop-band attenuation of the filters to obtain an attenuation of 80 dB and also removed the time delay between the filter input and output. Prior to filtering in the γ bands (32–200 Hz), we passed the ECoG signal through notch filters centered at 60, 120, and 180 Hz to remove the 60-Hz noise and its harmonics.

Hilbert amplitude.

We used the Hilbert transform (HT) to obtain the Hilbert amplitude (HA) of the ECoG signals. The HA was the modulus of the analytic signal z(n) = x(n) + j[HTx(n)], where x(n) could be either the filtered or the raw ECoG signal (Fig. 1B). The reason for using the HA for detecting reactivation was to reduce the potential impact of the difference in the phase of the signals in the two different conditions, which would have drastically influenced the correlation between signals. The filtered ECoG signals are essentially amplitude-modulated signals, for which most of the variability is captured by the modulation profile. It is possible that two amplitude-modulated sine signals with similar modulation profiles are in antiphase (e.g., one is positive while the other one is negative), which would lead to strong negative correlations and, implicitly, to a large correlation distance (see below). Thus, to diminish the effect of large phase differences on the computation of the correlation distance between signals corresponding to different conditions, we used HA, which extracted a positive modulation profile.

Quantitative measures of reactivation.

For each electrode, we computed the correlation distance between the HA of the ECoG signals (either filtered or unfiltered) corresponding to trials in Pre and Movie conditions, D(Pre,Movie), as well as the distance between HA sequences corresponding to trials in Movie and Blank conditions, D(Movie, Blank). The temporal sequences of HA used to compute the correlation distance started at the time when the letter task was presented on the screen and lasted 5 s. The correlation distance D(r,s) between two experimental conditions r and s is defined as

$$D\left(r,s\right)=1-\frac{\left(X_r-{\overline{X}}_r\right){\left(X_s-{\overline{X}}_s\right)}^T}{{\left[\left(X_r-{\overline{X}}_r\right){\left(X_r-{\overline{X}}_r\right)}^T\right]}^{\frac{1}{2}}{\left[\left(X_s-{\overline{X}}_s\right){\left(X_s-{\overline{X}}_s\right)}^T\right]}^{\frac{1}{2}}}$$ (2)

where ${\overline{X}}_r=\frac{1}{N}{\displaystyle {\sum }_{j=1}^N{X}_{rj}}$ and ${\overline{X}}_s=\frac{1}{N}{\displaystyle {\sum }_{j=1}^N{X}_{sj}}$ represent the mean values of the sequences X_r and X_s of HA computed for two different experimental conditions (e.g., r, s: Pre, Movie, Blank, Post) and superscript T denotes matrix transpose. The correlation distance between two sequences is simply defined as 1 minus the Pearson's correlation coefficient of the two sequences. A correlation distance between two sequences, X_r and X_s, is 0 when the sequences are identical, X_r = X_s.

To calculate the correlation distance, D(Pre, Movie), we paired the Pre and Movie trials that follow each other (of identical index, e.g., Pre₁ and Movie₁, Pre₂ and Movie₂, . . . , Pre₂₅ and Movie₂₅). Before pairing the trials, we discarded from each condition the trials that exhibited epileptic spiking. When trials were discarded, we reordered the trials to be unitarily increasing for each condition and then paired the trials Pre and Movie of the same index. For the computation of correlation distance D(Movie, Blank), we paired the interleaved Movie and Blank trials from the experiment. Prior to distance computation, we verified the epileptic spiking for each pair of interleaved trials (Movie and Blank) and discarded the pairs of trials with spiking activity.

After we computed D(Movie, Pre) and D(Movie, Blank) for each valid pair of trials, we performed a left-tail t-test. If the mean across trials of D(Movie, Blank) was significantly lower than the mean across trials of D(Pre, Movie), i.e., E[D(Movie, Blank)] < E[D(Pre, Movie)] with P < 0.05, left-tail t-test (E[.] denotes mean), we considered that the electrode exhibited reactivation. That is, the temporal response in the Blank trials was more similar to the temporal response in the Movie trials than to that in the Pre trials. A left-tail statistical test is the correct choice (Freund 1984) because we tested the statistical significance of the inequality E[D(Movie, Blank)] < E[D(Pre, Movie)] rather than the significance of the nonequality E[D(Pre, Movie)] ≠ E[D(Movie, Blank)], and in that case we would have used a two-tailed statistical test. Another way to compute the statistical significance of reactivation is to shuffle the Blank trials and for each electrode use 250 shuffles to compute the probability P that E[D(Movie, Blank)] < E[D(Pre, Movie)]. If P > 0.95 we will conclude that the electrode exhibited significant reactivation. For each subject and band, this allowed us to compute the percentage of reactivation electrodes by summing the reactivation electrodes in that band and dividing by the total number of implanted electrodes in that subject. However, the results are confirming those obtained by using the statistical left-tail t-test.

We examined whether reactivation can also occur in the backward direction. This was done by reversing the temporal order of the events within the 5-s blank trials and then comparing the mean across trials of D(Movie, Reversed-Blank) and D(Movie, Pre). If E[D(Movie, Reversed-Blank)] < E[D(Movie, Pre)] with P < 0.05, left-tail t-test, we considered that the electrode exhibited backward reactivation (see Fig. 4D).

Fig. 4.

Localization of reactivation effects across cortical regions. A and B: brain images represent the cumulative electrodes for all patients (white dots); the red dots represent electrodes exhibiting statistically significant reactivation in the γ frequency band. C: distribution of reactivation electrodes across cerebral lobes. We found most reactivation electrodes in the γ band and in the frontal (middle frontal gyrus) and temporal (inferior temporal gyrus and middle temporal gyrus) lobes. Table 2 shows the distribution of reactivation electrodes in all frequency bands in each brain area. D: electrode count for Reactivation, ITI, Cross, and Post groups across all frequency bands: θ, α, β, γ, hγ, and the HA of the ECoG signal (S) (see text). E: distribution of stimulus-selective reactivation electrodes across cerebral lobes. Reactivation electrodes were mainly found in the γ band and in the frontal and temporal lobes. F: stimulus-responsive reactivation electrodes for all subjects (S1–S4) represented for each specific frequency band: θ, α, β, γ, hγ, and the HA of the entire ECoG signal (S).

We also examined whether reactivation was modulated by the temporal order of the trials by shuffling the trials in each condition and then computing the reactivation for the shuffled data. The electrodes that exhibited reactivation for all 100 shufflings were considered to be reactivation electrodes in the shuffled condition. We computed for each subject the percentage of these reactivation electrodes with respect to implanted electrodes (Fig. 3E).

Fig. 3.

Reactivation of cortical activity in the human brain. A and B: 2 examples illustrating reactivation of evoked activity after exposure to the movie stimulus. A shows the Hilbert amplitude (HA) demonstrating reactivation in the γ band (subject 1, electrode 34). B shows the HA demonstrating reactivation in the raw (unfiltered) signal (subject 2, electrode 18). For both electrodes, the response in the Movie condition was more similar to the response in the Blank condition than the response in the Pre stimulus condition. A selected electrode exhibits reactivation if D(Movie,Pre) > D(Movie,Blank) with P < 0.05. C: histogram of the difference D(Movie,Pre) − D(Movie,Blank) in the γ band, for reactivation and nonreactivation electrodes. The reactivation electrodes have D(Movie,Pre) significantly greater than D(Movie,Pre), P < 0.05, t-test. D: % of reactivation electrodes for all subjects (S1–S4) represented for each specific frequency band: θ, α, β, γ, hγ, and the HA of the entire ECoG signal (S); % of reactivation electrodes (out of total implanted electrodes) was computed separately for each subject. E: % of electrodes exhibiting reactivation, after the trials were shuffled in each condition (Pre, Movie, and Blank). Trial shuffling abolishes reactivation in each subject.

To determine whether reactivation is stimulus specific, we performed the following comparisons after presenting two movies, Movie¹ and Movie², in block design in the same session: D(Pre¹, Movie¹) vs. D(Movie¹, Blank²) and D(Pre², Movie²) vs. D(Movie², Blank¹). If either E[D(Movie¹, Blank²)] < E[D(Pre¹, Movie¹)] or E[D(Movie², Blank¹)] < E[D(Pre², Movie²)], with P < 0.05, left-tail t-test, we considered the electrode as exhibiting stimulus-independent reactivation. The group of reactivation electrodes that exhibited stimulus-independent reactivation was labeled “Cross.”

We examined whether reactivation occurs in the Post trials by comparing, for each electrode, D(Movie, Post) and D(Movie, Pre). When E[D(Movie, Post)] < E[D(Movie, Pre)] with P < 0.05, left-tail t-test, we considered that the electrode exhibited Post reactivation. The group of reactivation electrodes that exhibited Post reactivation was labeled “Post” in Fig. 4D.

Coherence analysis.

We computed the coherence C_xy between two neural responses x(t) and y(t) as (Bokil et al. 2006)

$$C_{xy}\hspace{0.17em}(f\text{)}\hspace{0.17em}=\hspace{0.17em}\frac{{|G_{xy}\hspace{0.17em}(f)|}^2}{G_{xx}\hspace{0.17em}(f)G_{yy}\hspace{0.17em}(f)}$$

where G_xy(f) is the cross-spectral density between x and y, G_xx(f) and G_yy(f) represent the autospectral density of x and y respectively, and f is the frequency in Hz. The magnitude of the spectral density is denoted as |G|. To compute coherence in a specific frequency band (e.g., γ band) we averaged all the coherence values computed within the band limits (e.g., 32–128 Hz for the γ band). For each patient and for experimental conditions Pre, Movie, and Blank, we computed the coherence between all the electrode pairs. For all the trials in a given experimental condition, we computed the cross- and auto-spectral densities of electrode pairs in moving windows of 1-s length, which overlapped for 100 ms, and then averaged the spectral densities over trials in each condition. Finally, we computed the coherence in each time window, each frequency band, and each condition. Thus for each pair of electrodes, experimental condition, and frequency band, we obtained a coherence temporal sequence representing the change in the band coherence as a function of time (Fig. 5, A and B). To estimate the degree of similarity between the coherence values in different experimental conditions (Pre, Movie, Blank) we used the coherence distance (CD), which was calculated as a correlation distance with Eq. 2 in Quantitative measures of reactivation. We subsequently used the coherence distance to examine the differences between the temporal profile of coherence across experimental conditions for pairs of reactivation and nonreactivation electrodes (Fig. 5C).

Fig. 5.

Coherence analysis. A and B: coherence in γ band for a pair of reactivation electrodes (patient 4, electrodes 137 and 166) and nonreactivation electrodes (patient 4, electrodes 79 and 84), for the Pre, Movie, and Blank conditions. C: coherence distance CD(m, pre) is the distance between the coherence of Movie and Pre conditions, and CD(m, b) is the distance between the coherence of Movie and Blank conditions. The difference CD(m, pre) − CD(m, b), averaged across patients, is significantly higher for the reactivation pairs (R) than for the nonreactivation pairs (NR) in the α, β, and γ bands (P < 0.05).

RESULTS

We investigated whether successive, brief exposure to natural movies leaves a “memory trace” at individual recording sites in the absence of the stimulus (a phenomenon that we call reactivation). We measured brain activity as changes in ECoG signal in response to dynamic visual stimulation in four human patients (Conner et al. 2011) and assessed reactivation at individual recording sites (n = 443 electrodes) across multiple frequency bands. The experiment consisted of 25 Prestimulus trials, 50 interleaved Movie/Blank trials, and 25 Poststimulus trials (Fig. 1A), presented in that order. In each trial (irrespective of whether it was Prestimulus, Movie/Blank, or Poststimulus), subjects were required to maintain fixation in the center of a computer screen while viewing a rapidly alternating string of letters flashed for 250 ms each presented in the center of the screen. The only purpose of the letter task was to maintain the subject's fixation on the center of the screen throughout the trial. Each session began and ended with 25 blank fixation trials (Pre and Post conditions), each lasting 5 s. In each Pre, Post, and Movie/Blank trial subjects were required to attend to the letter task in the center of the screen by indicating, in a two-alternative forced-choice task at the end of the trial, whether the cued letter was present in the sequence. Briefly, after the presentation of the last image frame the screen turned gray, indicating that the subjects were required to respond. Importantly, the next trial would not be initiated unless a response had been recorded. Thereby, we ruled out the possibility that correct responses might not be registered simply because subjects responded too slowly.

In the Movie/Blank trials the subjects were passively exposed to either a short cartoon movie (without audio) or a blank screen, each presented for 5 s (see materials and methods), which were alternated for 25 trials each. The choice of our stimulus (cartoon movies) was motivated by the fact that colorful cartoons that contain rapid animation have been found to drive visually responsive neurons more effectively (Yao et al. 2007) than static natural scenes. During a given session, the same 5-s movie was presented binocularly on a computer screen of uniform gray background (35 cd/m²) at a viewing distance of 57 cm. Importantly, subjects continued to be exposed to the letter task in both the Movie and Blank trials, and thus the only difference across trials was the presence of the movie stimulus.

Reactivation of stimulus-evoked responses.

We hypothesized that after exposure to Movie trials the ECoG response in the Blank condition would be more similar to the response during the Movie trials than the response in the Prestimulus trials. The increased similarity for the Movie and Blank responses would support the memory-trace hypothesis, according to which brief stimulation with natural movies would cause a significant “memory trace” in a subsequent Blank trial and an increased similarity between the stimulus-evoked response and the ongoing response pattern. Based on this hypothesis we divided our recording sites between sites that showed this property (reactivation electrodes) and sites that did not show this property (nonreactivation electrodes).

Figure 2 illustrates the spectrograms of the ECoG signals at two example electrodes (Fig. 2, A and E). The difference between the spectrograms of the Movie and Blank responses (Fig. 2, C and G) indicates that the spectrograms of the Blank trials are more similar to the spectrograms of Movie trials than to the spectrograms of Pre trials (Fig. 2, B and F). Computing the SDs between the spectrograms in each condition, we found that, indeed, the SD between the Movie and Pre spectrograms, SD(Movie, Pre), was larger than the SD between the Movie and Blank spectrograms, SD(Movie, Blank), in all frequency bands (Fig. 2, D and H). We measured reactivation by computing the degree of similarity between the ECoG signals filtered in specific frequency bands (θ: 4–8 Hz, α: 8–16 Hz; β: 16–32 Hz, γ: 32–128 Hz, high γ: 128–200 Hz) in different stimulus/blank conditions (as described below). These filtered signals represented amplitude-modulated sinusoidal signals that were most informative in their modulation profile extracted from the signal envelope (Fig. 1B). Thus, for the entire study, we focused our analysis on the difference between the envelopes of the filtered and raw ECoG signals computed with the HT (namely, HA; Fig. 3, A and B, in different stimulus conditions).

We quantified the degree of similarity between the HA temporal sequences recorded during Pre, Movie, and Blank conditions by using the correlation distance. Briefly, a correlation distance between two signals close to 0 is equivalent to a high degree of similarity between the two signals (cf. Eq. 2, materials and methods). To measure reactivation, we computed (by averaging across trials) the correlation distance between the HA temporal sequences in the Pre and Movie conditions, D(Movie, Pre), and that between Movie and Blank conditions, D(Movie, Blank). If D(Movie, Blank) was significantly lower than D(Movie, Pre), the electrode was considered to exhibit reactivation, i.e., the temporal response in the Blank trials was more similar to the response in the Movie trials than that in the Prestimulus trials (P < 0.05, left tail t-test; see materials and methods). Figure 3, A and B, show examples of two electrodes exhibiting reactivation effects in the γ frequency band (Fig. 3A) and for the raw (unfiltered) ECoG signal (Fig. 3B). Both examples show that whereas the temporal responses in the Pre and Movie conditions are quite different, the response in the Blank condition is more similar to that in the movie condition [Fig. 3A: D(Movie, Pre) = 1.06, D(Movie, Blank) = 0.914; Fig. 3B: D(Movie, Pre) = 0.97, D(Movie, Blank) = 0.825].

We further examined the spatial distribution of the reactivation electrodes. In general, there were slight asymmetries in the distribution and coverage of electrodes across patients (Fig. 4, A and B), which was based solely on clinical assessment, and placement often favored the frontal and temporal regions. However, we found that, irrespective of cortical location, the highest percentage of reactivation electrodes (19.2%, i.e., 85 electrodes of 443 implanted) exhibited significant effects in the γ band (Fig. 3C). Importantly, the larger number of reactivation electrodes in the γ frequency band was an effect present in each subject (Fig. 3D)—of the total number of implanted electrodes, a large percentage exhibited reactivation effects in the γ band (S1 9.4%, S2 32.4%, S3 16.5%, and S4 9.6%). In contrast, reactivation was much weaker in the lower-frequency bands [5.2% reactivation electrodes in θ band, 7.8% in α band, and 6.1% in β band (Fig. 3D)]. This suggests a possible link between cortical reactivation and γ band synchronization reported in previous studies (Cardin et al. 2009; Chalk et al. 2010; Engel et al. 1991a, 1991b; Fries et al. 2001; Gray and Singer 1989; Gregoriou et al. 2009; Taylor et al. 2005; Womelsdorf et al. 2006). To examine the robustness of our measure for calculating reactivation, we shuffled the trials in each pre/stimulus/blank condition and recomputed our correlation distances (Fig. 3E). The percentage of reactivation electrodes in the shuffled condition (by averaging % of electrodes across 100 shuffling epochs) dropped dramatically for each subject and frequency band compared with the unshuffled condition.

Surprisingly, we found that γ band reactivation was not localized to a specific cortical region (Fig. 4, A and B) but was found in many areas (temporal: n = 22, 17.1% of recording sites; frontal: n = 34, 16.3%; parietal: n = 7, 11.1%; occipital: n = 5, 11.6%; Fig. 4C). Across subjects, γ band reactivation was mostly found in the temporal (S1: 5.7% of the total electrodes implanted in the temporal lobe, S2: 23.3%, S3: 29.43%, and S4: 14.7%) and frontal (S1: 0% of the total electrodes implanted in the frontal lobe, S2: 23%, S3: 13.8%, and S4: 16.7%) lobes.

Interestingly, ∼46% of the reactivation electrodes showed gradual trial-by-trial changes in correlation distance, i.e., D(Movie, Blank) computed for the γ band (the frequency band with the largest percentage of reactivation electrodes) had a negative slope across trials. This means that ECoG responses in the Movie and Blank conditions became gradually more similar as trials progressed. We further examined the persistence of cortical reactivation after the stimulus was no longer presented (in the Post condition). For our population of recording sites, we found that only 69 reactivation electrodes exhibited “post” reactivation in at least one frequency band (Fig. 4D), i.e., D(Movie, Post) < D(Movie, Pre), P < 0.05, left-tail t-test. The “post” reactivation was γ band specific, with 23 electrodes out of all 69 exhibiting statistically significant effects (P < 0.05).

We further tested whether reactivation is also present in the ITI between the movie presentation and subsequent blank when subjects were not exposed to the letter task. This was done by examining the resemblance between the responses elicited by the movie stimulus and those recorded in the ITI. We thus compared D(Movie, Pre) and D(Movie, ITI), as the only difference between the Pre and ITI conditions was the presence/absence of the letter task. However, we found only a small fraction of electrodes (7.6% as opposed to 42.8% for reactivation in all frequency bands) for which D(Movie, ITI) < D(Movie, Pre), P < 0.05 (ITI group in Fig. 4D).

Importantly, our results hold when only the electrodes responsive to the visual stimulus were included in the analysis (Fig. 4, E and F; an electrode was considered stimulus responsive when the mean HA in the Movie condition was significantly different from that in the Pre condition, using a Holm-Bonferroni correction for multiple comparisons with P = 0.05). We found that most of the reactivation electrodes were also stimulus-responsive electrodes (93.8% in θ band, 84.6% in α band, 50% in β band, 73.2% in γ band, and 100% in high γ band). As shown in Fig. 4, E and F, the stimulus-responsive reactivation electrodes were primarily identified in the frontal and temporal lobes and in the γ frequency band.

We also tested whether reactivation can occur in the reverse direction (backward reactivation; Foster and Wilson 2006). This was done by temporally reversing the ECoG signal for each electrode recorded in the 5-s blank trials and examining whether E[D(Movie, Reversed-Blank)] < E[D(Movie, Pre)] in each frequency band. However, we found that a much smaller fraction of electrodes exhibited backward reactivation (Fig. 4D), to indicate that awake reactivation primarily occurs in the forward direction. This finding is consistent with our previous study in monkey V4 (Eagleman and Dragoi 2012) reporting strong forward reactivation but only weak backward reactivation of stimulus-evoked activity.

Reactivation is stimulus specific.

To rule out the fact that reactivation could be due to a general change in brain responsiveness, possibly caused by a stimulus-independent increase in arousal or attention, we examined whether the effects described above exhibit stimulus specificity. Thus we exposed subjects (in a block manner) to a new Movie/Blank sequence (Movie²/Blank²) after initial exposure to Movie¹/Blank¹ (see materials and methods). We reasoned that if the reactivation observed during the presentation of the blank stimulus would simply reflect a general change in brain responsiveness independent of the movie stimulus the subjects were exposed to, then the response during Blank² would equally resemble Movie¹ and Movie². Thus we compared D(Pre¹,Movie¹) and D(Movie¹,Blank²), which is the correlation distance between Movie¹ and the response to the blank stimulus following Movie² [a similar analysis was performed for D(Pre²,Movie²) and D(Movie²,Blank¹)]. That is, we swapped Blank¹ and Blank² and repeated our “reactivation” analysis described above. If reactivation is due to a stimulus-independent change in arousal level or drift in the internal state of the patient, swapping the ECoG responses to Blank¹ and Blank² would yield the same degree of reactivation in our population of electrodes compared with the original (unswapped) stimulus conditions. However, we found that only a very small fraction of “reactivation” electrodes (4.9%, in all frequency bands), labeled “Cross” in Fig. 4D, exhibited stimulus-independent reactivation, i.e., D(Movie¹,Blank²) < D(Movie¹,Pre¹) or D(Movie²,Blank¹) < D(Movie²,Pre²), P < 0.05. We further examined this issue by directly comparing the correlation distance (in the γ band) between the Movie trials and the incongruent Blank trials [corresponding to a different movie; for instance, D(Movie¹,Blank¹) vs. D(Movie²,Blank¹)]. However, the correlation distance between congruent Movie-Blank sequences was significantly smaller than the correlation distance between incongruent Movie-Blank sequences (mean D_congruent = 0.9784; SE = 0.00396; mean D_incongruent = 0.9899; SE = 0.0034; P = 0.014, left-tail t-test). Altogether, these analyses indicate that the response in the Blank trials following the movie presentation is specific to the stimulus that the subject had just been exposed to. Presenting a new movie stimulus (Movie²) will generate a new reactivation response in the corresponding blank period (Blank²) that is unique to the movie that had just been presented. These results cannot be attributed to differences in the type or duration of the movies, as both movies were presented for the same duration and both were equally entertaining (we did not find any significant difference in the mean ECoG power in any frequency band between the brain signals associated to Movie¹ and Movie², P > 0.1, t-test).

Table 2 includes the distribution of electrodes across brain areas as well as the reactivation electrodes for all the patients. Also, for each subject we calculated the distribution of electrodes across brain areas, together with the reactivation in frequency bands for those respective brain areas. The reactivation electrodes shown in Fig. 4D were obtained by first applying the reactivation method and then removing from the group of electrodes exhibiting reactivation those electrodes exhibiting ITI and Cross reactivation (see materials and methods).

Table 2.

Distribution of reactivation electrodes in specific brain areas and frequency bands

Lobe	Brain Area	Electrodes (no.)	S	θ	α	β	γ	hγ
Frontal	Inferior frontal gyrus	12	3	0	0	1	2	1
Frontal	IFG-pars orbitalis	1	0	0	0	0	0	0
Frontal	IFG-pars triangularis	7	3	1	0	1	2	2
Frontal	IFG-pars opercularis	4	0	1	0	0	3	1
Frontal	Middle frontal gyrus	51	2	2	4	4	8*	2
Frontal	Superior frontal gyrus	27	1	2	3	0	7*	2
Frontal	Medial superior frontal gyrus	18	2	1	1	0	3*	2
Frontal	Precentral gyrus	26	0	2	2	0	2	0
Frontal	Pars opercularis Broca	1	0	0	0	0	0	0
Frontal	Pars orbitalis Broca	1	1	0	0	0	0	1
Frontal	Pars triangularis Broca	7	1	0	1	0	2*	1
Frontal	Orbitofrontal cortex	47	2	0	2	5	7*	1
Frontal	Gyrus rectus	3	0	0	0	0	0	0
Frontal	Medial precentral gyrus	2	0	1	0	0	0	0
Frontal	Anterior medial prefrontal	1	0	0	0	0	0	0
Temporal	Inferior temporal gyrus	20	1	0	1	0	5*	0
Temporal	Middle temporal gyrus	37	2	0	1	2	9*	3
Temporal	Superior temporal gyrus	16	1	2	1	0	1	1
Temporal	Anterior superior temporal gyrus	1	0	0	0	0	0	0
Temporal	Posterior superior temporal gyrus	5	0	0	0	0	2*	0
Temporal	Cingulate	5	0	0	0	0	0	0
Temporal	Medial occipitotemporal gyrus	1	0	0	0	0	0	0
Temporal	Lateral occipitotemporal gyrus	18	0	0	3	2	1	1
Temporal	Occitotemporal gyrus	6	1	0	0	1	2	1
Temporal	Temporal pole	9	1	0	1	0	2	2
Temporal	Parahippocampal gyrus	8	1	1	0	0	4*	2
Temporal	Fusiform gyrus	3	0	2	0	0	0	0
Parietal	Parietal lobule	6	0	0	0	1	0	0
Parietal	Inferior parietal lobule	14	1	1	3	0	3*	2
Parietal	Superior parietal lobule	12	1	0	1	0	0	0
Parietal	Angular gyrus	5	1	0	0	0	0	0
Parietal	Postcental gyrus	16	1	0	0	1	1	0
Parietal	Supramarginal gyrus	10	0	0	1	0	3	2
Occipital	Occipital lobe	31	1	0	1	4	4*	2
Occipital	Calcarine cortex	1	0	0	0	0	0	0
Occipital	Lateral occipital gyrus	4	0	0	0	0	0	0
Occipital	Lingual gyrus	7	0	0	0	2	1	0

^*We marked the brain areas for which the number of reactivation electrodes in γ band represented >10% of the subdural electrodes in that area (e.g., 5 electrodes, of 20 electrodes implanted in the inferior temporal gyrus, exhibited reactivation, so >10% exhibited reactivation). We considered for marking only the areas with >15 electrodes. Numbers in bold represent reactivation electrodes.

Reactivation sites are more strongly coupled.

Next, we evaluated the changes in functional connectivity (Fell and Axmacher 2011; Watrous et al. 2013) for the pairs of recording sites exhibiting reactivation and for those that did not exhibit reactivation. We reasoned that an increased functional connectivity between reactivation sites might contribute to an efficient propagation of the “memory trace” of the movie stimulus being presented during the exposure phase. As a measure of functional connectivity we used the spectral coherence between two recording sites in various frequency bands, computed in moving windows, for each of experimental conditions Pre, Movie, and Blank (see materials and methods). In other words, our measure of functional connectivity rested on the fluctuations in (frequency specific) coherence between two electrodes, over several seconds. For each frequency band, we determined whether the temporal profile of the coherence between two electrodes in the Blank and Movie conditions was more similar than the coherence in the Pre and Movie conditions (Fig. 5, A and C). This was done by evaluating the difference in the coherence distance, CD(m, pre) − CD(m, b), where CD(m, pre) was the correlation distance between the coherence sequences corresponding to Movie and Pre and CD(m, b) was the distance between the coherence in Movie and Blank conditions (Fig. 5C and materials and methods). The coherence distance is a measure that is uncontaminated by differences in coherence between electrodes, since Eq. 2 (see materials and methods) uses the normalized coherence (we found that coherence is significantly higher for the nonreactivation electrodes).

Overall, we found a significantly higher difference CD(m, pre) − CD(m, b) for the pairs of electrodes that exhibited reactivation than for the pairs not showing reactivation, and this result was robust in most frequency bands (α, β, and γ bands, P < 0.05; 100,000 bootstrap comparisons; Fig. 5C). Spectral coherence is a measure of similarity in the frequency domain, while the correlation distance is a measure of similarity in the time domain. Thus the reactivation electrodes exhibited a remarkable similarity between the responses in Movie and Blank conditions in both the frequency and time domains.

DISCUSSION

Our study demonstrates that exposure to sensory stimulation leaves a “memory trace” at individual recording sites in the absence of the stimulus. Since this memory trace is related to the history of stimulation, we describe this phenomenon as a reactivation of stimulus-evoked cortical responses. Indeed, we found that responses across cortical sites exhibit a stimulus-specific memory trace in the absence of sensory stimulation and that this reactivation is mainly found in the γ band frequency. A somewhat surprising result is that early visual areas do not show pronounced reactivation—larger percentages of recording sites exhibiting reactivation were found in cognitive and association areas (frontal and temporal lobes). This may possibly indicate that early visual areas may undergo adaptation by repeated stimulus presentation, and as a consequence they may show reduced reactivation. Importantly, we found that brain networks exhibiting statistically significant reactivation have a greater global functional connectivity across a wide range of frequencies compared with networks that did not exhibit reactivation.

Our study was performed in epileptic patients. Epilepsy is a disease marked by impaired cognitive and memory performance, often as a result of increased synchronized activity of large numbers of neurons. Do these factors confound our results? There are several reasons why this is unlikely to be the case. First, electrodes showing ictal and interictal discharge were systematically removed from our analysis based on evaluation by our clinical team, and all analyzed trials were inspected for artifacts related to epilepsy (see materials and methods). Second, disease-related low-frequency synchronization is likely to impair brain function and may interfere with reactivation. Thus we would expect more low-frequency activity across all trials in our experiment if ictal discharge alone accounted for our results, which is inconsistent with our findings. Third, it seems unlikely that epileptic activity would manifest itself as task-related (e.g., increased activity in the Movie/Blank trials) differences.

One issue of recent interest is whether the reactivation of previously evoked brain activity can be demonstrated in the awake state, not only during sleep or anesthesia. Our present investigations build on our previous demonstration of awake reactivation in visual cortical networks (area V4) of nonhuman primate (Eagleman and Dragoi 2012). Awake reactivation has also been demonstrated during quiescent periods in hippocampal cells (Carr et al. 2011; Davidson et al. 2009; Foster and Wilson 2006; Gupta et al. 2010; Karlsson and Frank 2009) and shown to be influenced by the animal's current location (Davidson et al. 2009; Diba and Buzsaki 2007; Foster and Wilson 2006; Karlsson and Frank 2009), to occur with elevated precision in novel environments (Diba and Buzsaki 2007; Foster and Wilson 2006), and to represent pathways not previously experienced by the animal (Gupta et al. 2010). Our results are also consistent with other studies (e.g., Tambini and Davachi 2013) reporting reactivation in human hippocampus in a memory task. Furthermore, another study (Destexhe et al. 1999) found “memory trace”-type responses in awake rat visual cortical cells in response to a moving dot stimulus swept across a linear path of adjacent receptive fields after a conditioning period.

Reactivation of natural image-evoked responses has been previously demonstrated in the human brain (Fuentemilla et al. 2010). For instance, Fuentemilla et al. found replay during the maintenance interval in a delayed match-to-sample working memory task that was coordinated by the phase of theta oscillations (4–8 Hz), and the amount of theta coordination was correlated with working memory performance. However, it was unclear whether reactivation could be manifested in the absence of a behavioral task when stimuli are only passively viewed. The issue of whether awake reactivation of cortical responses can be identified in the human brain during passive exposure to natural stimuli and whether the effects are manifested in specialized regions has been unclear.

Our measure of reactivation differs from previous studies. Replay and reactivation were previously quantified by examining the spiking responses of neuronal ensembles (e.g., Eagleman and Dragoi 2012; Karlsson and Frank 2009; Peyrache et al. 2009; Wilson and McNaughton 1994; Yao et al. 2007) or continuous signals, such as ECoG, EEG, or functional MRI signals. For continuous signals, the correlation between signals was used in one study (Tambini and Devachi 2013) and multivariate pattern classification in other two studies (Deuker et al. 2013; Fuentemilla et al. 2010). In our study, we introduce a novel method for quantifying reactivation based on the correlation distance between the HA of the stimulus-evoked and ongoing activity responses filtered in EEG bands. By analyzing reactivation in frequency bands we found that the γ band is associated with considerably stronger reactivation than the raw (unfiltered) ECoG signal (denoted by S in Fig. 3D).

Overall, our results demonstrate awake reactivation primarily in the γ frequency band across large brain areas. Importantly, we found that the cortical sites exhibiting reactivation are more strongly coupled than the sites that did not exhibit reactivation. This suggests the possibility of a functional network involved in reactivation that could mediate the propagation of stimulus-specific reactivation of response patterns across the network. Altogether, our study indicates a surprising degree of short-term plasticity in human cortical networks in the absence of visual stimulation as a result of repeated exposure to unattended, behaviorally irrelevant information (Gutnisky et al. 2009). Indeed, the capacity of individual recording sites to exhibit a memory trace-type response after the movie stimulus was extinguished can be attributed to transient plastic changes in the ongoing activity that shows detectable reactivation of the movie-evoked responses (Eagleman and Dragoi 2012; Yao et al. 2007). Our study supports the idea that reactivation is not exclusively a feature of neuronal responses in the hippocampus (Skaggs and McNaughton 1996; Tambini and Davachi 2013; Wilson and McNaughton 1994) but may constitute a more general dynamic response property across many brain regions. Our results show that even stimuli that are behaviorally and perceptually irrelevant can leave a memory trace. We describe a novel form of rapid cortical reactivation induced in the awake state precisely at the time when a stimulus is expected to occur. These results raise the possibility that the capacity of brain networks to reverberate may explain how the brain is able to learn and store events that occur in time following passive stimulus exposure during sensory experience (Gutnisky et al. 2009; Seitz and Watanabe 2003; Watanabe et al. 2001). Future studies will further explore whether the type of stimulus-specific reactivation demonstrated here can impact on subsequent cognitive performance, possibly on a trial-by-trial basis, in tasks involving the reactivated stimuli.

GRANTS

This work was supported by grants from the National Institutes of Health (NIH) EUREKA Program and the James S. McDonnell Foundation (V. Dragoi), an NIH Vision Training Grant (B. J. Hansen), and an NIH grant and Clinical and Translational Award from the National Center for Research Resources (N. Tandon).

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the author(s).

AUTHOR CONTRIBUTIONS

M.I.C. and B.J.H. performed experiments; M.I.C. and B.J.H. analyzed data; M.I.C. and V.D. drafted manuscript; B.J.H. and V.D. conception and design of research; N.T., C.R.C., J.D.S., and G.P.K. interpreted results of experiments; S.S. prepared figures; V.D. edited and revised manuscript; V.D. approved final version of manuscript.