Replay of cortical spiking sequences during human memory retrieval

Abstract

Episodic memory retrieval is thought to rely on the replay of past experiences, yet it remains unknown how human single-unit activity is temporally organized during episodic memory encoding and retrieval. We found that ripple oscillations in the human cortex reflect underlying bursts of single-unit spiking activity that are organized into memory-specific sequences. Spiking sequences occurred repeatedly during memory formation and were replayed during successful memory retrieval, and this replay was associated with ripples in the medial temporal lobe. Together, these data demonstrate that human episodic memory is encoded by specific sequences of neural activity and that memory recall involves reinstating this temporal order of activity.

Retrieving individual episodic memories in the human brain relies on our ability to internally replay neural patterns of activity that were present when the memory was first experienced (1–6). This suggests a link with a parallel line of work in rodents that demonstrated that individual neurons in the medial temporal lobe (MTL) fire in sequences when animals are navigating spatial environments, and that these sequences are replayed during awake periods of rest and during sleep (7–14). Replay of sequences of spiking activity has been interpreted to reflect memory retrieval and consolidation (13) and even memory-related planning (14), but no direct evidence exists that demonstrates that replay of neural spiking sequences could also underlie episodic memory retrieval in humans. Neuronal sequences replayed in the rodent MTL are associated with fast oscillations termed “ripples” (15–20). Ripples are also relevant for episodic memory retrieval in humans (21–26), raising the possibility that ripples may also be associated with memory-relevant replay of spiking activity in the human brain.

We therefore investigated the relationship between cortical ripples and single-unit spiking activity in six participants (two female; 34.8 ± 4.7 years of age; mean ± SEM). We implanted a microelectrode array (MEA) to collect single-unit and micro-local field potential (micro-LFP) data from the anterior temporal lobe in each participant (27, 28) while also collecting macro-scale intracranial electroencephalography (iEEG) signals from subdural electrodes placed over the lateral temporal cortex and along the MTL (Fig. 1, A and B). Recordings captured from the cortical iEEG contacts placed immediately above the MEA in the middle temporal gyrus (MTG) enabled us to simultaneously examine neural activity from the same brain region across spatial scales (Fig. 1A). We used the iEEG signals to detect ripple oscillations in the MTG and the MTL and any potential coupling between brain regions (25).

Fig. 1. Ripple oscillations reflect burst firing of cortical neurons.

(A) Intraoperative photograph showing the implanted MEA in the MTG (top) and after placement of a macro-iEEG grid over the MEA (bottom). (B) Locations of the MEAs in five participants. Colors represent each participant. One participant with a right hemisphere implant is not shown. (C) Example of a coupled ripple oscillation between the MTL and MTG detected in macro-iEEG (top) with simultaneous cortical ripples present in the micro-LFP (middle; instantaneous spike rate of each channel indicated by color of trace) and burst firing of cortical single units (bottom). All 151 units measured during this session are shown, and 82 of these units spiked during this cortical ripple event. Spikes are shown with width of 5 ms for visualization purposes. (D) Average population spike rate relative to onset of ripples in the macro-iEEG (left) and average single-channel spike rate relative to onset of ripples in the micro-LFP (right). t = 0 indicates start index of detected ripple oscillation. Error bars represent SEM across all participants. (E) A representative single-channel spike raster locked to the maximum trough of each ripple oscillation recorded in that channel (top), with average spike rate (middle) and LFP (bottom) across all ripple events. t = 0 indicates trough of detected ripple oscillation. (F) Phase preference of spikes with respect to ripple phase (80 to 120 Hz). Dotted line indicates uniform distribution that would be expected by chance given the number of histogram bins. Error bars represent SEM across all participants.

Ripples present in the iEEG recordings in the MTG were accompanied by ripples in the underlying micro-LFP signals and a burst of single unit spiking activity (Fig. 1C). Ripples exhibited band-limited power increases within 80 to 120 Hz at both macro-iEEG and micro-LFP spatial scales (fig. S1). Each ripple identified in each microelectrode was accompanied by an increase in single-unit spiking activity on that channel (Fig. 1C and figs. S2 and S3). Cortical spiking was tightly locked to the onset of the detected ripple oscillations at both the macro-iEEG and micro-LFP scales across participants (Fig. 1D and figs. S4 and S5). Within each micro-LFP ripple, spikes captured from the associated microelectrode channel in this cortical region were locked to the ripple trough, which is consistent with the relationship between spiking and ripple activity observed in rodents and humans (Fig. 1, E and F) (18, 29).

Each participant performed a paired-associates verbal memory task (25, 27) that required them to encode and subsequently retrieve new associations between pairs of randomly selected words on each trial (Fig. 2A and supplementary materials). Although we found a strong relation between cortical spiking and detected ripple oscillations, we focused our analyses on the bursts of single-unit spiking activity and their temporal structure during memory formation (9–12). We defined a burst event as the time indices during which the cortical spiking exceeded a population rate-based threshold for at least 25 ms (supplementary materials). Across all participants, burst events had a mean frequency of 1.4 ± 0.2 Hz, and each burst involved 39.9 ± 6.3% of all identified units within that session. Burst events occurred repeatedly throughout the duration of the word pair presentation (Fig. 2B). We reordered the units in each trial according to a template sequence that we derived from the relative timing of spiking activity between pairs of units during each encoding period. We used this template sequence to visualize, but not analyze, the temporal structure of unit activity across multiple burst events during both encoding and retrieval periods from the same trial (supplementary materials). Units within individual burst events appeared to preserve the same sequential order of firing throughout the duration of encoding (Fig. 2C).

Fig. 2. Burst events are organized into trial-specific sequences during successful memory formation.

(A) Memory-encoding portion of the paired-associates verbal memory task, during which participants are instructed to memorize associations between pairs of words. (B) Repeated cortical burst events during memory formation. (C) Same units as in (B) but reorganized according to the extracted average temporal sequence for that trial. Colors represent the average temporal ordering, with cooler and warmer colors representing earlier and later spiking in the template sequence, respectively. Red lines indicate the linear regression through the maximum spike rate times of all units within each burst event. The slope of the regression line indicates the temporal sequence of firing times for the respective burst event. (D) Spike sequences for burst events during two separate correct encoding trials. Units are colored according to the ordering in the first sequence (left) to demonstrate rearrangement of units to form the second sequence (right). Arrows indicate rearrangement for example individual units between the two sequences. Spikes are shown with width of 2 ms for visualization purposes. (E) Sequences in correct trials, but not incorrect trials, demonstrated higher within-trial compared with across-trial similarity (*P < 0.05).

Because we observed repeated sequences of neuronal firing while participants were encoding the word pairs, we quantified the extent to which the sequences of neuronal firing within the burst events were consistent within each individual trial and different between trials. For each burst event, we determined the sequence of spiking activity across units within that particular burst event by ordering each neuron according to when its maximum firing rate occurred in a ±75-ms window around the center index of the burst event (fig. S6). We found several examples of units that formed a sequence in one trial during word pair presentation and rearranged to form a different sequence in another trial (Fig. 2D). To examine how similar any sequence was to any other sequence, we computed the matching index (MI) (12). The MI compares the pairwise temporal relationships between all units that are common to both sequences and takes on a value of 1 for perfect forward replay and −1 for perfect reverse replay (supplementary materials). We computed the average pairwise MI between all sequences within each trial and compared this with the distribution of MI values that arises when comparing all pairwise combinations of sequences across different trials. Across participants, in correct, but not incorrect, encoding trials, sequences were significantly more similar to other sequences within the same trial than to sequences in other trials [n = 6 participants, paired t test; correct within-trial versus across-trial, t(5) = 3.26, P = 0.023; incorrect within-trial versus across-trial, t(5) = 1.47, P = 0.202; correct versus incorrect, t(5) = 3.68, P = 0.014] (Fig. 2E). This difference between correct and incorrect trials was not observed when calculating the similarity of unit identity (in which “identity” is defined as a binary vector on the basis of whether or not a unit fired within a burst event) (supplementary materials and fig. S7). This suggests that a necessary component of successful memory encoding, and later retrieval, involves repeated sequences of cortical spiking activity that are specific to each trial.

If successful memory encoding relies on the temporal order of neuronal firing, then we hypothesized that successful memory retrieval would involve replay of the same trial-specific sequence. In one trial, the sequence of cortical spiking observed in a single burst event during encoding was replayed in a burst event during memory retrieval (Fig. 3A and fig. S8). Across the entire trial, we observed repeated burst events during both the encoding and retrieval portions of the paired associates memory task (Fig. 3B). We calculated the average sequence similarity (MI) of each retrieval burst event to all encoding events. Over the course of memory retrieval, sequences appeared to become more similar to the encoding sequences until the moment when the participant vocalized their response (Fig. 3C and fig. S9). Across participants, we found that this pattern of increasing sequence similarity was recapitulated in correct, but not incorrect, trials (Fig. 3D and fig. S10). Before vocalization, retrieval sequences were significantly more similar to encoding sequences during correct compared with incorrect trials (P < 0.001, permutation test) (Fig. 3D).

Fig. 3. Memory-specific sequence replay occurs during successful memory retrieval.

(A) Example sequence replay event during memory retrieval. MI = 0.42 (P < 0.001) for this encoding-retrieval sequence pair. Colors represent the average temporal ordering during the encoding period, with cooler and warmer colors representing earlier and later in the sequence, respectively. (B) Encoding (left) and retrieval (right) rasters of the corresponding trial in (A) during the paired associates task. Inset text indicates the study pair to be memorized (CROW JEEP), the test probe (JEEP), and the verbalized response (CROW). The white dashed line indicates the time of test probe presentation. Red lines indicate the linear regression through the maximum spike rate times of all units within each burst event. (C) Average sequence similarity of each retrieval burst event to all encoding events for the trial shown in (B). (D) Average similarity of retrieval sequences to encoding sequences across all participants during memory retrieval. Bars indicate SEM across all participants. Correct retrieval demonstrated significantly increased sequence similarity compared with incorrect retrieval (***P < 0.001, permutation test). (E) Sequence replay for true encoding-retrieval comparisons versus replay after shuffling encoding labels for all trials (**P < 0.01). Each colored dot represents a replay value for one participant, and bars indicate SEM across all participants. (F) Sequence replay for correct trials versus shuffled correct trial conditions. True encoding-retrieval pairs were significantly more similar than shuffled correct and shuffled adjacent correct trials (*P < 0.05). (G) Increased sequence replay during correct retrieval (left) was accompanied by decreased population spike rate (middle) and decreased Fano factor (right). Each line indicates a single participant, and bars indicate SEM across all participants (*P < 0.05, **P < 0.01).

We replicated these results within individual participants (fig. S11) and performed an N-way (factorial) analysis of variance (ANOVA) with participants and correctness of response (binary between correct and incorrect) as the independent variables for each trial and the corresponding sequence replay value as the observation. We observed significant effects for correctness of response [F₆ = 4.55, P < 0.001] but not for participants [F₅ = 1.14, P = 0.340]. We also confirmed this difference between correct and incorrect trials by using an alternative metric of sequence similarity and by using nonparametric statistical tests (fig. S12). Sequence similarity was not correlated with the similarity in unit identity between burst events [correlation coefficient (r) = 0.07 ± 0.03; n = 6 participants, t(5) = 1.80, P = 0.132] or with the number of units that were common to both sequences [r = 0.06 ± 0.03; n = 6 participants, t(5) = 2.04, P = 0.096] (fig. S13), and the difference in sequence similarity between correct and incorrect trials did not arise owing to a difference in burst rates or ripple rates between conditions (figs. S14 and S15). We found significant differences in sequence similarity between correct and incorrect trials in both the presence of detected cortical macro-iEEG ripples and even when ripples were not explicitly detected when using our criteria (fig. S16). We did not find that sequences were replayed during the rest period between retrieval trials or during the math distractor period (fig. S17). We also did not find any evidence of significant reverse replay during correct retrieval and found that the duration of the sequences were not significantly different between correct encoding and retrieval [108.0 ± 10.4 ms versus 107.8 ± 10.8 ms; n = 6 participants, t(5) = 0.180, P = 0.865] (fig. S6).

Our data, which demonstrates that trial-specific sequences observed during encoding are replayed during retrieval, suggest that sequence replay should also be specific to each retrieval trial. We compared each sequence during the last second of the retrieval period with sequences from other encoding trials by using a shuffling procedure. We first shuffled all trials by calculating the replay of each correct retrieval trial for all nonmatching encoding trials. The true encoding-retrieval replay value was significantly greater than the values computed by using the shuffled pairs across all participants [n = 6 participants, paired t test, t(5) = 3.49, P = 0.018] (Fig. 3E). We next shuffled the encoding trial labels using only the correct trials and then by swapping only the encoding trial labels from adjacent correct trials. The true unshuffled average sequence similarity between retrieval and encoding was significantly greater than the average in each shuffled condition (one-way ANOVA across all categories: F₂ = 4.70, P = 0.026; post-hoc paired t test P <0.05 for each category pair) (Fig. 3F), demonstrating that the replay of cortical spiking sequences is specific for each retrieved memory. This memory specificity extended even to individual encoding-retrieval sequence pairs (fig. S18). Moreover, both correct encoding and retrieval had a lower population spike rate and lower Fano factor compared with those of incorrect trials (supplementary materials), suggesting that successful retrieval involves replaying precise sequences of sparse neural firing (Fig. 3G and fig. S19).

Burst events observed during retrieval were closely associated with ripple oscillations captured at both the macro-iEEG and micro-LFP scale (Fig. 1C), yet only some of these cortical events were also coupled to ripples in the MTL. Previous evidence has suggested that successful memory retrieval involves a ripple-mediated interaction between the cortex and the MTL (18, 25). Although each cortical ripple may reflect an underlying burst of spiking activity, only cortical burst events that are coupled to MTL ripples may be preferentially relevant for memory retrieval (Fig. 4A). We therefore hypothesized that sequences of spiking activity that occurred in association with MTL ripples during retrieval were more similar to encoding sequences than bursts of spiking that were not coupled with the MTL. In correct retrieval trials, we found several examples in which burst events associated with MTL ripples exhibited higher average sequence similarity to the encoding period than those burst events that occurred in the absence of a MTL ripple (Fig. 4B). We explicitly quantified replay in the cortex triggered to the onset of MTL ripples and found that maximal replay occurred ~100 ms after MTL ripple onset (significant replay compared with jittered ~100 to 175 ms) (Fig. 4C). We therefore used this temporal relationship to designate every cortical burst event as coupled or uncoupled to a MTL ripple (supplementary materials) and examined the replay content of each type of cortical burst event. In correct retrieval trials across all participants, burst events coupled to MTL ripples demonstrated significantly greater replay of the sequences present during encoding compared with uncoupled events [n = 6 participants, t(5) = 2.85, P = 0.036] (Fig. 4D).

Fig. 4. MTL ripples precede cortical sequence replay.

(A) Schematic of proposed mechanism for sequence replay in the cortex. Units are not sequentially organized when cortical bursts occur in isolation (left), whereas sequence replay occurs when MTL ripples occur in coordination with cortical bursts (right). (B) Example of dynamic coupling between MTL and cortical ripples (top), bursting events (middle), and sequence replay during successful memory retrieval (bottom). (C) Average cortical replay values relative to onset of MTL ripples in correct retrieval trials (blue), and the same data when MTL ripple temporal indices were jittered randomly (gray). Error bars represent SEM across all participants (***P < 0.001, permutation test). (D) Sequence replay during retrieval after dividing all sequences into those that were uncoupled and those that were coupled to MTL ripples. Each line indicates a single participant, and bars indicate SEM across all participants. Coupled sequences exhibited higher replay values than those of uncoupled sequences (*P < 0.05).

Together, our data demonstrate that ripple oscillations reflect bursts of spiking activity in the human cortex and that these bursts contain item-specific sequences of single-unit spiking that are established during memory encoding and replayed during memory retrieval. Cortical spike replay is enhanced during retrieval when bursts of spiking in the cortex are coupled with ripples in the MTL. Our data therefore suggest that successful memory encoding and retrieval of individual items involves a specific temporal ordering of cortical spiking activity and provide a link between previous evidence regarding the role of ripple oscillations in human memory (21–26) and evidence regarding the replay of neuronal sequences observed in rodents (7–14). Studies of spike replay in rodents have largely focused on sequences of spiking activity that emerge during spatial navigation and that are replayed during sleep or during awake periods of rest. These data have inspired models of memory that posit that memory formation involves an initial encoding state, in which the temporal order of spiking activity is established through sequential experience, and a subsequent consolidation state, during which these sequences of spiking activity are replayed in a temporally compressed manner in the MTL (17, 30). Our task requires participants to memorize abstract associations between word pairs and to retrieve those associations following only a brief distractor period. Hence, our data provide direct evidence that awake human memory retrieval involves replay of sequences of spiking activity in the cortex. Whether such cortical spike replay plays a similar role in longer-term memory consolidation in the human brain remains an open question. Moreover, whereas temporal compression and reverse replay may be common features of spike replay observed in rodents, their absence here may be related to our task involving neither sequential experience nor reward. Instead, our data suggest that spiking sequences may represent individual concepts in the human brain through a temporally stereotyped activation of a neural ensemble that emerges during the initial experience of an event and that is replayed when its memory is retrieved.