Reinstatement and transformation of memory traces for recognition

Abstract

Episodic memory relies on the formation and retrieval of content-specific memory traces. In addition to their veridical reactivation, previous studies have indicated that traces may undergo substantial transformations. However, the exact time course and regional distribution of reinstatement and transformation during recognition memory have remained unclear. We applied representational similarity analysis to human intracranial electroencephalography to track the spatiotemporal dynamics underlying the reinstatement and transformation of memory traces. Specifically, we examined how reinstatement and transformation of item-specific representations across occipital, ventral visual, and lateral parietal cortices contribute to successful memory formation and recognition. Our findings suggest that reinstatement in temporal cortex and transformation in parietal cortex coexist and provide complementary strategies for recognition. Further, we find that generalization and differentiation of neural representations contribute to memory and probe memory-specific correspondence with deep neural network (DNN) model features. Our results suggest that memory formation is particularly supported by generalized and mnemonic representational formats beyond the visual features of a DNN.

Direct brain recordings in humans show reinstatement and transformation of item-specific memory traces for recognition.

INTRODUCTION

Episodic memories are the sediments of previous experiences—they allow us to recall specific details of our lives and to recognize familiar events or items. This crucial cognitive function depends on the formation of memory traces or engrams (1, 2) that can be tracked via the coordinated activity of single cells and neural populations (3, 4). Although the formation of engrams relies critically on the hippocampus (5, 6), retrieving their full information content requires interactions with distributed neocortical networks that represent the various facets of an episode (7). In contrast to rodents, in whom engrams can be directly measured using optogenetics (8) and cellular imaging methods (9), cognitive neuroscience studies in humans have applied multivariate analysis methods such as representational similarity analysis (RSA) (10) to track item-specific activity patterns (11, 12). These methods allow for a comparison of the neural features, for example, time-frequency features in electrophysiological recordings, which represent item-specific contents during encoding with those during retrieval (encoding-retrieval similarity, ERS) (13, 14).

The cortical reinstatement hypothesis (15, 16) states that memory retrieval requires the reinstatement of neural activity patterns that were present during encoding—i.e., that levels of ERS are higher for remembered than for forgotten items. Previous studies demonstrated that retrieval depends on reinstatement of encoding-related activity patterns in medial temporal lobe (MTL) (11, 17) and neocortex (13, 18–20) that are specific to individual items. Whereas neuroimaging studies using functional magnetic resonance imaging (fMRI) combine relatively high spatial resolution with full-brain coverage, human intracranial electroencephalography (iEEG) recordings have provided deeper insights into the temporal dynamics of the neural patterns that constitute reinstatement such as the contribution of neural oscillations at specific frequencies (21–23). These studies showed that increased power of gamma-band activity (>30 Hz) in the first second of encoding correlates with higher levels of reinstatement in hippocampus (11) and in regions of the lateral temporal cortex (19). Reinstatement has been observed during both recall and recognition memory (24) and is related to both recollection and familiarity (16).

However, in addition to results showing reinstatement, memory traces have also been found to undergo considerable representational transformations between encoding and retrieval (25–27). More specifically, it has been shown that regions of lateral parietal cortex (LPC) represent event-specific information during retrieval but not during encoding (28). This suggests that LPC supports retrieval not through reinstatement of encoding-related activity patterns but through activation of transformed mnemonic representations (29), i.e., reflecting a dissociation of neuronal networks involved in sensory and mnemonic functions. The LPC has both direct and indirect connections to temporal lobe structures (30) and may thus be recruited during attempts to retrieve item-specific representations from memory (31). Thus, complement to reinstatement of sensory representations, brain regions such as the LPC are associated with the retrieval of mnemonic representations that result from transformations of encoded information.

ERS quantifies the similarity of item-specific neurophysiological features between encoding and retrieval and, therefore, does not inform about how a given item is represented relative to other items. Hence, observing transformation at the level of ERS is insufficient to indicate changes in the underlying representational geometry (32). We nevertheless hypothesized that successful memory may also depend on measures of between-item similarity during encoding (encoding-encoding similarity, EES) or retrieval (retrieval-retrieval similarity, RRS), reflecting representational distances. While ERS and EES/RRS do not directly relate to each other because of different underlying cross-correlation matrices, they may still both be associated with successful memory (33, 34). Specifically, the magnitude of similarity across items during encoding (EES) may relate to the magnitude of ERS in regions showing reinstatement, i.e., where items during encoding are represented in a way that benefits subsequent recognition. Accordingly, the magnitude of similarity between items during recognition (RRS) may relate to the magnitude of ERS in regions showing transformation, i.e., where item-specific representations need to acquire a format which differs from the one during encoding, to be recognized as old.

More specifically, the translation from sensory information to the formation of durable memory traces presumably affects the information that is available during retrieval, which may relate to semantization (35), i.e., the preferential retention of conceptual information at the expense of perceptual details. This process appears to start already during brief offline periods following encoding (36). Here, we conceptualize the different features that are available in a memory trace as different representational formats (37). For visual stimuli, these representational formats reflect the sensory processing hierarchy along the ventral visual stream (VVS) (38, 39). Along the posterior-to-anterior extent of the VVS, neural assemblies represent increasingly complex information: Whereas posterior VVS regions represent low-level perceptual features such as edges or colors, anterior regions represent more complex and categorical features including faces (40) and places (41). Previous studies showed that the VVS can be broadly divided into early (occipital, Occ), mid-level (posterior ventral temporal cortex, pTC), and high-level (anterior ventral temporal cortex, aTC) regions of interest (ROIs), which carry distinct representational formats of increasing complexity (42). Analytically, these distinctions can be captured via deep neural networks (DNNs): DNNs trained to classify visual stimuli have become powerful tools to study the neural representations underlying visual perception (38, 43, 44) and have more recently been applied to assess the representational formats of mnemonic representations (36, 37, 45). Thus, the potential to investigate content-based representational changes following mnemonic transformation using DNNs and the high spatiotemporal resolution of iEEG are a promising combination to study how correspondences between neural and DNN representational formats along the VVS unfold during the first few hundred milliseconds after stimulus presentation (39, 46).

Although reinstatement and transformation were primarily investigated in free or cued recall paradigms (22), they may also be relevant during recognition (47). While both types of memory tests involve the retrieval of mnemonic information, they differ in the types of cues that initiate the reactivation of previous experience. Whereas retrieval during free or cued recall is initiated via internal, partial, or associative cues, recognition involves the comparison of sensory inputs (identical to those during encoding) to mnemonic representations built during encoding, which allow recognizing that these sensory inputs had already been presented before. Hence, the concepts of representational reinstatement and transformation differ in that during recognition, reinstatement does not concern the active reproduction of individual sensory stimuli from memory but of information about the prior presentation of these stimuli. This information does not necessarily rely on item-specific sensory features but may rather reflect mnemonic information shared across multiple items. Vice versa, transformation during free or cued recall likely reflects deviations of mnemonic representations (during recall) from sensory-driven representations (during encoding); transformation during recognition presumably involves the change in processing of a previously seen item (during recognition) from the processing during the first encounter with that item (during encoding). During recognition, sensory input is identical between trials of successful and unsuccessful recognition, allowing the attribution of memory-specific differences to mnemonic effects. Both reinstatement and transformation may contribute to successful recognition, with differential functional relevance of representational formats of memory traces.

The functional compartmentalization of visual representational formats across the VVS exists already in children but continues to be refined through adolescence (48). Developmental changes in the representational format of a memory trace may affect the strategic usage of encoded information: Whereas encoding of event-specific perceptual formats may enhance the differentiation of memory traces and allow for a separation from memories of similar episodes (49), predominant retention of conceptual formats facilitates inferences to novel experiences with shared commonalities (50, 51). These two memory functions, i.e., to represent either specific or generalized information, have been ascribed to complementary processes and brain regions (52, 53) that undergo differential maturational trajectories (54–56), leading to an infantile bias toward generalization (57). Capitalizing on iEEG recordings, previous studies showed that while memory formation in children and adolescents depends on the same brain regions as in adults, there are developmental differences in the magnitude (56, 58–61) and intricate temporal dynamics of memory processing (62–64) that mediate age-related gains during adolescence (65). Hence, we would not expect that the levels of performance, although generally lower as compared to adults, are due to the recruitment of qualitatively different neural processes. More likely, the age-related variability in performance may be partially explained by the interplay and functional relevance of the involved mechanisms. Thus, the study of recognition memory during development is well suited to identify possible age- or performance-related differences in brain-behavior relationships.

How the reinstatement and transformation of representations supports recognition memory, whether they relate to differentiated versus generalized representations and on which representational formats they rely is not well understood. In addition, it is unclear whether these effects vary across development. Therefore, we sought to identify brain regions exhibiting reinstatement or transformation that are associated with successful recognition memory. Further, we aimed to relate these processes to generalized versus differentiated representations during encoding and retrieval. Last, using a convolutional DNN (cDNN) trained on a similar task as the participants performed during encoding (i.e., classify scenes into semantic categories), we aimed to characterize the relevance of specific visual representational formats in memory representations for reinstatement or transformation. To address these questions, we analyzed direct brain recordings in a sample of children, adolescents, and young adults (age range 6 to 32 years) with pharmaco-resistant epilepsy (N = 46) engaging in a simple scene encoding and recognition paradigm. Electrode coverage across the brain spanned large proportions of neocortex including early, middle, and late regions of the VVS and areas in the LPC putatively related to retrieval. Using RSA, we computed item-specific similarity estimates across electrophysiological features between pairs of items within (EES and RRS) and across (ERS) experimental phases and related these measures to memory success. ERS values were always calculated as the difference in similarity of same items during encoding and retrieval (ERS_Same) and different items during encoding and retrieval (ERS_Diff): ERS_Item = ERS_Same − ERS_Diff. These ERS_Item values were calculated separately for remembered and forgotten scenes and then subtracted: ERS_Item(remembered) − ERS_Item(forgotten). Higher values of ERS_Item(remembered) than ERS_Item(forgotten) values reflected a memory benefit of reinstatement, while higher values of ERS_Item(forgotten) than ERS_Item(remembered) values reflected a memory benefit of transformation.

Because of this operationalization, reinstatement and transformation may either rely on memory differences of ERS_Same and/or of ERS_Diff values, which were therefore compared separately in post hoc analyses: ERS_Same(remembered) values were compared to ERS_Same(forgotten) values, and ERS_Diff(remembered) values were compared to ERS_Diff(forgotten) values. Since ERS_Diff correlations reflect item-unspecific correlations between encoding and retrieval, we hypothesized that item-specific reinstatement and transformation effects are primarily reflected in differences of ERS_Same values of remembered versus forgotten items. Specifically, we hypothesized that reinstatement of item-specific representations would involve positive ERS_Same correlations of remembered trials, which should be absent for forgotten trials, and one may expect these memory effects not to occur for ERS_Diff correlations. In turn, a transformation of item-specific representations would lead to near-zero ERS_Same correlations of remembered trials but positive correlations of forgotten trials. Equivalently, we operationalized the generalization and differentiation of memory representations within experimental phases (encoding, EES; recognition, RRS) as the relative difference in pairwise similarity between (different) remembered and forgotten trials in each phase.

We hypothesized that regions of the VVS implicated in sensory processing of visual stimuli would engage in reinstatement, whereas parietal regions implicated in mnemonic functions would exhibit transformation. Therefore, we tested whether item-specific similarity between encoding and retrieval (ERS_Item) differed between remembered and forgotten items. We used time-frequency patterns across electrodes within predefined ROIs as features for the computation of representational similarities and investigated conditional differences in these features. Specifically, we expected effects of reinstatement and transformation to be located in ventral visual and parietal regions, respectively (13, 29). Similarly, we conducted planned analyses on memory-specific generalization and differentiation of neural representations, as well as statistical tests for the matching of neural and cDNN model predictions, without concrete expectations concerning the direction of memory-related effects. Further, given the subtle differences across childhood development described in earlier studies (56, 61–64, 66), we did not have strong expectations concerning age- or performance-dependent differences in the recruitment of reinstatement and transformation processes. We considered both the presence and absence of differential contributions across development informative about the neural mechanisms underlying memory in children and adolescents.

We found that both reinstatement in aTC and transformation in LPC contribute to successful recognition memory. Further, we present evidence suggesting that reinstatement concerns mnemonic rather than purely visual representational formats and is related to generalized representational patterns during encoding. Our results provide evidence for the contribution of reinstatement and transformation of distinct representational formats to recognition memory in a sample of children, adolescents, and young adults.

RESULTS

Behavioral performance

During encoding, participants were instructed to memorize visual scenes while completing an indoor/outdoor classification task (Fig. 1A). Scene classification performance was overall high [accuracy, 93.67 ± 8.07% (mean ± SD)], with response times of 1590 ± 533 ms. All subsequent analyses were restricted to trials with correct classification responses. During retrieval, participants classified old and new (foil) scenes as old/new. We used corrected recognition scores Pr (hits-false alarms) as a measure of recognition performance. Two participants with negative Pr scores were excluded from further analyses. In the remaining sample (N = 46), Pr was reliably above zero (0.41 ± 0.19; Fig. 1B). Corrected recognition scores were larger for indoor (Pr = 0.462 ± 0.207) as compared to outdoor scenes (Pr = 0.358 ± 0.224; T₄₅ = 3.742, P < 0.001). Response times during recognition were lower for hits (2116 ± 759 ms) as compared to misses (2552 ± 1.003 ms; T₄₅ = −5.108, P < 0.001). Recognition memory performance was numerically higher in older participants (Fig. 1C). This effect did not reach statistical significance (ρ = 0.234, P = 0.117); however, we additionally compared levels of performance in this scene memory task with data from a larger nonclinical population of healthy individuals reported earlier (56, 61, 66) who engaged in an fMRI version of the paradigm. We observed comparable trajectories across age despite substantial variability during adolescence (fig. S1) [see also (58–61, 64)]. This suggests that our data reflect typical memory development, allowing us to identify differences in the neuronal signatures of memory functions during childhood and adolescence.

Fig. 1. Task, sample, behavioral performance, and electrode coverage.

(A) Behavioral paradigm. Participants completed a visual scene memory recognition task with distinct encoding and retrieval blocks. (B) Sample distribution of age (15.61 ± 5.92 years; left) and recognition memory performance Pr (Hit – false alarm; 0.41 ± 0.19; right). (C) Corrected recognition memory performance (Hits – false alarms) across development. (D) Regions of interest (ROIs) based on Brodmann areas (BAs): Occ (BA17-19); pTC (BA37); aTC (BA20); LPC (BA39 + 40). (E) Heatmap of contributing electrodes in these ROIs. Coordinate axes labels: A, anterior; S, superior; L, lateral; P, posterior; V, ventral; ECoG, electrocorticography.

Reinstatement of item-specific representations in anterior temporal cortex supports memory

Following the reinstatement hypothesis of episodic memory, we computed ERS across regions in Occ, ventral temporal and parietal ROIs (Fig. 1, D and E) and tested for a larger magnitude of item-level ERS (ERS_Item) between remembered versus forgotten scenes (Fig. 2A). We extracted the distribution of power values across a broad frequency range (1 to 150 Hz) in individual time windows (300-ms width and 50-ms step) and correlated these item-specific features across electrodes of each ROI both within and between encoding and retrieval (Fig. 2, B to D). Using the same time-frequency features in each ROI and experimental phase, we also tested for conditional differences associated with successful memory and their interaction with age and performance, which may influence trial-level ERS (for details, see the Supplementary Materials; fig. S2). In the aTC (Fig. 3A), we found higher ERS_Item for remembered versus forgotten scenes (Fig. 3B; T_sum = 263.51, N_bin = 110, P_cluster = 0.018). This reinstatement cluster extended from 500 to 1000 ms during encoding and from 250 to 1150 ms during recognition. Follow-up analyses on the distinct contributions of ERS_Same and ERS_Diff to ERS_Item values showed that memory effects were selectively found for ERS_Same (T₃₃ = 2.508, P = 0.017) but not ERS_Diff (T₃₃ = −0.412, P = 0.682; Fig. 3C). Further, ERS_Same correlations were significantly larger than zero for remembered items (T₃₃ = 3.687, P < 0.001) but not for forgotten items (T₃₃ = −0.454, P = 0.652), indicating that reinstatement was indeed specific to same-item similarities during encoding and retrieval and not due to similarities between different items. ERS_Diff correlations were significantly larger than zero for both remembered (T₃₃ = 5.101, P < 0.001) and forgotten (T₃₃ = 3.463, P = 0.002) items.

Fig. 2. Representational similarity analyses.

(A) RSA. We computed pairwise similarities between different scenes within the same experimental phase (EES; RRS), and between the same and different scenes across experimental phases (ERS). (B) Time-frequency features used for RSA. We extracted item-specific distributions of power values across frequencies and electrodes within a given ROI in various time windows. (C) Schematic ERS matrix. Similarities during encoding and retrieval of the same scenes (ERS_Same) are depicted on the diagonal (blue) of this matrix, while similarities between different scenes (ERS_Diff) are depicted on the off-diagonal (orange). Item-specific ERS scores were computed by subtracting the average off-diagonal ERS score per row from its on-diagonal value: ERS_Item = ERS_Same – avg. (ERS_Diff). We compared ERS_Item values between remembered and forgotten trials, testing for a memory benefit of either reinstatement (remembered > forgotten) or transformation (forgotten > remembered). (D) Schematic pairwise similarity matrix during encoding (EES) or retrieval (RRS) reflecting pairwise representational similarities between different items. During both encoding and retrieval, we compared between-item similarities of remembered versus forgotten scenes. We compared the pairwise similarities between remembered and forgotten scenes, testing for memory benefits of generalization (remembered > forgotten) or differentiation (forgotten > remembered).

Fig. 3. Reinstatement of item-specific representations in anterior temporal cortex.

(A) Contributing electrodes in the aTC region of interest (ROI) (34 participants, 107 channels). (B) Left, higher ERS_Item scores for remembered versus forgotten scenes. Color reflects T values across subjects for each time point. Black contour indicates cluster of time points during which ERS_Item values were significantly larger for remembered versus forgotten scenes after cluster-based correction for multiple comparisons across time points. Right, illustrative plot of ERS_Item similarities (Spearman’s ρ) per subject for remembered (amaranth) versus forgotten (blue) scenes, averaged across time points of significant cluster. (C) ERS_Same (blue, diagonal) and ERS_Diff (orange, off-diagonal) values of remembered and forgotten items showing a memory effect of ERS_Same but not ERS_Diff scores. (D) No relationship between reinstatement effects on memory with either age (left) or performance (right). Circled * indicates P < 0.05 for one-sample T test. Line * indicates P < 0.05 for paired-sample T test.

Next, we tested whether reinstatement varied by age and whether it depended on the participants’ performance (Fig. 3D). We found no relationship between memory-specific ERS effects and age (ρ = 0.037, P = 0.832) or performance (ρ = 0.042, P = 0.810), indicating similar magnitudes of reinstatement effects on memory across development. To address influences of shared variance between our developmental measures, we also conducted partial correlation analyses between memory-specific ERS effects by controlling for mutual effects between age and performance. We found that neither age (ρ = 0.024, P = 0.891) nor performance (ρ = 0.032, P = 0.858) correlated with reinstatement. No significant memory effects of reinstatement were found in earlier VVS regions (i.e., Occ or pTC; all P values >0.187; fig. S3), suggesting that reinstatement of rather conceptual but not purely sensory representations is associated with recognition memory success.

Transformation of neural representations in LPC

We also investigated reinstatement or transformation effects in lateral parietal cortex (LPC) (Fig. 4A). In notable contrast to our results in aTC, we found that ERS_Item was reduced for remembered as compared to forgotten items in LPC (T_sum = −685, N_bin = 257, P_cluster = 0.003; Fig. 4B), suggesting transformation. This effect occurred from 0 to 1500 ms during encoding and from 0 to 900 ms during recognition. Notably, early onsets are most likely due to data smoothing, where the first time bin contains averaged data from a sliding window from 0 to 300 ms after stimulus onset. Again, we investigated the influence of same-item and different-item correlations and found memory differences in ERS_Same (T₃₇ = −2.488, P = 0.017) but not ERS_Diff scores (T₃₇ = 1.127, P = 0.266; Fig. 4C). Furthermore, ERS_Same scores were significantly larger than zero for forgotten (T₃₇ = 2.566, P = 0.015) but not for remembered items (T₃₇ = 1.252, P = 0.218). ERS_Diff correlations were significantly larger than zero for remembered items (T₃₇ = 3.499, P = 0.001) but not forgotten (T₃₇ = 1.900, P = 0.065) items.

Fig. 4. Transformation of item-specific representations in lateral parietal cortex (LPC).

(A) Contributing electrodes in LPC region of interest (ROI) (38 participants, 466 channels). D, dorsal. (B) Left, lower ERS_Item scores for remembered versus forgotten scenes. Color reflects T values across subjects for each time point. Black contour indicates cluster of time points during which ERS_Item values were significantly lower for remembered versus forgotten scenes after cluster-based correction for multiple comparisons across time points. Right, illustrative plot of ERS_Item similarities (Spearman’s ρ) per subject for remembered (red) versus forgotten (blue) scenes, averaged across time points of significant cluster. (C) ERS_Same (blue, diagonal) and ERS_Diff (orange, off-diagonal) values of remembered and forgotten items showing a memory effect of ERS_Same but not ERS_Diff scores. (D) No significant relationship between transformation benefit for memory and participants’ age. (E) Left, significant relationship between transformation benefit for memory and participants’ performance. Colors indicate grouping following median split. Right, transformation effects in low- and high-performing participants (median split). (F) Relationship between ERS_Item magnitude in aTC and LPC: The magnitude of ERS_Item values in aTC is associated with ERS_Item magnitude in LPC. Circled * indicates P < 0.05 for one-sample T test. Line * indicates P < 0.05 for paired-sample T test. * indicates uncorrected P < 0.05 for main effect of LPC ERS_Item predicting aTC ERS_Item.

We next tested whether subject-specific levels of transformation depended on age (Fig. 4D) or performance (Fig. 4E). While the relationship with age did not reach significance (ρ = 0.237, P = 0.151), we found a significant association between transformation and performance (ρ = 0.372, P = 0.022), indicating more pronounced transformation in low-performing participants. Partial correlation analyses confirmed our initial finding that when controlling for performance, the correlation of age and transformation remained nonsignificant (ρ = 0.144, P = 0.396), while the correlation of performance and transformation was significant when controlling for age (ρ = 0.325, P = 0.049). A similar result was obtained when we applied a median split of corrected recognition performance rates Pr (MD = 0.403; transformation effect in low-performing participants: T₁₈ = −6.308, P < 0.001; in high-performing participants: T₁₈ = −0.688, P = 0.499; Fig. 4E and fig. S4).

Relationships between aTC reinstatement and LPC transformation

Given the observed dichotomy of aTC reinstatement and LPC transformation effects, we asked whether the two effects were related on a trial-by-trial level rather than both being associated with recognition memory success (Fig. 4F). We focused our analysis on the subgroup of N = 28 participants with electrodes in both ROIs (N = 90 aTC electrodes; N = 362 LPC electrodes). We extracted ERS_Item values averaged across time windows in which the ERS effects in aTC and LPC overlapped (see Materials and Methods) and compared these values across trials using mixed linear models with trial-wise aTC ERS_Item as criterion; LPC ERS_Item, memory success, stimulus category, and the interaction of LPC ERS_Item, and memory as predictors; and participants as random slopes. We found a significant main effect of memory (F₁₃₅₄ = 5. 569, P = 0.018), no main effect of LPC ERS_Item (F₁₃₅₄ = 0.042, P = 0.837), but a significant interaction of LPC ERS_Item with memory (F₁₃₅₄ = 5.658, P = 0.017). Follow-up analyses revealed a significant effect of LPC ERS_Item on aTC ERS_Item for remembered (F₈₇₂ = 14.795, P < 0.001) but not forgotten items (F₄₈₂ = 0.044, P = 0.833). This relationship indicates that the magnitude of ERS_Item values in LPC is associated with lower values of ERS_Item in aTC, suggesting that the two processes are related across individual trials.

Generalization and differentiation of memory representations during encoding and retrieval

Next, we sought to test the relevance of generalized versus differentiated representations for memory. First, we estimated the similarity of scene representations to all other scenes within the same experimental phase, i.e., the between-scene similarity during encoding and retrieval (EES and RRS; Fig. 2D) and tested whether similarities between remembered scenes differed from similarities between forgotten scenes, indicating differences in representational similarity. We found memory effects of EES in all VVS ROIs (Fig. 5), indicating more similar representations of subsequently remembered compared to forgotten scenes (Occ: T_sum = 63.05, N_bin = 22, P_cluster < 0.001, onset 300 ms, EES_rem = 0.235, EES_forg = 0.205; pTC: T_sum = 53.32, N_bin = 18, P_cluster < 0.001, onset at 500 ms, EES_rem = 0.104, EES_forg = 0.086; aTC: T_sum = 7.14, N_bin = 3, P_cluster < 0.001, onset at 600 ms, EES_rem = 0.028, EES_forg = 0.013). EES in the LPC was not related to memory.

Fig. 5. Generalization and differentiation of memory representations.

Memory effects of generalization (i.e., EES/RRS remembered > forgotten) or differentiation (i.e., EES/RRS remembered < forgotten) during encoding (EES, top) and retrieval (RRS, bottom). Colored horizontal lines indicate time points of significant memory effects after cluster-based correction for multiple comparisons across time points, separately for each region of interest (ROI).

During recognition, between-item similarities (RRS) showed memory-specific generalization in pTC (T_sum = 13.08, N_bin = 5, P_cluster < 0.001) but not in Occ or aTC (Fig. 5). Effects in pTC during retrieval occurred from 300 to 500 ms after scene onset and thus during an earlier period as compared to encoding. LPC representations showed an opposite effect between 500 and 600 ms (T_sum = −6.999, N_bin = 3, P_cluster < 0.001, RRS_rem = 0.026, RRS_forg = 0.037): Pairwise similarities of remembered items were lower compared to forgotten items, indicating that differentiation but not generalization of LPC representations is associated with successful recognition. Further, we found no association of generalization or differentiation effects with development but with trial-level magnitudes of ERS_Item (for details, see the Supplementary Materials; fig. S5).

Representational formats during reinstatement and transformation

Last, we sought to quantify the occurrence of visual representational formats and their association with memory in aTC and LPC during timepoints where we previously reported reinstatement and transformation effects. For this, we used the neural representations between items encountered during encoding and retrieval. We capitalized on the representations in individual layers of the PlacesNet DNN (Fig. 6A and fig. S6) and compared them to the representations observed in the neural data (Fig. 6B). We specifically tested whether the correspondence between DNN and neural representations during encoding and retrieval was related to memory success.

Fig. 6. Deep neural networks (DNNs) reveal memory-relevant representational formats.

(A) Top, schematic of cDNN architecture trained to classify scenes (PlacesNet). Bottom, multidimensional scaling (MDS) plots for scene stimuli used in the experiment showing more pronounced categorical (indoor/outdoor) clustering of representations in higher versus lower DNN layers. (B) RSA comparing neural representational similarity matrices (RSMs) during encoding (EES) and retrieval (RRS) with RSMs from the individual DNN layers. During both encoding and retrieval, we correlated trial-wise RSMs observed in neural data with RSMs in each DNN layer and then conducted a follow-up test for differences of neural-DNN similarities depending on memory (i.e., remembered versus forgotten trials). (C) Time-resolved analysis of encoding-DNN correspondence for remembered versus forgotten items (left) and separately for only remembered (middle) or forgotten (right) items. Dashed vertical lines indicate on- and offset of encoding and retrieval time periods relevant for reinstatement in aTC. Horizontal colored bars indicate time points of cluster-corrected effects across time points, separately for each layer of the DNN.

To do so, we compared neural-DNN correspondences of remembered and forgotten items for all time points separately and performed cluster-based correction for multiple comparisons across time points. During encoding in aTC, we found higher correspondence for remembered versus forgotten items between 850 and 1000 ms with representations of layer fc7 (T_sum = 9.472, N_bin = 4, P_cluster = 0.032) and between 750 and 1000 ms with representations of layer fc8 (T_sum = 14.702, N_bin = 6, P_cluster = 0.008) (Fig. 6C, left). Both clusters fell into the encoding time periods showing reinstatement effects on memory. During retrieval, we did not observe any memory effects in neural-DNN correspondence (fig. S7). To better understand the memory-specific differences in neural-DNN correspondence in aTC, we repeated the analysis separately for remembered and forgotten items. For remembered items, we did not observe a significant correspondence between neural and cDNN representations across the entire encoding time period (Fig. 6C, middle, no time points with P < 0.05). In contrast, for forgotten items, most cDNN layers showed a significant negative correspondence (Fig. 6C, right): conv3 (T_sum = −27.893, N_bin = 10, P_cluster < 0.001), conv4 (T_sum = −26.657, N_bin = 10, P_cluster = 0.005), conv5 (T_sum = −29.464, N_bin = 11, P_cluster = 0.010), fc6 (T_sum = −16.321, N_bin = 6, P_cluster = 0.048), fc7 (T_sum = −18.209, N_bin = 7, P_cluster = 0.008), and fc8 (T_sum = −18.925, N_bin = 7, P_cluster = 0.012). These results indicate that correspondence of aTC representations to sensory representational formats as captured by the cDNN are associated with recognition memory failure, suggesting that other formats in this region are required for successful memory formation (for details, see the Supplementary Materials; fig. S8). Further, analyses in the other VVS regions showed that correspondence of DNN representations to pTC representations related to subsequent memory as well, while correspondence of DNN representations to Occ cortex representations were not related to memory (fig. S7). In LPC, we found no significant difference in the neural-DNN correspondence of remembered and forgotten trials during encoding (fig. S7). During recognition, we found that neural-DNN correspondence was significantly larger for forgotten as compared to remembered trials in layer conv5 (T_sum = −11.689, N_bin = 5, P_cluster = 0.022) between 400 and 600 ms (for details, see the Supplementary Materials).

Similar findings of neural-DNN correspondence in aTC during encoding and LPC during recognition were obtained when performing post hoc tests specifically on those time points showing reinstatement and transformation effects (fig. S9). In addition, we tested whether the negative neural-DNN correspondence of subsequently forgotten trials in aTC was associated with the magnitude of ERS_Same correlations in the ERS analysis. We found that the negative neural-DNN correspondence was more pronounced in the subgroup of forgotten trials that yielded low ERS_Same correlations, pointing toward shared influences across analyses (for details, see the Supplementary Materials; fig. S10). Notably, given the exploratory nature of these analyses, interpretation of effects based on individual layers should not be overemphasized. Nevertheless, the overall pattern of results of memory-specific alignment of separable visual formats during encoding and retrieval may provide additional insights into the nature of the underlying memories and how they unfold during encoding and recognition.

DISCUSSION

The formation and retrieval of memory traces is at the heart of adaptive behavior and involves the recruitment of multiple distinct neural mechanisms acting upon specific item-related feature formats (67, 68). Here, we used direct electrophysiological recordings to investigate the functional roles of reinstatement and transformation for recognition memory, the relevance of distinct representational formats, and the unfolding of these processes in a sample of children, adolescents, and young adults.

We tested whether item-specific representations between encoding and retrieval (ERS) indicated whether memory is supported by reinstatement or transformation, reflected in higher or lower levels of ERS between remembered and forgotten trials. In line with previous findings, we found beneficial effects of reinstatement for memory but also showed that these effects were specific to the aTC. Reinstatement related to temporally restricted encoding activity patterns (500 to 1000 ms), pointing toward the relevance of specific representational formats (36). Consistent with findings in previous studies (69), we found that reinstatement occurred during earlier time windows during retrieval as compared to encoding, in line with a reversed processing hierarchy in the VVS between perception (posterior-to-anterior) and memory (anterior-to-posterior) (70, 71). We did not observe any memory effects of reinstatement in early- and mid-level VVS regions, suggesting that only the reinstatement of mnemonic representations is associated with successful recognition memory. Further analyses on representational formats in aTC using a cDNN indicated a diminished relevance of sensory formats compared to pTC and that recognition memory decisions may be associated with the reinstatement of rather mnemonic but not purely sensory representations, whereas regions presumably carrying low-level sensory content were not associated with memory-specific reinstatement.

Further, reinstatement of item-specific representations in aTC as reflected in higher ERS_Item correlations of remembered as compared to forgotten scenes could potentially relate to recollection-based processes of recognition that have been shown in temporal cortex regions before (11, 16) and that presumably reflect the reactivation of contextual or item-specific information. Notably, the reinstatement effect appeared to be driven predominantly by a drop of ERS_Same correlations of forgotten items rather than particularly high ERS_Same correlations of remembered items—although the former did not differ from zero while the latter did, ERS_Diff correlations of both remembered and forgotten items were larger than zero as well, suggesting that the ERS of remembered items was at least partially due to reoccurrence of unspecific processes. In addition, follow-up analyses of the forgotten items suggest that the low ERS_Same values were related to their negative match with DNN representational formats, suggesting that they were ambiguously processed in the aTC. However, ERS_Diff correlations did not differ between remembered and forgotten items, and, indeed, the initial finding of reinstatement was defined by a significant interaction between ERS_Same versus ERS_Diff correlations of remembered versus forgotten items, pointing toward the specificity of memory effects to activity patterns elicited by specific items. Thus, while reinstatement appeared to be related to a peculiar drop in ERS_Same values of some forgotten items that showed negative correlations with DNN representational formats, these effects did not occur for remembered items, preserving the similarity of the activity patterns of these remembered items during encoding with their patterns during retrieval. In other words, while the successful engagement of memory processes for the remembered items is not associated with prominent item-specific representations in the aTC that reoccur during retrieval, the putative “mnemonic” representational formats of these items correspond to similar activity patterns during encoding and retrieval of individual items that are absent for forgotten items. Future studies should aim to further characterize the putative mnemonic representational formats of remembered items and to clarify the representational mechanisms underlying the unexpected negative alignment of forgotten items to visual representational formats captured by DNNs.

In contrast to the memory effects of reinstatement in aTC, we found evidence for a transformation of item-specific representations in LPC (28, 29). Stimulus-related activity patterns during retrieval resembled those during encoding if participants did not remember having seen the scene before, while successful remembering elicited activity patterns that were unrelated to those during encoding. LPC has emerged as a prominent region implicated in retrieval-related accounts of episodic memory, reflecting retrieval success (30, 72, 73). Accordingly, some studies reported negative subsequent memory effects in this area (74). Being centrally located at the temporo-parietal junction, the LPC receives inputs from various sources including the dorsal and ventral visual processing streams as well as prefrontal and MTL structures (30). Two theoretical frameworks on LPC function, the AtoM (attention-to-memory) (75) and the CoBRA (cortical binding of relational representations) (73) models, both emphasize its close link with MTL structures but diverge in their interpretation concerning its underlying function. The AtoM model focuses on the relevance of bottom-up versus top-down attentional accounts (76) and ascribes bottom-up–driven orientation toward mnemonic representations to the ventral posterior parietal cortex. However, this model does not assume that the parietal cortex represents specific contents itself, which has been challenged by recent findings of content-based information decoding (28, 77) and the representation of engrams (78) in this region.

The CoBRA model argues for an integrative role of LPC serving the binding of object-based, semantic, and contextual information into multimodal formats. This may fit to our findings since the higher difference between encoding and retrieval activity patterns of remembered items may reflect the integration of multimodal stimulus features that ultimately results in transformed representations during retrieval, which may be selectively recruited to aid recognition memory decisions. Following earlier regard that reinstatement in aTC may relate to recollection-based processes during recognition, transformation in LPC may possibly reflect familiarity-based decisions of recognition memory, which reflect a modified response to an item upon its repeated encounter in a signal-detection process (79). Presumably, familiarity would reflect low- or near-zero similarities between encoding and retrieval, a pattern that we observe for ERS_Same correlations. Further, earlier studies have already described LPC to be involved in familiarity-based decisions of recognition memory (80, 81) as well as post-retrieval mechanisms of cognitive control through interactions with temporal cortices (82).

Transformation of item-specific representations in LPC also contradicts previous ERS findings: Using fMRI to test emotional and neutral scene recognition memory, a previous study found that parietal reactivation rather than transformation predicted successful recognition memory (18). Nevertheless, meta-analytic results on parietal involvement in episodic memory suggest that inferior lateral parietal cortices are more strongly involved in retrieval- versus encoding-related processes (83) in line with distinct involvement during encoding and retrieval, suggesting transformation. On a more general level, an alternative account on the pattern of results in LPC (i.e., a lack of ERS_Same correlations for remembered items, positive ERS_Same correlations for forgotten items, and no link to DNN representational formats for either items) may be that activity patterns in this region do not reflect stimulus-specific memory traces in the first place but are rather due to item-unspecific cognitive processes that only show overall differences between remembered and forgotten items. For example, all forgotten items may induce similar activity patterns during encoding and retrieval that reflect, e.g., lack of attentional engagement or mind-wandering to unspecific associative contents, resulting in positive ERS_Same correlations. By contrast, activity patterns of remembered items may reflect successful engagement of encoding and retrieval modes during the respective experimental phases; these modes are associated with different activities in LPC (28), which may explain the low and nonsignificant ERS_Same correlations. However, while we cannot entirely rule out this interpretation, we believe that it is less likely to fully account for our data for several reasons. First, we did not observe memory-related differences in overall spectral power in the LPC, and, thus, the engagement of unspecific processes would need to manifest selectively in the distribution of power across electrodes, which may appear unlikely (though not impossible) given that different patients were implanted in different subregions of LPC. Second, although activity patterns of forgotten items did not match DNN representational formats, we only considered trials with correct category classification during encoding, ensuring allocation of at least some attentional resources toward the stimulus. Third, if encoding and retrieval processes were indeed entirely unspecific to individual items, one would expect similar differences between remembered and forgotten items for ERS_Diff correlations as well, which we did not find—our main finding was based on a significant interaction between ERS_Same and ERS_Diff correlations of remembered versus forgotten items.

Our findings show that both reinstatement and transformation promote successful recognition but are supported by distinct brain regions, suggesting the involvement of dissociable neural mechanisms. The aTC is located at the apex of the VVS and represents sensory information at a conceptual level, whose reinstatement has previously been shown to benefit memory (36, 45). By contrast, the integrated multimodal representations in LPC may be primarily accessible during retrieval rather than encoding, in line with our finding that this region contains information about scenes in a transformed representational format. Evidence for item-specific representations (ERS_Item) in LPC occurred particularly when the magnitude of reinstatement in aTC was low, indicating interrelated processes between the two regions. Further, comparisons of trial-level ERS_Item values in aTC and LPC revealed an association between the magnitudes of item-specific information specifically for remembered items: The magnitude of item-specific representations (ERS_Item) in LPC depended on aTC, suggesting interrelated processes between the two regions. Thus, temporal and parietal cortices may represent distinct types of information with unique contributions to successful recognition, which may flexibly and distinctly influence memory decisions. We did not find memory-related differences in spectral power in these two regions. This likely reflects content-specific representations of individual items within the distribution of power across frequencies quantifiable using RSA rather than conditional differences in predefined frequency bands. In addition, the lack of a memory effect in spectral power analyses suggest minor effects of differences in response times, which would equally affect memory-specific findings at all levels of analysis.

Are reinstatement and transformation processes necessary for memory? Of course, our study does not provide direct information on the causal role of any of the observed processes. Our findings suggest that reinstatement is a general and beneficial mechanism for memory retrieval independent of developmental gains in memory, while recognition memory may depend on transformation. Although we find that transformation of item-specific representations is associated with successful recognition on the trial-level, this does not necessarily imply that this association exists in every participant. We found this effect specifically in a subgroup of participants with low recognition memory performance. This suggests that recruitment of memory transformation in the LPC is a less effective and possibly compensatory strategy compared to reinstatement in aTC. Recognition memory can rely on different processes of recollection and familiarity (31, 79, 82, 84), which may be differently associated with VVS reinstatement and LPC transformation effects (11, 16) and differently recruited by high versus low-performing participants. Nevertheless, given the relationship between the two effects, we speculate that neither of them may be sufficient for memory retrieval alone but that they may flexibly aid retrieval by task-relevant recruitment of distinct representational formats. Here, the characteristics of the memory trace determine how and by which features a familiar scene may be recognized. Alternatively, both reinstatement and transformation enable the representation of information in complementary formats that contribute differently to memory decisions depending on performance. According to this perspective, transformation processes may, in principle, occur in high-performing participants as well but are less relevant for them to conduct the relatively simple recognition memory task in our study. Notably, performance in our sample is likely influenced by a diverse set of factors, such as age-related differences in metacognitive capacities, the utilization of mnemonic strategies, or demands on compensatory mechanisms to overcome limited memory capacity. Although structural or functional deficits due to epilepsy cannot be ruled out, previous findings point toward comparable memory processes in epilepsy and healthy control groups (64, 85, 86).

We hypothesized that because of reinstatement and transformation processes, aTC and LPC contain distinct representational formats that differ between the two regions. Although we found a subsequent memory effect in Occ high-frequency broadband activity, and previous research links Occ gamma to early visual DNN formats (39), correspondence of representations in the Occ lobe and in the DNN was functionally irrelevant to memory in our study. However, our DNN results show a contribution of intermediate DNN formats to memory traces in pTC and more abstract formats in aTC. These results suggest a dissociation between the role of overall processing in early visual cortex (that may, e.g., reflect selective attention and thereby support memory indirectly) and the formation of the memory trace itself (which appears to rely on representational formats of higher-level visual areas). Further, similar to our findings in aTC, a previous study applied DNNs to investigate perirhinal cortex, an MTL region situated at the apex of the VVS, and found that it enables perceptual behaviors beyond VVS contributions, arguing for more abstract and potentially mnemonic representations (87) in this region. During retrieval, we found no memory-specific DNN correspondence with representations in aTC or upstream VVS regions. One may speculate that this is due to a diminished comparability between behavioral and network representations due to different task demands in encoding (scene classification) and retrieval (recognition). However, the correspondence between neural and DNN representations in the LPC during recognition tentatively suggested a memory effect that was influenced by a negative matching of neural and cDNN representations. We are currently unable to provide a full explanation to this effect. While negative correlations between related items have previously been described as “repulsion” of mnemonic representations (14, 88–90), these effects were associated with a functional benefit for memory, different from our data. Alternative explanations should thus be considered, e.g., that later forgetting is associated with processing of noninformative features during encoding. Notably, from our aTC ERS and EES-DNN analyses, we cannot directly infer that the described representational formats were also relevant for ERS because both types of analyses consider different cross-correlation matrices: Whereas ERS compared neural features of same versus different items, EES reflects representational distances between different items. Nevertheless, we observe a temporal (latency) and spatial (ROI) overlap in memory-specific differences in both types of analyses, which suggest that both effects are supported by overlapping representational and neurophysiological features. In the future, it may be promising to compare neural representations during retrieval to representations in generative DNNs such as variational autoencoders (91) or in recurrent networks trained on memory tasks (92).

How are reinstatement and transformation related to the generalization or differentiation of memory representations? We find that regions in the VVS showed memory-specific generalization during encoding. Further, we additionally found a reinstatement of memory representations in aTC. In LPC, we observed memory-specific differentiation during recognition and a transformation of representations between encoding and retrieval. According to theoretical accounts, both generalization and differentiation could be beneficial: While generalized representations allow inferences to novel experiences (33), distinct representations improve their separation from the representations of similar events (93). We found that generalization but not differentiation of VVS representations during encoding related to memory success. This effect may be explained by pronounced extraction of conceptual information that overlaps across multiple items, in line with pronounced correspondence to visual representational formats of subsequently forgotten but not remembered trials. Alternatively, generalization of representations may happen either because of the integration of all subsequently remembered items into a unified episodic context, i.e., reflecting episodic binding (94–97), or because of increased similarity of memory representations between all studies items, reflecting global matching (33). Accordingly, it has been shown that increased between-item similarity in high-, but not low-level, sensory areas during encoding is associated with successful memory and vividness during recall and levels of cortical reinstatement (34). By contrast, during retrieval, we found that representations of remembered items in the LPC were more differentiated than those of forgotten items. Notably, we observe generalization and differentiation effects in regions and during task phases not showing conditional differences in spectral power (i.e., in aTC and pTC during encoding and in LPC during recognition), suggesting memory-specific differences at the level of item representations beyond domain-general effects.

There are a few limitations to our study that require mentioning and that could be addressed in future research. First, although we have described the functional relevance of distinct representational formats during successful memory formation and recognition, our paradigm did not involve the experimental manipulation of low- and high-level sensory formats. Explicit manipulation of such formats could deliver more direct evidence for which formats are necessary for reinstatement or transformation and where they are represented in the brain. Second, although recordings from grid electrodes offer extensive sampling from cortical surface structures, our dataset did not allow fine-grained analyses of deeper brain structures in MTLs such as the hippocampus. More extensive MTL recordings would provide valuable insights into the involvement of hippocampal activity and its mechanistic function in the coordination of reinstatement and transformation-related processes in neocortical areas. This would especially be interesting to further characterize the differential contribution of sensory and mnemonic effects during recognition memory. Third, sensory and mnemonic effects could be addressed in future studies using only partial cues, item-based, and source memory tests during recognition, more detailed behavioral responses in the form of continuous memory ratings or recollection/familiarity ratings.

To conclude, we demonstrate that successful recognition memory is associated with both the reinstatement and transformation of item-specific representations, which are located in the ventral temporal and lateral parietal cortices, respectively. Our findings further suggest that the generalization and differentiation of item representations across these regions contribute to successful memory formation and retrieval, likely due to the relevance of distinct representational formats. Our study sheds light on the multifaceted nature of recognition memory and shows that this seemingly simple cognitive function involves multiple operations acting on different aspects of the memory trace.

MATERIALS AND METHODS

Patient sample

In total, N = 48 patients participated in the experiment. All patients suffered from pharmacoresistant epilepsy and were implanted with invasive electrocorticography (ECoG) or stereo-EEG (sEEG) electrodes for presurgical localization of epileptic foci. Implantations were conducted at the Children’s Hospital of Michigan, hospitals of the University of California (UC) San Diego, UC Irvine, UC San Francisco, and the California Pacific Medical Center. Sites of implantation were chosen by medical staff and based solely on clinical needs of the patient. Two participants were excluded from all analyses because of poor memory performance. The final sample consisted of N = 46 patients (20 female, 26 male) aged 5.9 to 32 years (15.61 ± 5.92 years; for details, see Fig. 1). The institutional review boards of the Wayne State University (no. 048404MP2E), UC Irvine and UC San Diego (no. HS# 2014-1522), UC San Francisco (no. 10-03842), and the California Pacific Medical Center (no. 666687-17) approved the study in accordance with the Declaration of Helsinki. Written informed consent was obtained from patients aged ≥18 years and from the guardians of patients <18 years; written assent was obtained from patients aged 13 to 17 years; oral assent was obtained from younger children.

Behavioral task

Participants completed a scene recognition memory paradigm that has been used to study memory with pediatric samples (58–65). During encoding, the participants were presented with equal numbers of indoor/outdoor scenes (each scene presented for 3 s) and instructed to verbally indicate indoor/outdoor as well as to try and memorize the scenes for a later memory test.

Each trial started with a fixation cross of 500 ms, followed by 3 s of scene presentation. For all subsequent analyses, we only considered trials with correct indoor/outdoor classification responses during encoding. During the subsequent recognition memory test, the participants were presented with old and new (foil) scenes and instructed to verbally indicate their memory judgement (i.e., “old” or “new”). Each trial started with a fixation cross of 500 ms, followed by self-paced duration of scene presentation.

Each encoding block consisted of N = 40 trials and was followed by a recognition memory test in which the participants saw all 40 old scenes intermixed with 20 new (foil) scenes. Forty-three percent (N = 20) of the participants completed one block, and 57% (N = 26) completed two encoding-recognition blocks. As a measure of memory performance, we computed corrected recognition scores Pr (98), i.e., rate of hits – rate of false alarms. N = 2 participants with negative recognition scores were excluded from further analyses. For all analyses, we only considered trials with artifact-free neural data during both encoding and recognition (49 ± 18 trials per participant).

Data acquisition and preprocessing

ECoG and sEEG data were acquired using a Nihon Kohden, Natus, or Tucker-Davis Technologies system at a minimum sampling rate of 1 kHz, and data acquired at higher sampling rates were resampled to 1 kHz after filtering. Data were filtered offline using a finite impulse response filter (0.1-Hz high-pass and 300-Hz low-pass). Line noise (60 Hz) and its harmonics (120, 180, and 240 Hz) were removed by applying a narrow-band notch filter to a discrete Fourier transform as implemented in fieldtrip (99). We excluded electrode contacts that showed pronounced epileptiform activity, overlapped with clinically defined seizure onset zones, or contained distinct artefactual signals such as poor contact. Trial-level artefact rejection was done manually by trained experts. Continuous data were then epoched into individual trials from −1 to +3 s around scene onset, or longer if response times exceeded the duration of scene presentation.

Electrode implantation, localization, and bipolar referencing

ECoG patients (N = 34) were implanted with subdural platinum electrodes (10-mm inter-contact distance; 4-mm diameter) placed directly onto the surface of the cortex. Here, individual electrode contacts were contained in larger grids (e.g., 8 by 8 or 2 by 5) or in one-dimensional (1D) strips of various length (4 to 12 contacts). SEEG patients (N = 12) were implanted with stereotactic depth electrodes (5-mm inter-contact distance) of various lengths (4 to 16 contacts). Implantation sites varied between patients based on clinical considerations. Implantations were performed in left (N = 17), right (N = 18), or both (N = 11) hemispheres. In all analyses reported here, we collapsed across ROIs of both hemispheres and treated ECoG/subdural and sEEG/depth electrodes in all preprocessing and analysis steps equally.

For localization of electrode positions, 3D reconstructions were created by coregistering postimplantation planar CT images of the cortical surface with preoperative T1 MR images (100). After reconstruction of implantation schemes, we created patient-specific templates for bipolar referencing. To do so, we interpolated locations of virtual electrodes from pairs of adjacent electrodes lying on the same depth electrode or on the same strip of grid electrodes. For 2D-grid electrodes, we sampled across both rows and columns of the grid. Only virtual electrodes based on two artefact-free neighbors were included into further analyses. We excluded bipolar electrodes where both neighbors were in white matter or ventricles. Montreal Neurological Institute (MNI) coordinates of all virtual electrodes were used to derive labels of Brodmann’s areas using BioImage Suite MNI2TAL toolbox (101). The final coverage contained N = 3691 bipolar referenced, artefact-free virtual electrodes, distributed over 32 unique Brodmann areas. Whole sample coverage for all ROIs analyzed throughout this work is displayed in Fig. 1E.

We grouped Brodmann areas into distinct ROIs for further analysis: Occ cortex (Occ; Brodmann areas 17, 18, and 19) including calcarine sulcus with area striata (BA17; V1), as well as secondary (BA18; V2) and tertiary (BA19; V3-5) association cortices; pTC (Brodmann area 37), which corresponds to posterior parts of the fusiform gyrus; aTC (Brodmann area 20) posited rostral to pTC covering large parts of the ventral temporal cortex and bound medially but excluding hippocampus, peri-, entorhinal, and parahippocampal cortices (BA34, 35, 36); LPC (Brodmann areas 39, 40) corresponding to gyrus angularis (BA39) and gyrus supramarginalis (BA40). Note that our selection of ROIs was relatively broad to allow sufficient sampling of electrodes across participants. Similar divisions of VVS into functionally distinct subregions have been used in earlier studies (39, 46).

Spectral decomposition and time-frequency features for RSA

First, we applied a band-pass filter (0.1-Hz high-pass, 150-Hz low-pass) to raw and artefact-free trial segments to restrict the data to the frequency range of interest. To avoid edge artefacts in spectral decomposition, we mirrored individual time-series data segments by appending flipped trial-data to the beginning (from −5 to −1 s) and end (from 3 to 7 s) of each segment. For time-frequency decomposition, we applied a Morlet wavelet transformation to data segments from −500 to 1500 ms. We used wavelets from 1:1:150 Hz with a linear increase from 3 to 6 cycles for frequencies 1 to 30 Hz, and 6 to 12 cycles for frequencies 31 to 150 Hz. We normalized poststimulus periods by their relative change to a prestimulus baseline from −500 to −200 ms before stimulus onset. We then extracted the power of frequencies from 1 to 30 Hz in 1 Hz steps and averaged subsequent frequencies in 5 Hz bins from 31 to 150 Hz (31 to 35 Hz, 36 to 40 Hz, etc.), resulting in 54 distinct frequencies. We then averaged data in temporal windows of 300-ms length, sliding in 50-ms steps from 0 to 1500 ms. For all reports, each window is assigned to its middle time point (e.g., the first window expands from 0 to 300 ms and is indicated as 150 ms). For the range of 0 to 1500 ms, this resulted in 25 distinct windows. Electrode-specific time-frequency representations of individual trials resulted in n (channels) × 54 (frequencies) × 25 (time steps) values that were entered into subsequent RSA.

Representational similarity analysis

Here, we report results from two different sets of RSA (10), which are computed with the same neural features derived from preprocessed electrophysiological data. We estimated the pairwise similarity of trials either within the same experimental phase (EES; RRS) or between experimental phases (ERS) using Spearman correlation (Fig. 2, A to D). As features for similarity estimation, we used the distribution of power values across frequencies and electrodes in each ROI. If, for example, a participant contributed three (bipolar) electrodes to a certain Brodmann area, the trial-specific spectral power across frequencies of all three channels (3 electrodes × 54 frequencies) from encoding were concatenated into a 1D vector of 3 × 54 = 162 data points and correlated with the neural data of the same and of all different scenes in either the same or a different experimental phase. This analysis was done for various time windows (see above).

Cluster-based permutation statistic

To correct for multiple comparisons in all RSA, we applied cluster-based permutation statistics (102). We repeated each first-level computation during RSA (e.g., for a specific time bin of RSA feature data) after shuffling the trial-specific labels (remembered/forgotten trials) while keeping the amount of data per category intact. For example, when performing a first-level independent-samples T test of ERS_Item between remembered and forgotten trials, in each permutation, we randomly assigned the same ERS_Item values to each memory type while keeping the original number of trials per category intact. This procedure was conducted for 1000 iterations. We then corrected for multiple comparisons across time by extracting clusters of adjacent time points with uncorrected group-level P values <0.05 within the time (1D) or time × time (2D) maps. Depending on the type of analysis, we looked for temporally contiguous clusters in 1D temporal progression (EES, RRS, and DNN-EES) or in 2D temporally generalized data (ERS). For each identified empirical or surrogate cluster, we summed all T values contributing to it. Last, we computed the probability of observing the empirical cluster by extracting its rank from the distribution of all clusters in the same direction (positive or negative) in the surrogate data (P = rank/1000). Permutation statistics for memory-dependent differences in time-frequency spectra was conducted using fieldtrip toolbox. RSA analyses used custom MATLAB scripts.

Between-item similarity (EES/RRS)

For the between-item similarity analysis (see Fig. 3D), we computed the pairwise similarity of scenes within the same experimental phase (EES/RRS). Separately for each time point of the sliding window, we extracted trial-specific time-frequency features across electrodes and estimated their similarity using Spearman correlation. This yielded a symmetric identity stimulus × stimulus matrix where the diagonal is the similarity of a scene to itself. Obtained matrices were Fisher-Z transformed before further analysis. All analyses were performed on the lower-diagonal values of the symmetric similarity matrices. For each matrix and time point, we tested whether pairwise similarities of remembered versus forgotten scenes differed. To do so, we computed an independent-samples T test of pairwise similarities of scenes from the same memory category (all remembered-remembered and forgotten-forgotten scene pairs) for each participant. On the second level, we tested whether subject-specific first-level T values consistently deviated from zero. In the time series of group-level effects, we then looked for adjacent time points forming temporal clusters and estimated their statistical significance using cluster-based permutations statistics. Surrogate first-level data were obtained by shuffling the labels of trials (remembered versus forgotten) and repeating the analysis with shuffled data 1000 times.

Encoding-retrieval similarity

In the second line of RSA, we investigated memory-specific representational reinstatement or transformation. Specifically, we estimated the neural similarity between the representation of a specific scene during encoding and recognition (ERS). This analysis yielded a cross-correlation matrix of scenes (dimension: stimulus × stimulus). Matrices were Fisher-Z transformed before further analyses. In this matrix, the diagonal elements denote similarities between the same scene during encoding and recognition, while the off-diagonal elements denote similarities between different scenes. Since we applied a temporal sliding-window approach, the stimulus × stimulus matrix was calculated for all possible combinations of time points, yielding a 4D matrix of ERS values (dimensions: stimulus × stimulus × encoding time × recognition time). This time-resolved approach allowed us to investigate whether neural representations occurring at a specific time point during encoding correlated with representations at a specific—possibly different—time point during recognition.

Next, we quantified the item-specific neural similarity between encoding and recognition by taking the similarity score of the same scene during encoding and recognition (same item, diagonal element) and subtracting the average similarity of this scene during encoding with all other scenes during recognition (different item, row-wise off-diagonal elements), i.e. ERS_Item = ERS_Same – avg.(ERS_Diff). Crucially, ERS_Diff considered values from the same scene type (indoor/outdoor) and memory category (remembered/forgotten). We replicated ERS results for aTC and LPC when considering only column-wise off-diagonal elements, as well as row and column-wise off-diagonal elements (fig. S11).

Does the degree of scene-specific similarity between encoding and recognition support memory performance? To answer this question, we performed an independent-samples T test of ERS_Item scores, comparing their magnitude across all remembered and forgotten trials within each participant. This yielded one map of T values for each participant across the 25 encoding time points and the 25 recognition time points, which was used for subsequent group level analysis. On the group level, we performed a one-sample T test across the T values of all participants against zero. This analysis was conducted for all 25 × 25 encoding/recognition time points. We then extracted all T values corresponding to P values <0.05 and summed them across all adjacent time points for estimation of statistical significance (empirical cluster value).

Item specificity in aTC and LPC (ERS – ERS)

We tested the relationship of the magnitude of ERS_Item between aTC and LPC using hierarchical linear mixed models. This analysis was conducted on the subset of participants (N = 28) contributing electrodes to both regions. We averaged ERS_Item values across overlapping time points, i.e., encoding-retrieval time bins with significant ERS effects in aTC (Fig. 3B) and LPC (Fig. 4B), N_bins = 33, encoding time range 500 to 1000 ms, retrieval time range 400 to 900 ms) of reinstatement and transformation effects and then predicted the magnitude of aTC ERS_Item by LPC ERS_Item, included a main effect of memory and modeled subject-specific slopes [aTC ~ LPC * memory + (1|subject)].

DNN model

To classify scene images shown throughout the experiment, we used a pretrained convolutional neural network “PlacesNet (103).” PlacesNet was trained on the Places database (104), incorporating 2.5 million scene images of 205 category labels. cDNNs with the same network architecture [e.g., AlexNet (105)] trained on a variety of object classes have been found to exhibit representational geometries corresponding to neural representations observed throughout the VVS (38, 39, 43, 106). PlacesNet consists of five convolutional and three fully connected layers (Fig. 6A). Information is initially processed on pixel-by-pixel level in convolutional layer 1 and is then unidirectionally transformed to downstream layers, ultimately leading to the assignment of semantic labels in the output layer (softmax). We averaged the values in each convolutional layer for each image over the spatial dimension (107) (retinotopic units) resulting in one value per feature (images × features). Extracted pairwise Spearman’s ρ estimates based on internal feature activations yielded symmetric layer-specific representational similarity matrices (RSMs) of all 120 by 120 scene images, resulting in 8 RSMs of 120 by 120 images (see fig. S6C). RSMs were Fisher Z transformed for further analyses.

Item-specific alignment of neural and DNN model features

RSMs with corresponding structure to RSMs from DNN layers were obtained from EES and RRS analyses (scene*scene), thus allowing for a direct comparison of representational structure between iEEG ROIs and PlacesNet. EES/RRS matrices were obtained for each individual time point during encoding and retrieval, resulting in 25 scene*scene matrices per participant, ROI, and experimental phase. Since internal features of feed-forward models are insensitive to temporal dimensions, we obtained one RSM for each layer of the network, resulting in eight model matrices equally used for encoding- and retrieval related neural data.

To match the item-specific approach in the previous ERS analysis, we computed scene-specific alignment of neural features with each of the network layers [see (36, 45) for a similar approach]. For each scene, we extracted the vector of similarity values with all other scenes (vector of length N_scenes-1, e.g., one row in matrix) from neural and DNN matrices. These vectors describe the representational geometry of the current scene to all other scenes. We then computed the Spearman correlation of neural and model vectors separately for each time point and network layer, yielding a score describing the item-specific similarity of neural and network representations. The subsequent statistical procedure was similar to ERS analysis: First, we computed independent-samples T tests between the alignment scores of remembered and forgotten items to test whether shared similarities differed depending on memory at any time point. On the second level, we tested across participants whether these T values differed significantly from zero. Then, we identified adjacent time points with group-level T values corresponding to P values <0.05 and performed correction for multiple comparisons across adjacent time points by comparing the clusters in the empirical data with clusters from surrogate data, where the order of scene-specific similarity vectors from the neural data was randomly shuffled before comparison with scene-specific vectors from individual DNN layer matrices.

Supplementary Materials

This PDF file includes:

Supplementary Text

Figs. S1 to S11

References