Neural Activity Reveals Interactions Between Episodic and Semantic Memory Systems During Retrieval

Abstract

Whereas numerous findings support a distinction between episodic and semantic memory, it is now widely acknowledged that these two forms of memory interact during both encoding and retrieval. The precise nature of this interaction, however, remains poorly understood. To examine the role of semantic organization during episodic encoding and retrieval, we recorded intracranial encephalographic signals as 69 neurosurgical patients studied and subsequently recalled categorized and unrelated word lists. Applying multivariate classifiers to neural recordings, we were able to reliably predict encoding success, retrieval success, and temporal and categorical clustering during recall. By assessing how these classifiers generalized across list types, we identified specific retrieval processes that predicted recall of categorized lists and distinguished between recall transitions within and between category clusters. These results particularly implicate retrieval (rather than encoding) processes in the categorical organization of episodic memories.

The ability to remember events and facts depends on separate episodic and semantic memory systems respectively (Schacter & Tulving, 1994; Tulving, 1972, 2002). Neuroimaging (Kapur et al., 1994; Poldrack et al., 2001) and neuropsychological (Patterson, Nestor, & Rogers, 2007; Rogers et al., 2004) evidence implicates distinct neural bases for episodic and semantic memory: Damage to the hippocampal formation selectively impairs episodic memory (Vargha-Khadem et al., 1997), whereas injury to the anterior temporal lobe results in semantic memory deficits (Simons, Graham, Galton, Patterson, & Hodges, 2001).

Despite clear differences between episodic and semantic memory systems, interactions between them are ubiquitous: The semantic structure of remembered items affects performance in episodic memory tasks, such as free recall, where recall of semantically similar items tends to occur in clusters (Bower, Clark, Lesgold, & Winzenz, 1969; Bousfield, 1953; Howard & Kahana, 2002). A striking demonstration of semantic structure affecting episodic memory is the finding that for lists of words that are strongly related (e.g., “pillow”, “night”, “moon”, “bed”), a missing associate (e.g., “sleep”) is often confidently mis-remembered as having been studied (Roediger & McDermott, 1995). Likewise, in tasks asking participants to freely recall all studied items in any order (i.e., free recall), categorically structured study lists lead to increased performance and categorical organization of recall sequences (Bousfield, 1953; Bower et al., 1969).

Many studies have attempted to attribute such effects of semantic structure of the memoranda to either encoding or retrieval processes. Given that measures of memory performance reflect the joint effects of encoding and retrieval, any such distinction on the basis of these measures relies on strong assumptions about experimental manipulations or participant characteristics. Typical approaches include attempts to selectively disrupt or enhance either encoding or retrieval (Craik, Govoni, Naveh-Benjamin, & Anderson, 1996; Fernandes & Grady, 2008) as well as examinations of neuropsychological populations with well-defined memory deficits (Gershberg & Shimamura, 1995; Greenberg, Keane, Ryan, & Verfaellie, 2009). To the extent that experimental manipulations or neuropsychological deficits are not selective, however, associated conclusions are open to alternative interpretations. Indeed, there is little consensus across studies about the relative contributions of encoding and retrieval processes in the interplay between episodic and semantic memory (see Greenberg & Verfaellie, 2010, for a review).

Here we pursue a novel route to link specific interactions between semantic and episodic memory to encoding and retrieval processes. Rather than relying solely on memory performance, we leverage ongoing neural activity during study and recall phases of a free-recall task to identify such interactions. In this effort, we build upon prior work that has characterized neural biomarkers of encoding and retrieval: Structures in prefrontal, temporal, and posterior parietal cortices respond differentially during the encoding of subsequently remembered and forgotten items (Uncapher & Wagner, 2009; Wagner et al., 1998)—a subsequent memory effect (Paller & Wagner, 2002). Direct measures of neural activity obtained from intracranial electroencephalography (iEEG) have implicated the same neural structures in memory processing, exhibiting increases in high frequency (i.e., >70 Hz) power with concomitant decreases in power at lower frequencies during successful encoding (Burke, Long, et al., 2014; Long, Burke, & Kahana, 2014). Similar neural signatures during the recall period of free-recall tasks predict successful retrieval (Burke, Sharan, et al., 2014; Kragel et al., 2017).

We also take advantage of the statistical power of multivariate (“machine learning”) classifiers to quantify the neural signals during encoding and retrieval periods that are predictive of different aspects of memory performance or recall organization (Norman, Polyn, Detre, & Haxby, 2006). Specifically, we compare performance of such classifiers trained on features from iEEG recordings while participants studied and recalled lists of semantically categorized or unrelated words during a free-recall task. This approach allowed us to directly compare neural signals during encoding and retrieval phases that predict successful memory, as well as categorical and temporal clustering of recall sequences (which respectively index effects of semantic and episodic memory). In addition to highlighting encoding and retrieval effects that generalize across memoranda with varying semantic structure, our findings particularly implicate retrieval processes as drivers of the interactions between semantic and episodic memory systems.

Method

We analyzed direct neural recordings from cortical and deep brain structures from 69 participants as they performed free recall of categorized and unrelated words lists (see Figure 1). As part of regular clinical procedure for the monitoring of epileptic seizures, patients were implanted with electrodes (i.e., a combination of subdural grids, strips, and/or depth electrodes) to directly measure iEEG activity. To examine the influence of semantic structure on the neurophysiological markers of memory function, we examined changes in the spectral power of the EEG time series recorded during the encoding and retrieval of categorized and unrelated word lists in a free-recall task.

Figure 1.

Schematic of the task structure. Each study list comprised of 12 unrelated items (I) or of 12 items drawn from three distinct categories (A, B, C, with 4 items drawn per category). Analyses of retrieval organization examined temporal clusters based on “near” or “far” transitions (i.e., those from items presented adjacently in the study list or not). We analyzed categorical clustering by contrasting between and within-category recall transitions (i.e., those from items that belonged to a different or the same category respectively). See the online article for the color version of this figure.

Participants

We examined data from 69 neurosurgical patients (25 female, with an average age at implant of 36.3 years, ranging from 20 to 63 years of age) with medication-resistant epilepsy who had intracranial electrodes implanted for diagnostic reasons. The data were collected as part of a larger project in collaboration with Columbia University Medical Center (New York, NY), Darmouth-Hitchcock Medical Center (Hanover, NH), Emory Hospital (Atlanta, Georgia), Hospital of the University of Pennsylvania (Philadelphia, PA), Mayo Clinic (Rochester, MN), Thomas Jefferson University Hospital (Philadelphia, PA), and University of Texas Southwestern Medical Center (Dallas, TX). Institutional review boards at the respective hospitals approved our research protocol and we obtained informed consent from each participant. We selected data from patients who participated in both standard and categorized free-recall tasks (see below). From a total of 94 patients who participated in both versions of the free-recall task, we selected those for which we had at least 0.8 power to detect classifier performance of at least 0.7 area under the receiver operating characteristic (ROC) curve (AUC) for subsequent recall encoding and retrieval decoding and cross-decoding classifiers (see below). We used permutation analyses to estimate chance classification performance for these power analyses and selected the sample of 69 patients on which all analyses presented here are based. For neural analysis of temporal clustering, we analyzed data from a subset of these patients (n = 60) who had a sufficient number of observations to train classification models. Some of the analyzed data have been used in other studies (Ezzyat et al., 2017; Kragel et al., 2017; Long et al., 2017; Solomon et al., 2017), but the analyses reported here are novel.

Free-Recall Tasks

During their time in the epilepsy monitoring unit, patients participated in a range of memory tasks including two versions of a delayed free-recall task that differed in the semantic structure of the study lists (described in detail below). Within each experimental session we only presented one list type and our selection criteria (described above) required that all included participants contributed data from at least one session with each list type (i.e., each participant contributed data from at least two sessions). Each session consisted of up to 26 cycles of encoding, delay, and free recall; the first of which was used as practice and not further analyzed. A 10 s countdown preceded each study list which consisted of a total of 12 words, presented for 1600 ms each, and separated by blank interstimulus intervals of 750–1000 ms (randomly sampled from a uniform distribution). The two versions of the task differed only in the composition of the study lists (described below). Following each study list (and prior to free recall), participants were asked to solve simple math problems of the form A + B + C = ?? where, A, B, and C, were random integers in [1, 9]. Individual math problems were presented one at a time until a response was entered on a keypad when either a new math problem was presented or a signal to initiate free recall was given (math problems were presented until a delay of at least 20 s was achieved). A central row of asterisks presented together with an 800 Hz tone signaled the start of each recall period. Participants were given 30 s to recall as many words from the most recent study list as possible in any order. Vocal responses were recorded and annotated offline.

Study List Construction

Study lists consisted of either unrelated or categorized words with the same list type used throughout a given session. For unrelated word lists, a pool of 300 words was constructed by selecting words from a larger word pool used in a large-scale study of free recall on a separate set of participants. For this separate study, the effects of each individual word on recall performance were modeled while accounting for serial position, frequency, concreteness, imageability, and length. Words for which recall performance fell on either end of the resulting distribution were removed to yield 300 words with intermediate recall performance. From this pool of 300 words, individual lists of 12 items were constructed such that the mean pairwise semantic similarity within list was relatively constant across lists (with a Latent Semantic Analysis cosine similarity of around .2). All 300 words were presented exactly once across the 25 experimental lists in each complete session, but individual participants did not see the same list twice across sessions.

Categorized lists were generated by first asking a set of 40 participants on Amazon Mechanical Turk to generate lists of exemplars from 28 categories. From these responses the 25 exemplars that were generated most frequently from each category were shown to a second set of online participants, who rated their typicality for the respective category. The 12 most prototypical exemplars, according to these ratings, were selected and the three categories with the lowest mean prototypicality ratings across those 12 exemplars were discarded to yield a word pool of 300 words consisting of 25 categories with 12 highly prototypical exemplars each. Each experimental study list was constructed by randomly selecting four exemplars from each of three randomly selected categories. Words were sequentially presented in pairs (two exemplars from a given category), but no two pairs from the same category were presented consecutively. Each session began with the presentation of a practice study list which was always composed of the same 12 (unrelated) words in random order regardless of whether the words in the experimental study lists were unrelated or categorized; recall performance from this practice list was not analyzed.

Recording and Processing of Electrophysiological Signals

We recorded from subdural grids and strips (space between adjacent contacts: 10 mm) and from depth electrodes (space between adjacent contacts: 5–10 mm) on a variety of recording systems across clinical sites with sampling rates varying between 500 and 2000 Hz to accommodate the local recording environment. We rereferenced all recordings using a bipolar referencing scheme (Burke et al., 2013), applied a 4th order Butterworth filter with a 58–62 Hz stop-band to remove line noise, and convolved the resulting signals with Morlet wavelets (wave number 5; 8 center frequencies, log-spaced between 3 and 180 Hz) to obtain spectral power. For the analysis of encoding data, we averaged power over the entire 1600 ms stimulus presentation interval. We constructed patterns of neural activity from the recall period of the task by averaging spectral power from 900 to 100 ms preceding a correct recall response. We used mirrored buffering of recall period data to prevent spreading of low frequency signals produced during recall vocalization from mixing with potential signals of interest. To discriminate between successful and unsuccessful retrieval, we additionally constructed failed-retrieval events. The onsets of these events within the recall period were matched to valid recalls in different lists, with the constraint that no valid recall occurred in the following 2000 ms. To ensure that response production from prior recalls did not influence our estimates of retrieval-related activity, we eliminated recalls from our analysis in which the onset of the prior recall occurred within 1500 ms. We applied the same constraint to failed-retrieval events.

Univariate Analyses

To account for heterogeneous electrode coverage across participants, we aggregated electrodes into 9 regions of interest (ROIs; Figure 3a, top). For each ROI (and participant who contributed to that ROI) we calculated differences in z-transformed (by session) log-power during either the 1600 ms encoding period or the 800 ms retrieval period between event types of interest (e.g., subsequently recalled vs. forgotten items). We then divided the mean differences across participants by their standard errors to obtain the t-values shown in Figure 3. Preliminary analyses did not indicate significant differences as a function of hemisphere and we thus collapsed across hemispheres for each ROI to simplify presentation and increase power.

Figure 3.

Encoding and retrieval biomarkers from categorized and unrelated word lists (a). Regions of interest (ROIs) are displayed on an average cortical surface template. The number of participants contributing electrodes to each ROI is shown below, separately for left (L) hemisphere, right (R) hemisphere, or bilateral (B) coverage. Dashed line indicates the total number of participants. IFG = inferior frontal gyrus; MFG = middle frontal gyrus; SFG = superior frontal gyrus; HC = hippocampus; PHG = parahippocampal gyrus; TC = temporal cortex; IPC = inferior parietal cortex; SPC = superior parietal cortex; OC = occipital cortex (b). Statistical maps for changes in spectral power for successful versus unsuccessful encoding and retrieval. Statistically significant (false discovery rate [FDR] corrected, q < .05) increases and decreases are shown in red (top portion of the color bar) and blue (bottom portion of the color bar), respectively. See the online article for the color version of this figure.

Multivariate Classification

We trained L2-regularized logistic regression classifiers to distinguish patterns of brain activity associated with different memory states. The first classifier discriminated between neural activity associated with successful and unsuccessful memory function. During encoding, we determined successful memory function by whether or not each studied item was subsequently recalled. Likewise, during retrieval, we contrasted intervals in the recall period that lead to successful recall with those that did not in order to assess successful memory function. We used the second classifier to discriminate neural states that predicted whether recall of items from a categorized list would follow a recall from a same-category item (i.e., within-category transition) or a recall from a different-category item (i.e., between-category transition; see Figure 1). We constructed a third classifier to identify items that would be recalled within a temporal cluster (i.e., the preceding recall was presented in an adjacent serial position) or not (see Figure 1). We used these classifiers to decode patterns of spectral power observed during encoding (i.e., identifying biomarkers of subsequent clustering) and during recall periods.

For all classifiers, we fixed the regularization parameter C, as implemented in scikit-learn (Pedregosa et al., 2011), to 0.0007 based on explorations on a larger data set. We fit these classifiers separately to data from each participant using the z-transformed (by session) log-power in the 8 frequencies across all electrodes as features. For classifiers trained and tested on the same list type (i.e., decoding classifiers), we cross-validated classifier performance by holding out each list (across all sessions using that list type) once. For classifiers trained and tested on different list types (i.e., cross-decoding classifiers) we trained the classifier on all data from the respective list type and evaluated its performance on all data from the other list type. By examining the degree to which each classifier would generalize to held out lists with a different semantic structure (i.e., a cross-decoding approach; Kragel & Polyn, 2015; Kragel et al., 2017) compared to its performance on held-out test lists that matched the semantic structure of the training lists (i.e., a decoding approach), we were able to quantify neural signals that were specifically sensitive to the semantic structure of the memoranda. We used each classifier’s AUC as measure of its accuracy, with 0.5 corresponding to chance performance and 1.0 indicating perfect classification (Fawcett, 2006).

Statistical Analysis

We assessed significance of group level effects by treating participants as random effects. We evaluated differences in recall performance and classifier generalization across list types using repeated-measures ANOVAs, testing for significance at p < .05. For tests of a single factor (e.g., category clustering), we performed two-tailed one sample t tests except in cases where we tested whether classifier performance exceeded chance. For tests of univariate differences in activity across multiple ROIs and frequencies (see Figure 3), we adjusted the resultant p values using the Benjamini and Hochberg procedure (Benjamini & Hochberg, 1995) to control the false discovery rate (FDR, q < .05).

Results

As described above, the semantic structure of study lists profoundly affects subsequent recall with categorized lists generally producing increased rates of recall (relative to similar lists of unrelated items), semantically organized recalls, and reduced interrecall times when items are recalled within a semantic cluster (Bower et al., 1969; Bousfield, 1953; Howard & Kahana, 2002; Wingfield, Lindfield, & Kahana, 1998). We first demonstrate these effects in our data and then use analyses of neural signals to test the relative contributions of encoding and retrieval processes to recall performance and organization.

To verify that categorically organized lists improved recall performance in our patient population, we conducted a 2 (list type) × 12 (serial position) repeated measures ANOVA on the probability of recall (see Figure 2a). Participants recalled a higher percentage of items from categorized than unrelated word lists (F[1, 68] = 52.46, MSE = 0.04, p < .001), with a significant main effect of list position reflecting enhanced recall of early list items across both list types (primacy effect; F[11, 748] = 56.79, MSE = 0.02, p < .001). The distractor-filled retention interval limited the benefit of recency on memory performance to little (on categorized lists) or none (on unrelated lists), resulting in a significant interaction between list type and serial position (F[11, 748] = 1.93, MSE = 0.01, p = .03). These results replicate previous findings of improved recall performance for semantically structured memoranda (Bousfield, 1953; Bower et al., 1969).

Figure 2.

Recall performance and organization (a). Probability of recall for categorized (C) and unrelated (U) word lists as a function of serial position (top panel) and associated differences (bottom panel). Shaded regions denote 95% confidence intervals (b). Violin plots illustrating the distributions of interrecall times for transitions within and between category clusters. Means and 95% confidence intervals are indicated (c). Violin plots illustrating the distributions of adjusted ratio of clustering (ARC) scores for categorized and unrelated lists (we determined clustering for unrelated lists relative to the categorical structure of matched categorized lists). Means and 95% confidence intervals are indicated (d). Scatter plot showing differences in ARC for categorized versus unrelated word lists against corresponding differences in the probability of recall for each participant and associated regression line. Shaded region denotes the 95% confidence region of the regression line.

We next confirmed that presentation of categorized lists influenced recall dynamics by causing subjects to recall bursts of same-category items (Wingfield et al., 1998). As in previous studies (Pollio, Richards, & Lucas, 1969; Patterson, Meltzer, & Mandler, 1971), within-category recall transitions were significantly faster than between-category transitions (t[68] = 14.54, SE = 0.15, p < .001; see Figure 2b). This difference remained significant when eliminating transitions that were faster than 1500 ms and thus did not contribute to our analyses of brain activity (see Method; t[68] = 8.36, SE = 0.20, p < .001). We quantified the degree of clustering with the adjusted ratio of clustering (ARC) statistic (Roenker, Thompson, & Brown, 1971),¹ which was significantly greater for categorized than unrelated lists (using the category structure of matched categorized lists to quantify baseline levels of clustering; t[68] = 10.08, SE = 0.03, p < .001; Figure 2c). This measure confirms that category clustering exceeded what one would expect from reinstatement of the encoding sequence. Indeed, participants were also more likely to cluster same-category items that were not presented in adjacent serial positions in the study list (ARC calculated only for items in nonadjacent positions in the study lists was significantly greater for categorized lists than for matched unrelated lists; t[68] = 6.03, SE = 0.033, p < .001). Participants who exhibited greater category clustering also recalled a higher percentage of items on categorized as compared to unrelated word lists (Figure 2d, r = .56, p < .001). Having established that our participants exhibited the classic effects associated with the recall of categorized lists, we next analyzed neural activity to determine the relative contributions of encoding and retrieval processes to this behavior.

If the semantic relatedness of memoranda interacts with episodic encoding and retrieval processes, one would expect corresponding differences in associated neural biomarkers of memory function. To identify these differences, we partitioned electrode locations into the 9 ROIs shown in Figure 3a and contrasted average log-transformed power (z-transformed by session) from electrodes within each ROI as a function of subsequent recall (Figure 3b).² Specifically, we separately averaged power during encoding periods (i.e., over the 1600 ms study word presentation intervals) for words that were subsequently recalled and those that were not. To identify biomarkers of successful memory function during retrieval, we averaged power during 800 ms intervals ranging from 900 ms to 100 ms prior to successful recalls and contrasted these with matched intervals not preceding recalls (these intervals putatively index unsuccessful retrieval attempts). Despite the clear differences in recall performance and organization as a function of list type (see Figure 2), these encoding and retrieval biomarkers of successful memory were remarkably similar across both list types (Figure 3b): Increased power in higher frequencies and decreased power in lower frequencies indicated subsequent memory and imminent retrieval across a wide range of ROIs for both categorized and unrelated lists. We found no significant differences across list types even when we did not correct for multiple comparisons. While these findings may suggest that the same neural processes underlie successful memory encoding and retrieval independent of the categorical structure of the list, differences in neural signal may have been obscured due to aggregation across ROIs and differences in electrode placement across participants.

To increase our power to detect differences in the influence of semantic structure on episodic encoding and retrieval, we turned to multivariate classification of memory states within individuals. We trained L2-penalized logistic regression models on spectral iEEG features during study and recall of categorized and unrelated word lists. These models learned to discriminate between neural activity corresponding to differences in (subsequent) recall performance (e.g., between neural activity during encoding periods that predicted whether the studied word was subsequently recalled). We evaluated the ability of classifiers to generalize to lists with differing semantic structure, allowing us to determine whether the neural biomarkers of memory encoding or retrieval were specifically influenced by categorical list structure (see Figure 4 for an illustration of our cross-decoding approach).

Figure 4.

An illustration of decoding and cross-decoding approaches with associated receiver operating characteristic (ROC) functions and AUC differences (ΔAUC, indexing classifier specificity) for recall performance (a) and recall organization (b) classifiers. Subpanels are numbered using roman numerals (i–iv) to facilitate discussion of the results. Subpanels (i) and (iii) show results for encoding classifiers with (ii) and (iv) showing performance of retrieval classifiers. Subpanels (i)–(ii) and (iii)–(iv) show results from classifiers trained on categorized and unrelated lists respectively. Subsequent clustering classifiers distinguished within- from between-category recall transitions in categorized lists and near from far recall transitions in unrelated lists (see Figure 1). Error bars around the mean ΔAUC values indicate the 95% confidence intervals.

Figure 4a depicts average ROC functions showing decoding and cross-decoding performance for classifiers trained on neural activity during either encoding or retrieval periods in each list type to predict (subsequent) recall. To the extent that these ROC functions overlap, the corresponding classifier generalizes beyond the semantic structure of the training lists, whereas differences in these ROC functions indicate that the corresponding classifier capitalizes on neural processes that are specific to the semantic structure of the memoranda. To measure this specificity, we calculated the difference in the AUCs between the decoding and cross-decoding performance (ΔAUC), depicted alongside the corresponding ROC functions in Figure 4a.

The curvature of all ROC functions in Figure 4a along with the substantial degree of overlap between corresponding decoding and cross-decoding ROC functions suggests that both encoding and retrieval classifiers predicted (subsequent) recall reliably for both list types. Indeed, decoding performance was significantly above chance for all classifiers (all ts[68] > 14.16, SEs ≤ 0.01, ps < .001) with similar results for cross-decoding performance (all ts[68] > 13.18, SEs < 0.02, ps < .001). These findings, together with the remarkably similar univariate (subsequent) recall contrasts across both list types (Figure 3b), implicate general memory processes that determine recall performance irrespective of the semantic structure of the memoranda.

The difference between decoding and cross-decoding performance was significant in the encoding classifier for unrelated lists (t[68] = 2.27, SE = 0.01, p = .03; Figure 4a (iii)) and in the retrieval classifier for categorized lists (t[68] = 2.76, SE = 0.01, p = .007; Figure 4a (ii)), with no other ΔAUCs deviating significantly from zero (all ts(68) < 1.70, SEs ≤ 0.01, ps > .09). A 2 (training list type) × 2 (task phase; i.e., encoding vs. retrieval) repeated measures ANOVA on ΔAUC values revealed a significant interaction (F[1, 68] = 5.12, MSE = 0.006, p = .03) but no significant main effects of training list type (F[1, 68] = 1.47, MSE = 0.01, p = .23) or task phase (F[1, 68] = 0.002, MSE = 0.007, p = .97). Direct post hoc comparisons between ΔAUC values for encoding and retrieval classifiers did not show significant differences for either list type (ts(68) < 1.59, SEs ≤ 0.01, ps > .12). These findings are consistent with prior demonstrations of common neural signatures indexing episodic and semantic memory (Rajah & McIntosh, 2005), while also highlighting processes during retrieval of categorized lists that are sensitive to the semantic structure of the list.

Having identified specificity of the retrieval classifier when it is trained on categorized lists and (to a lesser extent) the encoding classifier when it is trained on unrelated lists, we next aimed to link neural activity during encoding and retrieval to the organization of recall sequences. The most prominent difference in recall behavior between the two list types is the tendency to recall categorized lists in category clusters (Figure 2c). As described in the introduction, category clustering is a reliable marker of interactions between episodic and semantic memory systems. The tendency to cluster recalls of items that were studied nearby in time (e.g., in adjacent positions in the study list), on the other hand, is a general feature of episodic memory (Kahana, 1996). To establish the contributions of encoding and retrieval processes to these types of recall organization, we trained multivariate classifiers to distinguish recall within and between such clusters based on neural activity during encoding and retrieval periods. Specifically, for categorized lists, we trained classifiers to distinguish neural signals for words that were recalled immediately following the recall of another word in the same category (i.e., within-category recall) from those that were recalled immediately following the recall of a word from a different category (i.e., between-category recall; see Figure 1). Likewise for unrelated lists, we trained classifiers to distinguish brain activity for words that were recalled immediately following the recall of an adjacently presented item (i.e., near recall) from those that followed the recall of an item that was presented further away in the study list (i.e., far recall; see Figure 1). The fact that we presented pairs of items from the same category in categorized lists partially confounded temporal distance in the study list with category membership. Our cross-decoding approach, however, is able to determine to what extent each classifier is specific to the categorical or temporal clustering it has been trained to identify.

Figure 4b shows the ROC functions for decoding and cross-decoding performance for classifiers trained to predict categorical or temporal clustering on the basis of neural activity during encoding and retrieval periods. Decoding performance was significantly above chance for all classifiers (t[59] = 2.5–11.93, SE =0.01–0.03, p ≤ .02) with the exception of the temporal clustering retrieval classifier (t[59] = 0.41, SE = 0.02, p = .68; Figure 4b (iv)). Additionally, decoding performance for the categorical clustering retrieval classifier was higher than that for the corresponding encoding classifier (t[59] = 6.65, SE = 0.03, p < .001) with no significant difference in decoding performance as a function of task phase for the temporal clustering classifiers (t[59] = 1.50, SE = 0.03, p = .13). A 2 (classifier type, predicting temporal or categorical clustering) × 2 (task phase; i.e., encoding vs. retrieval) repeated measures ANOVA exhibited significant main effects of classifier type (F[1, 59] = 10.72, MSE = 0.02, p = .002) and task phase (F[1, 59] = 4.50, MSE = 0.01, p = .04), as well as a significant interaction (F[1, 59] = 21.95, MSE = 0.01, p < .001). In line with our previous result that retrieval classifiers are specifically sensitive to the semantic structure of categorized lists, these findings particularly implicate retrieval processes in the production of categorical clusters in episodic recall.

Our cross-decoding approach further highlights the specific role of retrieval processes in producing categorically clustered recall sequences. Figure 4b shows the ΔAUC values for the clustering classifiers alongside the corresponding decoding and cross-decoding ROC functions. Only the retrieval classifier trained to identify categorical clustering exhibited significantly better decoding than cross-decoding performance (t[59] = 7.07, SE = 0.01, p < .001; Figure 4b (ii)). We performed a 2 (classifier type) × 2 (task phase) repeated measures ANOVA on the ΔAUC values and found a nonsignificant effect of task phase (F[1, 59] = 0.08, MSE = 0.03, p = .77), a significant effect of classifier type (F[1, 59] = 7.59, MSE = 0.02, p = .007), and a significant interaction between the two (F[1, 59] = 15.46, MSE = 0.02, p < .001). A post hoc test confirmed that the categorical clustering classifier was more sensitive to the semantic structure of the memoranda at retrieval than at encoding (t[59] = 3.83, SE = 0.02, p < .001), an effect which appears to be mostly driven by the superior decoding performance for the retrieval classifier described above. Indeed, given that the temporal clustering retrieval classifier’s decoding performance was not above chance, we would not expect the cross-decoding performance of the categorical clustering retrieval classifier to exceed chance. While encoding and retrieval processes clearly both contribute to category clustering, the categorical clustering retrieval classifier’s particularly high decoding performance corroborates our earlier finding that only processes at retrieval uniquely predict semantic clustering of episodic memory.³

Discussion

Prior knowledge about the world has profound effects upon how we encode and retrieve experiences, leading us to organize episodic memories along semantic dimensions (Bower et al., 1969; Bousfield, 1953). The relative contributions of encoding and retrieval processes that specifically drive this interaction between episodic and semantic memory systems, however, are poorly understood. By directly comparing recall performance and associated neural activity during encoding and retrieval periods for lists of categorized and unrelated words, we characterized how semantic structure of memoranda affects encoding and retrieval of events. Multivariate classifiers that predicted encoding and retrieval success generalized across categorized and unrelated lists (Figure 4a), identifying shared processes that support episodic memory irrespective of semantic content. By contrasting decoding and cross-decoding performance, we also found that specific processes support successful retrieval from categorized lists (Figure 4a, ii), suggesting that retrieval mechanisms are critical for producing semantically organized recall sequences. Our temporal and categorical clustering analyses supported this view: Neural signals during retrieval contained significantly more information about category clustering than corresponding signals during encoding (Figure 4b, i–ii). These classifiers indexed processes that were specific to categorical clustering; attempts to predict temporal clustering were less successful, even for classifiers trained to detect temporal clustering (i.e., not just for cross-decoding, but also for decoding classifiers; Figure 4b, iv). Taken together, these findings suggest that the semantic structure of the memoranda affects recall performance mainly through changes in retrieval (rather than encoding) processes.

Our use of multivariate classification to estimate cognitive states during encoding and retrieval allowed us to overcome a central challenge to understanding human memory: even though encoding and retrieval processes are both critical determinants of performance in memory tests, only their joint effects are observable in overt behavior. By quantifying neural signals that predict recall performance and organization, we were able to effectively track associated encoding and retrieval processes as they occurred. Previous studies of the relation between episodic and semantic memory systems (Greenberg et al., 2009; Vargha-Khadem et al., 1997; Wingfield et al., 1998), especially those attempting to resolve the relative contributions of encoding and retrieval processes (Gershberg & Shimamura, 1995; Greenberg & Verfaellie, 2010; Johnston, Griffith, & Wagstaff, 1972; Park, Smith, Dudley, & Lafronza, 1989), have yielded equivocal results. Our finding that neural signals at both encoding and retrieval predict recall performance and organization may help explain why different studies have identified either encoding or retrieval processes as responsible for the semantic organization of episodic memory. This finding is also consistent with previous neuroimaging work showing a high degree of overlap in brain regions engaged during tasks probing episodic and semantic memory (Burianova, McIntosh, & Grady, 2010) as well as the recruitment of episodic information even in putatively semantic tasks (e.g., category fluency; Ryan, Cox, Hayes, & Nadel, 2008; Sheldon & Moscovitch, 2012; West-macott & Moscovitch, 2003; see Moscovitch, Nadel, Winocur, Gilboa, & Rosenbaum, 2006, for a review). In line with these results, our findings contribute to the growing evidence for a general system that subserves memory function irrespective of task demands. Our separate identification of neural signals indexing retrieval processes that are specifically tuned to the semantic structure of the memoranda highlights the benefit of distinguishing processes associated with general memory function from those specifically responsible for categorical organization of recall.

Theories on the relation between episodic and semantic memory typically focus on the question of how semantic knowledge arises from individual episodes (Howard, Shankar, & Jagadisan, 2011; Mack, Love, & Preston, 2017; McClelland, McNaughton, & O’Reilly, 1995; Nelson & Shiffrin, 2013). Our work addresses the flip-side of this question: How does established semantic knowledge affect memory for specific events? In our study, categorized lists led to improved rates of recall and categorical clustering of recall sequences. There are at least three common approaches to modeling such effects: they can result from overlapping representations of individual memory traces (Abbott, Austerweil, & Griffiths, 2015; Katkov, Romani, & Tsodyks, 2017), processes at encoding that activate memory representations that are related to studied items (Collins & Loftus, 1975; Cooke, Durso, & Schvaneveldt, 1986; Underwood, 1965; Watkins & Gardiner, 1979), or adaptive retrieval mechanisms that differentially make use of categorical or contextual information as needed (Morton & Polyn, 2016; Patterson et al., 1971; Shiffrin, 1970). Below we consider the ability of each of these approaches to explain our data, even though they are clearly related (e.g., encoding and retrieval processes can leverage similarities in the representation of memory traces as discussed below).

The idea that the similarity between memory traces drives recall dynamics can explain basic patterns of behavior in free recall (Katkov et al., 2017) and semantic fluency tasks (Abbott et al., 2015). Contemporary models of free recall assume that each encoded item is associated with context information that reflects semantic information and recent history (Lohnas, Polyn, & Kahana, 2015; Polyn, Norman, & Kahana, 2009). Because context serves as a cue during retrieval, associating each item with such context information can result in categorically structured recall sequences and improved recall performance for categorized lists. While our analyses do not directly examine the neural activity evoked by individual items, previous electrophysiology studies have identified neural signals during encoding that reflected semantic information and were reinstated during retrieval, driving semantic organization of recall (Manning, Sperling, Sharan, Rosenberg, & Kahana, 2012; Morton et al., 2013). Our finding of neural signals during retrieval that specifically predict category clustering, however, suggest that additional retrieval processes guide the organization of recall sequences.

It seems natural to assume that the details of how items are represented in memory should also affect encoding. Spreading activation (Collins & Loftus, 1975), for example, refers to the idea that processing of any item also activates nearby items in a semantic network. This mechanism can explain a range of findings in semantic priming (Meyer & Schvaneveldt, 1971, 1976) and recall (Cooke et al., 1986) tasks. Similarly, a classic explanation for false memories of semantic associates to studied items is an implicit associate response during encoding, which creates a representation of the semantic associate that is later retrieved, just like memories for studied items (Underwood, 1965). To the extent that encoding processes adapt to the semantic structure of the memoranda in ways that result in differential recall performance or organization, we would have expected associated neural signals to specifically predict category clustering. Instead, neural signals during encoding periods generalized across different list types, suggesting that encoding processes are more sensitive to features of individual items rather than the semantic structure of a study list.

Despite the success of theories relying on semantic networks and encoding processes to account for a wide range of data, several theorists consider flexible retrieval mechanisms necessary to explain human recall performance. Modern theories of false memories, for example, invoke both encoding and retrieval processes (Meade, Watson, Balota, & Roediger, 2007; Roediger, Balota, & Watson, 2001), and some accounts of semantic fluency conceptualize retrieval from semantic memory as a dynamic, controlled process rather than a random walk on a semantic network (Hills, Jones, & Todd, 2012; Jones, Hills, & Todd, 2015). Various memory tests require the match of a probe item to memory. In these cases responses to new probe stimuli are often affected by the experience with other stimuli. Associated theories generally explain such context effects either by assuming changes in the mapping between the match signal and responses (i.e., a response bias) or by assuming that the match signal itself adapts to the context (an explanation akin to the flexible retrieval processes for which we present evidence here). For example, a range of short-term priming experiments use a perceptual identification task requiring the match of a probe to a briefly presented target while ignoring primes presented in the vicinity of the target (Huber, Shiffrin, Lyle, & Ruys, 2001; Pecher, Zeelenberg, & Raaijmakers, 2002; Weidemann, Huber, & Shiffrin, 2005, 2008). Some of these studies have found strong effects of prime-target contingencies (e.g., how likely the prime is to match the target). Accordingly, trials that match in all aspects except for the history of previous trials (and thus the learned prime-target contingencies) can produce large differences in performance (Pecher et al., 2002; Weide-mann, Huber, & Shiffrin, 2008). Whereas such effects are generally assumed to be the result of response biases (Pecher et al., 2002), there is evidence that the match signal adapts to these contingencies (Weidemann et al., 2008). Likewise, in tests of recognition memory requiring the classification of a probe stimulus as either previously studied (i.e., a target) or not (i.e., a lure), even responses to lures are affected by how targets are studied. Experimentally manipulating strength of encoding (e.g., by increasing study time; Ratcliff, Clark, & Shiffrin, 1990; Stretch & Wixted, 1998) not only increases hit rates, but also decreases the likelihood of false alarms to lures (i.e., a strength-based mirror effect; Glanzer & Adams, 1985). Whereas some accounts for this effect attribute it to response biases (Starns, White, & Ratcliff, 2012; Stretch & Wixted, 1998), it has been successfully modeled by assuming that the match signal aggregates evidence across the full set of targets, leading to a poorer match for lures when they are compared to more strongly encoded targets (Criss, 2010; Criss & McClelland, 2006; Kiliç, Criss, Malmberg, & Shiffrin, 2017; Shiffrin, Ratcliff, & Clark, 1990; Shiffrin & Steyvers, 1997). Our results provide converging evidence that the structure of encoded material can have profound effects that are specific to processes involved in probing memory (i.e., retrieval, rather than encoding, processes).

Whereas neural activity during retrieval predicted categorical clustering best, neural activity during the encoding of unrelated lists best predicted temporal clustering. Previous studies have interpreted increased activity within a putative “core memory network” during encoding as reflecting item to context associations, a central process in several models of episodic memory (Long & Kahana, 2015). Our ability to decode subsequent temporal clustering on the basis of encoding activity provides converging evidence for such encoding processes. Our attempts to decode temporal clustering on the basis of neural activity during retrieval failed, however, despite robust temporal clustering in the recall sequences for unrelated lists (see Kragel et al., 2017, for measures of temporal clustering in a superset of these data). Given the inherent limitations in our ability to intracranially record brain activity in humans (Parvizi & Kastner, 2018) and our focus on spectral power for a small set of frequencies as sole indices for this activity, it is likely that we missed relevant neural signals that could predict temporal clustering during retrieval. Nevertheless, the specificity of the categorical clustering classifier at retrieval implicates distinct processes in the semantic and episodic organization of memories.

The question about the relative contributions of encoding and retrieval processes to our ability to remember is often difficult to answer, because changes in encoding can be counteracted by changes in retrieval and vice versa. Using a multivariate cross-decoding approach, we tackled this problem by linking neural activity to encoding and retrieval processes that either generalized across different list types or specifically predicted recall performance and organization for categorized lists. We identified retrieval signals that specifically predicted categorical recall organization, whereas signals at encoding did not. The lack of encoding signals that specifically predict categorical recall organization is particularly striking, given that our experimental setting imposed a clear categorical structure at the time of encoding and allowed for the full anticipation of the memory test. These findings provide converging evidence for the importance of flexible retrieval mechanisms in models of human memory and challenge the notion that similarity of long-term memory representations are the primary drivers of recall performance (Katkov et al., 2017). Indeed, our results suggest that flexible retrieval is a basic feature of human memory that adapts memory search to task demands even when these are readily apparent at encoding.