Cortical Reinstatement Mediates the Relationship Between Content-Specific Encoding Activity and Subsequent Recollection Decisions

Abstract

Episodic recollection entails the conscious remembrance of event details associated with previously encountered stimuli. Recollection depends on both the establishment of cortical representations of event features during stimulus encoding and the cortical reinstatement of these representations at retrieval. Here, we used multivoxel pattern analyses of functional magnetic resonance imaging data to examine how cortical and hippocampal activity at encoding and retrieval drive recollective memory decisions. During encoding, words were associated with face or scene source contexts. At retrieval, subjects were cued to recollect the source associate of each presented word. Neurally derived estimates of encoding strength and pattern reinstatement in occipitotemporal cortex were computed for each encoding and retrieval trial, respectively. Analyses demonstrated that (1) cortical encoding strength predicted subsequent memory accuracy and reaction time, (2) encoding strength predicted encoding-phase hippocampal activity, and (3) encoding strength and retrieval-phase hippocampal activity predicted the magnitude of cortical reinstatement. Path analyses further indicated that cortical reinstatement partially mediated both the effect of cortical encoding strength and the effect of retrieval-phase hippocampal activity on subsequent source memory performance. Taken together, these results indicate that memory-guided decisions are driven in part by a pathway leading from hippocampally linked cortical encoding of event attributes to hippocampally linked cortical reinstatement at retrieval.

Introduction

When confronted with environmental stimuli, we often make decisions that are guided by the recollection of contextual and other associated details surrounding previous encounters with the stimuli. Much neuroimaging research is concerned with the neural mechanisms that support such recollective decisions (Davachi 2006; Diana et al. 2007; Mitchell and Johnson 2009; Danker and Anderson 2010; Rissman and Wagner 2012). In particular, 3 sets of findings concerning the functional neurobiology of recollection are of relevance to the present study. One examines how stronger content-specific cortical representations at encoding are correlated with more accurate decisions at retrieval (e.g., Kirchhoff et al. 2000; Davachi et al. 2003; Uncapher et al. 2006; Otten et al. 2007; Prince et al. 2009; Kuhl, Rissman, et al. 2012), suggesting that recollective decisions are partially dependent on the initial neural representation of associated details or event attributes. Another examines how content-specific cortical representations at encoding are reinstated at retrieval (e.g., Nyberg et al. 2000; Wheeler et al. 2000; Kahn et al. 2004; Polyn et al. 2005; Woodruff et al. 2005; Wheeler et al. 2006), suggesting that recollective decisions are also dependent on the reactivation of cortical patterns that initially represented associated details. A third examines how activity in the hippocampus supports both the encoding (e.g., Davachi et al. 2003; Kirwan and Stark 2004; Ranganath et al. 2004; Uncapher 2005) and retrieval (e.g., Eldridge et al. 2000; Kirwan and Stark 2004; Yonelinas et al. 2005; Montaldi et al. 2006) of associations. Although these 3 phenomena have been independently demonstrated, limited work has examined the relationship between the strength of neural representations at encoding, reinstatement of neural representations at retrieval, and memory performance. In the present study, we used functional magnetic resonance imaging (fMRI) and multivoxel pattern analyses (MVPA) (Haxby et al. 2001; for reviews, see Norman et al. 2006; Rissman and Wagner 2012; Tong and Pratte 2012) to test the hypothesis that cortical reinstatement at retrieval mediates the relationship between the establishment of cortical patterns at encoding and the later recollection of associated source (or contextual) information (operationalized as accurate responding on a 2-alternative forced-choice source memory task).

Numerous fMRI studies have used univariate analytic techniques to examine “subsequent memory effects” (SMEs), wherein the magnitude of encoding-phase blood oxygen level-dependent (BOLD) activity predicts later remembering or forgetting (e.g., Brewer et al. 1998; Wagner et al. 1998; Fernández et al. 1999; Henson et al. 1999; for reviews, see Paller and Wagner 2002; Blumenfeld and Ranganath 2007; Uncapher and Wagner 2009; Kim 2011). Of particular significance to the present experiment are studies that demonstrated content- or source-specific SMEs, wherein activation in regions thought to support the representation of specific classes of stimuli (e.g., Kirchhoff et al. 2000; Otten et al. 2007; Prince et al. 2009) or source features (e.g., Davachi et al. 2003; Uncapher et al. 2006; Staresina et al. 2011) was predictive of content- or source-specific subsequent memory. These SME studies suggest that stronger activity in cortical regions that selectively represent a specific stimulus or source category at encoding can lead to better memory performance at retrieval.

More recently, several neuroimaging studies have employed multivariate analysis strategies to examine how distributed patterns of trial-specific neural activity at encoding relate to later memory outcomes (Jenkins and Ranganath 2010; Xue et al. 2010; Watanabe et al. 2011; Kuhl, Rissman, et al. 2012; Zeithamova et al. 2012; LaRocque et al. 2013; Ritchey et al. forthcoming). Notably, Kuhl, Rissman, et al. (2012) found that, during the viewing of word-face or word-house pairs, classifier-derived estimates of the strength of face- or house-specific patterns of activity in temporal and prefrontal cortices were higher for pairs for which subjects could subsequently recollect that a face or house was previously associated with the word.

Univariate and multivariate methods have also been used to demonstrate the phenomenon of “cortical reinstatement” (reviewed in Mitchell and Johnson 2009; Danker and Anderson 2010; Schacter et al. 2012), whereby content-specific cortical activity at encoding is reinstated at retrieval (e.g., Nyberg et al. 2000; Wheeler et al. 2000; Kahn et al. 2004; Polyn et al. 2005; Woodruff et al. 2005; Wheeler et al. 2006; Johnson et al. 2009; Kuhl et al. 2010, 2011). Moreover, memory decisions accompanied by subjective reports of conscious recollective experience have been shown to be associated with stronger content-specific cortical activity at retrieval (e.g., Wheeler and Buckner 2004; Johnson and Rugg 2007), and classifier-derived estimates of the fidelity of cortical reinstatement have been observed to relate to retrieval performance (Johnson et al. 2009; McDuff et al. 2009; Kuhl et al. 2011; Kuhl, Rissman, et al. 2012; Staresina et al. 2012; Ritchey et al. in press). Collectively, these studies document a link between the strength or fidelity of cortical reinstatement and behavioral expressions of event recollection.

In addition to a relationship between cortical encoding and retrieval patterns and memory behavior, extensive data implicate the hippocampus as critical for event memory (e.g., Squire 1992; Cohen and Eichenbaum 1993; Squire et al. 2004). Several fMRI studies indicate that hippocampal activity during episodic encoding is greater for trials in which subjects encode associations in which specific event details or associations can be subsequently retrieved (e.g., Davachi et al. 2003; Kirwan and Stark 2004; Ranganath et al. 2004; Staresina et al. 2011). Additionally, hippocampal–cortical connectivity during encoding has been shown to be greater for subsequently retrieved stimuli (Ranganath et al. 2005; Gagnepain et al. 2011; Schott et al. 2011). During retrieval, hippocampal activity is greater both in contrasts of correct versus incorrect associative retrieval (e.g., Eldridge et al. 2000; Cabeza et al. 2001; Dobbins et al. 2003; Chen et al. 2011) and contrasts of recognition with versus without recollection (e.g., Eldridge et al. 2000; Yonelinas et al. 2005; Montaldi et al. 2006; c.f., Wais et al. 2010; Smith et al. 2011). These studies suggest that, in conjunction with cortical representations of event content, the hippocampus is involved in event encoding and retrieval such that subjects can recollect past event or source details when presented a partial cue.

While prior studies have established that (1) content-specific cortical activity at encoding can predict later memory behavior, (2) cortical reinstatement is a neural component of acts of recollection, and (3) hippocampal activity at encoding and retrieval predicts memory expression, no study has tested the critical synthesizing hypothesis that the relationship between content-specific cortical encoding activity and later retrieval of that event content is mediated through the fidelity of cortical reinstatement at retrieval. To this end, the present study sought to bridge subsequent memory and cortical reinstatement effects, by employing MVPA to investigate (1) how content-specific patterns of activity at encoding are related to content-specific patterns at retrieval, and (2) how together these quantities predict behavioral variables related to retrieval decisions. While undergoing fMRI during incidental encoding and subsequent source retrieval, subjects performed a mental imagery-based source memory task. Specifically, during encoding, subjects encountered individual words and imagined either a corresponding face or scene for each word. Following encoding, subjects reencountered the studied words individually and engaged in a source retrieval task, wherein they were required to indicate whether they had performed face or scene imagery during encoding of the word. Because correct performance on this 2-alternative forced-choice task is dependent on successful recollection, correct versus incorrect source judgments were taken to index retrieval with versus without recollection (while acknowledging that some trials may be correct due to guessing). MVPA analyses of data from both the encoding and retrieval phases provided continuous, trial-specific measures of encoding strength and cortical reinstatement.

Using this design and analytic approach, we provide new insights into the neurocognitive processes subserving episodic memory. First, we demonstrate that a neurally derived measure of content-specific cortical encoding strength varies with encoding-phase hippocampal activity and with later source retrieval accuracy and decision time. Secondly, we demonstrate that this neural measure of encoding strength scales positively with cortical reinstatement at retrieval, suggesting that the magnitude of cortical reinstatement at retrieval is dependent in part on the strength of established cortical patterns at encoding. Thirdly, we demonstrate that retrieval-phase hippocampal activity scales with this cortical reinstatement measure. Finally, and critically, and we demonstrate that the relationship of both cortical encoding strength and retrieval-phase hippocampal activity to retrieval accuracy is partially mediated by the effect of cortical reinstatement. Taken together, these results indicate that recollection-dependent memory decisions are driven by a pathway leading from content-specific representational strength at encoding through hippocampally linked content-specific cortical reinstatement.

Materials and Methods

Subjects

Thirty-three right-handed subjects were recruited from the Stanford University community. Written informed consent was obtained in accordance with procedures approved by the institutional review board at Stanford University; subjects received monetary compensation ($20/h). Because many of the correct trials were likely to be guesses for subjects demonstrating poor retrieval performance, the data from 5 subjects were excluded due to poor performance (<65% correct source retrieval; performance range of excluded subjects was 49–62%). Additionally, because this study depends on good estimates of the strength of source representations at encoding, data from one subject were excluded due to poor encoding task performance (mental imagery was subjectively reported as successful on <75% of trials). While arbitrary, these exclusionary criteria were determined before analyses of the questions of interest were conducted. Data from 27 subjects were submitted to all analyses (15 females; mean age = 20.1 years, range = 18–24 years).

Behavioral Procedures

During an incidental encoding phase, subjects were cued to engage in 1 of 2 mental imagery generation tasks for each of a set of 180 adjectives (Fig. 1). At the onset of each 14-s encoding trial, a source/task cue—either “Scene” or “Person”—was centrally presented for 500 ms, followed by 500 ms of a blank screen. Subsequently, an adjective (e.g., “loud”) was centrally presented for 750 ms and subjects were given an additional 3250 ms to imagine either a scene or a famous face that was compatible with the adjective, depending on the source task. After imagery, subjects were given 2 s to indicate whether or not they were able to successfully generate any details related to an appropriate scene or face. Subjects made this rating by pressing a button with either their left or right pointer finger. This rating was used to exclude encoding trials on which the subject could not properly perform the task (a mean of 10.3% of face and 5.8% of scene trials were excluded from the analysis). Adjectives were presented over 6 consecutive 7.2-min scanning runs; during each run, 15 adjectives were encoded via face imagery and 15 via scene imagery. To permit single-trial measurement of the observed hemodynamic response, a slow event-related design was implemented by including a 7-s fixation period following offset of the rating period. Condition order and button-press mapping were randomized; source assignment of words was counterbalanced across subjects.

Figure 1.

Schematic of behavioral tasks. (a) During encoding, subjects were presented with a source/task cue (“Person” or “Scene”) followed by an adjective, and instructed to imagine either the face of a famous person or a scene described by the adjective. (b) At retrieval, subjects viewed adjectives previously presented at the encoding phase and made a response indicating whether they remembered associating each item with a face or scene.

During the retrieval phase, subjects performed a 2-alternative forced-choice source recognition task on all of the words presented during the encoding phase. On each 10-s trial, a studied word was presented for 3 s, and subjects made a button with either their right or left index finger to indicate which imagery task (face or scene) they remembered having previously performed on that word. A 7-s fixation period followed word offset. Across six 4.9-min retrieval scans, subjects made source decisions for 168 adjectives; during each scan, 14 words had been encoded via face imagery and 14 via scene imagery. The remaining 12 studied adjectives, which corresponded to the first and last item of each encoding run, were used in a practice version of the retrieval task that immediately preceded the retrieval scans. Condition order was randomized, and button-press mapping was counterbalanced across subjects. The average lag between a word's study and test presentations was 49.7 min (range = 44.9–58.1 min).

fMRI Acquisition

Functional data were acquired on a 3-T Signa MRI system (GE Medical Systems) using a T₂*-weighted, gradient-echo spiral in–out sequence (repetition time TR = 2 s, echo time = 30 ms, 30 axial slices, 3.3 × 3.3 × 4 mm spatial resolution). Six initial volumes from each run were discarded to allow for T₁ equilibration. High-resolution, T₁-weighted spoiled gradient recalled echo structural images were collected for anatomical visualization. Visual stimuli were projected onto a screen and viewed through a mirror; responses were collected through a magnet-compatible button box.

fMRI Analysis

Data were preprocessed using SPM5 (http://www.fil.ion.ucl.ac.uk/spm, last accessed August 7, 2009). Functional images were corrected for differences in slice acquisition timing, followed by motion correction using sinc interpolation, and coregistration of structural images to each subject's mean functional image. The structural images were then segmented, and gray matter images were spatially normalized to a gray matter template image in Montreal Neurological Institute (MNI) stereotactic space. The resulting nonlinear transformation parameters were applied to all structural and functional images. Functional images were resampled into 3-mm isotropic voxels and smoothed with a 6-mm full-width at half maximum (FWHM) kernel.

Univariate Conjunction Analysis of Encoding/Retrieval Overlap

Separate statistical models were created for the encoding (imagery) and retrieval (source decision) phases, under the assumptions of the general linear model (GLM). The encoding model included event regressors corresponding to face and scene imagery trials for which subjects reported successfully performing the imagery task; imagery failure and no-response trials were coded as a condition of noninterest. The retrieval model included 4 event regressors of interest, corresponding to correct and incorrect face and scene source memory trials for which subjects had reported successful imagery during the encoding phase; retrieval trials for which imagery was unsuccessful at encoding or for which there was no response at encoding or retrieval were coded as a condition of noninterest. Session effects and individual volumes labeled as artifactual by the criteria of having TR-to-TR motion of >0.5 mm or global signal intensity values of >4 SD from the mean were modeled as nuisance covariates. A median 0.9% of encoding-phase volumes and 1.4% of retrieval-phase volumes met these criteria.

One-sample t-tests were used to identify voxels for which linear contrasts of GLM parameter estimates reliably differed from zero across subjects. To identify voxels at the group level that differed both during face versus scene encoding/imagery and during correct face versus scene source retrieval decisions, we performed a random-effects conjunction analysis across the encoding and retrieval data (P < 0.005, 5-voxel extent for each contrast). To correct for multiple comparisons, Monte Carlo simulations across 10 000 randomly generated data sets were run using AlphaSim (http://afni.nimh.nih.gov, last accessed Aug 13, 2010), using our full-brain anatomical mask to define the search space and assuming the empirical smoothness observed in our data (11.2 × 11.5 × 10.9 mm FWHM). This analysis revealed that a probability of false detection P < 0.05 corresponds to a voxel height threshold of P < 0.005 coupled with a spatial extent threshold of k≥ 40 voxels; these are the statistical and spatial thresholds at which data are reported. For visualization purposes, the conjunction maps are displayed on a mean structural image created from the normalized T₁ images from each of the 27 subjects.

MVPA Approach

Data preprocessing was performed using the Princeton MVPA Toolbox (http://www.pni.princeton.edu/mvpa/, last accessed Nov 1, 2012) and custom Matlab (Mathworks, Natick, MA, USA) scripts. Following realignment and spatial normalization, session-specific functional time series from each voxel were linearly detrended. Since we were interested in using MVPA to index the degree to which neural representations of stimulus features at encoding were reinstated at retrieval (i.e., the “fidelity of reinstatement”; Kuhl et al. 2011), data from occipital and temporal cortices, exclusive of the hippocampi, were used to train and test the classifier. The occipitotemporal mask (17 394 voxels) was created by drawing a mask over occipital and temporal cortices of the group-averaged anatomical brain and excluding the hippocampal mask of the Automated Anatomical Labeling toolbox (Tzourio-Mazoyer et al. 2002).

Trial-specific encoding and retrieval patterns in bilateral occipitotemporal cortex were computed by averaging the TRs corresponding to 6–10 s post-task cue onset for encoding trials, and 4–8 s postprobe onset for retrieval trials. These were the TRs where the amplitudes of the mean hemodynamic response function of regions that demonstrated source-specific activity were greatest (Supplementary Fig. 1). Analysis of variance-based feature selection was employed to select the 1000 voxels within the occipitotemporal cortex for which activity maximally differentiated the classes within the training set of each classification. Prior to classification, activity of each voxel across patterns was z-scored. Classification was performed using a logistic regression classifier with L2 regression regularization (penalization parameter = 0.01 for all classifications), as instantiated in the LIBLINEAR classification library (http://www.csie.ntu.edu.tw/~cjlin/liblinear/, last accessed Dec 17, 2012). Prior to each iteration of classifier training, the data were subsampled to ensure equal numbers of face and scene trials.

Encoding task classification was accomplished by randomly generated 10-fold cross-validation of occipitotemporal data from the encoding phase and, as described below, the value of probabilistic classifier output was further treated as a measure of trial-specific “encoding strength” (i.e., the degree to which the encoding pattern for a trial resembled the pattern associated with face or with scene imagery). Subsequently, reinstatement classification was accomplished by training a new classifier on all encoding-phase data and then testing on all retrieval-phase data; the value of probabilistic classifier output was further treated as a measure of the “fidelity of reinstatement” (i.e., the degree to which the retrieval pattern in occipitotemporal cortex resembled the encoding pattern associated with face or with scene imagery).

Trial-wise classifier accuracy was computed based on the value of probabilistic classifier output (ranging from 0, corresponding to a perfectly certain face classification, to 1, corresponding to a perfectly certain scene classification) relative to the chance value of 0.5. Classifier performance within each subject was calculated independently for faces and scenes; reported performance is the mean across these 2 values. The significance of classifier performance was calculated using permutation testing. First, within each subject, a null distribution of accuracy was generated, by shuffling the labels of the guesses over 10 000 iterations, and calculating the resulting accuracy for each iteration. A difference score was calculated by subtracting the median permutated classifier accuracy score from the observed classifier accuracy score. A t-statistic across subjects was computed by performing a 1-sample t-test of these difference scores across all subjects, against a null hypothesis of zero.

Encoding strength and fidelity of reinstatement were computed by taking the logit (log odds) of the trial-wise probabilistic classifier output for the encoding task and reinstatement classifications, respectively. Encoding strength and reinstatement are signed in the direction of the correct source for a given trial, such that, regardless of whether the trial was a face or scene trial, the evidence is positive when the classifier guessed correctly, and negative when the classifier guessed incorrectly. These variables are thus neutral with respect to which source class (face or scene) was retrieved.

To examine how the continuous measures of encoding strength and reinstatement scaled with decision-relevant behavioral variables—source retrieval accuracy and reaction time (RT)—we conducted mixed linear and logistic regression analyses involving these trial-wise neural measures of mnemonic evidence. Analyses were performed using mixed-effect GLMs as implemented by the lme4 statistical package (http://cran.r-project.org/web/packages/lme4/, last accessed Mar 2, 2012), in the R statistical environment (http://www.r-project.org/, last accessed Mar 2, 2012). A random intercept term modeling the mean subject-specific outcome value, along with a random slope term modeling the subject-specific effect of the independent variable of interest on the outcome, was included in each model. The approach of specifying random slopes and intercepts within mixed models allows for analysis of trial-wise variables across subjects, while also accounting for (1) subject-specific baselines in the dependent measure, and (2) subject-specific linear relationships between the independent variable of interest and the dependent variable.

Statistical models included relevant fixed-effect independent variables of noninterest. To control for the effect of retrieval accuracy status (correct or incorrect), which was moderately correlated with both encoding (mean r = 0.08) and reinstatement strength (mean r = 0.18), retrieval accuracy was included as a regressor of noninterest in all models, with the exception of models in which (1) only correct or incorrect trials were analyzed, or (2) retrieval accuracy was itself the dependent variable. We thus ensured that effects of interest could not only be attributed just to a trial being correct or incorrect, but rather could be attributed to the continuous estimates of encoding and reinstatement strength after controlling for retrieval accuracy. To control for the effect of stimulus content (face or scene) on various analyses of interest, stimulus category was included in all models as a regressor of noninterest. To test whether the effects of interest differed across face and scene categories, all reported mixed-effects analyses were repeated, testing for the significance of the relationship between the dependent variable and an independent variable modeling the interaction between the variable of interest and stimulus category. Regression equations are explicitly given in Supplementary Materials. The significance of effects within mixed-model regressions were obtained using log-likelihood ratio tests, resulting in χ² values and corresponding P-values.

In all reported statistical tests, classifier-derived variables were treated as continuous. For the purpose of visualization of data across subjects, however, when classifier-derived measures of encoding and reinstatement strength were used as independent variables, data within each subject were binned into quintiles based on the value of the independent variable of interest. The values of the dependent variable within each quintile were averaged across subjects. Error bars on these plots indicate ±within-subject error (Loftus and Masson 1994). Additionally, for each linear and logistic regression plot, subject-specific scatter and logistic histogram plots are included in the Supplementary Results (Supplementary Figs 3–10).

Mean Classifier Beta Map

To visualize which voxels in the occipitotemporal cortex most consistently drive a classifier to “face” or “scene” outcomes, a mean classifier beta map was computed. After the classifier was trained on the full set of encoding data, classifier beta values were assigned to each of the 1000 voxels used in the classifier. The mean beta maps across all subjects were then computed. For visualization purposes, these maps were arbitrarily thresholded at |β| > 2. Mean beta values that drove classification toward face outcomes were greater in magnitude in the positive direction; values that drove classification toward scene outcomes were greater in magnitude in the negative direction. Mean beta values that were small in magnitude were assigned to voxels that were less informative to classification, and voxels that were more frequently excluded from classification due to feature selection. For display, maps were projected onto inflated fiducial brains with the use of multifiducial mapping in Caret (http://brainvis.wustl.edu/, last accessed Jan 5, 2011); the extent of activation corresponds to the mean extent observed within template subjects from the PALS atlas.

Hippocampal Signal Analysis

To perform analyses linking hippocampal activity at encoding and retrieval to trial-wise encoding strength and cortical reinstatement measures, we extracted trial-wise hippocampal activity for each encoding and retrieval trial. Left and right hippocampal regions of interest (ROIs) were anatomically defined with the Anatomical Automatic Labeling toolbox for SPM (Tzourio-Mazoyer et al. 2002). Left and right hippocampal signal intensity values for each encoding and retrieval trial were extracted from each trial-wise pattern used in the MVPA approach described above. To remove effects correlated with the global signal, a linear regression of raw hippocampal signal against mean signal in the entire occipitotemporal lobe mask was run within each ROI, within each subject. The residuals from this analysis, representing the hippocampal activity controlling for the global mean signal, were used in all hippocampal analyses (note: analyses of hippocampal signal performed without this global signal residualization yielded results similar to that with the residualized hippocampal signal; see Supplementary Results).

Because left and right hippocampi were not hypothesized to have divergent functions, and because activity in these ROIs was highly correlated (mean r = 0.78 at encoding and 0.77 at retrieval), analyses combined activity across the hippocampal ROIs. This bilateral hippocampal signal was computed by first z-scoring each individual ROI within each subject, and then computing the mean across ROIs.

Multiple Comparisons Correction for Regression-Derived Effects of Interest

Regressions linking neurally derived variables were run among 6 variables of interest: Cortical encoding strength, encoding-phase hippocampal activity, cortical reinstatement, retrieval-phase hippocampal activity, RT, and accuracy. A priori hypotheses specified relationships among these variables (see Path Analysis). Nevertheless, because these variables are densely interconnected, any pair of these variables might have been related to each other, either directly or indirectly (through intermediate variables or through variables that relate independently to both members of the pair), so a significant relationship among any pair of variables would be of interest. Because of this, our hypothesis space spanned comparisons over all pair-wise relationships between 2 neurally derived variables, or between a neurally derived variable and a behavioral variable. Since there are 14 such pair-wise relationships, we performed multiple comparisons correction accounting for 14 comparisons within this family of variables (Supplementary Table 1), using false discovery rate (Benjamini and Hochberg 1995).

Path Analysis

To examine how our neural and behavioral variables at encoding and retrieval interrelated in a single statistical framework, we performed a path analysis. A generalization of regression methods to account for sequential effects, path analysis can be used to determine the strength of both “direct effects” between variables and “indirect effects” in which the relationship between variables is mediated by their relationship with intermediate variables.

Separate path structures were created for the dependent variable of memory accuracy (for which all trials were included; reported in Results section) and for that of retrieval RT (for which only correct trials were included; reported in Supplementary Results). The path structure was created by specifying a unidirectional path between each pair of variables shown to significantly covary with each other. Note that the directions of these path arrows are not meant to convey a strong causal claim about the relationship among variables. Rather, as with any regression analysis, path analysis requires one variable to be the dependent variable, and another set of variables to be the independent variables. Influenced by the sequence of events through time and by previous theoretical and empirical research, the direction of path arrows mirrored the directionality of all presented regression analyses, which were chosen based on the following principles: (1) encoding-phase variables preceded retrieval-phase variables; (2) neural variables at retrieval preceded behavioral output variables, given the hypothesis that hippocampal and cortical activity lead to response implementation; (3) cortical encoding strength preceded encoding-phase hippocampal activity, a choice made to fit within the theoretical framework in which, during perception, information propagates from cortex to hippocampus (Squire and Zola-Morgan 1991; Rolls 1996; Mishkin et al. 1997, 1998; Eichenbaum 2000); and (4) retrieval-phase hippocampal activity preceded cortical reinstatement, a choice made to fit with the theoretical framework that pattern completion in the medial temporal lobes leads to reactivation of associate representations in the cortex (Marr 1971; McClelland et al. 1995; McClelland and Goddard 1996; Rolls and Treves 1998; Wallenstein et al. 1998).

Prior to path estimation, all variables were z-scored, except for memory accuracy, which was treated as a binary categorical variable. Path coefficients are thus standardized and estimate the standard deviation change in the outcome variable per standard deviation change in the predictor variable, with the exception of path coefficients leading to memory accuracy, which reflect the change in the logit of memory accuracy per standard deviation change in the given predictor variable.

The goodness of fit of the path structure was computed using a directional-separation test (Pearl and Verma 1987; Shipley 2000) generalized for mixed-effects models (Shipley 2009). For each of k pairs of nondirectly connected variables X_i and X_j, the probability that X_i and X_j are independent, conditional on the set of variables with paths directly leading to either member of the pair, was calculated using a mixed-effects regression. These k probabilities were then combined with Fisher's combined probability test and compared with a χ² distribution with 2k degrees of freedom. A resulting probability value greater than an alpha of 0.1 signifies that we may retain the model. Optimality of the model was assessed by testing whether the deletion of any path present in the model resulted in a significantly worse model fit, and whether the addition of any path not present in the model resulted in a significantly better model fit. Comparison of fits between each set of 2 nested models was performed by assessing the significance of the difference in the nested model fits using a χ² distribution.

Path coefficients were estimated by decomposing the path structure into a set of regressions in which each variable was predicted by the set of all variables with a path leading toward it. As with the mixed-model regressions discussed earlier, regressions used in path analysis included random slopes and random intercepts of interest, and included stimulus category and, where appropriate, memory accuracy, as covariates of no interest.

The coefficient of the indirect path across direct paths A and B was computed as a× b, where a and b represent the direct effects for each corresponding path. The significance of this indirect effect was calculated with bootstrapping methods. A null distribution of the indirect effect a× b was calculated across 10 000 iterations of data sampled with replacement. Reported P-values constitute the proportion of this distribution greater than the null value of 0. The “index of mediation” [a× b× (σ_X/σ_Y)] is also reported (Preacher and Kelley 2011), providing a measure of effect size for the indirect effects.

Results

Behavioral Results

Subjects correctly remembered the source on 79.3% (SD = 8.77%) of retrieval trials. Mean retrieval decision RTs were faster on correct (mean = 1659 ms; SD = 174 ms) than on incorrect (mean = 1943 ms; SD = 234 ms) trials [t₍₂₆₎ = 7.22, P < 10^–4]. Accuracy did not differ across person versus scene trials [t₍₂₆₎ = 0.14, P > 0.5], but correct-trial RT was significantly faster for person (mean = 1622 ms, SD = 181 ms) versus scene (mean = 1712 ms, SD = 188 ms) imagery trials [t₍₂₆₎ = 3.01, P < 0.01].

fMRI-Derived Trial-Wise Quantification of Encoding Strength

We initially sought to quantify the strength of source encoding patterns from BOLD data in the bilateral occipitotemporal cortex on a trial-by-trial basis. To do so, we first conducted a 10-fold cross-validated multivariate pattern analysis on the fMRI data from the occipitotemporal cortex during encoding trials for which subjects reported correctly performing the source imagery task. Mean classifier accuracy in decoding the encoding task from occipitotemporal cortex (81.1%) was significantly above chance [t₍₂₆₎ = 18.4, P < 10⁻⁴]. Subsequently, we computed a continuous metric of cortical “source encoding strength” for each encoding trial, taking the log odds of the classifier's probabilistic estimate of the encoding source associated with each classification attempt. This procedure provided an estimate of the extent to which each encoding pattern resembled encoding patterns of one versus the other class, as such it can be construed as a measure of source-specific activity during encoding. Logistic regression revealed that a greater magnitude of encoding strength predicted a higher likelihood of classifier accuracy (Eq. 1, χ²(1) = 65.3, P < 10⁻⁴, Supplementary Fig. 2), providing validation that our continuous measure of encoding strength is tightly coupled with the extent to which a classifier can use the neural information to correctly determine the imagery task being performed at encoding. Across subjects, the occipitotemporal cortical features that significantly drove classification (1) toward faces included voxels in fusiform gyrus, and (2) toward scenes included voxels in parahippocampal and superior occipital gyri (Fig. 2a).

Figure 2.

Patterns of BOLD response in the occipitotemporal cortex during face and scene imagery can be used to generate a metric of encoding strength that is related to later source retrieval performance. (a) Mean classification beta map, thresholded at |β| > 2. Red voxels more consistently drive classifier output toward “face” classification, and green voxels more consistently drive classifier output towards “scene” classification. (b) Plot of the probability of correct source retrieval as a function of encoding strength. (c) Plot of encoding strength against RT for correct retrieval trials. Error bars indicate ±within-subject SEM.

The multivariate approach employed in the present data set is an extension of univariate approaches in which mean activity is extracted across ROIs found to code for one versus other sources. To ascertain the extent to which our measure of encoding strength was driven by such univariate amplitude differences (vs. patterns), a univariate analysis was run on the encoding-phase voxels submitted to classification (Supplementary Methods). For each classifier iteration, sets of face- and scene-responsive voxels were defined from the training data, and trial-wise mean signal amplitude within these regions were extracted from the testing data. Analyses revealed that the measure of encoding strength scaled positively with univariate amplitude in voxels responsive to the correct class (Eq. S3, χ²(1) = 57.8, P < 10⁻⁴) and negatively with univariate amplitude in voxels responsive to the incorrect class (Eq. S3, χ²(1) = 56.3, P < 10⁻⁴). Taken together, univariate amplitude effects accounted for approximately 63% of the variance in our measure of encoding strength. Similar effects were found for the measure of cortical reinstatement, for which univariate amplitude effects accounted for 47% of the variance (Supplementary Results). These results indicate that the multivariate measures of encoding strength and cortical reinstatement are highly related to coarse-scale univariate amplitude effects. However, the remaining variance in the multivariate measures that is not explained by the univariate measures may be due to contributions to this multivariate signal from more fine-scale patterns. Finally, an exploratory analysis was conducted to determine whether source-specific activity patterns were also present in the hippocampus at encoding, using trial-wise data from the hippocampus. Classification of source within the hippocampus was significantly above chance [mean accuracy = 62.3%; t₍₂₆₎ = 7.61, P < 10⁻⁴, Supplementary Results].

Encoding Strength Predicts Subsequent Recollection Decisions

Motivated by recent data indicating that encoding patterns elicited by the perception of faces versus scenes predict later cued recall (Kuhl, Bainbridge, et al. 2012), we next sought to determine whether the strength of encoding patterns in the bilateral occipitotemporal cortex during face versus scene imagery was predictive of later source recollection decisions and decision latencies. Consistent with this possibility, logistic regression revealed that stronger occipitotemporal encoding patterns, as indexed by the classifier, predicted a higher likelihood of accurate retrieval (Eq. 2, χ²(1) = 20.3, P < 10⁻⁴, Fig. 2b).

For encoding trials for which subjects subsequently recollected the correct face/scene source associate, greater encoding strength also predicted faster retrieval RTs (Eq. 3, χ²(1) = 5.23, P < 0.05, Fig. 2c). In contrast, for subsequently incorrect decisions, encoding strength did not predict retrieval RTs (Eq. 3, χ²(1) = 1.60, P = 0.21). An encoding strength by retrieval accuracy interaction confirmed that the predictive relationship between encoding strength and retrieval RT differed across subsequently correct versus incorrect recollection decisions (Eq. 4, χ²(1) = 10.8, P < 0.005). Collectively, these results indicate that stronger, source-specific encoding patterns in the bilateral occipitotemporal cortex during visual imagery predict more accurate and faster source memory decisions at retrieval.

Encoding Strength Predicts Hippocampal Activity at Encoding

Motivated by prior observations that greater hippocampal encoding activity (i.e., BOLD amplitude) predicts later recollection accuracy (e.g., Davachi et al. 2003; Ranganath et al. 2004), we investigated whether encoding-phase hippocampal activity predicted subsequent memory accuracy. In contrast to these prior studies, which used recognition tests that discriminated between recollected versus familiar versus forgotten trials, hippocampal activity at encoding did not significantly predict subsequent memory performance on our 2-alternative forced-choice retrieval task (P > 0.1), a finding that may reflect the presence of guesses in the source correct condition of our study. We next examined whether encoding-phase hippocampal activity scaled with our trial-wise measure of cortical encoding strength. Indeed, cortical encoding strength scaled positively with mean hippocampal encoding activity (Eq. 5, χ²(1) = 17.5, P < 10⁻⁴, Fig. 3). This result indicates that greater strength of cortical representation at encoding scales with greater concomitant hippocampal activity, suggesting a role for hippocampus in the representation and/or binding of source features.

Figure 3.

Encoding strength predicts encoding-phase hippocampal activity. (a) Location of anatomically defined left and right hippocampal ROIs, overlayed on a mean normalized group anatomical image. (b) Plot of hippocampal activity against encoding strength for correct retrieval trials. Error bars indicate ±within-subject SEM.

Cortical Reinstatement—Univariate Analyses

Having established that source encoding patterns predict subsequent memory accuracy and decision RTs, we next tested whether source-specific activity at encoding was reinstated at retrieval, using univariate GLM analyses of the fMRI data at encoding and retrieval. These analyses confirmed 2 effects: (1) The face and scene imagery tasks were associated with differential cortical activity at encoding, and (2) source-specific regions active at encoding were reactivated during correct source memory decisions at retrieval. Specifically, a conjunction analysis of face versus scene activity during encoding and retrieval (P < 0.005 for each contrast; corrected P < 0.05) revealed that initial imagery and subsequent retrieval of faces differentially activated multiple regions, including the ventromedial prefrontal cortex, whereas imagery and retrieval of scenes differentially activated posterior parahippocampal cortex, among other regions (Fig. 4a). These results add to a rich literature establishing that recollection is associated with reinstatement of activation in regions that demonstrated content-specific responses during initial learning (e.g., Nyberg et al. 2000; Wheeler et al. 2000; Kahn et al. 2004; Johnson et al. 2009; for a review, see Danker and Anderson 2010).

Figure 4.

Cortical reinstatement tracks encoding strength and subjective memory decision. (a) Conjunction map of source-specific activity at encoding and retrieval, thresholded at P < 0.005 for both encoding and retrieval (conjoint P < 0.05, corrected for multiple comparisons). Red: face > scene; green: scene > face. Activations are displayed on the mean (N = 27 subjects) normalized anatomical image. (b) Classification performance for correct retrieval trials (blue) and incorrect retrieval trials (orange). Error bars indicate standard error of the mean. (c) Plot of the probability of correct source retrieval as a function of cortical reinstatement. (d) Plot of cortical reinstatement by encoding strength. Error bars indicate within-subject SEM.

Cortical Reinstatement Predicts Retrieval Decision Variables—Multivariate Analyses

Multivariate methods can extend univariate reinstatement results by providing a quantitative estimate of cortical reinstatement on a trial-wise basis (Polyn et al. 2005; Johnson et al. 2009; McDuff et al. 2009; Kuhl et al. 2011; Staresina et al. 2012; Ritchey et al. forthcoming). One question of interest is how multivariate estimates of encoding pattern reinstatement relate to memory decisions. Accordingly, to generate trial-specific estimates of cortical reinstatement on each retrieval trial, a classifier was trained to dissociate face- and scene encoding trials and tested with the independent data from the retrieval phase. This procedure provides an estimate of the extent to which retrieval patterns resemble encoding patterns of one versus the other class, as such it can be construed as a trial-wise measure of the reinstatement of source-specific activity. We then asked whether the classifier's probabilistic prediction of the retrieved source, based on the obtained retrieval pattern, tracked the subject's encoding experience or the subject's source memory decision at retrieval.

Our cortical reinstatement measure strongly correlated with retrieval accuracy (Eq. 6, χ²(1) = 50.6, P < 10⁻⁴, Fig. 4c), such that greater cortical reinstatement predicted a greater likelihood of correct retrieval. For trials on which subjects made a correct source decision, the classifier predicted the face/scene encoding source with a mean accuracy of 64.1%, which is significantly above chance [t₍₂₆₎ = 10.1, P < 10⁻⁴, Fig. 4b]. In contrast, for trials on which subjects made an incorrect source decision, the classifier performed modestly, but significantly below chance [mean accuracy = 45.7%, t₍₂₆₎ = −2.47, P < 0.05, Fig. 4b], suggesting that retrieval errors may be driven in part by the activation of occipitotemporal cortical patterns tied to the erroneous source context. We also found that greater cortical reinstatement correlated with faster decision RTs, but with marginal significance (P < 0.1) and with differential contributions from face and scene trials (Supplementary Results). Finally, an exploratory analysis was conducted to investigate whether reinstated source-specific patterns of activity were also present in the hippocampus. The classification rate of reinstated face/scene information in the hippocampus was modestly but significantly higher than chance [mean accuracy = 54.3%; t₍₂₆₎ = 2.96, P < 0.01, Supplementary Results]. Taken together, these data demonstrate that reinstatement tracked subjective memory decisions, even when the decisions were incorrect (e.g., Kahn et al. 2004).

The Strength of Encoding Patterns Predicts the Subsequent Fidelity of Cortical Reinstatement at Retrieval

To the extent that the fidelity of a retrieved representation is dependent on the strength of the initially encoded representation, then the fidelity of cortical reinstatement should be predicted by the strength of source-specific patterns at encoding. Consistent with this hypothesis, a mixed-effects linear regression of encoding strength on cortical reinstatement demonstrated that events that gave rise to greater encoding strength were associated with a higher fidelity of cortical reinstatement at retrieval (Eq. 7, χ²(1) = 19.7, P < 10⁻⁴, Fig. 4d). Critically, these data constitute novel evidence that future states of cortical brain patterns (1) are partially dependent on the strength of the cortical patterns established in the past and (2) have consequences for memory expression.

Hippocampal Activity at Retrieval Scales with Retrieval Accuracy and RT

Previous work has demonstrated that, relative to incorrect source decisions and correct rejections, correct recollection decisions are associated with greater hippocampal activity, implicating hippocampal computations in the recollection of past event details (e.g., Eldridge et al. 2000; Cabeza et al. 2001; Dobbins et al. 2003). Extending these prior categorical observations, mixed-effects linear regression analyses (Eqs 8 and 9) demonstrated that greater hippocampal retrieval activity was associated with (1) a higher likelihood of correct source recollection (Fig. 5a; χ²(1) = 14.4, P < 0.0005), and (2) faster retrieval decision RTs for correct trials (Fig. 5b; χ²(1) = 13.7, P < 0.0005). Taken together, these results demonstrate that greater hippocampal activity at retrieval is associated with faster and more accurate source retrieval decisions.

Figure 5.

Hippocampal activity at retrieval tracks performance and strength of cortical reinstatement. (a) Plot of the probability of correct source retrieval as a function of hippocampal activity. (b) Plot of RT for correct trials as a function of hippocampal activity. (c) Plot of cortical reinstatement strength as a function of hippocampal activity. Error bars indicate within-subject SEM.

The preceding analyses indicate that hippocampal retrieval activity varies with recollection accuracy and decision RT. Given that (1) hippocampal activity during encoding is thought to scale with greater binding of event details, (2) hippocampal activity at retrieval is thought to scale with greater recollection of the event details necessary for correct source decisions, and (3) stronger binding of details at encoding is thought to lead to stronger recollection of details at retrieval, we hypothesized that greater encoding-phase hippocampal activity would be associated with greater retrieval-phase hippocampal activity. Consistent with this hypothesis, a mixed-effects linear regression revealed that events associated with greater hippocampal activity at encoding were also associated with greater hippocampal activity at retrieval (Eq. 10, χ²(1) = 4.94, P < 0.05).

Hippocampal Activity at Retrieval Scales with Cortical Reinstatement

Recollection is thought to partially depend on hippocampal retrieval computations (pattern completion) that serve to reinstate source-specific cortical patterns (Marr 1971; McClelland et al. 1995; McClelland and Goddard 1996; Rolls and Treves 1998; Wallenstein et al. 1998). Supporting this hypothesis, we found that trial-wise retrieval activity in the hippocampus (Eq. 11, χ²(1) = 7.35, P < 0.01, Fig. 5c) predicted the classifier-measured strength of cortical reinstatement. Overall, these results provide evidence that retrieval events accompanied by greater hippocampal activity are associated with greater cortical reinstatement of content-specific representations observed at encoding.

Effects of Stimulus Content

To determine whether each reported effect of interest was differentially driven by the face or scene stimulus category, we tested for an interaction between the effect of interest and stimulus category within each analysis. Specifically, within each mixed linear model (Eqs 1–11; Supplementary Table 1), we tested the significance of the relationship between the dependent variable and an independent variable modeling the interaction of the effect of interest and stimulus category. Of these analyses, one stimulus category interaction proved significant: The relationship between activity in the hippocampus and reinstatement strength was significantly greater for face versus scene trials (χ²(1) = 3.87, P < 0.05). Besides this result, no other predictor–class interaction was found to be significant (P > 0.1, for all tested interactions) for any effect of interest. These analyses thus demonstrate that, for 8 of 9 reported regressions of interest involving measures of hippocampal activity and distributed cortical source representations, there is no evidence for a difference in these relationships across face and scene trials.

Multiple Comparisons Correction Across Regressions of Interest

To account for multiple comparisons among the regressions of interest, false discovery rate analysis (Benjamini and Hochberg 1995) was performed. An N = 14 was used, encompassing both the 8 regressions of interest reported above and the remaining 6 pair-wise comparisons (reported in Supplementary Table 1) that met the criteria of being a pair-wise comparison between variables of interest, and containing at least one neural variable. All above reported effects of interest (Eqs 1–11) have a q-value of <0.05, indicating that the expected rate of false discovery, or incorrectly rejecting the null hypothesis, across this family of reported effects is 5% (Supplementary Table 1). All nonreported effects of interest have a q-value of >0.05. Thus, all regression effects reported above are significant when accounting for multiple comparisons, whereas all other pair-wise comparisons among these variables do not survive correction.

Synthesizing Regressions of Interest into a Path Analysis

As a final analysis approach, we conducted path analyses that combined all analyses of interest into a single sequential statistical model, which allowed us to test for indirect effects (i.e., examining whether an independent variable predicts a dependent variable through its action on an intermediate variable). In a first path model, the variables of “cortical encoding strength,” “cortical reinstatement,” “hippocampal encoding activity,” “hippocampal retrieval activity,” and “memory accuracy” were included. The path structure was created (Fig. 6a) by specifying paths between each pair of variables for which a significantly predictive relationship was found in the preceding analyses (see above). A directional-separation test revealed that we could retain this model as a reasonable fit to the data (Eqs 12a–c, χ²(6) = 8.26, P = 0.22). Further analysis revealed that this model had a locally optimal fit; adding any path that was not already present in the structure did not significantly improve the model fit (χ²(2) < 3.09, P > 0.13 for each added path), while subtracting any link that was present in the structure significantly decreased the model fit (χ²(2) > 8.34, P < 0.05). All direct relationships between variables remained significant in the path model (Eqs 12a–d, P < 0.05 for each path; Fig. 6), indicating that each reported effect of interest was significant even when controlling for other reported effects of interest. A second path model encompassing only correct retrieval trials was created to model all variables from the first model with the exception of memory accuracy and the addition of RT (Supplementary Results).

Figure 6.

Path analysis reveals how pathways of neural activity at encoding and retrieval lead to source retrieval. (a) Path analysis relating neurally derived mnemonic variables and retrieval accuracy. Numeric labels indicate standardized path coefficients. Path thickness indicates the statistical significance of each given direct effect. (b) The indirect path from encoding strength to cortical reinstatement to retrieval accuracy (dark gray) and the indirect path from retrieval-phase hippocampal activity to cortical reinstatement to retrieval accuracy (light gray).

Cortical Reinstatement Partially Mediates the Effects of Encoding Strength on Retrieval Accuracy

Importantly, path analysis allowed us to examine indirect (mediated) pathways of interest. First, we examined whether the effect of encoding-phase variables on retrieval behavior was mediated in part by retrieval-phase behavioral variables. Our path analyses contained 2 sets of indirect paths that fit this profile: (1) A path from encoding strength to retrieval strength to accuracy, and (2) a path from encoding strength to cortical reinstatement to RT. Indirect effect 1 was significant [P < 10⁻⁴, Z = 3.72; index of mediation = 0.036, 95% confidence interval (CI) = 0.026–0.044, Fig. 6b], demonstrating that distributed activity at encoding predicted subsequent memory accuracy in part through its effect on distributed activity at retrieval. Indirect path 2 did not reveal a significant indirect pathway from encoding strength to cortical reinstatement to RT (P = 0.19, Z = 1.32; Supplementary Results). Taken together, these analyses demonstrate that the effect of encoding strength on retrieval accuracy is partially mediated by the variable of cortical reinstatement.

Cortical Reinstatement Partially Mediates the Effects of Retrieval-Phase Hippocampal Activity on Retrieval

A second set of path analyses sought to examine how hippocampally supported pattern completion may support memory retrieval in part through the intermediate effects on cortical representations. Specifically, we tested whether the effect of retrieval-phase hippocampal activity on (1) retrieval accuracy and (2) RT was mediated by cortical reinstatement. We found that the indirect path predicting accuracy was significant (P < 10⁻⁴, Z = 3.72; index of mediation = 0.025, 95% CI = 0.010–0.043, Fig. 6b). The indirect path predicting RT was not significant (P = 0.20, Z = 1.28; Supplementary Results). These analyses demonstrate that retrieval-phase hippocampal activity predicted retrieval accuracy in part through its effects on cortical reinstatement.

Discussion

The present study demonstrated 5 key findings. First, a neurally derived measure of trial-wise content-specific “encoding strength” in the occipitotemporal cortex predicted (1) subsequent accuracy in remembering whether a face or scene had been imagined at encoding, and (2) subsequent retrieval decision latency. Secondly, cortical encoding strength scaled with encoding-phase hippocampal activity. Thirdly, cortical encoding strength and retrieval-phase hippocampal activity predicted the strength of cortical reinstatement at retrieval. Fourthly, the magnitude of cortical reinstatement predicted retrieval accuracy and RT. Finally, the effects of cortical encoding strength and retrieval-phase hippocampal activity on retrieval accuracy were partially mediated by the effect of cortical reinstatement. Collectively, these results document how the recollection of event details depends on cortical–hippocampal interactions during event encoding and subsequent retrieval.

Content-Specific Cortical Activity at Encoding

Regions of the human occipitotemporal cortex, such as fusiform gyrus and posterior parahippocampal cortex, demonstrate increased BOLD activity during the perception of specific visual categories such as faces and houses (e.g., Kanwisher et al. 1997; Epstein and Kanwisher 1998) and the generation of mental images specific to such categories (e.g., O'Craven and Kanwisher 2000; Davachi et al. 2003). Studies pairing categorical perception with a subsequent memory task demonstrate that content-specific activity during perception is greater for category members that are subsequently remembered relative to those subsequently forgotten (e.g., Kirchhoff et al. 2000; Davachi et al. 2003). In particular, greater activation in face-responsive regions of fusiform gyrus during face encoding is predictive of greater subsequent memory for faces (Sergerie et al. 2005; Nichols et al. 2006; Prince et al. 2009), whereas greater activation in scene-responsive regions of the parahippocampal cortex during scene encoding is predictive of subsequent memory for scenes (Brewer et al. 1998; Kirchhoff et al. 2000; Turk-Browne et al. 2006; Hayes et al. 2007; Awipi and Davachi 2008; Prince et al. 2009; Preston et al. 2010).

While nearly all prior studies of content-specific SMEs utilized univariate measures of encoding activity, the present study exploited multivariate techniques to quantify the “strength” of content-specific occipitotemporal cortical patterns at encoding and to relate this measure of cortical encoding strength to subsequent associative recollection (i.e., source memory). Through this approach, we demonstrated that when mental imagery at encoding is accompanied by stronger content-specific patterns across the occipitotemporal cortex, subjects are more likely to be able to subsequently remember having imagined an exemplar from the respective category (see also Kuhl, Rissman, et al. 2012 for similar findings with a perceptual-based encoding paradigm). Additionally, we observed that greater cortical encoding strength is predictive of faster correct retrieval decisions. As both correct decisions and faster RT have been linked to judgments made with greater decision evidence, these results support the view that greater encoding strength leads to greater source decision evidence. Future studies can further document the decision process by incorporating formal computational decision models into the analysis of the relationship of neural measures of reinstatement to behavioral retrieval variables.

A number of mechanisms may account for why cortical encoding strength predicts subsequent recollection. First, although analyses were restricted to encoding trials for which subjects reported successful mental imagery, variability in cortical encoding strength may nevertheless reflect differences in prior semantic knowledge that differentially enabled mental imagery given the conceptual cue (i.e., the word). Encoding trials for which a subject had greater semantic knowledge may have resulted in generation of a richer or more vivid mental image (Kozhevnikov et al. 2005) and may have been associated with “deeper” (i.e., more semantically rich) encoding of the word/image association. Both factors would facilitate subsequent retrieval. Secondly, variability in cortical encoding strength may also reflect variability in task-focused attention. Episodic encoding has been shown to decrease in the presence of a competing task (e.g., Craik et al. 1996; Foerde 2006). Trials in which subjects deployed more focused attention on the mental imagery task versus other endogenous goals or thoughts may be associated with a more richly detailed mental image and a greater likelihood of subsequent accurate retrieval. As such, the relationship between cortical encoding strength and subsequent memory may partially reflect the role of attentional processes during encoding (e.g., Craik and Lockhart 1972; Rock and Gutman 1981; Chun and Johnson 2011; Uncapher et al. 2011).

Cortical–Hippocampal Interactions at Encoding

Much research suggests that event encoding depends on hippocampal–cortical interactions. Anatomically, the outputs of neocortical regions, including occipitotemporal cortical areas, project to the perirhinal and parahippocampal cortex in the medial temporal lobe, which project, via entorhinal cortex, to the hippocampus (Suzuki and Amaral 1994; Lavenex and Amaral 2000). The hippocampus is thus a convergence zone for a broad range of cortically represented event features (Squire and Zola-Morgan 1991; Rolls 1996; Mishkin et al. 1997, 1998; Eichenbaum 2000). Computationally, associative encoding of events (episodes) is thought to require the cortical representation of event features, which converge on and are bound in the hippocampus (Marr 1971; Squire 1992; Cohen and Eichenbaum 1993; McClelland et al. 1995; Norman and O'Reilly 2003).

A number of studies indicate that greater encoding-phase functional connectivity between hippocampus and cortical regions involved in perception predicts a higher likelihood of subsequent memory. For example, fMRI data suggest that greater hippocampal–medial occipital cortical connectivity at encoding predicts subsequent recognition memory for previously viewed objects (Ranganath et al. 2005) and subsequent free recall for viewed words (Schott et al. 2011). During the presentation of auditory stimuli, subsequent recollection is associated with greater BOLD signal connectivity between hippocampus and superior temporal gyrus (Gagnepain et al. 2011) and greater theta-phase intracranial electroencephalography (EEG) synchronicity between hippocampal and occipitotemporal regions (Babiloni et al. 2009). Intracranial EEG data also indicate that encoding-period gamma-phase synchronization and theta coherence between hippocampus and perirhinal and/or entorhinal cortex predict subsequent free recall of visually presented words (Fell et al. 2001, 2003).

We observed a positive correlation between trial-to-trial fluctuations in the magnitude of hippocampal univariate activity and our multivariate measure of content-specific cortical encoding strength. Two mechanisms may account for this finding. First, stronger cortical representation of the content-specific information may trigger greater hippocampally supported encoding operations. Specifically, the propagation of stronger perceptual information to the hippocampus may have a higher likelihood of eliciting hippocampal neural responses and ultimately mnemonic binding (i.e., hippocampal encoding). Secondly, greater hippocampal activity during the encoding phase may signify that subjects are engaging in hippocampally supported retrieval processes while performing the mental imagery task. Greater hippocampally supported retrieval may contribute to stronger mental imagery, which, in turn, would promote greater cortical encoding strength. These 2 purported mechanisms are not mutually exclusive; it is possible that stronger content-specific cortical representations are associated with both greater hippocampal encoding and retrieval operations during mental imagery.

Cortical Reinstatement

Cortical reinstatement, the reactivation of content-specific encoding patterns at retrieval, has been observed in a broad array of paradigms (Nyberg et al. 2000; Wheeler et al. 2000; Kahn et al. 2004; Polyn et al. 2005; Woodruff et al. 2005; Wheeler et al. 2006; Johnson et al. 2009; Kuhl et al. 2011; Kuhl, Bainbridge, et al. 2012). The present data revealed that the strength of cortical reinstatement is predictive of retrieval decision times and decision accuracy. As such, our results indicate that the fidelity of cortical reinstatement can affect multiple decision variables at retrieval, which suggests that the reinstated cortical patterns serve as a source of evidence driving the mnemonic decision. Previous research indicates that cortical reinstatement scales with many conditions that are thought to be associated with greater strength of recollection, including contrasts of (1) source recollection versus item recognition (e.g., Kahn et al. 2004), (2) subjective reports of recollection versus familiarity (Wheeler and Buckner 2004; Johnson and Rugg 2007), (3) conditions indexing graded memory strength (Johnson et al. 2009), (4) greater amounts of associated event details (Khader et al. 2005), (5) stronger subjective ratings of vividness of retrieved details (Daselaar et al. 2008), and (6) more specificity of retrieved details (e.g., Kuhl et al. 2011).

Memory-guided decisions are thought to depend on retrieved mnemonic evidence that is evaluated in relation to decision criteria (Ratcliff 1988; Dunn 2004; Wixted and Stretch 2004). The present data revealed that the strength of cortical reinstatement is predictive of retrieval decision times and decision accuracy. As such, our results indicate that the fidelity of cortical reinstatement can affect multiple decision variables at retrieval, which suggests that the reinstated cortical patterns serve as a source of evidence driving the mnemonic decision.

Our data also indicate that source-related cortical activity patterns during retrieval are more closely tied to subjective retrieval experiences (or decisions) than to the objective experiential history associated with a stimulus (Kahn et al. 2004; Slotnick and Schacter 2004; Rissman et al. 2010 for related findings concerning item memory). That is, rather than tracking the true mnemonic history of an item, the pattern classifier's prediction about the subject's memory state tracked the subject's reported decision, even when the decision was incorrect. This observation suggests that retrieval errors are driven in part by the activation of distributed cortical patterns that represent features of the incorrect source or content. This finding complements prior results derived from univariate fMRI analyses, which demonstrate that erroneous memory for novel items (i.e., false recollection) is sometimes accompanied by reactivation of cortical regions selective for the content of the false memory (e.g., Kahn et al. 2004; Slotnick and Schacter 2004).

While a growing body of evidence links cortical reinstatement to retrieval decision variables, it is important to note that cortical reinstatement effects can arise for reasons not directly tied to the retrieval decision process. For instance, in the present data, it is possible that each retrieval trial is accompanied by subjects engaging in imagery of face and scene information related to the retrieval cue. Subjects who experience a greater familiarity for any imagined face versus scene information may be more prone to respond that a given memory cue was originally associated with a face at retrieval. In this way, it would be possible, in principle, to correctly identify a source associate without engaging in recollection. Given the relatively high level of source retrieval accuracy in this study, we believe that, if present, such a generate-recognize process was likely used in a minority of trials. Nevertheless, future work is needed to dissociate how different forms of task cognition may contribute to “cortical reinstatement” signals.

Hippocampal–Cortical Interactions at Retrieval

Consistent with the notion that hippocampal retrieval processes subserve the recollection of event details, we observed that greater hippocampal activity during retrieval correlated with source memory accuracy and correct retrieval decision time. This accords with an extensive literature demonstrating that hippocampal activity is greater for correct retrieval events (e.g., Eldridge et al. 2000; Cabeza et al. 2001; Dobbins et al. 2003), and extends this relationship to retrieval decision latency.

We also found that retrieval-phase hippocampal activity correlated with the strength of cortical reinstatement. These data complement findings in humans, demonstrating that hippocampal activity is correlated with measures of cortical reinstatement during retrieval (e.g., Kuhl et al. 2010; Staresina et al. 2012; Wimmer and Shohamy 2012; Zeithamova et al. 2012; Ritchey et al. in press). Taken together, our findings indicate that hippocampal BOLD signal amplitude during source recollection covaries with a multivariate measure of cortical reinstatement (a neural measure of retrieval strength), as well as with retrieval accuracy and decision latency (behavioral measures of retrieval strength).

Furthermore, we synthesized the results linking hippocampal activity to reinstatement and to memory performance by creating a path model to relate all neural and behavioral variables for which significant relationships were documented into a single statistical model (Fig. 6). This model is consistent with the perspective that a cascading series of neural responses, including cortical encoding strength, hippocampal univariate encoding activity, cortical reinstatement, and hippocampal univariate retrieval activity, drive memory behavior.

In particular, this model revealed that the effect of retrieval-phase hippocampal activity on subsequent retrieval was partially mediated by the magnitude of cortical reinstatement. This result provides evidence for theories positing that retrieval is supported by hippocampal processes that drive the reinstatement (or replay) of cortical patterns established at event encoding. Specifically, cue-related information represented in the cortex is thought to propagate to the hippocampus, where neurons linked to associates of the cue become active, a process known as pattern completion (Marr 1971; McClelland et al. 1995; McClelland and Goddard 1996; Rolls and Treves 1998; Wallenstein et al. 1998). Signals resulting from hippocampal pattern completion are thought to ultimately project back to cortical regions that code for the associated event details, reinstating cortical patterns that were established at encoding, and enabling retrieval task performance. The results of the present path analysis indicate that cortical reinstatement partially mediates the effect of retrieval-phase hippocampus activity on retrieval accuracy, thus providing a compelling account for how interactions between the hippocampus and cortex can lead to successful memory-dependent behavior.

From Cortical Encoding Strength to Cortical Reinstatement to Retrieval Accuracy

Critically, the path model also revealed the existence of a significant indirect path from cortical encoding strength to cortical reinstatement to subsequent memory performance. This finding documents one mechanistic pathway for content-specific SMEs: Content-specific activity at encoding predicts subsequent memory in part through the intermediate variable of cortical reinstatement strength. More generally, this result suggests that the state of the cortical pattern at encoding affects memory through its affect on the state of the cortical pattern elicited at retrieval. This finding complements those of Ritchey et al. (in press), who observed that correct item recognition is predicted by the similarity of the pattern of occipital cortical activity elicited by an item at encoding and retrieval, and that this similarity is related to retrieval-phase hippocampal activity.

It is worth noting that, while an indirect path through the variable of cortical reinstatement partly accounted for the relationship between cortical encoding strength and subsequent memory, the direct effect of encoding strength on subsequent memory remained significant. We note that complete mediation of the effect of encoding strength on retrieval behavior is unlikely to be observed because of the presence of multiple potential mediator variables (Baron and Kenny 1986), including variables that are not measured by our cortical encoding and cortical reinstatement assays. In this study, for instance, pattern typicality effects likely contributed to our measures of encoding strength and cortical reinstatement, but were not modeled out of our analyses. Thus, for a pattern in which the reinstated activity was strong but was more atypical of encoding trials of the same class, the “true” strength of reinstatement will be underestimated by our measurement. In a related point, item-specific representations at encoding and retrieval likely played important roles in establishing memory accuracy, but were not included in our analyses, because our approach measured only category-general representational strength. Future studies examining how item-specific neural patterns at encoding and retrieval affect memory performance (e.g., Staresina et al. 2012; Ritchey et al. in press) may further clarify how neural activity at encoding predicts retrieval behavior through an intermediate effect on that at retrieval.

Conclusion

This study demonstrates that retrieval of event details can be predicted by the strength of cortical representations at both encoding and retrieval. Furthermore, the strength of these representations scales with hippocampal activity at both encoding and retrieval. Finally, path analysis indicates that the state of occipitotemporal cortex at encoding influences retrieval performance in part through its effects on the state of occipitotemporal cortex at retrieval. The approach of relating classifier-derived measures of cortical representational strength and hippocampal activity at encoding and retrieval promises to yield future insights into how cortical–hippocampal interactions support memory-based decisions.

Supplementary Material

Supplementary material can be found at: http://www.cercor.oxfordjournals.org/.

Funding

This work was supported by a grant from the National Institute of Mental Heath (R01–MH080309) and a National Science Foundation Graduate Research Fellowship to A.M.G.