Decoding the content of recollection within the core recollection network and beyond

Abstract

Recollection – retrieval of qualitative information about a past event – is associated with enhanced neural activity in a consistent set of neural regions (the ‘core recollection network’) seemingly regardless of the nature of the recollected content. Here, we employed multi-voxel pattern analysis (MVPA) to assess whether retrieval-related fMRI activity in core recollection regions - including the hippocampus, angular gyrus, medial prefrontal cortex, retrosplenial/posterior cingulate cortex, and middle temporal gyrus – contain information about studied content and thus demonstrate retrieval-related ‘reinstatement’ (‘reactivation’) effects. During study, participants viewed objects and concrete words that were subjected to different encoding tasks. Test items included studied words, the names of studied objects, or unstudied words. Participants judged whether the items were recollected, familiar, or new by making ‘remember’, ‘know’, and ‘new’ responses, respectively. The study history of remembered test items could be reliably decoded using MVPA in most regions, as well as from the dorsolateral prefrontal cortex, a region where univariate recollection effects could not be detected. The findings add to evidence that members of the core recollection network, as well as at least one neural region where mean signal is insensitive to recollection success, carry information about recollected content. Importantly, the study history of recognized items endorsed with a ‘know’ response could be decoded with equal accuracy. The results thus demonstrate a striking dissociation between mean signal and multi-voxel indices of recollection. Moreover, they converge with prior findings in suggesting that, as it is operationalized by MVPA classification, reinstatement is not uniquely a signature of recollection.

1. Introduction

Recognition memory is widely held to be supported by two distinct memory signals, recollection and familiarity (Mandler 1980; Wixted & Mickes, 2010; Yonelinas, 2002). Recollection refers to the retrieval of qualitative information about a past event such as its spatiotemporal context. Familiarity refers to a memory signal that supports recognition in the absence of qualitative information about the event. In the laboratory, recollection has been dissociated behaviorally from familiarity in several ways (for review, see Yonelinas, 2002). For example, recollection can be operationalized by memory for a specific contextual feature from a study episode (source memory), as when a test item elicits the subjective experience of recollection rather than an acontextual sense of familiarity (a ‘Remember’ rather than a ‘Know’ judgment; Tulving, 1985), and by successful memory for an inter-item association. Concurrently acquired functional magnetic resonance imaging (fMRI) data indicate that successful recollection is associated with enhanced neural activity in a consistent set of neural regions, sometimes referred to as the ‘core recollection network’ (Hayama, Vilberg, & Rugg, 2012; Rugg & Vilberg, 2013; King, de Chastelaine, Elward, Wang, & Rugg, 2015). The network, which overlaps with the well-studied ‘default mode network’ (Buckner, Andrews-Hanna, & Schacter, 2008), comprises the left angular gyrus, retrosplenial/posterior cingulate cortex, medial prefrontal cortex, middle temporal gyrus, parahippocampal cortex, and hippocampus. Recollection-related enhancement of activity within these regions appears to be largely independent of how memory retrieval is cued or the nature of the recollected content (for reviews see Kim, 2010, 2016; Rugg and Vilberg, 2013).

While the hippocampus has long been recognized as important to episodic memory by virtue of its role in supporting memory for associations between the different elements of an episode, studies have only recently begun to identify possible roles for other members of the core network. Hayama et al. (2012) provided evidence that regions within the core network are sensitive to the amount of information recollected, reporting that neural activity in most regions was enhanced when recall of a study episode was accompanied by retrieval of a source feature relative to recall of the study episode alone (see also, Vilberg & Rugg, 2007, 2009a, 2009b). Relatedly, recollection-related activity within each region of the network has been reported to be elevated for recollection judgments made with high confidence relative to those made with low confidence (Leiker & Johnson, 2015; Thakral, Wang, & Rugg, 2015; Yu, Johnson & Rugg, 2012a, 2012b). Together, these findings suggests that the network is sensitive to the both the amount and fidelity of retrieved information.

Vilberg and Rugg (2012, 2014) assessed whether recollection-sensitive regions can be dissociated according to the time-courses of their activity. Whereas recollection-related activity in most members of the network demonstrated a transient time-course, activity in the angular gyrus and middle temporal gyrus was sustained, tracking the period over which recollected information was maintained prior to a decision about its content. These findings were interpreted as evidence that these two regions contribute in some way to the representation of recollected content (see also, Bonnici, Richter, Yazar, & Simons, 2016; Hutchinson, Uncapher, Weiner, Bressler, Silver, Preston, & Wagner, 2012; Kuhl & Chun, 2014; Shimamura, 2011; for a review, see Rugg, Johnson, & Uncapher, 2015). This proposal stands in contrast to the view that recollection-related activity within the angular gyrus reflects an automatic re-orientating of attention to the products of retrieval (for a review, see Cabeza, Ciaramelli, & Moscovitch, 2012).

Additional evidence in favor of a representational account of the role of both the angular gyrus and other cortical regions in recollection comes from studies employing multi-voxel pattern analysis (MVPA). In these studies, MVPA was employed to examine whether patterns of neural activity within different cortical regions, in particular the angular gyrus, carry information about the content of recollected information (Bonnici, et al., 2016; Leiker & Johnson, 2015; Kuhl & Chun, 2014; Kuhl, Johnson, & Chun, 2013). In the study of Kuhl and Chun (2014), for instance, participants studied word-face or word-scene pairings. During the subsequent test phase, participants were cued with the studied words and required to recall the paired face or scene. A multi-voxel classifier was trained using patterns of study phase activity in several regions of interest, including three members of the core recollection network (angular gyrus, medial prefrontal cortex, and the hippocampus), to discriminate face and scene trials. The trained classifier was then used to assess whether face and scene information could be decoded from patterns of activity within those regions during test trials (a measure of ‘cortical reinstatement’; for reviews, see Rissman & Wagner, 2012 and Rugg et al., 2015). Kuhl and Chun (2014) reported that the classifiers were able to reliably discriminate the class of the studied associate of test words eliciting recall. In a secondary analysis confined to the angular gyrus and the ventral temporal cortex, classification did not differ according to whether recall was associated with a ‘vivid’ or a ‘weak’ memory for the study event. These findings converge with those of other studies in suggesting that retrieved content can be decoded from multiple regions of the core network (Chadwick, Hassabis, Weiskopf, & Maguire, 2010; Chadwick, Hassabis, & Maguire, 2011; Johnson, McDuff, Rugg, & Norman, 2009; Ritchey, Wing, LaBar & Cabeza, 2013; St-Laurent, Abdi, & Buchsbaum, 2015; Tompary, Duncan, & Davachi, 2016; Wing, Ritchey, & Cabeza, 2015).

There are a number of caveats attached to the findings from several of the above-mentioned studies. Some of the studies either employed multiple study-test cycles that each comprised only small numbers of items (Kuhl et al., 2013; Kuhl & Chun, 2014), or they repeated items across multiple cycles (Bonnici, et al., 2016; Chadwick et al., 2010, 2011; St-Laurent, et al., 2015). When study lists and study-test delays are short, studied content is highly accessible. Therefore, the results from such studies may not generalize to studies that employ a more extensive study phase and hence more demanding retrieval conditions (cf., Elward & Rugg, 2015). When the same studied items are repeated across multiple test phases, it becomes difficult to dissociate effects reflecting the reinstatement of the original encoded information from those reflecting the retrieval of ‘re-encoded’ information from an earlier test phase. Moreover, both of these experimental designs necessitate that encoding is intentional, running the risk of the engagement of content-specific encoding strategies that are not representative of the incidental encoding typical of ‘everyday’ memories.

A further limitation of prior studies that employed MVPA to examine reinstatement in the core recollection network is that findings have rarely been contrasted according to whether retrieval was recollection- or familiarity-driven. For example, in the study of Kuhl and Chun (2014) reinstatement was assessed as a function only of subjective ratings of the vividness of recall. In other studies, MVPA was conducted for trials exclusively associated with accurate recollection (source memory) judgments (Leiker & Johnson, 2015) or classifier accuracy for accurate recollection judgments was compared with a condition comprising a mixture of trials associated with inaccurate recollection and recognition failure (Tompary et al., 2016). The question of whether reinstatement is sensitive to familiarity as well as recollection is however highly relevant to what might be called the ‘standard’ model of episodic retrieval (e.g., Alvarez & Squire, 1994; McClelland, O’Reilly, & McNaughton, 1995; Rugg et al., 2015). According to this model, cortical reinstatement provides access to the qualitative information that comprises recollected content, and this occurs when a retrieval cue activates a hippocampally-stored memory representation. By this account, therefore, reinstatement should be stronger, and perhaps only evident, when recognition is based on recollection, not least because familiarity is widely held to be independent of the hippocampus (Eichenbaum, Yonelinas, & Ranganath, 2007).

The present study goes some way to addressing these issues. In a further analysis of a previously published dataset (Wang, Johnson, de Chastelaine, Donley, & Rugg, 2016), we sought evidence for content sensitivity in retrieval-related activity elicited during recollection- and familiarity-based recognition judgments in members of the core recollection network. Participants completed a single study-test cycle. In the study phase, participants viewed objects and concrete words in association with different study tasks. At test, participants viewed words that corresponded to studied words, the names of studied objects, or unstudied words and judged whether the items were recollected, familiar, or new by making ‘Remember’, ‘Know’ and ‘New’ responses, respectively. To assess whether recollected content could be decoded from regions of the core network, multi-voxel classifiers were trained on the study phase data to discriminate picture and word trials. The classifiers were then used to assess whether studied content could be decoded from recollection- and familiarity related patterns of neural activity (i.e., Remember and Know test trials, respectively) within the core recollection network.

In addition to examining activity patterns within the core recollection network, we also examined activity within the left dorsolateral prefrontal cortex. Although mean BOLD signal within left dorsolateral prefrontal cortex is rarely enhanced as a function of recollection success (Kim, 2010, 2016; Rugg and Vilberg, 2013), we have previously reported that this region is recollection-sensitive when its involvement in recollection is assessed by modulation of event-related functional connectivity rather than mean signal change (King et al., 2015). In that study, psychophysiological interaction analysis revealed that left dorsolateral prefrontal cortex exhibited a recollection-related increase in connectivity with each member of the core network. In light of this finding, we examined whether, despite the insensitivity of mean BOLD signal in the region to successful versus unsuccessful recollection, retrieval-related activity in the dorsolateral prefrontal cortex carries information about retrieved content.

Lastly, we note that the present dataset comprises data from samples of both young and older participants, reflecting the fact that the aim for the original study was to examine cortical reinstatement and recollection effects as a function of age (Wang et al., 2016). In agreement with other recent work from our laboratory (de Chastelaine, Mattson, Wang, Donley, & Wang, 2016), Wang et al. (2016) reported that age did not exert a detectable influence on either of these classes of effects. While not the primary focus of the present analysis, we took advantage of the dataset to examine whether, in contrast to our prior findings for content-selective cortical regions, reinstatement effects in members of the core recollection network are age-sensitive.

2. Materials and Methods

2.1 Participants

The experimental protocol was approved by the Institutional Review Boards of the University of Texas at Dallas and the University of Texas, Southwestern Medical Center. Informed consent was obtained prior to participation. Forty-eight participants were included in the analyses reported below (24 older participants (13 female; mean age 68 years) and 24 younger participants (13 female; mean age 24 years)). All participants reported that they were right handed, had normal or corrected-to-normal vision, and scored 26 or higher on the Mini Mental State Examination. Participants completed a neuropsychological test battery in advance of the scanning session (see Supplemental Table 1 for young and older participants’ characteristics and neuropsychological test performance).

2.2 Experimental materials

Experimental items comprised 216 colored pictures of common objects and their names. The objects were drawn from Hemera Photo Objects 50,000 Volume 2 (http://www.hemera.com/index.html). The names of each picture were between 3 and 12 letters long, with a mean frequency of occurrence of 14.3 counts/million (Kucera & Francis, 1967). For each participant, the 216 object-name pairs were randomly subdivided into those to be used at study and those to be used as new items (names only) on the test task. For each participant, 144 picture-name pairings were used to create a single study list. The study list was further subdivided into three sub-lists, one for each fMRI study session. In each sub-list, half of the items were presented as pictures and half were shown as names (24 pictures and 24 names). A corresponding test list comprised the 144 items that were presented during study (all now presented as names) and 72 new names. The test list was subdivided into three sub-lists, one for each fMRI test session. Each sub-list contained a pseudorandom ordering of 48 studied items (24 names previously presented as pictures and 24 re-presented names) and 24 new names. In addition to the items described above, 15 names and 9 pictures were used as fillers for the study and test lists and 18 names and 12 pictures were used for practice.

2.3 Experimental procedures

Participants completed a single study-test cycle. The study and test phases were each split across three fMRI scanning sessions. During study, participants performed one of two tasks (Figure 1A). For pictures, participants judged whether the illustrated object would fit into a shoebox. For words, participants judged whether the object denoted by the word would more likely be found inside or outside a house. Study items were presented in ‘mini-blocks’ (cf., Johnson et al., 2009; McDuff, Frankel, & Norman, 2009), each of which comprised either three word or three picture trials. Each mini-block began with the presentation of a red fixation cross at the center of the screen for 500 ms. The cross was replaced by a task cue for 2 s (‘Used inside a house?’ or ‘Fits inside a shoebox?’). The three items were then presented. Presentation duration was 2 s per item, with an inter-item interval of 2s. During the interval, a black fixation cross was presented for 1.5 s followed by a red cross for the final 500 ms. ‘Yes’ and ‘No’ responses were made using the left and right index fingers, and finger assignment was counterbalanced across participants. By presenting study trials as mini-blocks we aimed to maximize the segregation of the BOLD activity associated with each study task (taking advantage of the summation of activity across successive trials) while avoiding the employment of a slow event-related design, which would have led to an unacceptably prolonged scanning session. A similar approach was adopted by both McDuff et al., (2009) and Johnson et al., (2009). Participants practiced the study task before entering the scanner.

Figure 1.

A. Study task (representative trials from each of two study tasks shown to the left and right). B. Test task. For each test cue, participants were asked to make a ‘Remember’, ‘Know’, or ‘New’ response.

Five minutes after the last study session, participants completed a short practice version of the test task while remaining in the scanner, following which the first test fMRI session commenced. Each test trial began with the presentation of a red fixation cross for 500 ms followed by the presentation of a word for 3 s (Figure 1B). Following word offset, a black fixation cross was presented for a variable duration (~66% of trials at 4.5 s, ~25% of trials at 6.5 s, and ~9% of trials at 8.5 s). A remember/know/new task was employed (Tulving, 1985). A ‘Remember’ response was to be made when recognition of the test word was accompanied by retrieval of at least one specific detail from the study episode. Participants were instructed that recollection of any contextual detail from the study episode, and not only the study material or task, justified a ‘Remember’ response. A ‘Know’ response was to be made when recognition of the test item was not accompanied by retrieval of any detail about the study episode. A ‘New’ response was to be made when a test item was judged unstudied or when participants were unconfident of the study status of the item. ‘Remember’, ‘Know’, and ‘New’ responses were made with the index and middle fingers of one hand, and the index finger of the opposite hand. Finger assignments were counterbalanced across participants.

2.4 Image acquisition and analysis

Functional and anatomic images were acquired with a 3 Tesla Phillips Achieva MRI scanner (Philips Medical System, Andover, MA USA) equipped with a 32-channel head coil. Functional images were acquired with an echo-planar imaging sequence (SENSE factor 2, TR 2 s, TE 30 ms, flip angle 70°, field-of-view 240 × 240, matrix size 80 × 79, 30 slices, 3 mm isotropic voxels, 1 mm inter-slice gap). Slices were acquired in ascending order and oriented parallel to the anterior-posterior commissure plane. For each study session, 170 volumes were acquired and for each test session 338 volumes were acquired. T1-weighted anatomic images were acquired with a magnetization-prepared rapid gradient echo sequence (matrix size 220 × 193, voxel size 1 mm isotropic voxels, 150 slices).

fMRI data were analyzed using both a univariate general linear model (GLM) and classification-based MVPA. Univariate analyses were conducted using Statistical Parametric Mapping (SPM8, Wellcome Department of Cognitive Neurology, London, UK). Multi-voxel analyses were conducted using the Princeton MVPA Toolbox (https://code.google.com/p/princeton-mvpa-toolbox/) and custom MATLAB scripts.

Functional image preprocessing included two-stage spatial realignment (first to the mean image across sessions and then with manual reorientation to approximate the orientation of the Montreal Neurological Institute (MNI) template), slice-time correction (using the middle slice as reference), and normalization to a sample-specific template that was unbiased with respect to age group (de Chastelaine, Wang, Minton, Muftuler, & Rugg, 2011; Mattson, Wang, de Chastelaine, & Rugg, 2014). For univariate analyses, volumes were smoothed with an 8 mm full-width-half-maximum Gaussian smoothing kernel to the normalized volumes. The time series in each voxel was high-pass filtered at 1/128 Hz and scaled to a constant mean within session. Anatomic images were normalized using a procedure analogous to that adopted for the functional images (see above).

2.4.1 Mean Signal Analyses

Mean signal analyses for both the study and test phase data were conducted using a two stage mixed effects model. In the first stage, neural activity associated with each event was modeled at event onset by an impulse response (delta) function. The associated BOLD response was modeled by convolving the functions with a canonical hemodynamic response function and its temporal and dispersion derivatives (Friston et al., 1998) to yield regressors in a GLM that modeled the BOLD response for each event type. For the analysis of the study data, there were two events of interest: picture trials and word trials. An additional event of no interest included filler trials and missed study task responses. For the test analysis, there were 3 events of interest: Remember-picture, Remember-word, and Know (collapsed across word and picture trials, see below). Remember-picture trials corresponded to Remember responses to test words previously presented as pictures. Remember-word trials corresponded to Remember responses to test words previously presented as words. Know trials corresponded to words previously presented as either words or pictures (due to low trial numbers, Know responses were collapsed across study category). Three further categories of test events comprised correct rejections (new words correctly judged as new), misses (studied items incorrectly judged to be new), and an event of no-interest (false alarms, filler trials, and trials without a response). Both study and test analyses also included six regressors representing movement related variance (three for rotation and three for rigid-body translation). Regressors modeling each scan session were also entered into the design matrix.

In the second stage of the analysis, parameter estimates for the events of interest were estimated for each participant. Participant-specific parameter estimates were used to perform linear contrasts as implemented in SPM8, with participants serving as a random effect. To identify ‘core recollection’ regions, we inclusively masked the contrast between Remember and Know test trials (Remember > Know, p < 0.01 after correction for family-wise error rate (FWE), with a 22 voxel cluster extent threshold) with the separate contrasts for each class of study category (Remember-word > Know and Remember-picture > Know, each thresholded at p < 0.01 uncorrected). This analysis procedure identified regions where across-content recollection effects were both reliable and not driven solely by one class of content. To identify content-selective activity during study, we contrasted the two classes of study trials (word > picture and vice versa, each contrast thresholded at p < 0.01, FWE, with a 22 voxel cluster extent threshold).

2.4.2 Multi-voxel analyses

2.4.2.1 Feature selection

Classification analyses were conducted on regions of the core recollection network as well as the left dorsolateral prefrontal cortex (see below). To ensure that the voxels in each ROI were restricted to gray matter, recollection effects were inclusively masked with the default SPM gray matter probability map thresholded at p > 0.2. For each participant, identification of the recollection network ROIs was conducted in a manner that avoided circularity. For each of 24 randomly selected pairs of older and younger participants, ROIs were identified by performing the above-described univariate analysis on the data from the remaining 46 participants. The mean peak MNI coordinates for the different ROIs were as follows: angular gyrus: −46, −74, −26, with a mean (standard deviation) size of 348 (36.3) voxels, medial prefrontal cortex: −3, 65, 22, 732 (61.5) voxels; middle temporal gyrus: −54, −49, −11, 146 (41.3) voxels; retrosplenial/posterior cingulate cortex: −14, −53, 33, 190 (23.6) voxels; and hippocampus: −27, −11, −23, 55 (4.2) voxels. In light of the small number of hippocampal voxels that were identified, we elected to employ an anatomically defined anterior hippocampal mask which encompassed the peak identified by the above analysis. A mask consisting of 283 voxels was created by manually tracing the anterior hippocampus in both hemispheres using the across-participant mean anatomic image based on standard anatomical landmarks (Frisoni et al., 2015). Anterior hippocampus was defined as that portion of the hippocampus anterior to the site of the uncal apex (y = −21; Poppenk, Evansmoen, Moscovitch, & Nadel, 2013). Findings for the functionally defined hippocampal ROI did not differ from those reported below for the anatomically defined region.

The left dorsolateral prefrontal cortex ROI was defined as Brodmann area (BA) 46 according to the Wake Forest University PickAtlas v3.0 (Maldjian, Laurienti, Kraft, & Burdette, 2003). After inclusive masking with the gray matter mask (see above), the ROI comprised 268 voxels. Figure 2A illustrates the 6 ROIs for a representative participant. Figure 2B illustrates the results of the analysis of the study phase data for the univariate contrasts of word > picture study trials and vice versa.

Figure 2.

A. Features employed in the multi-voxel analyses for one representative participant overlaid on the across-participant mean anatomic image. Those features identified with the univariate analysis are shown in blue (i.e., angular gyrus (AG), middle temporal gyrus (MTG), retrosplenial/posterior cingulate cortex (RSP/PCC), and medial prefrontal cortex (MPFC)), those in cyan correspond to the hippocampus (HIPP), and in green are the features for the dorsolateral prefrontal cortex (DLPFC). B. Content-selective effects during study (regions where activity elicited by word exceeded that on picture trials in magenta, and the reverse contrast shown in red).

2.4.2.2 Multi-voxel analyses

Functional data from each ROI were subjected to several preprocessing steps prior to MVPA (see also, Koen & Rugg, 2016; Wang et al., 2016). First, functional image preprocessing was conducted as described above, but with omission of the spatial smoothing stage. Second, the data from each ROI were de-trended to remove linear and quadratic trends, and z-scored across voxels within each scanning session. Third, estimates of the voxel-wise BOLD response on each study and test trial were obtained by averaging the z-transformed BOLD signal between 6–10 s (corresponding to TRs 4–5) following the onset of each mini-block and test item, respectively¹. The single-trial BOLD signals in each voxel of the ROIs for the study data were regressed on reaction times (RTs) for the first trial of each mini-block to reduce possible RT-related confounds (Todd et al. 2013). An analogous procedure was conducted for the test data. The residualized single trial estimates for voxels in each ROI were then subjected to further z-scoring across both trials and voxels (the latter step removing mean signal differences between response categories). The resulting z-transformed values were used in the classification analyses described below.

MVPA was employed to assess whether the category of the study trial (picture or word) could be decoded from the patterns of neural activity elicited during retrieval within each of the six ROIs: angular gyrus, medial prefrontal cortex, middle temporal gyrus, retrosplenial/posterior cingulate cortex, hippocampus, and dorsolateral prefrontal cortex. We accomplished this by first training a classifier on the study phase data to discriminate between picture and word study trials (study-study classification). The classifier was then employed to classify Remember test trials as a function of the studied category (study-test classification). Classification was implemented with regularized logistic regression (L2 regularization) with a penalty parameter (λ) of 0.05 (Wang et al., 2016). For study-study classification, a three-fold cross-validation procedure was used in which data from two of the three study blocks were used to train the classifier, and the remaining study block was used to test classifier accuracy. Accuracy was defined as the proportion of correct study trial assignments (picture vs. word) across the three iterations of training and testing. Classification was deemed accurate if the returned classifier evidence for the correct study category was greater than 0.5 (i.e., chance). Accuracy was binarized to give a score of 1 for correct and 0 for incorrect classification. An ANOVA of the study-study classifier accuracies with factors of region and study category failed to reveal a region by category interaction or a main effect of category (Fs < 1, this and all following degrees of freedom were corrected for non-sphericity with the Greenhouse-Geisser procedure). Thus, there was no significant bias in the study-study classifier in favor one or the other category.

For study-test classification, a classifier trained on the study-phase data was applied to the test phase data. The mean (standard deviation) number of Remember-picture and Remember-word trials were 39 (14.5) and 44 (12.6), respectively. To ensure that the classifier was not biased toward the response category with the greater number of trials, for each participant, the response category with the larger number of trials was randomly subsampled 100 times to equate numbers with the response category containing the lower number of trials. Classifier accuracy was taken as the mean accuracy across the 100 iterations. As with the study-study classification, there was no bias in study-test classifier performance in favor of one of the study categories (Fs < 1).

2.4.2.3 Multi-voxel analyses, Remember and Know trials

The foregoing analysis assessed whether patterns of activity elicited by Remember test trials resembled those elicited during the study phase, as would be expected if content-dependent effects at test reflect the reinstatement of study processing (Danker & Anderson, 2010; Rissman & Wagner, 2012; Rugg et al., 2015). A second classifier-based analysis was conducted to assess whether classification of study history differed according to whether a test item was endorsed as Remember or Know (cf., Johnson et al., 2009; Wang et al., 2016). This analysis was restricted to those participants (14 young, 16 old) with at least 10 trials in each of the four response categories of interest (Remember-picture, Remember-word, Know-picture, and Know-word). The mean (standard deviation) number of trials of Remember-picture, Remember-word, Know-picture, and Know-word trials were 35 (12.3), 39 (12.5), 17 (6.6), and 26 (10.1), respectively. As in the analysis of the Remember trials described above, these analyses were conducted after the trial numbers had been equalized for each participant across the four response categories, again using 100 iterations of randomized sub-sampling.

To ascertain whether discrimination of study history differed according to recollection or familiarity, we conducted classification analyses identical to those described in the previous section (see Multi-voxel analyses, Remember trials), but separately on matched sets of Remember and Know trials. For the study-test classification, the trained study phase classifier was separately tested to discriminate Remember-picture and Remember-word trials and to discriminate Know-picture and Know-word trials. Classifier accuracy was then compared as a function of response category (Remember vs. Know). Consistent with the analyses described above, an ANOVA with factors of region and study category conducted on the study-test accuracy values for each classifier (Remember and Know), failed to reveal either region by category interactions (Fs(4.47,129.49) < 1.07, ps > 0.05) or main effects of category (Fs (1,29) < 2.96, ps > 0.05), indicating that the classifiers were not biased in favor of one of the categories.

3. Results

3.1 Behavioral results

These results were first reported in Wang et al., (2016). Figures 3A and 3B illustrate estimates of recollection and familiarity as a function of study category and age. Estimates of recollection were calculated as the probability of a Remember response to a studied item minus the probability of a Remember response to a new item. Estimates of familiarity were calculated in an analogous fashion after correction for independence between recollection and familiarity (Yonelinas & Jacoby, 1995). An ANOVA on the recollection estimates with factors age and study category revealed main effects of age (F(1, 46) = 4.70, p < 0.05) and category (F(1, 46) = 5.71, p < 0.05). The effects reflected higher recollection estimates for younger participants and for picture relative to word trials, respectively. An analogous ANOVA on the familiarity estimates revealed no effect of age but a main effect of category (F(1, 46) = 74.03, p < 0.001), reflecting greater familiarity estimates for words than for pictures.

Figure 3.

A. Recollection estimates as a function of study category and age. B. Familiarity estimates as a function of study category and age. C. RTs for Remember responses as a function of study category and age. D. RTs for Know responses as a function of study category and age.

Figures 3C and 3D illustrate the reaction times associated with Remember and Know responses as a function of study category and age. ANOVA with factors age, category, and memory judgment revealed a main effect of category (F(1, 46) = 18.53, p < 0.001), a main effect of memory judgment (F(1, 46) = 90.98, p < 0.001), and an interaction between category and judgment (F(1,46) = 9.26, p < 0.005). The interaction was driven by the significantly longer reaction times for picture than word trials given Know responses (t(47) = 6.31, p < 0.001), with no reliable difference for the analogous contrast for Remember trials.

3.2 fMRI results

Consistent with our previously reported findings from the same dataset (Wang et al., 2016), we found no hint in any of the analyses reported below of differences between the two age groups in classifier performance (across ROIs and analyses of both Remember and Know trials, pairwise contrasts between classifier accuracy values were uniformly non-significant; minimum p (uncorrected) = 0.12, two-tailed). Thus, for clarity’s sake all analyses are reported for classifier performance collapsed across the two groups (N = 48) as well as, for the reasons noted in the Methods section, study category. Study-study classification accuracies for each ROI are illustrated in Figure 4A. Here and in the analyses reported below, one-tailed tests were employed to assess whether accuracy was greater than the chance value of 0.5. The outcome of each test was considered significant only if the p value survived Bonferroni correction for multiple comparisons; 0.05/6 regions, giving a corrected p < 0.0083. In every case other than the hippocampus, classifier accuracy was significantly greater than the chance value of 0.5 (ts(47) > 4.72).

Figure 4.

A. Study-study classification accuracy for each of the regions analyzed. B. Study-test classification accuracy (in this and subsequent figures, * denotes corrected p < 0.05, ⁺ denotes uncorrected p < 0.05).

3.2.1 Remember trials

We next assessed whether the study phase classifier (now trained on all three study phases) could discriminate Remembered test trials according to their study history (Figure 4B). As is evident from the figure, accuracy was significantly greater than chance (ts(47) > 4.15) in every region except for the medial prefrontal cortex, and of necessity, the hippocampus, where study-study classification had failed. Given the significant study-study classifier performance together with the fact that previous MVPA studies have observed reinstatement effects within the medial prefrontal cortex (Kuhl & Chun, 2014; see also, Johnson et al., 2009, 2013), we are inclined to attribute the lack of a reliable reinstatement effect in this region to type II error (in support of this possibility, we note that study-test classification in this region was reliable before correction, p = 0.02). Equivalent findings to those just described were obtained when the continuous variable of classifier evidence, rather than the binarized variable of classifier accuracy, was employed as the dependent variable.

3.2.3 Remember and Know trials

As noted in the Methods section, we performed additional analyses to assess whether study-test classifier accuracy differed between Remember and Know trials (cf., Johnson et al., 2009; Wang et al., 2016). Echoing the findings from our analyses described above, as is illustrated in Figure 5, study-test accuracy for Remember trials was significantly greater than chance in the angular gyrus, middle temporal gyrus, retrosplenial/posterior cingulate cortex, and dorsolateral prefrontal cortex (ts(29) > 3.68). Study-test accuracy for Know trials was also significantly greater than chance in the angular gyrus, middle temporal gyrus, retrosplenial/posterior cingulate cortex, and dorsolateral prefrontal cortex (ts(29) > 2.87). A two-way repeated measures ANOVA (factors of region and memory judgment) conducted on the accuracy values failed to reveal either a main effect of judgment or a region by judgment interaction (Fs < 1). Thus, classifier accuracy was statistically indistinguishable for test trials associated with recollection- or familiarity-based recognition. As in the prior analyses, equivalent findings were obtained when examining classifier evidence.

Figure 5.

Study-test classification accuracy for Remember and Know trials.

4. Discussion

We employed MVPA to assess whether patterns of neural activity in members of the core recollection network and the left dorsolateral prefrontal cortex demonstrate recollection-related cortical reinstatement. With the exception of the hippocampus, an identical pattern was observed. Specifically, in angular gyrus, middle temporal gyrus, retrosplenial/posterior cingulate cortex and dorsolateral prefrontal cortex and, less robustly, medial prefrontal cortex, study-study and study-test classification of recollection-related neural patterns were reliably above chance. In follow-up analyses, we examined whether these findings were specific to recollection, or whether they extended to familiarity-based (Know) test trials. Strikingly, classifier accuracy did not differ as a function of memory judgment. Thus, despite the fact that the core recollection network is defined by its sensitivity to recollection success, we could find no evidence that reinstatement effects within members of the network – at least when these effects were operationalized by the accuracy of a multi-voxel classifier – were selective or enhanced for recollection.

The present findings replicate prior results in that they indicate that retrieved episodic content can be decoded from regions belonging to the core recollection network (see Introduction). Intriguingly, the findings are also consistent with previous studies indicating that patterns of neural activity within the ‘general semantic network’ (Binder, Desai, Graves, & Conant, 2009; Binder & Desai, 2011) – a set of regions that, with the exception of the hippocampus, closely overlaps the core recollection network – carry information about semantic content (e.g., Fairhall & Caramazza, 2013; Fernandino, Humphries, Conant, Seidenberg, & Binder, 2016).

Importantly, as noted above, here we examined whether study content could be decoded not only from recollection-related patterns of neural activity, but also from neural activity associated with test items endorsed as familiar only. In two prior studies we also assessed whether study content could be decoded from test trials associated with familiarity-based recognition (Johnson et al., 2009; Wang et al., 2016), although in neither of these cases was the analysis of reinstatement effects targeted at the core recollection network (see Introduction). In both of these prior studies familiarity-based recognition was associated with robustly above-chance classification, albeit at a somewhat lower level than was the case for recollection. In the present experiment, accuracy of study-test classification for Know judgments not only significantly exceeded chance in several regions, but in addition did not significantly differ from trials attracting Remember judgments. Thus our prior findings of stronger reinstatement for recollection-than for familiarity-based judgments appear not to extend to regions demonstrating content-independent recollection effects. The present findings are somewhat reminiscent of those reported by Kuhl and Chun (2014). As was noted in the Introduction, in the case of retrieval-related activity in the left angular gyrus and ventral temporal cortex, these authors reported that the study category (face vs. scene) of the recalled associate of a test item could be classified with equivalent accuracy regardless of the vividness with which the item was recalled. Together with the present findings, this suggests that reinstatement effects indexed by MVPA classifier accuracy can be largely insensitive to the subjective experience accompanying successful episodic retrieval.

Together with the reliable cortical reinstatement effects for familiarity-based recognition judgments obtained in our prior studies, the present findings are, at first sight, inconsistent with the commonly held assumption that cortical reinstatement is exclusively associated with recollection, and is mediated by hippocampal retrieval operations (Gordon, Rissman, Kiani, & Wagner, 2014; Leiker and Johnson, 2015; Ritchey et al., 2013; for review see Rugg et al., 2015). At the least, our prior and present findings suggest that although reinstatement - as it is operationalized by multi-voxel classifiers - may be necessary for the subjective experience of recollection, it is not sufficient. These observations may not, however, extend to ‘objective’ indices of successful recollection such as source memory. In two prior studies (Gordon et al., 2014; Leiker & Johnson, 2015), it was reported that MVPA classifier accuracy co-varied with the strength of the memory signal supporting accurate source memory judgments, findings consistent with idea that reinstatement contributes to the accuracy and confidence of such judgments (see Thakral et al., (2015), for analogous findings from a univariate fMRI analysis approach). Should this apparent test-based dissociation in the relationship between MVPA-derived reinstatement effects and strength of recollection be confirmed, it would add to the evidence that subjective and objective indices of recollection are functionally distinct (Duarte, Henson, & Graham, 2008; Rugg et al., 2012; Slotnick, 2010; Spaniol, Davidson, Kim, Han, Moscovitch, & Grady, 2009).

It should also be noted that the present findings may not generalize to MVPA approaches that assess similarity of study-test overlap at the single item level rather than the categorical level as in the present case. In two studies (Ritchey et al., 2014; Wing et al., 2015), the strength of item level reinstatement was reported to co-vary with subjective ratings of the strength or vividness of retrieval (although, in a finding somewhat consistent with the present findings and those of Kuhl and Chun (2014), in Wing et al., (2015) reinstatement effects in the retrosplenial/posterior cingulate cortex were insensitive to vividness ratings). Unfortunately, the present experimental design, which employed mini-blocks during the study phase, precludes an item-level similarity analysis of the current data. It will be of considerable interest to directly compare the outcomes of categorical and item-level MVPA in the future (see Koen and Rugg (2016) for evidence that the two approaches identify functionally distinct reinstatement effects).

All that said, the question remains as to why classifier accuracy failed to differentiate recollection- and familiarity-based recognition judgments in the present experiment. One possibility arises from a theoretical perspective that supposes that, like the memory signal that supports familiarity, recollection depends on a ‘continuous’ rather than a ‘threshold’ signal (Ingram, Mickes, & Wixted, 2012; Slotnick, 2013; Wixted & Mickes, 2010). By this account, all test items elicit a recollection signal to some extent, and whether an item is endorsed as recollected or familiar largely depends on the placement of a response criterion. From this perspective, it is possible that our MVPA approach was simply unable to discriminate between relatively ‘strong’ and ‘weak’ recollection signals (i.e., Remember and Know trials, respectively). A related possibility comes from the fact that, as already alluded to, MVPA classifiers are sensitive exclusively to patterns of neural activity that discriminate between different study trials at the categorical level, and hence are sensitive only to patterns that are shared between trials belonging to a given category of trials. This means that a classifier-based analysis is insensitive to neural patterns that individuate study trials, but is highly sensitive to something akin to differences in the ‘gist’ information shared by the trials belonging to a given study condition. Thus, to the extent that familiarity-based recognition judgments are driven by the retrieval of gist information (e.g., Brainerd, Reyna, & Mojardin 1999; Brainerd & Reyna, 2001), the present findings could be interpreted as evidence that cortical regions belonging to the core recollection network (along with the dorsolateral prefrontal cortex) are sensitive to gist as well as more highly differentiated, episode-specific information. An attractive feature of this account is that it offers an explanation for our failure to find evidence of reinstatement effects in the hippocampus: unlike the cortical regions with which it interacts, the hippocampus is held to represent information in the form of sparse, highly differentiated representations that support the individuation of specific episodes at the expense of generalization across episodes (e.g., Marr, 1971; McClelland et al., 1995; Norman and O’Reilly, 2003). A very different possibility of course is that, counter to most prevailing theories (see Introduction), familiarity-based recognition, like recollection, depends on hippocampally-mediated reinstatement of patterns of encoding-related neural activity.

One final possibility, shared with most, if not all, prior fMRI studies of reinstatement, is worth noting. According to this account, the reinstatement effects identified here reflect a pre-condition for, rather than a consequence of, successful retrieval. By this argument, the reinstatement effects are a reflection of the benefit to memory retrieval that accompanies recapitulation of study processing elicited by the presentation of a retrieval cue (as exemplified in the principles of ‘transfer appropriate processing’ (Morris, Bransford, & Franks, 1977), and ‘encoding specificity’ (Tulving & Thomson, 1973)). A similar notion – context reinstatement – is also found in the ‘temporal context’ model of memory (Sederberg, Howard, & Kahana, 2008; see Polyn, Natu, Cohen, and Norman (2005) for MVPA evidence of context reinstatement). If it is assumed that both recollection- and familiarity-based recognition benefit when there is overlap between study and cue processing, the present findings can be explained without recourse to the idea that recollection and familiarity share retrieved content.

The four possibilities outlined in the preceding two paragraphs are not mutually exclusive. Further research, including studies that employ methods with greater temporal resolution than that afforded by fMRI (c.f., Johnson, Price, & Leiker, 2015), will be required to disentangle their respective contributions to the present and prior findings. Importantly, these uncertainties do not detract from the finding that patterns of retrieval-related activity within members of the core recollection network (and in the left dorsolateral prefrontal cortex) contain information sufficient to allow decoding of the study history of a retrieval cue.

The question remains as to what drove the phenomenological experience of recollection in the present study. A seemingly obvious answer lies in the enhancement in mean signal in the core recollection network that accompanies Remember judgments. That is, differences in mean signal between Remember and Know test trials may reflect neural activity uniquely associated with processes that support the experience of recollection. However, this possibility is unlikely in light of evidence that mean signal within the core network varies as a function not of the subjective experience of recollection, but of the amount of contextual information retrieved (Yu et al., 2012a, 2012b; Rugg et al., 2012). An alternative possibility is that univariate reinstatement effects in regions where mean signal varies as function of studied content support the phenomenal experience of recollection (Rugg, Johnson, Park, & Uncapher, 2008; Thakral et al., 2015; Wang et al., 2016; see also, Thakral & Slotnick, 2015). Another possibility is that the neural correlates of recollective experience are manifest in patterns of neural activity that are difficult or impossible to detect with fMRI, such as changes in the inter- or intra-regional synchrony of oscillatory activity (cf. Melloni, Molina, Pena, Torres, Singer, & Rodriguez, 2007).

The present findings add to the existing evidence that univariate and multi-voxel analyses can give rise to dissociable results (e.g., Emrich, Riggall, LaRocque, & Postle, 2013; Epstein & Morgan, 2012; Hsieh, Colas, & Kanwisher, 2012; Jimura & Poldrack, 2012), albeit not necessarily for theoretically incisive reasons (Davis, LaRoque, Mumford, Norman, Wagner, & Poldrack, 2014). While a wealth of univariate evidence indicates that the angular gyrus, retrosplenial/posterior cingulate cortex, and middle temporal gyrus are sensitive to the ‘occurrence’ and not to the content of recollection, MVPA identified retrieval effects in these regions that were sensitive to the nature of the retrieved content but insensitive to whether retrieval was recollection- or familiarity-driven. Moreover, MVPA revealed content-sensitivity within the dorsolateral prefrontal cortex, despite the absence of detectable recollection-related mean signal change in the region (although see King et al., (2015) for other evidence implicating the dorsolateral prefrontal cortex in recollection-related processing). As has been previously noted, there are several reasons why mean signal and multi-voxel findings might diverge (Davis et al., 2014). Notably, multi-voxel and mean signal analytic approaches are differentially sensitive to voxel-level and participant-level variability, respectively. Whatever the reasons for the dissociations between the approaches observed here, the current results highlight the complementary nature of multi-voxel and mean signal analyses.

Finally, we briefly note that as in the original analysis of the same dataset (Wang et al., 2016), we failed to find any evidence of age-related differences in the strength of cortical reinstatement effects. To the best of our knowledge, prior to Wang at al. (2016), only three studies have examined cortical reinstatement effects as a function of age (Dulas & Duarte, 2012; McDonough et al., 2014; St-Laurent et al., 2014), with mixed findings (see Wang et al., (2016) for review and discussion). The null effects of age reported here clearly corroborate our earlier result (Wang et al., 2016). Nevertheless, these null findings should be treated with caution. As was discussed in Wang et al., (2016) and above, the employment of a classifier-based approach precludes assessment of reinstatement effects at the trial- rather than the categorical level (cf. Ritchey et al., 2014; Wing et al. 2015). It remains to be established whether episode-specific reinstatement is as insensitive to age as categorical reinstatement effects appear to be.

5. Conclusion

The current findings add to evidence that members of the core recollection network, as well as at least one neural region where mean signal is seemingly insensitive to recollection success, carry information about the study history of a retrieval cue. Importantly, the study history of recognized items endorsed with a Remember or a Know judgment could be decoded with equal accuracy. Thus the results demonstrate a striking dissociation between univariate and multi-voxel indices of recollection. Moreover, they indicate that, as it is assessed with classifier-based MVPA, cortical reinstatement is not uniquely a signature of successful recollection, at least when recollection is operationalized by phenomenal report. Our findings demonstrate the importance of employing both univariate and multi-voxel analysis methods to identify neural correlates of episodic retrieval and, ultimately, their functional significance.