Temporal Memory Is Shaped by Encoding Stability and Intervening Item Reactivation

Abstract

Making sense of previous experience requires remembering the order in which events unfolded in time. Prior work has implicated the hippocampus and medial temporal lobe cortex in memory for temporal information associated with individual episodes. However, the processes involved in encoding and retrieving temporal information across extended sequences is relatively poorly understood. Here we used fMRI during the encoding and retrieval of extended sequences to test specific predictions about the type of information used to resolve temporal order and the role of the hippocampus in this process. Participants studied sequences of images of celebrity faces and common objects followed by a recency discrimination test. The main conditions of interest were pairs of items that had been presented with three intervening items, half of which included an intervening category shift. During encoding, hippocampal pattern similarity across intervening items was associated with subsequent successful order memory. To test for evidence of associative retrieval, we trained a classifier to discriminate encoding patterns associated with faces versus objects and applied the classifier on fMRI patterns during recency discrimination. We found evidence that the category content of intervening items was reactivated during recency judgments, and this was related to hippocampal encoding-retrieval similarity. A follow-up behavioral priming experiment revealed additional evidence for intervening item reinstatement during temporal order judgments. Reinstatement did not differ according to whether the items occurred within a single context or across context boundaries. Thus, these data suggest that inter-item associative encoding and retrieval mediated by the hippocampus contribute to temporal order memory.

Introduction

Understanding events from our past requires remembering the broad temporal context during which the events occurred as well as the specific temporal relationships between representations. Recent research has begun to characterize the hippocampal processes that support the encoding of temporal relationships (Eichenbaum, 2013; Ezzyat and Davachi, 2014; Davachi and DuBrow, 2014). These mechanisms focus on how temporal gaps can be bridged during an experience to facilitate the associative binding of representations occurring across time. However, cognitive theories have proposed that temporal memory decisions may be supported by the memory strength of individual items and item-location associations as well as associations between items across time (Friedman, 1993; Marshuetz, 2005).

Human neuroimaging investigations into the mechanisms supporting the encoding of temporal information have implicated the hippocampus and surrounding medial temporal lobe (MTL) cortex with subsequent order memory (Jenkins and Ranganath, 2010; Tubridy and Davachi, 2011). Recent work has indicated that multivariate hippocampal patterns contain temporal information (Schapiro et al., 2012; Ezzyat and Davachi, 2014; Hsieh et al., 2014), yet it remains unclear whether and how hippocampal activity across multiple items in a sequence is related to explicit temporal order judgments. We recently provided behavioral evidence that temporal order discriminations between items separated in time can be supported by intact associative relationships between the intervening representations that span the sequence (DuBrow and Davachi, 2013). Specifically, we showed that temporal order memory retrieval was reduced when inter-item associative binding during encoding was disrupted by either a dual task or a context shift. Since it is well established that the hippocampus is critical in associative encoding (Davachi, 2006; Eichenbaum et al., 2007), hippocampal processes may play an important role in establishing and retrieving temporal order across extended sequences.

A number of prior fMRI studies have also implicated the hippocampus and surrounding parahippocampal cortex in the successful retrieval of temporal order (Konishi et al., 2002, 2006; Dudukovic and Wagner, 2007; St Jacques et al., 2008; Lehn et al., 2009). Critically, the network of regions implicated in the recovery of temporal order overlaps with the regions involved in associative or source recollection (Dudukovic and Wagner, 2007), consistent with the notion that memory for temporal information involves associative retrieval or the recollection of temporal source information. Thus, one possibility is that the hippocampus supports the retrieval of temporal information through the reinstatement of event details, including the sequence of items. However, while the network of brain regions that shows order memory success effects is consistent with this associative retrieval account, neural evidence for the reinstatement of intervening information during temporal order retrieval has not been tested directly.

The current fMRI study aimed to understand the nature of the representations used during temporal order memory judgments of items learned across temporal gaps with intervening items. Specifically, using a multivoxel pattern similarity approach, hippocampal pattern maintenance during encoding was analyzed as a function of subsequent order memory. To the extent that inter-item associative encoding supports the ordering of items within an extended sequence, we predicted that greater hippocampal pattern maintenance would be associated with successful order memory. During retrieval, we tested the hypothesis that recency discrimination involves intervening item reinstatement by measuring category-level evidence for intervening representations. In a follow-up behavioral study, we further characterized intervening item reinstatement using a recognition-priming task following recency discrimination. By inducing context changes or “boundaries” during sequence encoding, which have been shown to disrupt inter-item associative binding in episodic memory (Ezzyat and Davachi, 2011; DuBrow and Davachi, 2013), we were able to compare mechanisms involved in temporal order memory for sequences that varied in the content of intervening associations.

Materials and Methods

Experiment 1 participants.

Twenty-five right-handed, native English speakers (17 female; age range: 18–28, mean = 22) participated for pay ($25/h). Participants were recruited from New York University and the broader community. Informed consent was obtained in a manner approved by the University Committee on Activities Involving Human Subjects. One participant was excluded from analysis due to chance performance on the memory task. We additionally excluded one participant for excessive head motion (>1 voxel) and six for low trial numbers (<7 in at least one condition) from fMRI analysis.

Experiment 1 procedure.

Stimuli consisted of 200 color images each of celebrity faces and nameable objects modified from sets used in previous studies (Polyn et al., 2005; Kuhl et al., 2011; DuBrow and Davachi, 2013). Assignment of stimulus order and condition was randomized for each participant.

The experiment consisted of a short practice, 16 study-test rounds in the scanner, and a final test outside of the scanner. During each study round, participants were presented with a series of 25 images (faces and objects; Fig. 1A) with their corresponding verbal label. Participants were instructed to memorize the order of the images and were encouraged to use an associative encoding strategy. Additionally, for each image presentation, participants were instructed to make a category-specific judgment for up to 2 s. On a scale of 1 to 4, participants rated how much they liked each celebrity face and how often they had encountered each object. Each list contained five boundary items for which the category of the image, and thus the category-specific judgment, switched from the preceding trial. The intertrial interval (ITI) was pseudorandomized to be 4, 6, or 8 s to maximally orthogonalize conditions for fMRI analysis while maintaining the temporal lag between pairs of interest constant.

Figure 1.

Schematic of behavioral task and predictions. A, During encoding, participants were presented with lists of 25 images and instructed to remember their order. Images were either celebrity faces, for which participants made likability judgments, or objects, for which participants made commonality judgments. B, Following a distractor task, participants were presented with pairs of images and were instructed to select the image that had been presented more recently (R2, arrows) and indicate their confidence. The main conditions of interest were no switch (blue), which came from sequences of same-category items, and switch (orange), which came from sequences that had intervening context switches.

After each study list, a 45 s arrow distractor task was presented followed by the recency test period. During this test, 12 image pairs were tested. Each trial consisted of two images (henceforth called the R1 and R2 items, where R1 occurred first in the sequence) randomly assigned to the left and right sides of the screen. The prompt “more recent?” appeared, and participants were instructed to select the side of the screen with the more recent (R2) image and indicate their confidence in their response. Thus, they selected one of four buttons (high and low confidence for each of the two images) with their right hand. The recency test was self-paced with a maximum response time of 8 s per trial followed by a fixed ITI of 9 s during which the arrow task was again performed. All items presented in the recency test had appeared in the immediately preceding study list. The two main conditions of interest were image pairs that had been presented with three intervening images (“lag-3”) that were either all from the same category train (“no switch”) or had been presented across two category switches (“switch,” Fig. 1). The switch condition consisted of images that occurred across two boundaries so that the images were drawn from the same stimulus category (i.e., either two faces or two objects). In addition to the no switch and switch conditions, neighboring items from the same category train (“neighbors”) and same-category items presented across seven to nine intervening images that included switches (“long”) were also tested. Last, four test pairs that consisted of images presented across a single boundary, and thus were of different categories, were included so as not to discourage sequential binding across category switches. The lag-3 no switch and switch conditions were matched for serial position across lists. The other distance conditions could not be perfectly matched for list position because they were different lengths. Thus, we chose to match according to their mean serial position (e.g., items occurring at positions 3 and 11 would be considered to be in position 7). Pairs were then matched according to their mean positions within list fifths. That is, each item pair in one condition had a “matching” counterpart in each of the other conditions of interest that was within five positions. Each condition of interest (no switch, switch, neighbors, and long) was tested twice per round, and the order of test trials was randomized.

Following recency discrimination, participants performed serial recall. Their responses were recorded via the intercom in the MRI scanner on a handheld recording device. Specifically, participants were instructed to recall the names of the images from the preceding list in the order in which they had been presented. If they failed to recall an image, they were instructed to proceed to the next remembered image. They were given up to 90 s to complete serial recall for each list. Serial recall data are not reported here.

fMRI data analysis.

MRI data were acquired on a 3T Siemens Allegra head scanner. Functional data were collected using an EPI pulse sequence (34 contiguous slices oriented parallel to the AC–PC axis; TR = 2000 ms; TE = 15 ms; flip angle = 82°). The first four volumes of each run were discarded to allow for T1 stabilization. A high-resolution, T1-weighted anatomical scan (magnetization-prepared rapid-acquisition gradient echo sequence, 1 × 1 × 1 mm) was also obtained for each subject. Images were preprocessed in SPM8 (Wellcome Trust Centre for Neuroimaging, London UK). Functional images were slice time corrected, realigned to the first images across all runs to correct for head motion, coregistered to the high-resolution anatomical image, and high-pass filtered (<3 cycles per run removed). No smoothing or normalization was performed.

Individual trials were modeled in MATLAB (The MathWorks) using an iterative general linear model approach (Hsieh et al., 2014), which has shown some improvement in the estimation of single trials in fast event-related designs (Mumford et al., 2012). To this end, a separate model was run for every trial in the experiment in which one regressor corresponded to the trial of interest and additional regressors were included for each trial type. Trials were grouped according to category (face vs object) and memory stage (encoding vs retrieval) and entered into the model as nuisance regressors in addition to the trial of interest. Each trial was modeled with a 2-TR boxcar starting at the onset of a trial convolved with a canonical hemodynamic response function. This procedure resulted in a parameter estimate for each trial in the experiment for every voxel in the brain.

The first analysis examined hippocampal subsequent order memory effects. The bilateral hippocampi of each subject were segmented using an automated tool in FSL (Patenaude et al., 2011), and manually edited to ensure correct identification. Parameter estimates for each encoding event corresponding to the to-be-discriminated R1 and R2 items were extracted across the hippocampus. To calculate univariate responses, parameter estimates were averaged across the hippocampus for each recency item. To calculate multivoxel pattern similarity, the pattern of parameter estimates across the hippocampus in response to the R1 and R2 items was correlated. Note the overall time between these items was fixed at 30 s. The resulting r values were z-transformed. Univariate responses and correlation values were sorted according to condition and subsequent order memory. High confident (HC) correct responses were considered successful memory trials, and low confident (LC) correct responses were binned with incorrect responses to increase power. To assess the relationship between hippocampal encoding activity and subsequent memory, a mixed-effects logistic regression was run with memory success as the dependent measure. For each trial, univariate responses to the first and second to-be-discriminated items and the correlation between the two were entered into the model. The logistic regression was run separately for the switch and no switch conditions.

Reactivation of intervening representations during recency discrimination was assessed using a multivoxel pattern classification approach. A penalized logistic regression classifier with L2-norm regularization (penalty parameter = 100) was implemented using the Princeton Multi-Voxel Pattern Analysis Toolbox (http://www.pni.princeton.edu/mvpa). Before classification, the top 1000 voxels from the whole brain that best differentiated faces versus objects were selected by running an ANOVA on the univariate data. The classifier was then trained to discriminate face versus object trials during encoding and was tested on the recency discrimination data. This procedure resulted in a probability estimate for each of the two categories that ranged between 0 and 1 for every test trial. The log odds associated with the on-screen category was then taken to produce an estimate of category evidence for every recency discrimination trial.

To probe whether category evidence on each retrieval trial exhibited information related to the reinstatement of specific intervening representations, we computed a correlation between category evidence during recency discrimination and hippocampal encoding-retrieval similarity (ERS; Staresina et al., 2012; Ritchey et al., 2013). Classifier analyses were rerun with the hippocampal data removed so that the relationship between the hippocampal activity patterns and category evidence in the rest of the brain during recency discrimination could be tested on a trial-by-trial basis. Hippocampal ERS was assessed by computing the correlations between the activation patterns corresponding to the three intervening encoding trials and the response pattern to the corresponding retrieval trial. The resulting Pearson correlation r values were z-transformed and averaged to produce a single ERS value for which the items from the encoding sequence of interest did not overlap perceptually with its corresponding retrieval trial. This measure was entered as a predictor of category evidence into a mixed-effects regression implemented in R (http://www.r-project.org/). The model additionally included univariate hippocampal encoding and retrieval activations computed by averaging parameter estimates across all hippocampal voxels. For encoding, the estimates associated with the three intervening stimuli were averaged to produce a single estimate of the hippocampal response to the sequence spanning the to-be-discriminated items. Finally, a category-level ERS, computed by taking the average pairwise correlation between the retrieval trial and all the same category encoding trials other than the five items in the corresponding sequence, was included in the model to account for correlated hippocampal patterns due to shared category content.

Experiment 2 participants.

Thirty-two, native English speakers (28 female; age range: 18–24, mean = 20) participated for credit. Participants were recruited from the New York University Psychology study pool. Informed consent was obtained in a manner approved by the University Committee on Activities Involving Human Subjects. Three participants were excluded from analysis due to chance performance on the memory task.

Experiment 2 procedure.

The experimental procedure was largely similar to that of Experiment 1 with a couple of modifications. Most importantly, each recency discrimination trial was followed by an old/new recognition trial to test priming of intervening items. Participants were instructed to make their recency response within 8 s with their left hand. As soon as a response was made, a cue displayed “old/new” for 0.5 s followed by a recognition probe. Participants were instructed to make a response with their right hand indicating either “old” with their index finger or “new” with their middle finger as quickly as possible without sacrificing accuracy. Following switch and no switch trials (two each per list), participants were either presented with the middle item between the R1 and R2 items or a control item. The recognition probes following no switch trials were always the same category as the recency items, whereas the recognition probes following switch items were always the other category. Since we were interested in recognition priming of intervening items in the two conditions, recency items and probes were approximately matched for list position across lists and conditions (within three items), which required expanding the lists to 31 items. In addition to the four trials of interest tested for recency in each list, four additional across-category recency pairs were tested so as not to discourage sequential encoding across boundaries. Of those, two were followed by new recognition probes. Two additional same-category pairs followed by new items were also tested. Thus, 10 recency pairs were tested in each list, and of these 6 were followed by old recognition probes and 4 by new recognition probes. The ITI following each recognition trial was 1.5 s followed by a 0.5 s recency cue before the next recency pair was presented.

Results

Experiment 1: behavioral results

Encoding

Response times (RTs) during encoding trials were on average 887 ms (SD = 212 ms). Importantly, a significant RT cost was evident for “boundary” items, the first items of each category train (boundary M = 922 ms, SD = 214 ms; nonboundary M = 864 ms, SD = 222 ms; t₍₂₃₎ = 3.97, p < 0.001).

Retrieval

Mean accuracy for recency discriminations across all conditions was 0.767 (SD = 0.098). As predicted, there was an effect of the lag between recency pair items such that recency performance increased as a function of increasing lag between presented items. Specifically, performance on neighboring items (M = 0.657, SD = 0.130) was less than lag-3 items (no switch M = 0.785, SD = 0.119; no switch> neighbors, t₍₂₃₎ = 5.94, p < 0.001; switch M = 0.731, SD = 0.117; switch > neighbor t₍₂₃₎ = 3.77, p < 0.001), and performance on long lag items (M = 0.845, SD = 0.120) was greater than on lag-3 items (no switch < long, t₍₂₃₎ = 2.92, p = 0.008; switch < long, t₍₂₃₎ = 5.66, p < 0.001). Critically, replicating our prior results (DuBrow and Davachi, 2013), we observed that even when lag was the same (lag-3), recency performance on no switch pairs (M = 0.785, SD = 0.119) was significantly better than performance on switch pairs (M = 0.731, SD = 0.117; t₍₂₃₎ = 3.08, p = 0.005; Fig. 2).

Figure 2.

Behavioral performance for Experiment 1. Proportion correct on recency discrimination in the four same-category conditions. Error bars reflect SEM; *p < 0.05.

The average RT for correct recency discriminations was 2.45 s (SD = 0.45 s). Paralleling the accuracy data, some lag effects were present (neighbors M = 2.63 s, SD = 0.53 s; no switch M = 2.50 s, SD = 0.44 s; switch M = 2.57 s, SD = 0.50 s; long M = 2.49 s, SD = 0.49 s; neighbors > no switch, t₍₂₃₎ = 2.10, p = 0.047; neighbors > long, t₍₂₃₎ = 2.19, p = 0.039), but not others (lag-3 > long, neighbors > switch, ps > 0.242). Notably, there was no significant difference between the no switch and switch conditions (t₍₂₃₎ = 1.42, p = 0.170).

Experiment 1: fMRI results

Encoding: hippocampal correlations

To test whether the maintenance of hippocampal BOLD activity patterns across extended sequences is related to subsequent memory for the order of items in the sequence, we ran a mixed-effects logistic regression to assess encoding factors that predict memory (LaRocque et al., 2013). For each tested sequence, we included the mean trial-evoked univariate activity for subsequently tested R1 and R2 items as well as the similarity between R1 and R2 multivariate response patterns, as predictors of later recency memory outcome (HC correct vs LC/incorrect). In the left hippocampus, R1-R2 pattern similarity was a significant predictor of memory success (B = 0.99, SE = 0.39, X²(1) = 6.31, p = 0.01; Fig. 3), while univariate responses did not predict memory outcome (ps > 0.24). Furthermore, R1-R2 pattern similarity was a significantly better predictor than the mean trial-evoked responses (B = 1.98, SE = 0.79, X²(1) = 6.32, p = 0.01). In the right hippocampus, neither univariate activity nor pattern similarity was a significant predictor of memory outcome (ps > 0.25).

Figure 3.

Hippocampal pattern similarity predicts subsequent order memory. Parameter estimates from a logistic regression including the predictors of hippocampal univariate activation (“act”) for the first (R1) and last (R2) items from each sequence, as well as the multivoxel pattern correlation between them (“sim”). Hippocampal pattern similarity predicted subsequent recency discrimination success. Error bars reflect SEM; *p < 0.05, ∼p < 0.10.

To assess whether this effect differed between the switch and no switch conditions, we ran the same logistic regression within condition in the left hippocampus. In the switch condition, pattern similarity was significantly related to memory success (B = 1.26, SE = 0.63, X²(1) = 4.04, p = 0.04) and in the no switch condition, this relationship was marginal (B = 1.12, SE = 0.67, X²(1) = 2.81, p = 0.09). Thus, no interaction was observed between switch condition and pattern similarity (B = −0.05, SE = 0.89, X²(1) = 0.003, p = 0.95).

Retrieval: intervening category evidence during recency discrimination

To ask whether there is evidence for reinstatement during recency discrimination, we trained a whole-brain classifier on the encoding data to distinguish face versus object encoding and applied it to each recency trial during retrieval. The retrieval trials of interest (no switch and switch trials) were always drawn from the same category and, thus, were perceptually matched but differed mnemonically in the content of the intervening items (Fig. 4A). Thus, differences in classifier evidence between conditions should be related to mnemonic representations of the category of the intervening items presented during encoding, which may be related to the reactivation of the sequence. To match for memory status, we compared on-screen category evidence for HC correct trials across the two conditions. This comparison revealed significantly greater on-screen category evidence in the no switch condition than the switch condition (t₍₁₆₎ = 2.32, p = 0.034; Fig. 4B). That is, using the example in which the R1 and R2 items are both faces, this result suggests that there is relatively more face evidence in the no switch condition in which intervening items are exclusively faces compared with the switch condition in which two of the intervening items are objects (Fig. 1A). This difference between conditions was not present for LC and incorrect trials (t₍₁₆₎ = 0.367, p = 0.719). The accuracy by condition interaction, however, was not significant (F_(1,13) = 1.93, p = 0.184). To ensure that subthreshold differences in RTs could not account for the condition differences in classifier evidence, we assessed the effects of recency RTs on the classifier output by correlating these two measures within subjects for each of the single-category conditions (neighbors, no switch, switch, and long). The average z-transformed correlation value was 0.006 (SD = 0.085; t₍₁₆₎ = 0.30, p = 0.768).

Figure 4.

Classification of category during recency judgments. A, Classifier predictions. The perceptual account (left) predicts no difference in classifier evidence because the presented images are both faces. The memory account (right) predicts differences in classifier evidence between conditions based on the category content of the intervening items spanning the R1 and R2 items (no switch: all faces; switch: two objects and one face). B, Classifier evidence for intervening item reinstatement. Whole-brain classifier output in log-odds (logits) for the on-screen category by condition for HC correct recency trials (left) and LC and incorrect recency trials (right). Classifier evidence for the on-screen category was greater in the no switch compared with the switch condition for HC correct recency judgments. Error bars reflect SEM; *p < 0.05.

Since we used a binary classifier, evidence for the on-screen and off-screen categories was inversely related to each other and nonindependent. Thus, it is unclear whether the observed difference between conditions is a result of greater on-screen category evidence in the no switch condition compared with the switch condition, greater off-screen category evidence in the switch condition, or some combination of these effects. To partially address this question, we compared classifier evidence in the lag-3 conditions to the neighbors condition, in which there were no intervening items. Thus, this condition can serve as a baseline in which category evidence on successful trials would not be due to intervening reactivation because no items were presented between them. On-screen category evidence in the neighbors condition fell between evidence in the no switch and the switch conditions, but was not significantly different from either (ps > 0.16). This suggests that the difference between no switch and switch may be the result of intervening category reactivation in both conditions.

Because follow-up analyses compared classifier output to hippocampal activity on a trial-by-trial basis, we also reran the same classifier analysis excluding the hippocampus and replicated the condition differences (no switch > switch HC correct, t₍₁₆₎ = 2.41, p = 0.028; no switch > switch LC/incorrect, t₍₁₆₎ = 0.337, p = 0.741; condition by accuracy interaction, F_(1,13) = 2.27, p = 0.15). This suggests that these results are not exclusively driven by the reinstatement of category-level information in the hippocampus.

Encoding-retrieval interactions: hippocampal encoding-retrieval similarity is related to on-screen category evidence

While the prior analyses suggest that category-level intervening information is reinstated during successful recency discrimination, it leaves open the question of whether hippocampal processes might mediate this reactivation. Previous work has shown that hippocampal activation at retrieval is related to and may mediate cortical reinstatement for information presented on the same trials (Staresina et al., 2012; Gordon et al., 2013; Ritchey et al., 2013). However, it is unclear how hippocampal activity relates to reactivation of sequences that span multiple items. Furthermore, reactivation of intervening items may be related to hippocampal processes both at encoding and retrieval. To address these questions, we tested whether category evidence during recency judgments was related, on a trial-by-trial basis, to four different measures of hippocampal function: hippocampal univariate encoding-related activity for intervening items, hippocampal univariate retrieval activity, hippocampal ERS for intervening items, and hippocampal ERS for same-category items other than those from the corresponding sequence (Fig. 5A). To this end, we included these factors as predictors of on-screen category evidence in mixed-effects models for the no switch and switch conditions separately. We found that sequence-specific ERS was a significant predictor of on-screen category evidence in the no switch condition (X²(1) = 3.13, p = 0.039, one-tailed; Fig. 5B). Other than a marginal negative relationship between the univariate response at retrieval and on-screen category evidence (X²(1) = 3.18, p = 0.074, two-tailed), no other factors emerged as significant predictors of on-screen category evidence in this condition (ps > 0.86). Thus, pattern similarity in the hippocampus between recency discrimination trials and their corresponding intervening encoding sequence was significantly correlated with classifier evidence of the on-screen category in the no switch condition, consistent with the notion that whole-brain category evidence during recency discrimination may be related to specific intervening item reinstatement in the hippocampus.

Figure 5.

A, Possible predictors of category evidence during recency judgments. The four predictors were hippocampal encoding (“enc”) and retrieval (“ret”) univariate activation (“act”), as well as ERS for the intervening items (“int”) and all nonsequence same-category items (“cat”). B, Hippocampal ERS for intervening items is related to category evidence. Parameter estimates from the regression of hippocampal responses on classifier output for the presented category. Hippocampal ERS was positively related to classifier output in the no switch condition. Error bars reflect SEM; *p < 0.05, one-tailed.

Interestingly, the same analysis conducted on the switch trials did not reveal any significant effects (all ps > 0.23), suggesting that hippocampal responses during encoding and retrieval did not explain category-level evidence in the switch condition. This may suggest that reinstatement of intervening sequences is more limited in the switch condition. However, it is important to note that the intervening items in the switch condition contain one item from the on-screen category and two from the off-screen category, and therefore category evidence of reactivation within the switch condition alone is not a good index of sequence reinstatement. Thus, we ran a follow-up study to test for behavioral evidence of sequence reactivation in the two conditions. Recognition priming immediately following recency judgments was used to index whether intervening representations were active.

Experiment 2: behavioral results

Encoding

RTs for the encoding tasks were on average 624 ms (SD = 166 ms). There was a nonsignificant RT cost for “boundary” items (boundary M = 700 ms, SD = 186 ms; nonboundary M = 639 ms, SD = 225 ms; t₍₂₈₎ = 1.65, p = 0.111).

Retrieval

The mean proportion correct on recency discriminations across all conditions was 0.675 (SD = 0.079) with an average correct RT of 2.42 s (SD = 0.50 s). Based on prior data, we predicted that recency discrimination would be relatively better in the no switch condition compared with the switch condition. We found that this effect was marginally significant in these data (no switch: M = 0.687, SD = 0.107; switch: M = 0.653, SD = 0.116; t₍₂₈₎ = 1.68, p = 0.053, one-tailed). There was also a significant effect of switch condition on RT (no switch: M = 2.40 s, SD = 0.52 s; switch: M = 2.52 s, SD = 0.49 s; t₍₂₈₎ = 2.66, p = 0.013).

Mean hit rate on the old/new recognition-priming task was 0.90 (SD = 0.06) and false alarm rate was 0.07 (SD = 0.07). The mean RT for hits was 1.39 s (SD = 0.18 s). To assess whether there was evidence for priming of intervening items following recency discrimination and whether this differed across conditions, a two-way repeated-measures ANOVA was conducted on hit RTs following accurate recency discrimination with condition (switch or no switch) and position (intervening or control) as factors. This revealed a significant effect of position (F_(1,28) = 4.28, p = 0.048; Fig. 6) and no effect of condition nor a condition by position interaction (ps > 0.4).

Figure 6.

Recognition priming of intervening items. RTs on correct hits following correct recency discrimination in Experiment 2. Error bars reflect SEM; *p < 0.05.

Follow up pairwise analyses revealed that RTs to intervening items following accurate no switch discriminations (M = 1.26 s, SD = 0.18 s) were marginally faster than those to control items (M = 1.32 s, SD = 0.22 s; t₍₂₈₎ = 1.97, p = 0.029, one-tailed). RTs to intervening items following accurate switch discriminations (M = 1.29 s, SD = 0.21 s) were not significantly faster than switch control items (M = 1.31 s, SD = 0.19 s; t₍₂₈₎ = 0.71, p = 0.243, one-tailed). Replicating our prior results, this shows that reinstatement of intervening items is more likely to occur in the no switch condition. However, one possibility is that reinstatement can also occur in the switch condition when those sequences are well encoded. To assess this, we limited the analysis to recognition trials following HC recency discriminations. This analysis revealed a significant priming effect in the switch condition (intervening: M = 1.28 s, SD = 0.19 s; control: M = 1.34 s, SD = 0.20 s; t₍₂₈₎ = 2.01, p = 0.027, one-tailed). This was not the case for the no switch condition (intervening: M = 1.28 s, SD = 0.21 s; control: M = 1.35 s, SD = 0.31 s; t₍₂₈₎ = 1.19, p = 0.123, one-tailed), but note that memory for LC trials was above chance in the no switch condition (t₍₂₆₎ = 3.18, p = 0.004) but not in the switch condition (t₍₂₆₎ = −0.80, p = 0.43).

Discussion

Using both behavioral and neural measures during encoding and retrieval, we showed evidence that temporal order memory is shaped by processes that occur over extended sequences of events. First, at encoding, we found that the stability in hippocampal multivoxel patterns from the first to the second items later tested in recency discrimination was positively related to later accuracy on those judgments. Second, during later recency discrimination, there was evidence at both the behavioral and neural levels for reinstatement of intervening representations. Together these results suggest that judgments of relative temporal order are associated with reactivation of intervening associations encoded by the hippocampus. Finally, context boundaries were found to modulate the associability of items in memory such that the probability of successfully remembering the order of within-event sequences was higher than that of items spanning boundaries. However, our data suggest that the processes identified here support fine-grained order memory both within and across events.

The role of the hippocampus in temporal order and sequence memory is now well established (for reviews, see Davachi and DuBrow, 2014; Eichenbaum, 2013). Previous fMRI data has shown that hippocampal and surrounding MTL cortical univariate activation is related to the ability to remember temporal order both during sequence encoding (Jenkins and Ranganath, 2010; Tubridy and Davachi, 2011) and retrieval (Konishi et al., 2002, 2006; Dudukovic and Wagner, 2007; Lehn et al., 2009). Furthermore, lesions to rodent hippocampus have been shown to impair temporal order discriminations (Chiba et al., 1994; Gilbert et al., 2001; Fortin et al., 2002; Kesner et al., 2002, 2010) as well as temporal distance judgments (Jacobs et al., 2013), demonstrating the necessity of the hippocampus for temporal memory. In humans, distributed patterns of hippocampal BOLD activity have recently been shown to be sensitive to item-position information in well learned sequences (Hsieh et al., 2014), and to be related to later subjective mnemonic measures of temporal proximity (Ezzyat and Davachi, 2014). Our results extend this work by demonstrating that stability in hippocampal patterns across extended sequences is related to measures of successful temporal order memory.

The relationship between the time passed between items and the ability to discriminate their order raises an important question of the type of information used to remember order. Distance and position-based theories of temporal memory make the prediction that the more distinct two items are in time (i.e., the farther apart they occur), the easier it should be to discriminate their order, and indeed, this effect has been well established in the psychological literature (Friedman, 1993). However, this temporal lag effect conflates the distance between items and the absolute primacy and recency of the tested items. This is due to the fact that as the distance between the first and second items increases, their position in the list also moves earlier and later, respectively. Therefore item strength can change as a function of distance, and distance may bias order memory to rely more on item information than sequential information. Indeed, there is fMRI data consistent with familiarity-based recency judgments for relatively long lags versus recollection-based recency decisions for shorter lags (Dobbins et al., 2002; Konishi et al., 2002; Suzuki et al., 2002; St Jacques et al., 2008; Kwok et al., 2012). In the current paradigm we tested order memory within and across boundaries, which have been shown to influence subjective distance (Ezzyat and Davachi, 2014), while keeping actual distance constant, allowing us to test order memory processes while matching item strength. Under these conditions, with a relatively short lag between tested items, temporal order encoding appears to be supported by associative information. This is evidenced by the recency performance cost for items presented across versus within boundaries in conjunction with evidence that cued recall and serial recall are relatively impaired across boundaries compared to within events (Ezzyat and Davachi, 2011; DuBrow and Davachi, 2013).

It is also clear that in certain cases, item information can be sufficient to support some forms of temporal memory, as in the lag-distance effect described above. In one investigation of temporal context memory, Jenkins and Ranganath (2010) found that univariate BOLD activation in the anterior hippocampus and a number of PFC regions during encoding was positively associated with coarse temporal memory for the position of individual items occurring over the course of the entire experiment (Jenkins and Ranganath, 2010). Additionally, trial-to-trial multivoxel pattern change in the rostrolateral PFC predicted this measure of coarse temporal position, suggesting that the extent to which a trial or item is distinct from surrounding items contributes to this form of temporal memory. Consistent with a role for discriminability in temporal memory, an electrophysiology study conducted in rodents found that greater change in the ensemble responses across a trial sequence was associated with better order memory (Manns et al., 2007). The present data, however, offer a complementary perspective that challenges the notion that pattern distinctiveness is always good for temporal memory and raises questions regarding under what conditions pattern similarity versus pattern change is beneficial for recovering temporal order.

If relative order decisions rely on associative information, then associative retrieval processes should be engaged during recency discrimination. Previous neuroimaging studies have observed activations associated with recency judgments in regions of the recollection network (Konishi et al., 2002, 2006; Suzuki et al., 2002; Dudukovic and Wagner, 2007), suggesting that retrieval of contextual details plays a role in resolving temporal order. Here, using two methods, we directly tested the hypothesis that the associations that intervened between the discriminated items are reinstated in the service of order memory decisions. First, we used an MVPA decoding approach that tested category evidence during each recency discrimination trial. Because we tested two conditions of temporally matched sequences in which the to-be-discriminated items were drawn from the same category (faces or objects) but varied only in the category content of their intervening items during encoding, differences in category evidence between conditions suggest reinstatement of the encoding context, perhaps reflecting reactivation of the intervening representations. Indeed, we observed that category evidence for the viewed category was greater in the no switch condition, in which the intervening items were all of the same category, than in the switch condition, in which two of the three intervening items belonged to the other category. Importantly, this was only observed for accurate recency discrimination trials on which participants were confident. Interestingly, we also found that hippocampal pattern similarity across the intervening encoding sequence and its corresponding recency discrimination trial was positively correlated with category evidence in the no switch condition. This is the condition for which the category evidence of the viewed category should be positively related to intervening item reinstatement. This relationship did not hold for the switch condition in which the intervening sequence is composed of both categories. ERS on individual trials has been previously linked to successful memory retrieval (Staresina et al., 2012; Ritchey et al., 2013). Our data extend this by showing that multitrial ERS is related to evidence of cortical sequence reinstatement.

Given the classifier and similarity results alone, it is difficult to determine whether the intervening items themselves are reinstated or whether our neural measures are driven by some more general context representation. To address this question behaviorally, we tested the accessibility of the intervening items using recognition probes that immediately followed each recency judgment. Given that recency discrimination performance is, on average, worse in the switch condition, we reasoned that this relative impairment may be related to the reduced availability of item–item associative information. In an earlier behavioral study (DuBrow and Davachi, 2013), we observed significant recognition priming for intervening items from a sequence only in the no switch condition, although there was no interaction between conditions. However, that design was not optimized to assess recognition priming for the switch condition since we did not assess priming for items from the opposite category. It is possible, for example, that retrieval in the switch condition involves a coarser level of associative information for which the category switch items serve as anchors. Thus, here we tested the switched-to category following switch recency judgments instead of testing only same-category items. The present data revealed priming in both conditions, suggesting that temporal order decisions, at least at this fine temporal resolution, involve reinstating the intervening representations both within and across events. While the level of the associative structure (i.e., item-to-item or more hierarchical item-to-anchor or item-to-context) may differ between the switch conditions, further work would be necessary to test whether the event structure of temporal sequences modulates the level of associative information that is retrieved at test.

The data presented here suggest that the hippocampus can support temporal order memory by associatively linking items in a sequence that are later recovered during judgments of temporal order. Event structure and shared context influence order memory, at least in part, by modulating associative encoding such that binding is reduced across boundaries (DuBrow and Davachi, 2013). However, when associations are formed, the present data suggest that the reinstatement of intervening representations underlies judgments of order for items both within and across events. While there are many types of temporal information that can be stored in memory perhaps by distinct neural processes, the work presented here suggests that hippocampally mediated sequential associations play an important role in temporal order memory.