A Specific Role of the Human Hippocampus in Recall of Temporal Sequences

Abstract

There is a growing interest in how temporal order of episodic memories is represented within the medial temporal lobe (MTL). Animal studies suggest that the hippocampal formation (HF) is critical for retrieving the temporal order of past experiences. However, human imaging studies that have tested recency discrimination between pairs of previously encoded items have generally failed to report HF activation. We hypothesized that recalling a naturalistic sequence of past events would be particularly sensitive to HF function, attributable to greater involvement of associative processes. To test this prediction, we let subjects watch a novel movie and later, during functional magnetic resonance imaging, asked them to rearrange and “replay” scenes from the movie in correct order. To identify areas specifically involved in retrieval of temporal order, we used a control condition where subjects logically inferred the order of scenes from the same movie. Extensive MTL activation was observed during sequence recall. Activation within the right HF was specifically related to retrieval of temporal order and correlated positively with accuracy of sequence recall. Also, the bilateral parahippocampal cortex responded to retrieval of temporal order, but the activation here was not related to performance. Our study is the first to unequivocally demonstrate that correct sequence recall depends on HF.

Introduction

Memories of past experiences (episodic memories) are thought to be organized by order of occurrence (Tulving, 1983; Eichenbaum, 2004). The medial temporal lobe (MTL) supports episodic memory, and accumulating evidence from animal research (Skaggs and McNaughton, 1996; Fortin et al., 2002; Kesner et al., 2002; Dragoi and Buzsáki, 2006; Pastalkova et al., 2008) suggests that temporal order is represented in the hippocampal formation (HF). This view is consistent with theoretical models of hippocampal function that propose a critical role in associating events across time (Rawlins, 1985; Wallenstein et al., 1998), possibly drawing on the recurrent connections in area CA3 (Levy, 1996; Lisman, 1999). Human imaging studies are less conclusive about hippocampal involvement in memory for temporal order. Most studies do not report preferential HF responses but instead emphasize the importance of the prefrontal cortex (Nyberg et al., 1996; Cabeza et al., 1997; Suzuki et al., 2002; Dobbins et al., 2003; Fujii et al., 2004; Hayes et al., 2004).

Previous imaging studies may not have been sensitive to the type of temporal order memory that involves HF. Typically, subjects were asked to make recency discriminations between pairs of stimuli, in which they could rely on feelings of relative trace strength or familiarity (Yonelinas and Levy, 2002; Hintzman, 2005). Familiarity judgments are most likely mediated by parahippocampal structures, in particular the perirhinal cortex (Brown and Aggleton, 2001; Eichenbaum et al., 2007). This interpretation is supported by observations of more parahippocampal than hippocampal activation during recency judgments (Rekkas et al., 2005; Dudukovic and Wagner, 2007; St. Jacques et al., 2008). Prefrontal involvement in recency judgments may reflect familiarity monitoring (Henson et al., 1999; Rajah and McIntosh, 2006), temporal integration (Fuster, 2001), or logical ordering (Knutson et al., 2004).

The HF is proposed to selectively support associative memory, like recollection (Davachi, 2006; Eichenbaum et al., 2007; but see Squire et al., 2007), and appears to be preferentially active when recency judgments involve retrieval of temporal relationships, rather than item familiarity (Konishi et al., 2006). Sequence recall represents a special case of temporal order memory which likely depends in particular on hippocampal function. Remembering the temporal order within a series of related events increases the demand for relational processing and may involve reactivation of “order codes” (Friedman, 1993). Recent functional magnetic resonance imaging (fMRI) experiments have demonstrated hippocampal contributions to temporal order mismatch detection and disambiguation of overlapping sequences during sequential exposure to unrelated, single items (Kumaran and Maguire, 2006a,b, 2007).

The aim of the present study was to assess the contribution of subregions in the human MTL to recall of the temporal sequence of related events. Unique to our study is the use of naturalistic and meaningfully related stimuli that resemble real-life event sequences in episodic memory. We let subjects watch a novel movie and later, during fMRI, presented sets of four scenes from the movie and asked subjects to rearrange these in correct order. Based on previous work, we expected this task to specifically activate HF, as opposed to parahippocampal structures.

Materials and Methods

Subjects

Twenty-three healthy females (23–29 years; all right handed according to self-report) without a history of neurological or psychiatric disease participated in this study. The subjects were recruited among students and staff at the Norwegian University of Science and Technology and St. Olavs Hospital in Trondheim, Norway. Two subjects were excluded from the analysis because of excessive head motion, and one subject was excluded because of poor task compliance, resulting in a final sample of 20 subjects. All subjects provided written informed consent before participation, and the study was approved by The National Committee for Medical Research Ethics in Norway.

Overview of the experimental procedure

Subjects took part in two experimental sessions, organized across two consecutive days. On day 1, subjects watched a commercial movie (encoding). On day 2, they were asked to remember the temporal order of scenes from this movie (retrieval) during fMRI scanning. None of the subjects had seen the movie before the experiment. Before encoding, subjects were instructed to concentrate on the movie and memorize as much of it as possible. They were made aware of the intention to test their memory of the movie the following day; however, they were not informed about what type of information they would be tested on. Before retrieval, subjects were told that they would be shown sets of four pictures from the movie and that their task was to indicate the temporal order of these pictures. They were not instructed to use a particular strategy, nor informed about our expectations regarding choice of strategy (see below). Subjects received detailed task instructions with examples of the different screen displays and familiarized themselves with using the MRI compatible joystick. Example stimuli were not used in the fMRI experiment. The total scan session lasted ∼60 min. Immediately after scanning, subjects were debriefed to obtain additional information about their task responses.

Stimuli

At encoding, subjects watched an 89-min movie from the Swedish television series Beck (Movie 4: Øye for øye, by Kjell Sundsvall, 1997). The movie portrays professional and personal events in the life of chief inspector Martin Beck and his colleagues during a murder investigation. The events take place over a few weeks and are both of the ordinary kind (e.g., breakfast at home, office meeting) and more exceptional (e.g., autopsy, knife attack). The movie depicts events in a realistic Scandinavian environment with true-to-life characters and has a sensible plot; thus, watching it can be viewed as mimicking “real-life” events unfolding over time. Advantages of using cinematic material to probe episodic memory have previously been recognized in the literature (Furman et al., 2007; Hasson et al., 2008).

The retrieval test made use of 120 unique movie scenes pictures. The pictures were whole-screen captures taken at different time points during the entire movie, distinguishable by the specific action, situation, and/or setting in which the persons and/or objects were depicted. The pictures were grouped into 30 fixed sets of four pictures each, of which one-half was used for Retrieve trials and the other half for Infer trials (see below, Cognitive paradigm). On Retrieve trials, pictures were of a kind that promoted the use of memory. There was no apparent or logical order among the pictures, i.e., subjects were expected to retrieve temporal relationships from memory to reconstruct the sequence of events (Fig. 1A). On Infer trials, pictures were of a kind that promoted the use of logic. The order of the pictures could be observed as a chain of causality or a stereotypical script, i.e., subjects were expected to apply logical rules to infer the correct sequence of events (Fig. 1A). Pilot studies were conducted to ensure that the selected pictures were easy to recognize and that the two trial types were matched on level of difficulty. The temporal and spatial distance among the scenes varied across trial conditions (both were typically shorter on Infer trials). The event sequences were taken at face value to require either retrieval or inference of temporal order. The validity of the operationalization was assessed in a separate behavioral experiment (see below).

Figure 1.

Outline of the cognitive paradigm used in the fMRI experiment. A, Example of stimuli used in conditions Retrieve (left) and Infer (right). B, Four conditions with fixed order of presentation, from left to right: Retrieve/Infer (shown here, Infer), Retrieve-r/Infer-r (shown here, Infer-r), Evaluate (two separate displays for effort rating and indication of strategy), and Fixation. C, Example of stimuli used in the baseline condition Calculus with either a correct (right) or an incorrect (left) sum. See Materials and Methods for further details. The pictures are reprinted with permission from Filmlance International AB (Stockholm, Sweden).

Cognitive paradigm

The retrieval task (Fig. 1) included one experimental condition (Retrieve temporal order), one control condition (Infer temporal order), and one baseline condition (Calculus). In addition, Retrieve and Infer trials were always followed by a response condition (Retrieve-r and Infer-r) and an evaluation condition (Evaluate). Brief periods of rest (Fixation) were included before the onset of each Retrieve, Infer, and Calculus trial.

Retrieve.

Subjects were shown four pictures of movie scenes, randomly placed in each quadrant of the screen and with the question “Which order?” written above. Subjects were instructed to figure out the correct order of the scenes and to reconstruct the sequence of events in their mind. They were told to focus on the temporal order of the pictures until prompted to respond. Given the nature of the pictures that were used (see above, Stimuli) (Fig. 1A), the condition intended to measure retrieval of temporal order information (sequence recall). All trials had a fixed duration of 32 s.

Retrieve-r.

This condition followed immediately after each Retrieve trial. The four pictures remained visible, but now with the request to “Indicate order:” written above, and with a green cross hair in the center of the screen. Subjects used the joystick to indicate the correct order of the pictures, moving the cross hair and clicking on each picture in turn. This phase was included to obtain a continuous record of performance that could be used in subsequent analyses. All trials were self-paced with a max duration of 20 s each.

Infer.

Screen layout, timing, and instructions were identical to Retrieve trials, but a different type of pictures was used (see above, Stimuli) (Fig. 1A). This condition intended to measure logical inference of temporal order (sequence reasoning) and was used as a high-level control condition in the experiment. Subjects were expected to perform the task mainly without specifically retrieving information about temporal order. This enabled us to identify brain activation specifically related to temporal sequence recall while subtracting the impact of other cognitive operations, such as visual perception/re-encoding, scene recognition, recollection of visuo-spatial details, and general ordering of information.

Infer-r.

This condition followed immediately after each Infer trial but was otherwise identical to Retrieve-r.

Evaluate.

This condition followed immediately after each Retrieve-r and Infer-r trial. First, subjects indicated the amount of cognitive effort required by the preceding trial. The question “How much effort did you exert?” appeared on the screen together with a five-point rating scale (1 = very little, 5 = very much), and the subjects used the joystick to indicate the appropriate rating. Next, subjects indicated what strategy was used on the preceding trial. The question “How did you arrive at your answer?” appeared on the screen together with three response alternatives: “Tried to see a logical order,” “Tried to remember the order,” and “Other.” The Evaluate phase was included to obtain continuous records of perceived effort and strategy, to be used in subsequent analyses. All trials were self-paced with a max duration of 20 s (10 s for each rating).

Calculus.

Subjects were shown a series of simple sums (e.g., 3 + 6 = 9; 4 + 5 = 11) on top of four scrambled pictures, and their task was to indicate with a right or left button press whether or not the sum was correct. The presentation of each sum was self-paced with a max duration of 4 s. This condition was included to measure the baseline activation level, as a common reference for the other task conditions. All trials had a fixed duration of 32 s.

Fixation.

A white cross hair centered on a black screen was shown for a variable duration of 0.2–2.0 s to allow synchronization with the scanner.

In total, there were 15 Retrieve trials, 15 Infer trials, and 33 Calculus trials, equally divided across three experimental runs. A Retrieve or Infer trial was always followed by a Calculus trial. The order of Retrieve and Infer trials was randomized across subjects and runs. Presentation (Neurobehavioral Systems) was used for stimulus presentation, response collection, and logging of trial events during the fMRI experiment. Stimuli were presented to the subjects through fiber-optic goggles (NordicNeuroLab AS), mounted on the head coil. Subjects responded to the task using a fiber-optic joystick with two side buttons (Current Designs). A SyncBox (NordicNeuroLab AS) was used to synchronize stimulus presentation with image acquisition.

Image acquisition

Scanning was performed with a 3T Siemens Trio magnet, equipped with an eight-channel head coil for parallel imaging [GRAPPA (Generalized Autocalibrating Partially Parallel Acquisition)] (Griswold et al., 2002). A vacuum pillow and foam pads were used to minimize head motion. During the retrieval task, echo planar images (EPIs) sensitive to blood oxygen level-dependent (BOLD) contrast were acquired with a single-shot gradient-echo pulse sequence [rectangular field of view (FoV), 96 mm; acquisition matrix, 64 × 64; 26 coronal/oblique slices; in-plane resolution, 1.5 × 1.5 mm; slice thickness, 3.0 mm, repetition time (TR), 2.0 s; echo time (TE), 30 ms; interleaved slice acquisition; GRAPPA, 2; 410–474 volumes per run]. A restricted FoV (i.e., no whole-brain coverage) was chosen to enable high-resolution sampling within the MTL. This allowed a more accurate localization of activation within hippocampal and parahippocampal subregions. The slices were oriented perpendicular to the long axis of the hippocampus, or as close to perpendicular as possible without shifting the phase-encoding direction (head–feet). The most posterior slice was located just behind the tail of the hippocampus, and the most anterior slice was taken through the temporal pole. For anatomical reference, a T1-weighted three-dimensional (3D) volume was acquired with an MPRage pulse sequence (192 slices; TE, 2.94 s; TR, 2300 ms; FoV, 256 × 256; in-plane resolution, 0.5 × 0.5 mm; slice thickness, 1.0 mm). In addition, three T2-weighted images were acquired to optimize the registration of the small FoV BOLD images to the anatomical 3D volume (see below): two turbo spin-echo (TSE) scans acquired coplanar to the EPIs (42 slices; TE, 77 ms; TR, 4270 ms; FoV, 200 × 200; in-plane resolution, 0.78 × 0.78 mm; slice thickness, 2.0 mm) and one sagittal reference scan [22 slices, TE, 89 ms; TR, 4500 ms; FoV, 220 × 220; in-plane resolution, 0.69 × 0.69 mm; slice thickness, 4.0 mm (with a 1.2 mm gap)].

Postscan assessments

Immediately after scanning, we asked the subjects whether they had failed to recognize any of the individual pictures used in the task, and if so, to point out the critical picture(s). Next, subjects were given a questionnaire to report (1) how often they had experienced recollection of the pictured event sequences (“mental replay”), using a scale from 1 (never) to 5 (always), and (2) how often they had figured out the answer well within the time limit, on a scale from 1 (never) to 5 (always). The questions were answered separately for Retrieve and Infer trials.

Data analysis

All image analyses were performed in FSL 4.0 (Smith et al., 2004) (FMRIB Software Library, Oxford; www.fmrib.ox.ac.uk/fsl/). First, the functional images were motion corrected, spatially smoothed (Gaussian kernel full-width at half-maximum, 3 mm), and high-pass filtered (cutoff, 170 s). Next, time course statistical analyses were performed using the general linear model. The expected signal time courses were modeled with a box-car stimulus response function, convolved with a two-gamma hemodynamic response function (Boynton et al., 1996) and its temporal derivative. The model included five predictors, corresponding to the different conditions in the retrieval task (Calculus + Fixation = implicit baseline). Within-subjects parameter estimates were obtained separately for each run and then pooled across runs with a fixed effects model of variance. Group statistics were calculated with a mixed effects model of variance, as implemented in FLAME1 + 2 (FMRIBs local analysis of mixed effects) (Beckmann et al., 2003). Before computing the group statistics, a mask was applied to the functional images to exclude all non-MTL structures. The mask was based on the Harvard–Oxford probabilistic atlases (http://www.fmrib.ox.ac.uk/fsl/fslview/atlas-descriptions.html) and included the left and right hippocampus, parahippocampal gyrus anterior division, and parahippocampal gyrus posterior division (all with max probability >0.25). For significance testing, individual voxels were first thresholded at z > 3.1, and voxels that survived this threshold were used to define clusters of activation. Each cluster's significance was then estimated based on random field theory (Friston et al., 1994) and compared with the cluster probability threshold (p < 0.05, corrected for multiple comparisons). This threshold reflects the probability under the null hypothesis of obtaining a cluster of a particular size, given the voxel z threshold. The threshold was lowered to voxel p < 0.005 (uncorrected for multiple comparisons) and a minimum cluster size of five voxels in exploratory analyses.

Registration of functional to anatomical images was performed by the following procedure, using FLIRT (FMRIBs linear registration tool) (Jenkinson et al., 2002). First, the mean functional image (the average of all scans within a single run) was registered to the mean of the two coplanar T2 TSE scans. Next, the T2 TSE mean was registered to the T2 reference scan, which in turn was registered to the T1 anatomical image. The T1 anatomical image was registered to the Montreal Neurological Institute (MNI)-152 standard template. Finally, the statistical maps were registered to standard space by combining the transformation matrices obtained in the preceding steps. Peak activations are reported in MNI coordinates. To explore potential differences among hippocampal subregions, peak voxels were also localized within the probabilistic maps of the Jülich histological atlas (Amunts et al., 2005), warped to MNI152 space in FSL (http://www.fmrib.ox.ac.uk/fsl/fslview/atlas-descriptions.html). We quantified the responses within six regions-of-interests (ROIs), defined according to the Jülich probabilistic maps (max probability >0.5): left and right cornu ammonis (CA), dentate gyrus (DG), and subiculum (SUB). Subject-specific parameter estimates were obtained from each ROI and compared at group level with repeated measures statistics.

Behavioral data were analyzed in SPSS. Statistical comparisons were made with t tests, or a nonparametric equivalent in cases where the data were not normally distributed. Among the behavioral measures is a sequencing score that reflects the grade of accuracy on Retrieve and Infer trials. Each correct response (i.e., picture in correct temporal position) was awarded one point relative to each remaining response, such that the maximum sequencing score on each trial was 6 (3 + 2 + 1 + 0). Similar scores have been used previously with sequencing tasks (Kumaran and Maguire, 2006b).

Results are presented as mean ± SD in the text.

Behavioral experiment

A separate behavioral experiment was conducted to assess the validity of our paradigm as a measure of temporal sequence recall. Fifteen subjects (23–29 years, all female) were recruited from the same population as the fMRI subjects. All subjects gave oral consent before participation. The subjects were not shown the movie (and assured they had not seen it on any previous occasion) yet completed the same retrieval test as the fMRI subjects. A PC version of the paradigm was used with similar stimulus and timing properties, except that the baseline and evaluation conditions were not included. Subjects indicated the order of the pictures by a series of button presses, and the critical outcome measure was the difference in performance between Retrieve and Infer trials, tested for significance with the Wilcoxon signed-rank test. Performance on retrieve trials was compared with chance level performance using the exact binomial distribution of the number of correct answers by chance (0.625).

Results

Task validation

Subjects in the unexposed control group were able to identify the correct temporal sequence on 2.60 ± 1.88 of the 15 Retrieve trials and on 10.67 ± 1.84 of the 15 Infer trials (Fig. 2). Their performance on Retrieve trials was significantly lower than on Infer trials (Wilcoxon z, −3.42; p < 0.001) yet significantly above chance level (0.625; p < 0.005). Subjects in the fMRI group were able to identify the correct temporal sequence on 10.10 ± 1.86 of the Retrieve trials and 11.35 ± 1.31 of the Infer trials. This difference is (marginally) significant (Wilcoxon z, −1.96; p = 0.050). fMRI subjects performed significantly better than control subjects on Retrieve trials (Mann–Whitney U; z, 5.03; p < 0.001) but not on Infer trials (Mann–Whitney U; z, 1.15; p = 0.248). These results indicate that performance on Retrieve trials relies more on memory for temporal order than performance on Infer trials. Still, the fact that control subjects performed above chance level on Retrieve trials suggests that, at least on some trials, it is possible to determine the correct temporal sequence without the retrieval of temporal order information.

Figure 2.

Performance in control versus fMRI subjects. Mean performance in control and fMRI subjects on Retrieve and Infer trials. Performance was measured as the number of trials (out of 15 Retrieve and 15 Infer trials in total) where subjects identified the correct temporal sequence. Error bars indicate SEM. Means were compared with the Wilcoxon signed ranks test (within groups) and Mann–Whitney test (between groups); *p < 0.05; ***p < 0.001; n.s., not significant.

fMRI task performance

The trial-by-trial self reports obtained during the fMRI experiment revealed that subjects adopted the intended strategy on the majority of both Retrieve (Tried to remember the order, 88.67 ± 9.45%) and Infer trials (Tried to see a logical order, 96.00 ± 5.88%). The fMRI subjects attempted to infer a logical order on 1.33 ± 2.74% of Retrieve trials and used “Other strategy” on 10.00 ± 9.55%. Subjects reported attempts to retrieve the temporal order on 3.33 ± 5.92% of Infer trials and the use of any Other strategy on 0.67 ± 2.05%. To be able to provide an unambiguous interpretation of our data, we decided to include only trials where the intended strategy was used (on average, 13.0 Retrieve trials and 14.4 Infer trials per subject) in all subsequent analyses.

Subjects were able to identify the correct temporal sequence on 68.90 ± 13.06% of Retrieve trials and 76.89 ± 9.94% of Infer trials. The difference is not significant (t₍₁₉₎ = 2.04; p = 0.055). The total sequencing score was significantly higher on Infer trials (70.10 ± 7.24) than on Retrieve trials (63.70 ± 10.14) (t₍₁₉₎ = 2.19; p = 0.041). The average rating of effort level was 2.60 ± 0.46 on Retrieve trials and 2.49 ± 0.53 on Infer trials, not significantly different (t₍₁₉₎ = 1.24; p = 0.231).

Postscan assessments

The recognition rate of individual pictures was high for both Retrieve trials (99.42 ± 1.13%) and Infer trials (97.46 ± 4.25%), although slightly higher in the former (Wilcoxon z, 1.96; p = 0.050). Subjects reported to have “mentally replayed” the sequence of events more often on Retrieve trials (3.35 ± 1.18) than on Infer trials (2.80 ± 0.83). The difference is significant (Wilcoxon z, 2.67; p = 0.008). Ratings of required processing time were similar for both types of trial (Retrieve, 3.55 ± 0.61; Infer, 3.75 ± 0.44; Wilcoxon z, 1.41; p = 0.157).

The behavioral data are summarized in Table 1.

Table 1.

Behavioral data

Measure (units)	Condition
	Retrieve	Infer	Retrieve versus Infer
Accuracy I (% correct)	68.90 ± 13.06	76.89 ± 9.94	n.s.
Accuracy II (sequencing score)	63.70 ± 10.14	70.10 ± 7.24	p = 0.041
Scene recognition (% hits)	99.42 ± 1.13	97.46 ± 4.25	p = 0.050
Effort (rating 1–5)	2.60 ± 0.46	2.49 ± 0.53	n.s.
Mental replay (rating 1–5)	3.35 ± 1.18	2.80 ± 0.83	p = 0.008
Processing time (rating 1–5)	3.55 ± 0.61	3.75 ± 0.44	n.s.

Mean behavioral performance in the group of fMRI subjects (n = 20). Data were obtained during scanning (accuracy and effort) or immediately afterwards (scene recognition, mental replay, and required processing time). Means were compared with the paired samples t test (accuracy and scene recognition) and the Wilcoxon signed ranks test (effort, mental replay, and processing time); n.s., Not significant. See Materials and Methods for further details.

fMRI data

Retrieve versus Baseline

To determine which MTL regions were engaged during sequence recall, we first compared Retrieve trials to Baseline. Three clusters of significantly increased activation were detected (Table 2, clusters A–C; Fig. 3). Cluster A covered parts of both the HF (mainly medial portion of the anterior half) and the parahippocampal cortex (PHC) (throughout most of its extent) in the right hemisphere. Cluster B covered parts of both the HF and PHC in the left hemisphere and was widely similar to cluster A in localization and extent. Cluster C covered a smaller area laterally in the head of the left HF. For a detailed description of the cluster localizations, see the supplemental text, available at www.jneurosci.org as supplemental material.

Table 2.

Clusters of activation during sequence recall

Contrast	Cluster
	Index	Size	z max	x	y	z	Localization
Retrieve > Baseline	A	2584	5.78	25	−21	−13	Right HF and PHC
	B	2195	5.82	−29	−39	−13	Left HF and PHC
	C	50	4.21	−35	−17	−18	Left HF
Retrieve > Infer	D	134	4.27	21	−21	−15	Right HF
	E	114	3.99	−24	−40	−15	Left PHC
	F	50	4.26	27	−31	−18	Right PHC

Clusters with significant activation in the contrasts Retrieve > Baseline and Retrieve > Infer. Significance level, p < 0.001 with cluster correction for multiple comparisons (p < 0.05). Size, Number of voxels; z max, maximum z score within cluster; x, y, z, MNI coordinates of peak voxel; localization, anatomical region covered by the cluster. The clusters are referred to in Results using the same indices as here (A–F). The clusters are shown in Figure 3 (A–F) and Figure 4 (D–F).

Figure 3.

Maps of activation during sequence recall. Areas of significant activation during sequence recall. Areas in blue show increased activation in Retrieve trials relative to Baseline and correspond to clusters A–C in Table 2. Areas in red show increased activation in Retrieve trials relative to Infer trials and correspond to clusters D–F in Table 2. R, Right. Color bars indicate voxel z scores. Voxels are significant at p < 0.001 with cluster correction for multiple comparisons (p < 0.05).

Retrieve trials evoke a range of mnemonic processes that are likely to be reflected in the comparison with Baseline. To identify irrelevant activations, we used a high-level control task (Infer) where the order of the four scenes was not retrieved but determined through logical inference but which was otherwise similar to the sequence recall task (Retrieve). Relative to Baseline, Infer trials correlated with increased activation that was similar to that in Retrieve trials but more restricted (see supplemental Table 1 and supplemental text for further details, available at www.jneurosci.org as supplemental material).

Retrieve versus Infer

The primary aim of the present study was to identify MTL regions that were specifically involved in the retrieval of temporal order information. To achieve this, we compared the level of activation in Retrieve and Infer trials directly. Three clusters of increased activation were detected (Table 2, clusters D–F). All clusters overlapped with the activation observed in Retrieve > Baseline but were more limited to specific parts of the HF and PHC (Figs. 3, 4).

Figure 4.

Activation specifically related to retrieval of temporal order (Retrieve > Infer). Areas marked in red show clusters of voxels with increased activation in the contrast Retrieve > Infer. The clusters are referred to in Results using the same indices as here (D–I). Each cluster is shown on a separate row with one sagittal section (leftmost picture) and five coronal sections taken in the anterior–posterior direction (from left to right). The numbers below the pictures are MNI coordinates. Clusters D–F are significant at p < 0.001 with cluster correction for multiple comparisons (p < 0.05). Clusters G–I are significant at p < 0.005, uncorrected.

Cluster D peaked in the center of the right HF body, medially near the border toward the entorhinal cortex (EC) (Jülich histological atlas probability, 87% SUB). The cluster extended 4 mm in the posterior direction and 5 mm in the anterior direction. At the most posterior levels, the activation was restricted to the HF–EC border area. At more anterior levels, the activation extended slightly in the superior and lateral direction.

Cluster E peaked at the posterior end of the left PHC, at the medial bank of the collateral sulcus (CS). The cluster extended 2 mm in the posterior direction and 5 mm in the anterior direction. At all levels, the activation was confined to the medial bank of the CS.

Cluster F peaked anteriorly in the right PHC, at the medial bank of the CS. The cluster extended 5 mm in the posterior direction and 1 mm in the anterior direction. At all levels, the activation was confined to the medial bank of the CS.

Because we had no a priori prediction of lateralized hippocampal activation, we re-examined the contrast Retrieve > Infer with a more liberal statistical threshold (p < 0.005, uncorrected; minimum cluster size = 5). With this threshold, five clusters of activation were identified in the left HF (supplemental Table 2 and supplemental text, available at www.jneurosci.org as supplemental material), in addition to clusters at the same locations, but of larger extent, as in the initial maps. Notably, the two largest clusters in the left HF (Fig. 4, clusters G and H) appeared at a similar location as the activation within the right HF (cluster D). An additional cluster (Fig. 4, cluster I) was found at a more lateral position, primarily coinciding with the left HF activation in Retrieve > Baseline (cluster C). The fact that the hippocampal activations were widely similar across hemispheres but stronger on the right is likely to reflect the nature of our stimuli. Previous studies have suggested a right lateralization of memory for visual (Kelley et al., 1998) and nonverbal (Golby et al., 2001) stimuli.

The contrast Infer > Retrieve yielded no significant activation, i.e., no MTL regions responded specifically to logical inference of temporal order. This implies that this function is supported by brain regions located outside the MTL, not scanned in this experiment, most likely in the prefrontal cortex (Knutson et al., 2004).

Correlation with behavioral performance

Because of the low number of incorrect responses and little trial-to-trial variation in sequencing scores, we were unable to obtain a reliable estimation of within-subject, parametric effects of performance. We did analyze a subset of data where different performance levels were represented within the same run and observed significant activation in left and right HF (threshold p < 0.005, uncorrected; data not shown). However, having 15 trials per subject and an average level of 69% correct leaves only five trials for the lower levels of performance (sequencing scores 0–5), and typically only two or three different levels were represented within a given run. Therefore, we also performed a between-subjects analysis in selected ROIs, i.e., in clusters that were active in Retrieve > Infer. A significant, positive correlation was observed between the accuracy of sequence recall (sum of sequencing scores across all trials) and level of activation in the right HF (cluster D, Pearson r = 0.60; p < 0.005) (Fig. 5). Activation in the left and right PHC was not related to accuracy (left PHC: r = −0.20, p = 0.41; right PHC: r = 0.04, p = 0.86). There was no significant correlation between accuracy and the level of (subthreshold) activations in the left HF.

Figure 5.

Significant positive correlation between level of activation within the right HF (cluster D, Results) and accuracy of sequence recall. Accuracy was measured as the sum of sequencing scores on all Retrieve trials. Level of activation was measured as the peak voxel's percentage signal change in Retrieve > Infer. Each dot represents an individual subject, and the correlation is shown with the line of best fit.

To examine whether the performance effect was confounded by amount of recollection or vividness, we used the rating of mental replay obtained with the postscan questionnaire. Mental replay was not correlated to accuracy of sequence recall (r = −0.01; p = 0.972), nor to the level of activation in right HF (r = 0.05; p = 0.82). The correlation between the hippocampal activation and sequence accuracy remained significant after controlling for mental replay (r = 0.60; p = 0.007).

Comparison of activation within hippocampal subfields

To explore potential differences between hippocampal subfields, we compared the peak activations (averaged across participants) in the left and right CA, DG, and SUB, as defined by the Jülich histological atlas (Fig. 6). In the right hemisphere, the level of activation was significantly higher in the subiculum (Wilcoxon z, 3.81; p < 0.001) and CA fields (Wilcoxon z, 3.92; p < 0.001), compared with DG. Also in the left hemisphere, the level of activation was significantly higher in the subiculum (Wilcoxon z, 3.92; p < 0.001) and CA fields (Wilcoxon z, 3.88; p < 0.001) than in DG. The level of activation was not significantly different in the subiculum and CA fields in either hemisphere.

Figure 6.

Activation during sequence recall in subregions of the hippocampal formation. Subregions were defined with the probabilistic maps of the Jülich histological atlas. Activation was measured as the peak voxel's percentage signal change in the contrast Retrieve > Infer and averaged across participants. Error bars indicate SEM. Means are compared with the Wilcoxon signed ranks test; ***p < 0.001; n.s., not significant.

Discussion

The aim of this study was to examine the contribution of MTL subregions to a particular manifestation of episodic memory, recall of the temporal sequence of past events. We used high-resolution fMRI to quantify levels of MTL activation while subjects reconstructed the temporal order of life-like events, derived from a movie they had watched the day before. Extensive activation was observed, and effects specifically related to retrieval of temporal order were localized bilaterally in HF and PHC. Across subjects, the right hippocampal activation correlated positively with performance, whereas this was not observed in PHC.

Sequence recall involves HF and PHC

The contrast Retrieve > Baseline revealed that recall of temporal sequences involves the coordinated operation of HF and PHC bilaterally. Unlike in previous imaging studies of memory for temporal order, our paradigm was designed to measure retrieval of complex, naturalistic event sequences, in keeping with theoretical accounts of MTL involvement in episodic memory (Tulving, 1983; Eichenbaum, 2004), and our results clearly support a role for the MTL. The activation in Retrieve > Baseline reflects several mnemonic processes, including scene recognition, recollection of spatial context, and retrieval of temporal order. With exception of the latter, these processes are likely to occur also on Infer trials, which explains the similar activation in Infer > Baseline. However, Infer trials did not require memory for successful performance. Control subjects who had not seen the movie performed well above chance, implying that temporal order could be inferred directly. It is still possible that retrieval contributed to performance; however, the influence hereof would be small because we included only trials where logic was the reported dominant strategy.

HF restores the correct order of events

The critical measure in this study was the comparison between Retrieve and Infer trials, where temporal order was reconstructed from memory or derived from logical rules, respectively. Importantly, the two conditions were comparable in terms of performance and effort and used the same stimulus material to evoke similar processes of scene recognition and context recollection. The observed increase in right hippocampal activation during Retrieve trials, and left hippocampal activation at a lowered statistical threshold, indicate that HF responds particularly to retrieval of temporal order. This result is consistent with current theories of MTL function, suggesting that HF associates episodic items with their spatio-temporal context (Davachi, 2006; Diana et al., 2007).

fMRI studies of temporal order memory have generally failed to observe hippocampal activation, most likely because of the use of recency discrimination paradigms. For example, previous studies that have used naturalistic stimuli, such as persons encountered in a virtual reality game (Ekstrom and Bookheimer, 2007), and photographs of locations that subjects visited before scanning (St. Jacques et al., 2008), measured recency judgments and did not observe significant HF activation specific to temporal order retrieval. We hypothesized that sequence recall would be more sensitive to hippocampal function, because of greater involvement of associative processes. A sequence involves a series of events, rather than a pair; hence, a higher number of temporal associations must be determined. Also, the use of naturalistic events that are meaningfully related may contribute to encoding of their temporal relationship, whereby order judgments can be based on associative rather than item-based retrieval (Friedman, 1993). Partly in support of this is the report that relational encoding of words enhanced hippocampal activation during subsequent recency judgments (Konishi et al., 2006). Moreover, HF participates in sequence learning and mismatch detection of sequences of unrelated items (Kumaran and Maguire, 2006a,b, 2007). Our study is the first to combine a complex sequencing task with naturalistic, related stimuli, and to our knowledge, the first to present robust evidence for HF involvement in retrieval of temporal order.

Memory sequences enable mental replay: vivid recollection of how past experiences unfold over time (Tulving, 1983). Previous fMRI studies reported hippocampal activation when subjects re-experienced autobiographical episodes (Piefke et al., 2003; Addis et al., 2004; Steinvorth et al., 2006), and that hippocampal activation increases with ratings of vividness (Gilboa et al., 2004). In our experiment, autobiographical experiences were mimicked with events from a movie, and subjects were encouraged to replay the event sequences mentally. Replay occurred most often on Retrieve trials, which possibly contributed to stronger hippocampal activation in this condition. Admittedly, mental replay not only involves recollection of temporal but also visuo-spatial information. However, recollection of visuo-spatial details alone is unlikely to explain the observed hippocampal activation because we found a positive correlation between accuracy of sequence recall and level of right HF activation that remained significant after controlling for mental replay. This strongly suggests that HF contributes specifically to correct retrieval of temporal order. The lack of a similar correlation in left HF is possibly related to the stimulus material being visuo-spatial and nonverbal.

Computational models suggest that HF codes associations across time by strengthening synaptic connections between representations of successive events (Levy, 1996; Wallenstein et al., 1998). Gelbard-Sagiv et al. (2008) recently described a human analog of sequence replay in rats (Skaggs and McNaughton, 1996; Foster and Wilson, 2007). Single human hippocampal neurons showed selective and sustained responses to television episodes that later recurred during free recall. Theoretical concepts relate the potential to code sequential information either to the presence of recurrent connections in CA3 (Levy, 1996; Lisman, 1999) or their absence in CA1 (Eichenbaum et al., 1999; Manns and Eichenbaum, 2005; Rolls and Kesner, 2006). It is clear, though, that CA3 and CA1 contribute to memory in fundamentally different ways (Lee et al., 2004; Leutgeb et al., 2004; Hartley et al., 2005; de Almeida et al., 2007). Compelling connectional and functional evidence discriminates between DG and CA3 on the one hand, and CA1 possibly together with subiculum on the other (Witter et al., 2000; Knierim et al., 2006). Our observation of increased activation mainly in the CA field and subiculum argues in favor of a stronger involvement of CA1 in temporal/sequence coding. This is consistent with animal data showing that temporal order memory depends in particular on CA1 (Hoge and Kesner, 2007), possibly by adding a temporal “tag” to subsequent events (Manns et al., 2007). Although it cannot yet be excluded that CA3 is involved (Hoang and Kesner, 2008), temporal order memory does not require DG (Gilbert et al., 2001).

PHC reinstates visuo-spatial contexts

The contrast Retrieve > Infer also showed increased activation in bilateral PHC, consistent with previous findings that both HF and PHC support associative memory (Davachi et al., 2003; Düzel et al., 2003; Kirwan and Stark, 2004). However, unlike the hippocampal activation, the parahippocampal activation was not directly related to task performance. This indicates that HF and PHC have distinct roles during recall of past event sequences. It is possible that PHC is involved in retrieving temporal context, as suggested by studies of recency discrimination (Dudukovic and Wagner, 2007; St. Jacques et al., 2008), albeit in a manner less critical for performance than HF. A more likely interpretation, with more support in the literature, is that the PHC activation reflects processing of visuo-spatial information. Ample evidence suggests specialization of PHC in perception of visuo-spatial scenes (Epstein and Kanwisher, 1998; Bar and Aminoff, 2003; Epstein et al., 2007) and retrieval of spatial context (Burgess et al., 2001; Kahn et al., 2004; Ekstrom and Bookheimer, 2007; but see Bar et al., 2008). In our experiment, Retrieve trials had higher stimulus complexity than Infer trials, in that the scenes had larger temporal spacing and more often depicted different settings. It is conceivable that PHC responded to these properties and that the increased activation during Retrieve trials reflects higher demands for (re-)encoding of visuo-spatial scenes. Alternatively, PHC may have been involved in retrieval of visuo-spatial contextual details that are associated with the scenes, supporting visual imagery and reconstruction that take place during mental replay of past events (Hassabis and Maguire, 2007).

Visuo-spatial contextual details can be combined with general knowledge of temporal patterns to infer the correct order of events (Friedman, 2004). Although we included only trials where subjects reported to have remembered the order of the events, we cannot exclude the possibility of additional inference. This is illustrated by the fact that control subjects performed above chance level on Retrieve trials. However, the use of inferential processes to reconstruct temporal order is believed to occur mainly during recency judgments, especially when indices of relative trace strength are indistinguishable, such as when the temporal distance between items is short (Friedman, 1993, 2004). Notably, a recent fMRI study measured recency judgments under these conditions and found significant activation in PHC but not in HF (St. Jacques et al., 2008). Increased parahippocampal activation as observed in the present study thus most likely reflects reinstatement of visuo-spatial contexts, not related to temporal ordering.

Conclusion

This study provides the first fMRI data on the role of MTL structures in recall of temporal sequences. We have used naturalistic, complex stimuli, and based on our findings, we argue that correct sequence recall depends in particular on the hippocampal formation.