Similarity breeds proximity: Pattern similarity within and across contexts is related to later mnemonic judgments of temporal proximity

Summary

Experiences unfold over time, but little is known about the mechanisms that support the formation of coherent episodic memories for temporally-extended events. Recent work in animals has provided evidence for signals in hippocampus that could link events across temporal gaps, however it is unknown whether and how such signals might be related to later memory for temporal information in humans. We measured patterns of fMRI BOLD activity as people encoded items that were separated in time and manipulated the presence of shared or distinct context across items. We found that hippocampal pattern similarity in the BOLD response across trials predicted later temporal memory decisions when context changed. By contrast, pattern similarity in lateral occipital cortex was related to memory only when context remained stable. These data provide the first evidence in humans that representational stability in hippocampus across time may be a mechanism for temporal memory organization.

Introduction

The episodic memory system allows us to flexibly relive past events through the recovery of details such as the people that were present, the setting in which the event took place, and the temporal relationships that structured the event. The ability to mnemonically navigate a past experience, bringing to mind a series of events in the order in which they occurred highlights the intuition that temporal information is a fundamental organizing principle of episodic memories (Tulving, 1983). However, while it is clear that experiences unfold over time, it is less clear how the memory system handles this ebb and flow of experience. In particular the mechanisms that support the encoding of temporally-extended events are relatively underexplored. Prior work has investigated the perception and on-line segmentation of temporally-extended events (Zacks et al., 2007) as well as some aspects of how memory is influenced by event boundaries (DuBrow and Davachi, 2013; Ezzyat and Davachi, 2011; Swallow et al., 2009). However, there has been less focus on how the memory system codes temporally-extended experiences such that the temporal structure can later be recovered. Furthermore, we can remember details from past events that spanned minutes or even hours, which raises questions about how mnemonic associations are formed between event details separated in time. Thus, a more complete understanding of episodic memory will require a mapping of the mechanisms by which temporally-extended events become integrated into episodic memory representations.

Theoretical models of hippocampal function have emphasized its role in binding representationally-distinct elements of an experience, including temporally-separated events (Cohen and Eichenbaum, 1993; Davachi, 2006; Eichenbaum, 2013; O’Reilly and Rudy, 2001). Such models are motivated in part by the finding that hippocampal lesions result in impaired performance on a range of tasks, including trace conditioning (Clark and Squire, 1998; Solomon et al., 1986), spatial navigation (Eichenbaum et al., 1990), and working memory (Hannula et al., 2006; Olson et al., 2006), that all share the common requirement to bind elements that are either temporally or spatially separated. In humans, fMRI has been used to show that the hippocampus is differentially activated when people are asked to bind (spatially or temporally) separated elements of an experience (Hales and Brewer, 2010; Kirwan and Stark, 2004; Qin et al., 2007; Staresina and Davachi, 2008, 2009). However, these effects have been examined only on short time scales with the to-be-bound stimuli presented within single trials and have not been related to temporal memory.

Computational models have proposed that a time-varying representation of temporal context might serve as a substrate for linking events across time (Estes, 1955). These models propose that items encountered in close temporal proximity share more similar temporal context representations than items that are further apart; the more similar the shared context, the more likely items are to temporally cluster during recall (Howard & Kahana, 2002; Polyn, Norman, & Kahana, 2009). Recent results from hippocampal neurophysiological recordings in animals have indicated the presence of encoding signals that are consistent with a representation of temporal context. One study reported that population activity in the hippocampus becomes more dissimilar as time passes, perhaps representing a time-drifting context signal (Mankin et al., 2012). The level of dissimilarity in hippocampal population activity during encoding of two odors has also been shown to predict later recency discrimination performance in rodents, supporting the notion that such a signal may serve to organize temporal information in a behaviorally meaningful way (Manns et al., 2007). Beyond patterns of population activity, recent work has also identified single-unit responses in rodents (MacDonald et al., 2011) that appear to code for specific temporal ‘moments’ between behaviorally-salient events. Although it is not clear whether the single unit and population data represent the same or distinct mechanisms, they suggest possible means by which the hippocampus may associate temporally-separated events.

Despite prior animal work, there is little evidence for a relationship between a hippocampal context signal during an experience and later temporal memory in humans. One prior experiment has shown that the magnitude of hippocampal BOLD activation during single trials is related to later memory for temporal information presented on those trials (Tubridy & Davachi, 2011), but it is not known how the hippocampus contributes to the mnemonic binding of events beyond the confines of single trials. Moreover, in our everyday experience the passage of time between events is often accompanied other changes in context—new locations, people and objects. To date it remains unknown how such contextual changes might influence the hippocampal mechanisms of associative memory across time.

The present study was designed to investigate how stability in the pattern of hippocampal activation across trials is related to temporal memory for information encountered during those same trials. Specifically, we asked whether multi-voxel patterns of BOLD activation across the hippocampus might be a substrate for encoding the temporal structure of events. Participants were scanned using fMRI while they encoded trial-unique faces and objects each presented with a scene. The scene stimuli were repeated across two or four consecutive trials, serving as our manipulation of ‘environmental’ context across trials and capturing the intuition that context varies slowly across time. After encoding, participants made temporal memory judgments in which they indicated the temporal proximity of two items (a face and an object) from the encoding phase. Pairs of tested trials were divided based on whether or not they were encountered with an intervening scene change (context boundary), or whether they were encountered with no intervening scene change (same context). If the similarity of encoding activity patterns is a substrate for linking temporally-separated events, we predicted that pattern similarity should be higher for pairs of trials later rated as having occurred closer in time. Our design also allowed us to compare how fMRI activity patterns differentially relate to temporal memory in situations of stable and changing environmental context.

Results

Behavioral Results

Encoding

On each encoding trial participants were required to imagine the face/object in the paired scene (Figure 1A). Participants reported successful imagery on a majority of encoding trials (M = 88.6 ± 2.0%), indicating that they were able to generate mental images for the object/face and scene pairings. The proportion of ‘yes’ imagery responses did not differ based on whether the item was an object or face [object M = 88.4 ± 2.1%; face M = 88.4 ± 1.9%; F < 1], but was higher for repeated-presentation scenes relative to first-presentation scenes [repeated M = .89 ± .02; first M = .88 ± .02; F(1,19) = 4.42, p < .05; interaction F < 1].

Figure 1. Experimental Design.

(A) Participants performed an associative encoding task on each trial. Scene stimuli repeated across either 2 or 4 consecutive trials, while object and face stimuli were trial unique. Objects and faces that were separated by a change in the associated scene made up the context boundary condition, while objects and faces that were paired with the same scene made up the same context condition.

(B) Following each encoding block, participants performed a memory test in which they were asked to rate the temporal proximity of pairs of items from the encoding block. Participants made their responses on a 4-point scale: very close, close, far, very far. For the purposes of the behavioral and fMRI analyses, responses were collapsed into two bins: close and far. Temporal memory was analyzed based on whether the item pairs were from the context boundary or same context conditions during encoding.

(C) Following the scan session, participants performed a surprise source memory test for the associations between the trial-unique faces and objects, and their paired scene stimuli. Participants were presented on each trial with a face or object from encoding and two alternatives for the associated scene stimulus. Participants chose either the left or right scene stimulus while indicating high or low confidence.

Participants were marginally faster to make ‘yes’ responses on object trials relative to face trials [object M = 719 ± 27 ms; face M = 740 ± 27 ms; F(1,19) = 3.92, p < .07] and were significantly faster at making ‘yes’ responses for repeated-presentation scenes relative to first-presentation scenes [repeated M = 717 ± 25 ms; first M = 741 ± 27 ms; F(1,19) = 5.37; p < .04; interaction F < 1]. We also examined whether response times were faster on the third and fourth trials of same context quartets relative to the third and fourth trials of context boundary quartets. This was important in order to determine whether the time devoted to processing the stimuli at encoding differed across conditions, which could lead to differences in ‘subjective’ time between items that could influence responses on the later temporal memory test. However, a direct comparison of response times on the second-half of context boundary and same context quartets did not reveal any significant differences [context boundary M = 731 ± 26 ms; same context M = 718 ± 26 ms; t(19) = 1.20, p > .24].

Temporal Distance Memory

Responses from the temporal distance memory test were analyzed to examine whether the context boundary manipulation influenced participants’ later subjective mnemonic ratings of distance. On each trial, participants were asked to indicate the temporal distance between two items from the encoding phase (see Experimental Procedures). Critically, the actual distance between the presented items was always the same: at encoding, there were always two intervening trials between the presented retrieval probes. Thus, any differences in temporal distance responses across conditions reflect subjective mnemonic assessments of distance. Participants were significantly more likely to label same context pairs close compared to far [t(19) = 4.28, p < .002; interaction: F(1,19) = 16.5, p < .0007], but there was no difference between the proportion of close and far responses in the boundary condition (p > .80). Importantly, the proportion of pairs labeled close in the same context condition was higher than the proportion of pairs labeled close in the boundary condition [same context close M = .67 ± .05; boundary close M = .49 ± .04; t(19) = 4.06, p < .0007]. Further, the proportion of pairs labeled far was greater in the context boundary compared to the same context condition [boundary far M = .51 ± .04; same context far M = .33 ± .05; t(19) = 4.06, p < .0007]. (Figure 2A).

Figure 2. Behavioral Results.

(A) Mean proportion of pairs that participants labeled close compared to far. Participants were more likely to rate same context pairs as close compared to context boundary pairs. Within the same context condition, participants were also more likely to rate pairs as close compared to far.

(B) Mean response times to label pairs as close and far. For pairs labeled close, participants were slower to label context boundary pairs compared to same context pairs. Error bars denote standard error of the mean.

Response times on the temporal distance memory test were also analyzed to look for evidence of context boundary-related modulations. We predicted that participants would be slower to make decisions about the temporal distance between items when they had been separated by a context boundary, compared to items in the same context condition. A context condition (boundary/same) X response (close/far) ANOVA revealed that participants were slower to make responses to boundary pairs relative to same context pairs [boundary M = 3044 ± 266 ms; same context M = 2818 ± 253 ms; F(1,19) = 7.35, p < .02]. Planned comparisons showed that, for close responses, participants were significantly slower to respond for boundary pairs compared to same context pairs [boundary close M = 2916 ± 262 ms; same context close M = 2629 ± 199 ms; t(19) = 2.34, p < .04], indicating an influence of the boundary when the response was matched across conditions (Figure 2B). Collapsing across context condition, participants were also significantly slower when labeling a pair of items as far compared to close [far M = 3089 ± 300 ms; close M = 2773 ± 224 ms; F(1,19) = 6.01, p < .03].

Source Memory Test

Following four study-test runs of the encoding and temporal memory tasks, participants were removed from the scanner and administered a surprise source memory test for the association between the trial-unique stimuli (faces and objects) and their associated scenes. The overall proportion correct was significantly above chance [M = .77 ± .03; t(19) = 10.4, p < 10⁻⁸]. We analyzed source memory as a function of temporal memory response and context condition to determine any relationship between source memory, temporal memory and context. A context condition X temporal memory response ANOVA showed a marginal interaction [F(1,19) = 4.26, p < .06] and no main effects. Planned comparisons revealed this to be driven by marginally higher source memory for same context pairs rated close compared to far [same context close M = .80 ± .03; same context far M = .75 ± .02; t(19) = 2.05, p < .06; Figure S1]. These data are consistent with the idea that participants may be more likely to rely on source memory (i.e. the two items were paired with the same scene) to make temporal memory decisions in the same context condition than in the context boundary condition (see Discussion).

fMRI Results

Pattern Similarity in Hippocampus

We used pattern similarity (PS) as a measure of the extent to which stable patterns of activation were seen across trials and investigated whether hippocampal PS is related to context condition and temporal memory response. A context condition X memory response ANOVA in left hippocampus (see Experimental Procedures for ROI definition) showed that PS was significantly higher for pairs subsequently rated as close compared to far [close z = .047 ± .012, far z = .021 ± .014; F(1,16) = 4.7, p < .025 one-tailed: Figure 3A]. This effect was not evident in the right hippocampus where PS did not vary with memory response (p > .35) or context condition (p > .48). Planned comparisons revealed greater PS for pairs later labeled close compared to far in the context boundary condition [t(16) = 1.91, p < .04 one-tailed: Figure 3B] but not in the same context condition (p > .46), although there was not a significant interaction between factors (p < .3). A complementary logistic regression analysis (LaRocque et al., 2013; Ritchey et al., 2012) also showed that hippocampal PS predicted temporal memory in the context boundary condition and revealed that PS was a significantly better predictor of temporal memory than mean activation on either trial (see Supplementary Information).

Figure 3. Encoding Pattern Similarity in Hippocampus and LO is Associated with Temporal Memory.

(A) Pattern similarity for pairs of items labeled close was higher than for pairs labeled far in left hippocampus.

(B) Left hippocampal pattern similarity was enhanced for close pairs relative to far pairs in the context boundary condition. See also Figure S1 and Table S1.

(C) Pattern similarity was also enhanced for close pairs compared to far pairs in left LO.

(D) In contrast to left hippocampus, pattern similarity in left LO differentially predicted close/far memory in the same context condition, leading to a region X context condition X memory response interaction (p < .03). Error bars denote standard error of the mean (see also Figure S2).

The PS and logistic regression analyses are consistent with the idea that PS in left hippocampus across trials is related to later temporal memory and suggest a distinct role for the similarity measures in temporal memory beyond the overall level of univariate activation. To determine whether univariate hippocampal activation was related to other forms of memory (Davachi, Mitchell, & Wagner, 2003; Kirwan & Stark, 2004; Ranganath et al., 2004), we conducted a GLM analysis in which we modeled each encoding trial according to whether the participant later showed correct high-confidence memory for the association between the presented stimulus (face or object) and scene (Univariate Source Memory Analysis; see Experimental Procedures). A contrast of source correct > source misses revealed a cluster in anterior left hippocampus (Figure S1) and suggests that, at least in our task, univariate activation in hippocampus was related to successful associative binding of representations presented within trials, while PS in hippocampus was related to later temporal memory for information presented across trials (full list of regions in Table S1).

Pattern Similarity in Category-Selective ROIs

To determine if the relationship between PS and later temporal memory is specific to the hippocampus, we also investigated if PS was modulated by context condition and temporal memory in category-selective ventral temporal and occipital cortex ROIs. Because of its known responsiveness to images of scenes, we first analyzed PS in PPA (Epstein & Kanwisher 1998). All effects were the same in right and left PPA so we display the data collapsed in Figure S2. Interestingly, we did not see any evidence that PS differed by context condition, [left PPA: boundary z = .149 ± .047, same context z = .171 ± .043; p > .41; right PPA: boundary z = .206 ± .048, same context z = .212 ± .367; p > .80]. There was also no difference in PS in left or right PPA based on participants’ subsequent temporal memory responses (p > .13).

Several other stimulus-category selective functional ROIs were also defined (FFA, LO, and RSC; see Experimental Procedures). The majority of these regions in both the left and right hemispheres showed a pattern of results similar to that shown by PPA, that is, no significant PS differences as a function of context condition or memory (see Figure S2). The one exception was left LO (Malach et al., 1995), a region known to be sensitive to both objects and faces, the trial-unique memoranda employed in the current paradigm. A context condition X memory ANOVA revealed a main effect of memory such that close pairs showed higher PS relative to far pairs [close z = .054 ± .021; far z = .019 ± .026; F(1,12) = 3.28, p < .05 one-tailed: Figure 3C]. LO also showed a main effect of context condition such that PS was significantly higher for same context pairs (z = .075 ± .030) compared to boundary pairs [z = −.002 ± .026; F(1,12) = 4.76, p < .025 one-tailed]. There was also an interaction between conditions [F(1,12) = 7.67, p < .02: Figure 3D] driven by significantly greater PS for close pairs (z = .128 ± .031) compared to far pairs (z = .022 ± .038) within the same context condition [t(12) = 3.18, p < .004 one-tailed], but not the boundary condition (p > .13). Thus, PS was greater for trials remembered to be in close temporal proximity if the trials occurred in the same context but not if they were separated by a context boundary. A complementary logistic regression analysis also showed that PS in LO was a significant predictor of temporal memory in the same context condition, and revealed that PS was a significantly better predictor of temporal memory than mean activation on either trial (see Supplementary Information). Finally, we also examined univariate activation in left LO for a relationship with source memory. Using an ROI-based approach in which data were extracted and averaged across left LO, we did not observe any difference in encoding activity between subsequent source correct and source miss trials (p > .79).

The PS results in LO are distinct from those in the hippocampus where greater PS was related to later temporal distance judgments for items separated by a context boundary. To statistically test this distinction, a comparison of PS effects in left LO and left hippocampus revealed a significant region X context condition X memory interaction [F (1,11) = 7.75, p < .02], showing that PS in these regions differentially contributes to mnemonic judgments of temporal distance depending on the presence or absence of shared context across trials (i.e. the context boundary/same context manipulation). Follow-up tests comparing the two regions showed that the difference in PS for close and far trials was significantly greater in hippocampus in the context boundary condition [t(11) = 2.27, p < .05], while the reverse was true in the same context condition [t(11) = 2.35, p < .04]. In other words, PS in left LO appears to be more related to how close participants remember stimuli that occurred within the same context while the same measure in the left hippocampus is more related to how close participants remember stimuli that occurred in distinct contexts.

Pattern Similarity Searchlight Analysis

As a final step to determine the specificity of the effects described above in hippocampus and LO, we performed a searchlight analysis (Kriegeskorte et al., 2006) on the whole brain designed to reveal other clusters showing similar effects. We first ran the searchlight to identify regions showing differential PS for boundary compared to same context pairs. This analysis revealed several regions in which PS was higher for context boundary pairs compared to same context pairs, including right intraparietal sulcus (Figure 4A), as well as multiple regions showing enhanced PS for same context pairs relative to context boundary pairs as observed in LO, including right medial PFC (Figure 4B; full list in Table S2). However, none of these regions showed PS effects that differentially predicted later temporal memory judgments in either context condition.

Figure 4. Pattern Similarity Searchlight Analysis.

(A) A whole-brain searchlight analysis revealed a number of regions such as right intraparietal sulcus (pictured) in which pattern similarity was enhanced for context boundary pairs compared to same context pairs (p < .05 corrected).

(B) The searchlight analysis also identified regions such as right medial prefrontal cortex (pictured) in which pattern similarity was enhanced for same context pairs relative to context boundary pairs. Critically, the searchlight analysis identified no regions in which overall pattern similarity was related to later temporal memory (p < .05 corrected). Error bars denote standard error of the mean (see also Table S2).

We next used the searchlight to identify regions showing enhanced PS for close pairs relative to far pairs (collapsed across context condition) as observed in hippocampus and LO. At a map-wise corrected threshold of p < .05, no regions emerged showing significantly greater PS for close pairs compared to far pairs. We then conducted a searchlight analysis looking for differences between close and far pairs within each of the context boundary and same context conditions individually. Again, at a corrected threshold no regions showed significant differences between close/far PS in either context condition (see Supplemental Information).

Repetition Suppression

Because we used repeating scenes as our context manipulation, we investigated repetition suppression (RS) effects in the regions of interest to ask whether RS was related to our PS results. In particular, it is possible that differences in RS between the context boundary and same context conditions might be related to our observation of PS differences across these conditions, particularly in scene-responsive regions such as hippocampus. Thus, we conducted two analyses to quantify any relationship between PS and RS in our regions of interest. The first compared RS effects in hippocampus and LO; the second examined whether there was any trial-by-trial relationship between the level of RS and the level PS.

In left hippocampus, there was marginal evidence for RS [S1 beta = .034, S2 beta = .019; F(1,17) = 3.77; p < .07] and a follow-up t-test showed that RS was higher in the same context condition [same context S1 beta = .028, same context S2 beta = .003; t(17) = 2.02; p < .03 one-tailed]; however there was no interaction between factors (p > .37). We found similar effects in left LO, which showed a non-significant trend consistent with RS [S1 beta = .022; S2 beta = −.002; F(1,11) = 3.1; p < .11] and a significant RS effect in the same context condition [same context S1 beta = .020, same context S2 beta = −.018; t(11) = 2.13; p < .03 one-tailed]. As in left hippocampus, the interaction between factors was not significant (p > .43). We then directly compared the hippocampal same context and LO same context RS effects and observed no difference in the size of the effect across regions (p > .36). These data suggest the presence of similar RS in left hippocampus and LO (Figure S4) and, thus, support the notion that across-region differences in PS are not driven by differences in RS. Finally, because of its known role in scene processing, we also examined RS effects in PPA and observed significant RS [S1 beta = .083 ± .010, S2 beta = .054 ± .006; F(1,15) = 8.73, p < .01]. PPA also showed a trend for an interaction between context condition and trial position [F(1,15) = 3.49, p < .09] that was supported by significant RS in the same context condition [same context S1 beta = .075 ± .016, same context S2 beta = .025 ± .009; t(15) = 2.89, p < .01 one-tailed].

To quantify the relationship between RS and PS in left hippocampus and LO, we also computed trial-by-trial within-participant correlations between RS and PS. The goal of this analysis was to determine whether RS on individual trial pairs was related to PS on the same trial pairs, and to see whether this relationship differed across context and memory conditions in a way that could explain our PS results. This analysis showed that PS and RS were not significantly correlated and that the PS-RS correlations did not significantly differ across context or memory conditions in either hippocampus or LO. Our final analysis examined whether RS itself is related to temporal memory, and also found no significant main effects or interactions in either region (see Supplemental Information).

Discussion

The present study examined the neural mechanisms supporting memory for temporally-separated information and how these mechanisms are influenced by changes in context. The behavioral data showed that the presence of context boundaries during encoding influenced later judgments of proximity. Specifically, even when actual temporal distance was matched, participants were less likely to rate items as having occurred close together, and were slower to make their memory judgments, when items had been separated by a context boundary during encoding, relative to items encountered in the same context. Analysis of the fMRI data showed that increased hippocampal PS across trials was associated with participants remembering those trials as having occurred closer together in time, suggesting that the pattern of activity in hippocampus may serve as a substrate for linking temporally-separated events in memory. We found that PS in hippocampus significantly differed for close and far trials within the context boundary condition but not the same context condition, consistent with the idea that the hippocampus plays a special role in associating information across representational gaps (e.g. changes in context). Although the interaction within hippocampus did not reach significance, we did observe a significant interaction between the hippocampal pattern and the pattern in left LO where PS was also related to temporal proximity judgments but only when no context boundary intervened between stimuli, suggesting a role in binding temporally-separated information that is distinct from that of the hippocampus.

The presence of shared context across items is thought to be fundamental to memory organization and is an idea that has played a prominent role in computational models created to explain temporal clustering in free recall tasks—that is, the ubiquitous observation that items that are studied as neighbors also tend to be recalled as neighbors (Howard and Kahana, 2002; Kahana, 1996). These propose that during successful item recall, the item’s encoding context is reinstated, which facilitates the subsequent retrieval of neighboring items that share similar context representations. There is now good evidence in humans showing that successful retrieval is accompanied by reinstatement of brain activity patterns that were present at encoding (Kuhl et al., 2012; Manning et al., 2011; Morton et al., 2012; Polyn et al., 2005; Ritchey et al., 2012; Staresina et al., 2012) and that the similarity between neural patterns present during retrieval and encoding is related to clustering in recall (Manning et al., 2011; Morton et al., 2012). By contrast, there are much less empirical data that examine contextual similarity across presented items during encoding and how the stability of this context signal might be related to later memory for those sequentially-presented items. Thus, the current work makes a critical contribution to theories of contextual similarity and memory organization by suggesting that stability in neural patterns across trials during encoding may serve to link unique episodic representations and experiences occurring across time.

In order to introduce shared context representations across trials, we presented repeating scenes paired with trial-unique stimuli. Importantly, context boundaries were operationalized as those trials on which the presented scene stimulus changed from the previous trial. This manipulation allowed us to not only examine the influence of shared context on behavioral responses and BOLD activation patterns but to further compare a shared context condition to one that included a context shift, holding constant the actual temporal gap in both cases. The goal of the repeated-scene manipulation was to capture the intuition that contextual information varies with lower temporal frequency than non-contextual (e.g. item) information. The slow temporal nature of context is a critical element of computational models of memory search (Estes, 1955; Howard and Kahana, 2002) and has been adapted in the form of blocked encoding tasks in prior work examining contextual reinstatement during memory retrieval (Johnson et al., 2009; McDuff et al., 2009).

Prior work has also used temporally-extended stimulus sequences to study how responses in high-level visual areas such as PPA are modulated by temporal context (Turk-Browne et al., 2012). Interestingly, in the present paradigm, while we did observe significant repetition suppression in PPA across repeating scenes, we did not observe PS differences in PPA related to the context manipulation (i.e. changing the scene stimulus). PPA also did not show PS differences on the basis of memory, and although there is evidence for a relationship between PS in PPA and recognition memory (Ward et al., 2013), to our knowledge a relationship between PS and temporal memory has not been demonstrated. Our goal in using scene stimuli was to structure the encoding experience into blocks of stable environmental context, with scene changes initiating shifts in participants’ mental context. Although one might expect scene changes to lead to PS differences in PPA, it is important to note that on each trial participants were asked to perform a complex and mentally vivid imagination task involving both of the presented stimuli. The scenes were therefore important to our contextual manipulation but were not the sole focus of participants’ mental activity during encoding, as has often been the case in prior work examining PPA responses to scenes and may have contributed to the absence of PS effects in PPA.

We observed a dissociation between hippocampus and LO with respect to each region’s relationship between PS and temporal memory by using repeating scenes to define shared context across trials. However, it is also well established that stimulus repetition leads to suppression of neural activity at multiple spatial scales (Miller et al., 1991; Stern et al., 1996) and across regions of the brain (Grill-Spector et al., 2006). Repetition suppression in visual cortical areas has been related to implicit measures of memory such as priming (Epstein et al., 2008; Maccotta and Buckner, 2004; Turk-Browne et al., 2006), however we did not observe a relationship between RS and temporal memory in either hippocampus or LO. At the same time, PS in hippocampus and LO were related to temporal memory (differentially across the context conditions). There is growing evidence that RS and PS may index different types of information and that they may therefore show distinct relationships with memory depending on how memory is assessed (Moore et al., 2013; Ward et al., 2013). PS across item repetitions has been shown to support explicit item memory and is hypothesized to do so via reactivation of the initial encoding event (Xue et al., 2010). In the present data, similarity in the representation of context across trials might support temporal memory for unique items separated by temporal gaps. One question for future work will be to determine how similarity in the neural representation of encoding context is modulated by internal mechanisms such as attention and expectation, and how these mechanisms can influence temporal memory between events.

Prior work, primarily in rodents, has shown that the hippocampus is critical for temporal learning and memory (Agster et al., 2002; Eichenbaum, 2013; Fortin et al., 2002; Jacobs et al., 2013; Kesner et al., 2002) and recent neurophysiological data suggest potential mechanisms that could bridge the gap between temporally-separated events, including sequential firing of ‘time cells’ (MacDonald et al., 2011, 2013) and slowly changing patterns of activation in hippocampal networks (Mankin et al., 2012; Manns et al., 2007; Naya and Suzuki, 2012). Our fMRI data are consistent with a mechanism whereby temporally-evolving patterns in hippocampal neuronal populations are a substrate for determining the temporal distance between events in memory. Enhanced similarity in hippocampal patterns could influence memory by facilitating associative processes that link temporally-separated events via intervening representations (DuBrow and Davachi, 2013). Alternatively, similarity in hippocampal encoding patterns could reemerge at retrieval to influence temporal memory decisions; for example, through a process that compares reinstated patterns in order to derive a mnemonic estimate of temporal distance (Friedman, 1993; St. Jacques et al., 2008).

The reported relationship between neural similarity and later temporal memory is novel in humans and suggests that similarity in hippocampal activity across experiences contributes to mnemonic binding between unique, episodic events that are experienced over extended time periods. Prior fMRI work in humans has shown that univariate hippocampal BOLD activation on individual trials is generally enhanced during temporal retrieval tasks (Ekstrom and Bookheimer, 2007; Lehn et al., 2009) and is greater during successful order encoding of stimuli presented on the same trial (Tubridy and Davachi, 2011) as well as successful recency judgments (Dudukovic & Wagner, 2007). Hippocampal activity is also sensitive to repeated exposures to temporal sequences (Kumaran and Maguire, 2006; Paz et al., 2010; Schapiro et al., 2012; Turk-Browne et al., 2010); our data further shows that hippocampal activity is related to temporal memory for events experienced only once, and suggests that this function derives from stability in measured patterns of activity across time.

The hippocampal PS data suggest that the relationship between PS and temporal memory may be more robust for items separated by context boundaries, an effect that has not been previously reported. Interestingly, we did not observe this relationship for items that shared a scene context where environmental context was more stable (although there was not a significant interaction). This result in the same context condition may reflect the fact that temporal memory judgments can be supported by different processes and types of information that participants may emphasize more or less depending on how memory is probed. For example, when asked to judge the temporal distance between two events, it is often possible to infer temporal distance based on one’s knowledge of the world (Block and Zakay, 1997; Friedman, 1993). In the present experiment, judgments of temporal proximity for same context items may have been supported by the use of source memory for the shared scene during retrieval. This interpretation is supported by the results from the source memory test in which participants showed marginally higher source memory for same context items that were remembered as having occurred close together in time than same context items remembered as far apart. By contrast, items separated by a context boundary did not, by definition, share the same scene thus removing the possibility of using such information to infer temporal proximity. Thus, our data suggest that performance for context boundary items was linked with neural similarity that was present during encoding, such as was observed in hippocampus, consistent with the notion that similarity served to more tightly link those representations in memory.

In contrast to hippocampus, left LO showed PS that was related to mnemonic distance judgments only when context was stable. Prior work has linked patterns of activation in LO to perception; for example, similarity in LO is related to ratings of perceptual similarity between items (Weber et al., 2009) and category membership (Williams et al., 2007). One possibility in our data is that PS in LO is directly related to the perceptual aspects of our contextual manipulation, namely the repeating scenes. However, if the perceptual aspects of the scenes were directly driving our similarity measure, one might have expected similarity in PPA to also be greater for repeated scenes, which we did not observe. An alternative possibility is that the shared context established by the repeating scenes may have modulated the trial-unique object and face representations in LO. That is, similarity in LO may reflect the maintenance and integration of trial-unique object and face representations within a stable context, although future work will be needed to explore this possibility.

The results presented here advance our understanding of how stability in environmental context and measures of neural similarity at encoding are related to temporal memory. Pattern similarity in both hippocampus and LO across sequential trials at encoding was predictive of later judgments of temporal proximity. These data bridge prior work in animals with human computational modeling to reveal encoding mechanisms that support memory for temporally-separated events across context shifts. Future investigations could extend this work to determine the common and unique ways that neural similarity in context and item representations might contribute to associations in episodic memory.

Experimental Procedures

Participants

Twenty-one right-handed native English speakers (13 female; age range: 18–31, mean = 24) were recruited from the New York University and New York City communities and participated for payment ($25/hr). Informed consent was obtained from each participant in a manner approved by the University Committee on Activities Involving Human Subjects. One participant was excluded from behavioral and imaging analyses due to sleepiness during the encoding task. Two other participants were excluded from the imaging analyses due to excessive head motion during scanning.

Stimuli

Stimuli for the encoding and retrieval tasks consisted of nameable objects and famous faces drawn from an online database and Internet image search, and outdoor scene stimuli drawn from an online database (http://cvcl.mit.edu/database.htm, Oliva & Torralba, 2001). Stimulus lists were created for each participant from 128 object, 128 face and 96 scene stimuli and were counterbalanced in such a way that no object or face was paired with the same scene more than once across participants. Across participants, all stimuli were presented equally often in the context boundary and same context conditions.

For the localizer task, non-famous faces from the AR face database (Martinez and Benavente, 1998) were used. Object and scene stimuli for the localizer task were drawn from the same databases as the encoding stimuli, however the localizer stimulus set did not overlap with encoding stimulus set. Scrambled object stimuli were created by dividing images of objects into a 20 x 20 pixel grid and randomly scrambling the locations of each 20 x 20 block in the grid.

General Procedure

Participants began outside the scanner with instructions and a practice session before entering the scanner and performing four runs of the encoding and retrieval tasks. Each run consisted of the encoding task, followed by a recognition memory test and finally the temporal memory test. Because it is not the focus of the present report, we only briefly describe the recognition task here. Participants were shown the faces and objects from the preceding encoding run (along with new lure stimuli) and were asked to make an old/new judgment on each stimulus. The goal of the recognition task was to measure response priming between items as a measure of associative strength in memory. Participants followed each recognition test with the temporal memory test. After completion of the fourth task run, participants performed the localizer task followed by an anatomical scan. Following a ten-minute delay after being removed from the scanner, participants performed a surprise source memory test.

Encoding Task

On each encoding trial, participants were presented with an outdoor scene paired with either an object or famous face for 4 s and were instructed to imagine a scenario in which the object/person was in the scene. Participants were told to elaborate on their imagery throughout the stimulus presentation. After stimulus offset a screen appeared that cued participants to make a yes/no judgment indicating their imagery success (1.5 s). Participants performed 64 encoding trials per run (four runs total). During encoding we attempted to manipulate the extent to which temporally-separated trials shared stimulus context. To set up a shared context across trials, scene stimuli were repeated across either two or four consecutive trials while the object/face stimuli were trial unique (Figure 1A). The experimental trials were therefore organized into quartets, or trial groups, consisting of four consecutive trials such that for half of the quartets the same scene was presented across all four trials (same context groups) and for the other half the scene switched on the third trial (context boundary groups).

The instruction phase stressed that participants should imagine a novel scenario for each object/face in spite of the scene repetition. In addition, within each quartet, two object and two face stimuli were presented consecutively. For context boundary groups this meant that the object/face category switched at the same time as the scene switched; for same context groups this meant that the object/face category switched without a simultaneous scene switch (Figure 1A). There were an equal number of object to face and face to object switches in both context conditions. A pseudorandom inter-trial interval (ITI) (2–20 s) occurred between each trial and was specified using an algorithm designed to optimize estimation of responses from the trial types of interest (Dale, 1999). The ITI between the second and third items of each event was fixed to 2 s; this was important for the recognition task (not discussed).

Temporal Memory Test

Following each recognition memory test, participants performed a retrieval test in which they were presented with two stimuli from the preceding encoding phase and asked to indicate how far apart in time the two items were at encoding. Participants were given up to 8 s to choose from a scale with four response options: very close, close, far, and very far. Participants could also respond don’t know and were instructed to use this option if they did not remember seeing one or both of the stimuli during the encoding phase. However participants were instructed to make a distance judgment in any situation in which they remembered seeing the items, even if they did not explicitly remember the inter-item distance. Participants performed 32 temporal distance judgment trials per block for a total of 128 trials across all blocks (Figure 1B). Half of the distance judgments were on pairs of trials that were consecutive at encoding (not discussed); the remaining half were on pairs of trials that were separated by two intervening trials at encoding. Behavioral and fMRI analyses of only these non-consecutive pairs are reported here.

Localizer Task

After the last retrieval block participants performed two runs of a one-back task that was used as a functional localizer for category-specific ventral visual ROIs. The task consisted of eight blocks each of faces, scenes, objects scrambled objects and fixation. Blocks were presented such that each stimulus block was preceded and followed equally often by every other stimulus block and no blocks of the same stimulus type ever occurred in succession. Each block lasted 16 s and consisted of 20 stimuli each presented for 300 ms followed by a 500 ms fixation cross (a central fixation cross was presented for the duration of fixation blocks). To ensure attention, participants were required to press a button when a stimulus was presented twice in succession; these repeats occurred once per block.

Source Memory Test

Following the scan session participants returned to the lab to perform a surprise memory test in which their memory was probed for the object-scene and face-scene pairings. On each trial participants were presented with one object or face from encoding and two scenes from encoding (one target, one lure). Participants were asked to indicate which scene was paired with the given object/face during encoding and to indicate their confidence (high/low) in their decision. The test was self-paced and divided into four blocks (64 trials/block) that corresponded to blocks from the encoding phase (i.e. stimuli from the first block of encoding were tested in the first block of the source memory test). The lure stimulus on each trial was always one of the other scenes presented during the same encoding block and every scene served equally often as a target and a lure within-subject. Thus, scenes that were paired with two stimuli at encoding appeared twice as a target and twice as a lure in the source memory test (similarly for scenes that were paired with four stimuli at encoding). The left/right screen position of the target and lure stimuli was equated within participants and counterbalanced across participants.

fMRI Data Acquisition

Functional imaging was performed using a Siemens Allegra 3T head-only scanner with a custom head coil (NM-011; Nova Medical) located at the Center for Brain Imaging at New York University. Functional data were collected using an echo-planar (EPI) pulse sequence (34 contiguous slices; TR = 2000 ms; TE = 15 ms; flip angle = 82°) with slices oriented parallel to the AC-PC axis. Slices were positioned ventrally to provide full coverage of the anterior temporal lobes and prefrontal cortex; this resulted in omission of parts of the superior parietal cortex and, occasionally, parts of motor cortex. A high-resolution T1-weighted anatomical scan (magnetization-prepared rapid-acquisition gradient echo sequence, 1 x 1 x 1 mm) was also obtained for each subject following the final block of the localizer task.

Preprocessing of fMRI Data

Images were preprocessed using SPM8 software (Wellcome Trust Centre for Neuroimaging, London UK). Functional images were realigned to the within-run mean to correct for head motion (one run from one participant was discarded due to head motion > 1 voxel). Realigned images were corrected for slice acquisition time and were then coregistered to the anatomical image to correct for between-run motion. For definition and analysis of subject-space anatomical (e.g. hippocampus) and functional (e.g. FFA, LO, PPA) ROIs, the coregistered images were smoothed using a 6 mm FWHM isotropic Gaussian kernel. For group-level analyses, the coregistered images were first spatially normalized to an EPI template in Montreal Neurological Institute space, resliced to 2 x 2 x 2 mm voxels and finally smoothed with a 6 mm FWHM isotropic Gaussian kernel. Low frequencies (< 2 cycles/run) were removed from the functional data in both the subject-specific and group analyses.

ROI Definition

Anatomical hippocampal ROIs were defined for each participant using FSL’s FIRST automatic segmentation tool (Patenaude et al., 2011). The resultant ROIs were then visually inspected and edited by hand to ensure that left and right hippocampus were correctly identified. In the cases of two participants for whom automatic hippocampal segmentation failed, left and right hippocampal ROIs were hand drawn on the participants’ T1-weighted anatomical images using an in-house drawing tool written in Matlab (Mathworks, Sherborn MA).

Category-specific ventral visual functional ROIs (fusiform face area, FFA; parahippocampal place area, PPA; the posterior portion of the lateral occipital complex, LO) were defined using data from the localizer scans. Blocks of each stimulus (faces, scenes, objects and scrambled objects) were modeled as boxcars convolved with a canonical hemodynamic response function (HRF). FFA was defined as a region in the fusiform gyrus that responded significantly more to face blocks compared to object blocks (p < .005; Kanwisher, McDermott, & Chun, 1997), PPA was defined as a region in the parahippocampal gyrus that responded more to scene blocks than to face and object blocks (p < 10⁻⁴; Epstein & Kanwisher, 1998), LO was defined as a region in the posterior lateral occipital cortex that responded more strongly to object blocks than to scrambled object blocks (p < 10⁻⁴; Grill-Spector, Kourtzi, & Kanwisher, 2001; Malach et al., 1995), and RSC was defined as a region in the retrosplenial cortex that responded more to scene blocks than to face and object blocks (p < 10⁻⁴). Localizer data were not collected for one participant; in addition, some ROIs could not be defined in all participants at the given thresholds. The number of participants that contributed to any particular ROI analysis is reported in the Results (range N = 11 for left FFA to N = 18 for right PPA).

Behavioral Analysis

The proportion of all responses excluding don’t know responses (M = 2.3 trials/participant) that fell into each bin (very close/close/far/very far) was computed, as was mean response time. In order to ensure a sufficient number of trials in each condition for the fMRI analyses (described below), the four responses from the temporal distance judgment memory test were collapsed into two (close: very close/close; far: very far/far). The proportion of responses and response time were analyzed using ANOVAs with context condition (boundary/same context) and response (close/far) as within-subject factors. For the source memory test, responses were sorted into high-confidence correct, low-confidence correct, and source incorrect (miss) trials.

Univariate Source Memory Analysis

To determine the relationship between univariate encoding activation in the hippocampus and subsequent associative memory between the face/object stimuli and scene stimuli (source memory), we conducted two analyses. In both cases encoding activation was estimated using a general linear model (GLM) in which trials were modeled as boxcars convolved with a canonical HRF. Separate regressors were included for high-confidence source correct, low-confidence source correct and source incorrect trials.

In the first analysis, a fixed-effects GLM was estimated for each participant using subject-space data. Mean parameter estimates for the high-confidence source correct and miss conditions were then extracted from each participant’s left hippocampal anatomical ROI and were entered into a paired t-test across participants. In the second analysis, a random-effects GLM was estimated across the group using data normalized to the MNI template. A contrast of high-confidence source correct > source incorrect was conducted across all voxels in the temporal lobes. This second analysis was done to identify any subregions of the hippocampus showing a source memory effect that were not revealed by the ROI-based approach. A voxel-wise threshold of p < .005 combined with a cluster size minimum of 5 voxels was used to threshold the whole-brain contrast.

Pattern Similarity Analysis

Pattern similarity (PS) analyses were conducted on subject-space functional data from the encoding runs. We used a modeling approach that has been shown to improve activation estimates for multivariate analyses conducted on fast event-related designs (Mumford et al., 2012). In this approach, a separate GLM is estimated for each trial of the experiment. Each model includes one regressor for the trial itself, modeled as an impulse at trial onset convolved with a canonical HRF, and one regressor that models all remaining N-1 trials as impulses at the trial onsets (also convolved with a HRF). This model is used to obtain an activation estimate in every voxel for the trial of interest. This procedure is then iterated for all trials to produce one GLM for each trial of the experiment.

Patterns of single-trial activation estimates were then extracted for pairs of encoding trials from the first and last position of each quartet (i.e. all PS comparisons are between trials that were separated by two intervening trials). The Pearson correlation between patterns on each trial was used to measure PS and these values were grouped both (1) according to whether the quartet contained a scene switch (boundary/same context) and (2) according to what response participants gave during the memory test (close/far). The correlation values for each trial pair were transformed using Fisher’s r-to-z transformation and were then averaged within subject before being used as input for statistical tests of differences between conditions. One-tailed tests of significance were adopted based on the a priori prediction that increases in PS should be seen for close pairs compared to far pairs and, at the same time, that PS increases should be observed for same context pairs relative to context boundary pairs regardless of temporal memory. One participant with PS values greater than 2.5 SDs from the mean in one condition was excluded from statistical analysis of the hippocampal data and comparisons between hippocampus and LO.

Pattern Similarity Searchlight Analysis

While our primary aim was to examine PS effects in a priori regions of interest, we also conducted an exploratory whole-brain searchlight analysis (Kriegeskorte et al., 2006; Xue et al., 2013) to examine whether other regions show differential PS based on context condition and temporal distance judgment. For each participant we used a 5x5x5 voxel cubic searchlight to compute local PS for every voxel in the brain. These subject-space maps of correlation values were then normalized to the MNI standard space and transformed using Fishers r-to-z transformation. The normalized maps of PS values were then used as inputs to t-tests at each voxel for differences between conditions. A permutation procedure was used to set a mapwise corrected threshold of p < .05 to identify significant clusters (see Supplemental Information).

Univariate Repetition Suppression Analysis

To assess whether there was evidence for repetition suppression (RS) in our ROIs, we conducted a univariate analysis in which we modeled all trials using a 4 s boxcar convolved with a canonical HRF. Trials were modeled according to whether they were part of a context boundary or same context pair and also according to their relative temporal position in the pair (first item in a pair—S1; second item in a pair—S2). Fixed-effects GLMs were conducted using each participants’ native-space data. The resulting parameter estimates for each participant were then extracted from individual-participant ROIs and analyzed using ANOVAs to probe for RS main effects (e.g. S1 > S2) and RS interactions with the context boundary/same context manipulation. Planned comparisons investigating RS in the same context condition were conducted as one-tailed tests, given the a priori expectation that RS effects would be especially prominent in this condition.

We conducted a second analysis to determine whether the level of RS on each trial pair was related to the level of PS across the trials. For this analysis, we extracted trial-specific parameter estimates (see Pattern Similarity Analysis above) averaged across the voxels in our regions of interest. We computed RS and PS for each trial pair and then correlated these trial-specific measures within-participant. These correlations were z-transformed before being analyzed using ANOVAs to determine whether these RS-PS correlations differed by context manipulation and by temporal memory response.