Transcending time in the brain: How event memories are constructed from experience

Abstract

Our daily lives unfold continuously, yet when we reflect on the past, we remember those experiences as distinct and cohesive events. To understand this phenomenon, early investigations focused on how and when individuals perceive natural breakpoints, or boundaries, in ongoing experience. More recent research has examined how these boundaries modulate brain mechanisms that support long-term episodic memory. This work has revealed that a complex interplay between hippocampus and prefrontal cortex promotes the integration and separation of sequential information to help organize our experiences into mnemonic events. Here, we discuss how both temporal stability and change in one’s thoughts, goals, and surroundings may provide scaffolding for these neural processes to link and separate memories across time. When learning novel or familiar sequences of information, dynamic hippocampal processes may work both independently from and in concert with other brain regions to bind sequential representations together in memory. The formation and storage of discrete episodic memories may occur both proactively as an experience unfolds. They may also occur retroactively, either during a context shift or when reactivation mechanisms bring the past into the present to allow integration. We also describe conditions and factors that shape the construction and integration of event memories across different timescales. Together these findings shed new light on how the brain transcends time to transform everyday experiences into meaningful memory representations.

INTRODUCTION

Our daily lives consist of a continuous stream of information. Yet like chapters in a book, we usually remember past experiences as distinct and meaningful events. For instance, a typical morning might be remembered as a series of discrete activities linked to a specific place and time, such as eating breakfast at home and then driving to work. The features of these autobiographical episodes are also not represented equally in memory: someone might recall a torturous commute to work as taking much longer than simply eating breakfast in their living room, even if the actual duration of these events was the same. These scenarios emphasize the fact that our memories are not veridical records of the past. Rather, they reflect discrete “units” of subjective experience. But what is it about these situations that lead to differences in how they are represented in memory? How do our thoughts, feelings, and surroundings integrate the elements of ongoing experience into temporally organized events?

Influential models of event perception posit that individuals perceive shifts in spatial or perceptual context, such as stepping through a doorway, as “event boundaries” (Radvansky, 2012; Zacks et al., 2007). It is thought that the ability to segment continuous sensory inputs is highly adaptive, because it unburdens the mind of fleeting and potentially obsolete working memory representations. By helping reorient attention to salient environmental changes, such as a sudden switch in one’s actions, intentions, or surroundings (Bailey et al., 2017; Khemlani et al., 2015; Zwaan & Radvansky, 1998), these boundaries are theorized to trigger brain mechanisms that update ongoing mental representations of the current state, or context (Richmond & Zacks, 2017; Zacks & Sargent, 2010). The updating of these active ‘event models’ may in turn promote the selection of behaviors best suited to the current environment. Prior research has largely focused on cognitive and neural processes that enable us to perceive discrete events. More recently, progress has been made in identifying how boundaries impact the long-term organization of episodic memory (Clewett & Davachi, 2017).

At the behavioral level, many types of context shifts, including narrative (Zwaan et al., 1995; Zwaan & Radvansky, 1998), spatial, (Radvansky & Copeland, 2006), motion (Zacks, 2004), and other perceptual shifts (Sridharan et al., 2007; Swallow et al., 2011; Swallow et al., 2009) have been demonstrated to not only influence how we perceive discrete events but also to influence how we remember the temporal aspects of those prior episodes (Davachi & DuBrow, 2015; DuBrow & Davachi, 2013, 2014, 2016; Ezzyat & Davachi, 2011, 2014; Heusser et al., 2018; Horner et al., 2016; Lositsky et al., 2016; Sols et al., 2017; Figure 1). For instance, when studying a list of information, items that appear sequentially are more likely to be “bound” together, facilitating memory for the order in which they occurred. However, if items appear on either side of an event boundary (e.g., moving from one room to another), their sequential binding is reduced. That is, individuals are more likely to forget the precise order of item pairs if they spanned an intervening context shift (DuBrow & Davachi, 2013, 2014, 2016; Ezzyat & Davachi, 2011; Heusser et al., 2018; Horner et al., 2016; Sols et al., 2017).

Figure 1.

The diverse effects of various context shifts, or event boundaries, on different episodic memory outcomes. Experiencing a shift in the current context, such as moving from a park to a city street, can cause individuals to perceive a boundary between one episode and the next. In the long-term, this boundary influences how those prior episodes become represented and organized in memory, with different influences on both the temporal and non-temporal aspects of episodic memory.

Event boundaries can also modulate how we remember time itself, particularly by dilating subjective time duration. Items spanning boundaries tend to be remembered as happening farther apart in time, despite having the same true temporal distance (Ezzyat & Davachi, 2014; Lositsky et al., 2016). In a similar vein, the number and complexity of context changes (e.g., how elaborate ongoing changes in stimulus features are) have been shown to increase duration estimates for the entire length of recent events (Faber & Gennari, 2015, 2017; Waldum & Sahakyan, 2013). Together these findings have emphasized the importance of contextual overlap in determining whether incoming information becomes integrated into a unified memory representation.

At the neural level, decades of lesion, physiology and imaging methods have implicated the hippocampus, medial temporal cortical regions (MTL), and the prefrontal cortex in distinct aspects of memory formation (Eichenbaum, 2004, 2017; Howard & Eichenbaum, 2013; Morton et al., 2017; Polyn & Kahana, 2008; Tulving, 1972, 2002). In particular, these regions have been implicated in different aspects of relational memory binding by which individual items are encoded with specific details about when they occurred, where they occurred, and their perceptual features. More broadly, research also implicates these neural mechanisms in being essential for memory integration, the concept that experiences with overlapping contextual or featural information also become stored as overlapping neural representations (Schlichting & Preston, 2015). This overlap in turn enables similar information, including both existing memories and novel information, to become linked together in memory. There has been a surge of work in recent years highlighting the individual and coordinated roles of these structures in binding sequential representations of discrete mnemonic events (DuBrow & Davachi, 2016; Ezzyat & Davachi, 2011, 2014; Fortin et al., 2002; Howard & Eichenbaum, 2013; Hsieh et al., 2014; Hsieh & Ranganath, 2015; Jenkins & Ranganath, 2010; Kalm et al., 2013; Kesner et al., 2002; Schapiro et al., 2012; Schapiro et al., 2013).

What has emerged from this body of work is a remarkable synergy between findings in humans and animals, thereby underscoring the evolutionarily conserved roles of these structures in episodic memory organization (Eichenbaum, 2017; Panoz-Brown et al., 2016; Panoz-Brown et al., 2018; Preston & Eichenbaum, 2013). This research has also drawn intriguing parallels between the brain mechanisms that link events together either during learning or after longer delays (Schlichting & Frankland, 2017), raising the possibility that similar or complementary mechanisms are involved in the formation and integration of episodic memories at different timescales (e.g., Mau et al., 2018; Nielson et al., 2015; Wirt & Hyman, 2017).

In humans, functional magnetic resonance imaging (fMRI) has provided an invaluable tool for investigating how boundaries shape memory representations in the brain. By inserting even the simplest change in sequence learning tasks, such as changing the location of an item or changing the visual category of a stimulus, researchers have begun to characterize encoding patterns of brain activity that can predict the temporal organization of events in long-term memory (for reviews see Brunec et al., 2018; Clewett & Davachi, 2017). Much of this work has focused on identifying neural measures of event organization to understand how the brain forms memories for distinct episodes. These methods include examining how event structure modulates average BOLD activation signal both in individual brain regions as well as in the functional coupling between regions that contribute to attention and memory processing (e.g., DuBrow & Davachi, 2016; Ezzyat & Davachi, 2014).

The advent of multivoxel pattern analyses, which target similarities and differences in neural representations across different stimuli or time points, has also provided an effective measure of neural encoding and retrieval signatures that may be obscured by more spatially coarse-grained univariate BOLD signal analyses (DuBrow & Davachi, 2014; Ezzyat & Davachi, 2014; Lositsky et al., 2016; Ritchey et al., 2012). Through these diverse techniques, neuroimaging studies have shed new light on the kinds of brain activity that respond to event boundaries and that define ‘events’ themselves. These neural measures also reveal the neural processes that may influence the temporal aspects of remembering, including memory for temporal order, temporal duration, and temporal distance between items from recent sequences (Davachi & DuBrow, 2015; Ranganath & Hsieh, 2016).

In this review, we synthesize evidence that temporal stability and change in context representations influence the neural and computational processes that integrate and separate episodic memories across time. We begin with findings suggesting that, at relatively short timescales of learning, hippocampal and cortical memory integration processes track regularities in experience, such as similarities in perceptual features over time, to support the encoding of order and temporal distance between sequential items. This formation of meaningful episodes can occur both proactively, or as a new or familiar event unfolds, as well as retroactively, or after a context shift has occurred. We primarily focus on evidence from human fMRI studies and, when relevant, rodent studies that inform the causal relationships how different brain regions communicate during sequence learning. We also foreground evidence that goal states may play a key role in regulating the influence of context shifts on temporal memory processes.

Next, we discuss work examining memory integration and separation processes and their behavioral correlates at relatively longer timescales. This includes new research showing that temporal proximity helps determine whether gradually evolving patterns of hippocampal and PFC activity integrates or separate memories for events that share overlapping information. We also review fMRI findings suggesting that hippocampal retrieval processes may serve to transcend larger gaps in time to bind context-appropriate information in memory. We conclude with research showing that boundaries also modulate non-temporal aspects of episodic memory, including memory for individual items and their surrounding source information (Figure 1). Through this review, we aim to provide a holistic view of the factors and neural processes that shape the long-term organization of episodic memory.

1. PROACTIVE MECHANISMS OF BINDING SEQUENTIAL MEMORY REPRESENTATIONS AT SHORT TIMESCALES

There are at least two distinct mechanisms by which event memories may emerge from experience. On the one hand, maintaining a stable context representation may link sequential elements into a unified memory. On the other hand, adjacent episodes can also become actively separated in memory when those underlying neural representations shift.

Consider driving along a new route to work. In order to remember your route in the future, it is essential to bind sequential information into unified yet discrete mental representations, such as the name and location of a specific street, its trajectory, and its other defining perceptual features. In turn, successful navigation relies on your ability to link together different segments of your drive. If you want to take that route again in the future, you will need to recall the order in which different parts of your drive occurred to successfully navigate from one location to the next. By chunking individual streets into meaningful memories, these sub-events can also be recombined in the future to predict and/or navigate alternative routes through space.

What this scenario exemplifies is that it is essential to identify what conditions lead to separation (i.e., keeping sub-events distinct) versus integration (i.e., combining contextually-relevant information) of unfolding experiences in memory. In the following sections, we discuss research suggesting that temporal stability in sensory or contextual features can modulate memory integration over time. Here, we define ‘contextual features’ broadly to encompass perceptual features, space, goal states, and internal representations of time. We also describe conditions that bias neural processes toward integrating versus separating sequences of information in memory as well as the cognitive factors that, when appropriate, moderate the impact of event boundaries on temporal memory.

1.1. Contextual stability over time promotes sequential memory integration

1.1.1. Experience shapes local neural representations of time and its organization in memory

Increasingly, research suggests that the stability of an unfolding context plays an important role in linking successive items in memory, particularly when those item sequences haven not been encountered before (Davachi & DuBrow, 2015; Robin et al., 2018; Robin, 2018). For instance, memory for the order of events has been shown to be better for information experienced within the same stable context (e.g., a series of similar percepts, such as face images) compared with information experienced across a change in context (e.g., a series of images that includes a switch from faces to object images; (DuBrow & Davachi, 2013, 2014, 2016; Ezzyat & Davachi, 2011; Heusser et al., 2018; Heusser et al., 2016; Horner et al., 2016).

Like spatial information in the environment (Brunec et al., 2018; Robin et al., 2016), internal representations of time may serve as important organizational principle of episodic memory integration. This idea is inspired by a large body of work showing that, in short lists, stimuli that appear close together in time tend to cluster in free recall (Kahana, 1996; Polyn et al., 2009). Temporal context models propose that this emergence of temporal associations arises from successive stimuli becoming associated through a slowly evolving temporal context signal (Estes, 1955; Landauer, 1975; Moscovitch, 1992; Norman et al., 2008; Polyn & Kahana, 2008; Polyn et al., 2009). In this way, temporally adjacent items have more similar temporal context signals, and are thus more likely to become integrated compared to more temporally distant items. Broadly speaking, this class of models has effectively explained both temporal clustering and recency effects in free recall memory (Norman et al., 2008; Polyn & Kahana, 2008; Polyn et al., 2009). The current work on event memory further suggests that temporal context representations may shift more rapidly at event boundaries rather than drift uniformly with passing time (DuBrow et al., 2017; Horner et al., 2016), with important consequences for how temporally extended events become organized in memory (see Section 3.2).

Interestingly, context stability over time has also been shown to temporally compress memory representations. For example, two items from a stable context are later remembered as having appeared closer together in time than two items encountered across a context change, despite the amount of elapsed time being the same (Ezzyat & Davachi, 2014). In this fMRI study, participants on each trial viewed a pair of images, either a face and a scene or an object and a scene. Across four ‘trials’, or a quartet, the same scene image was presented or the scene image changed after two trials (Figure 2A). In this way, the scene images provided either a stable temporal context or served as an event boundary when a change occurred. Following study, participants were shown a face and an object from the sequence. Unbeknownst to participants, these items had always appeared 3 trials apart (with 2 intervening trials). Participants were asked to rate how far apart those images were using one of four options for their temporal distance rating, which ranged from ‘very close’ to ‘very far.’ The findings revealed that participants were more likely to rate two images as appearing farther apart in the prior sequence when there had been a scene change between those items compared to no scene change between those items during encoding (Figure 2B). This demonstrates that event boundaries led to elongated representations of time in long-term memory.

Figure 2.

Remembering items as appearing closer together relates to more stable encoding patterns of hippocampal activity across time. (A) In this fMRI study, participants view a series of images that were organized as “quartets.” A face or object was paired with a scene that either remained the same for four trials (same context condition) or switched after two trials (boundary context condition), creating a stable or less stable context, respectively. Memory was then tested for the temporal distance between items as well as source memory for an item and its paired scene. (B) Results showed that participants were more likely to remember items from the same context as appearing closer together in the sequence; by contrast, participants were more likely to remember items that had spanned a context shift as having appeared farther apart in the sequence. (C) Ratings of closer temporal proximity were associated with greater hippocampal pattern similarity across event boundaries (adapted from Ezzyat & Davachi, 2014).

Similar findings have been reported in behavioral studies manipulating the number and types of perceptual changes (e.g., color, direction, shape) in short animations. In these studies, the number and diversity of perceptual changes that occurred during the animations, which can be thought of as complex shifts in context, led to larger retrospective estimates of clip duration (Faber & Gennari, 2015). Likewise, larger estimates of temporal duration have been observed when participants prospectively attend to the duration and number of perceptual changes that occur in dynamic animations (Faber & Gennari, 2017). Similarly, a study that manipulated the number of background songs in a time-based prospective memory task (e.g., indicating when 10 minutes have elapsed) found that people respond earlier when more song changes occur (Waldum & Sahakyan, 2013). Together these findings indicate that contextual stability provides scaffolding for linking together representations, leading to both improved objective temporal order memory and more compressed subjective estimates of temporal distance.

Using a virtual reality navigation paradigm, Brunec et al. (2017) examined how actively attending to boundaries might modulate such temporal memory effects (Brunec et al., 2017). While navigating city routes, participants paused at intersections prior to turning, which served as intermittent spatial boundaries during the task. In the active wait condition, participants were instructed to hold down a button during the entire duration of the stop, whereas in the passive condition, participants were simply taken along the routes and passively waited at stop points. The results revealed that, compared to passively waiting, actively waiting at stoplights led to over-estimations of temporal duration between two images of intersections experienced along the route. These data expand upon prior event boundary work by showing that elongated memory representations of time may be driven, in part, by increased attention to event boundaries, which is consistent with the attentional gate model of time perception (Zakay & Block, 1995).

Recent neuroimaging data have asked more directly if similarities in patterns of brain activity across sequential items relates to later memory for temporal order and temporal distance. In particular, temporal pattern stability in the hippocampus predicts both better temporal order memory on a later memory test (DuBrow & Davachi, 2014) and closer retrospective estimates of the temporal distance between two items from a recently encountered sequence (Ezzyat & Davachi, 2014; Figure 2C). Thus, stable hippocampal activity patterns over time may serve as a substrate for event formation as successive items will become ‘linked’ through a shared underlying neural representation.

In a different fMRI study, it was shown that activity in three brain areas - the PFC, MTL cortex, and ventral striatum - exhibited fluctuations in the fMRI BOLD response that mirrored the event structure of a narrative as it unfolded (Ezzyat & Davachi, 2011). Namely, activity in these structures gradually increased as an event unfolded and decreased at event boundaries. Further, participants whose vmPFC, hippocampal, and ventral striatal activity more closely mirrored the event structure of the narratives also exhibited stronger mnemonic clustering of sentences within each event. In a related finding, one fMRI study showed that BOLD signal in the mPFC is sustained throughout the duration of an implicit event, which was defined by higher transition probabilities between sub-clusters of images (Schapiro et al., 2013). In this case, sub-clusters refers to those that tended to appear close together in time, thereby giving rise to a common underlying representation of an event.

The finding that mPFC represents or tracks the contextual or sequential structure of experiences fits in well with past research showing that neuronal responses in mPFC are associated with other cognitive processes similar to our operationalization of ‘context’. For example, single-unit recordings in rodents demonstrate that mPFC neurons carry contextually-rich representations about current and past environmental contexts as well as actions (Hyman et al., 2012). Extending this finding, evidence in rodents suggests that ventral mPFC (medial orbitofrontal cortex) represents not only sensory or spatial context but also all of the features relevant to the current task (Schuck et al., 2016; Wikenheiser et al., 2017; Wilson et al., 2014). As such, these mental representations may promote the selection of context-appropriate behaviors. These data align with a large body of research focusing on how context or task representations are learned, stored, and used in future tasks. What has emerged from this research is that mPFC represents overlapping prior experiences (Tompary et al., 2017), or ‘schemas’, which can also provide scaffolding for integrating and learning new information (Gilboa & Marlatte, 2017; Preston & Eichenbaum, 2013; Tse et al., 2007; Tse et al., 2011; van Kesteren et al., 2014).

1.1.2. Hippocampal-prefrontal cortex dialogue mediates the influence of context on memory integration and behavior

Network-level communication between these regions may also be necessary for integrating memories within the same task or context (for reviews see (Eichenbaum, 2017; Preston & Eichenbaum, 2013; Schlichting & Preston, 2015). For instance, later serial recall for within-context sequential information, an index of successful temporal memory integration, is associated with greater vmPFC-hippocampal functional connectivity during sequence learning (DuBrow & Davachi, 2016). Critically, this association was specific to within-context recall but not across-context recall (Figure 3D). Serial recall of items that spanned a boundary, instead, was associated with increased univariate BOLD activation in lateral PFC and hippocampus at those event boundaries (Figure 3C). This finding suggests that - at least with respect to memory integration - perhaps more localized neural processes, such as active retrieval, contribute to preserving temporal order memory across context shifts.

Figure 3.

Local and global patterns of hippocampal and prefrontal cortex (PFC) activity/connectivity differentially relate to serial recall of items encountered within versus across events. (A/B) Following a sequence-learning paradigm, participants were more likely to accurately report an item and then its successor if those items had appeared within the same context (i.e., were both faces) compared to if those items that had spanned an event boundary. (B) Participants were also more likely to ‘jump’ to an event boundary during free recall, suggesting that context shifts enhance the memory strength of items constituting a boundary. (C) FMRI analyses revealed that the successful serial recall of items that spanned a context shift were associated with increased univariate BOLD signal in the lateral PFC. (D) Successful serial recall of items encoded within the same context was instead associated with increased functional connectivity between the hippocampus and medial PFC (adapted from DuBrow & Davachi, 2016).

One limitation of human fMRI studies is that they cannot speak to the directionality of these hippocampal-PFC interactions during sequence learning. Research in rodents, however, does provide some clues as to the necessity and directionality of vmPFC-hippocampal communication during associative learning (Place et al., 2016). In one study, functional connectivity, as indexed by time-shifted theta synchronization, revealed bidirectional patterns of informational flow between the hippocampus and mPFC during distinct task phases: upon context entry, the hippocampus conveyed context information to mPFC. This interpretation was drawn from evidence that hippocampal activity preceded mPFC activity on trials when rodents successfully identified a reward location associated with a specific spatial context. On the other hand, during the subsequent object-sampling phase, mPFC activity instead preceded hippocampal activity, suggesting that the mPFC may have retrieved context-appropriate representations in hippocampus when goal-relevant actions are underway.

The notion that the mPFC processes represent and modulate context-specific representations in hippocampus aligns with recent electrophysiological and chemical infusion-induced inactivation work in rodents (Guise & Shapiro, 2017). In this study, mPFC activity preceded activity in CA1 after the rules of a spatial learning task suddenly changed. By contrast, CA1 activity preceded activity in mPFC when performance was more stable, or times when the rule changes were learned quickly. These findings are also consistent with rodent work demonstrating that lesioning ventral hippocampus, an important output region to orbitofrontal cortex regions slightly ventral to mPFC, impairs the ability of the OFC to abstract information about task states and expected outcomes (Wikenheiser et al., 2017). Although these studies did not query sequence learning directly, they do reveal important information about how dialogue between the hippocampus and ventral/medial PFC regions supports contextual binding.

In summary, temporal stability in hippocampal activity over time supports sequential memory integration. Behaviorally, it is easier to remember the order item pairs if they had been encountered in the same learning context than if they spanned a change in context. This temporal memory effect is associated with more stable temporal activation patterns in hippocampus and coordinated activation profiles in hippocampus, vmPFC, and the ventral striatum over the course of an event. The implication of this work, along with findings in animals, is that a neural ensemble whose activity is maintained over time may be allocated to the current event as it unfolds. Interestingly, this neural co-allocation process can even emerge across much longer timescales (e.g., on the order of hours) and has important consequences for memory and behavior (see Section 4.1). As contextual shifts occur at boundaries, there may be a concomitant shift in the underlying neural ensemble allocated to the next event and so forth, leading to memory separation. Furthermore, it may be the case that top-down influences from ventral and mPFC promote temporal stability of context representations in the hippocampus across time. In this way, new sensory information may be prospectively allocated to a specific memory representation based on its goal relevance or congruence with existing memories.

Although beyond the scope of the current review, much work suggests that neural oscillations and their coupling, particularly in the theta (~3–8 Hz) and gamma (> 30 Hz) frequency bands, also support memory integration (for a review, see Buzsáki & Tingley, 2018). For instance, it has been shown that local theta-gamma oscillation coupling in hippocampus promotes temporal order memory for within-context representations (Heusser et al., 2016), consistent with phase coding models of sequence learning (Jensen & Lisman, 1996; Lisman & Idiart, 1995). Further, theta coherence between hippocampus and mPFC has also been shown to promote memory integration processes more generally (Backus et al., 2016). Together these findings highlight the importance of hippocampal and hippocampal-prefrontal networks in binding associative information in memory, particularly when information shares featural or contextual overlap.

1.2. Hippocampal prediction signals may proactively influence memory integration and separation

The temporal stability of an encoding context benefits the binding of sequential representations in memory; on the other hand, retrieving previously learned associations might also serve to reinforce sequential memory integration when those events are re-encountered in the future (Davachi & DuBrow, 2015).

Studies manipulating temporal violations within familiar sequences have uncovered evidence of hippocampal prediction mechanisms during learning that may be involved in this process. For instance, hippocampal activity has been shown to increase when the latter portion of learned sequences are violated. That is, the hippocampus responds when a different stimulus appears in the middle of a previously learned sequence (Kumaran & Maguire, 2006) A separate fMRI study showed that the hippocampal CA1 subregion is sensitive to temporal context information, such that hippocampal representations of the same target item differed depending on whether it was preceded by its two original images or two slightly similar images (Wang & Diana, 2016). This finding accords with other work showing that different hippocampal sub-regions respond to temporal sequence (Chen et al., 2015; Kim et al., 2017) and temporal order violations (Azab et al., 2014), as well as expected stimulus violations more broadly (Duncan et al., 2012). Furthermore, the hippocampus has also been shown to be sensitive not only to item violations but also temporal duration violations; however, in this instance, hippocampal activity was greater when the expected item appeared (Barnett et al., 2014).

Although not explicitly linked to temporal memory formation, per se, evidence suggests that hippocampal prediction signals may support associative memory binding (Figure 5). For instance, in one seminal study measuring electrophysiological activity in humans watching repeated video clips, researchers showed that patterns of hippocampal CA1 neuron activity over the course of learning gradually became more correlated across successive time-points within each movie clip (Paz et al., 2010). This time-shifted correlation became stronger with increasing repetitions of each movie clip, and the strength of correlated activity during the final iteration was linked to better free recall for different aspects of those events. Encountering a familiar sequence or previously associated items also appears to trigger a hippocampal forward prediction signal that may be important for learning temporal relationships (Hindy et al., 2016; see also Jafarpour et al., 2017; Schapiro et al., 2012).

Figure 5.

Summary of neural processes that support proactive and retroactive memory integration at shorter timescales. Different cognitive and neural processes are engaged either proactively as time unfolds (1 and 2) or retroactively when a context shift occurs (3 and 4; top panel). (1 and 2) First, memory integration can be facilitated by contextual overlap, either through (1) the extraction of statistical regularities as experiences unfold (e.g., contiguities in space, item color etc.) or (2) familiarity with a given sequence. In the latter, hippocampal forward prediction signals and its functional connectivity with mPFC may prime representations of learned sequences, providing the contextual and representational overlap that facilitates integration. Together, these neural processes may allocate context-appropriate information to a meaningful memory representation. (3) Although event boundaries typically impair temporal order memory, the immediate retrieval of pre-boundary information via PFC and hippocampal activity may counter these effects. This process may help to maintain a sense of continuity despite context changes (greenish hue persists into what would otherwise be represented as Episode 2). Goal-directed attention and associative learning strategies may trigger these processes that preserve memory integration across time. (4) Neural reactivation or replay processes in hippocampus, frontoparietal networks, and basal ganglia at a context shift might also retroactively integrate recent information into a coherent memory representation.

In sum, increasing work shows that hippocampal prediction signals may emerge with repetition. However, the relationship between this forward prediction signal and temporal memory integration is less clear. One function may be to bring the past into the present to promote context-appropriate memory integration. That is, it may serve to reinforce temporal relationships between successive items within a familiar sequence. One interesting possibility is that this hippocampal forward signal might also serve to diminish the typical mnemonic effects of boundaries. For instance, the disruptive effects of boundaries on temporal binding may be overcome with learning, or ‘bridged’, such that certain boundaries no longer impair order memory. If supported, this idea has important implications for the hierarchical organization of experience and subsequent memory. For instance, if fine-grained boundary effects fade away with learning, once discrete memory representations may become broadened into a higher-order, overarching representation of a temporally extended experience.

1.3. Proactively bridging event boundaries: the importance of goals in integrating memories across contexts shifts

As reviewed above, a reliable finding across studies is that event boundaries impair temporal order memory and serial recall for novel sequential information encountered across those boundaries. Event boundaries also tend to inflate retroactive estimates of temporal distance, such that pairs of items encountered across a boundary are later remembered as happening farther apart. Critically, however, hippocampal processes may counter these effects of boundaries on temporal memory. If ongoing patterns of hippocampal activity, as measured using fMRI, remain stable across an event boundary, memory for temporal order is preserved for two items spanning that boundary (DuBrow & Davachi, 2014). Hippocampal neural stability also predicts closer ratings of temporal distance for across-context items, a memory outcome typically observed for within-event information (Ezzyat & Davachi, 2014). These findings raise critical questions: under what conditions is temporal memory integration preserved across event boundaries and what mechanisms support integration across event boundaries?

One strategy that can promote the linking of information across context shifts is a deliberative, top-down associative encoding, such as creating a meaningful narrative between sequential items (DuBrow & Davachi, 2013, 2014, 2017). In this way, a cognitive context, or goal state, may supersede the influence of other context shifts on event segmentation, a concept known as ‘event prioritization’ (Khemlani et al., 2015). According to this framework, current goals play a leading role in event perception, such that the desire to integrate incoming information into an active event model prevents lower-level perceptual changes, including locations or objects, and even subordinate goals from eliciting segmentation (Magliano et al., 2014). Thus, knowing which features of experience should be prioritized may enable individuals to control the structure of their own memories, at least to some degree. As mentioned in the prior section, retrieving familiar sequential information could also theoretically support this proactive memory-structuring process. Returning to a previous example of driving to work, proactive binding could perhaps explain why, despite chunking your commute into memories of individual streets (sub-events), these events are not completely separated from each other in memory. Through a broader hierarchical representation of events, you are still able to remember how to navigate from street-to-street, or event-to-event, to reach your destination.

Studies using implicit memory tests also support the idea that associative encoding strategies promote linkages between successive information, irrespective of context shifts (DuBrow & Davachi, 2014). In one fMRI study, DuBrow and Davachi (2014) examined patterns of brain activity when participants made recency discriminations between two same-category (e.g., two faces) probe items from a prior sequence. Some of these encoding pairs occurred on either side of an event boundary (e.g., a category switch to objects), whereas other encoding pairs occurred within the same context (e.g., within a set of face images). A pattern classifier was trained to distinguish whole-brain activity patterns corresponding to viewing faces versus objects. The trained face/object classifier was then tested on brain activation patterns while participants made recency discriminations between two images from the previous sequence. Importantly, the images shown during these recency discriminations were always from the same visual category (e.g., two faces), so differences in classifier performance would not simply reflect perceptual information.

The classifier revealed greater evidence for images belonging to the specific category (i.e., objects or faces) that had appeared between the two memory probe images during encoding; that is, if the two faces had appeared on either side of a context shift, the classifier indicated greater evidence of the visual category that defined that boundary. On the other hand, if no context switch had occurred between the faces, the classifier showed more evidence for faces. These findings suggest that the sequential links between a series of memoranda are reinstated during retrieval irrespective of intervening boundaries so long as across-boundary links were successfully formed during encoding (see Section 2 for potential mechanisms).

While there was no direct comparison of different encoding strategies in this study, prior behavioral work indicates that, during novel sequence encoding, the use of an item-focused versus associative encoding strategy diminishes boundary-related impairments in temporal memory (DuBrow & Davachi, 2013). Interestingly, one fMRI study reported that using associative versus item-focused encoding strategies was also related to greater MTL activation during subsequent recency discriminations (Konishi et al., 2005).

2. RETROACTIVE MECHANISMS OF BINDING SEQUENTIAL MEMORY REPRESENTATIONS AT SHORT TIMESCALES

Research on temporal memory integration has largely focused how items are proactively bound together across time. Interestingly, however, emerging findings also implicate retroactive mechanisms in facilitating memory integration across different events. These retroactive memory processes are most evident at event boundaries, and may reflect goal-directed processes that preserve memory for the order of event sequences. In contrast to mPFC and mPFC-hippocampal interactions that guide schema-related integration and encoding processes, dorsolateral PFC regions seem to be important for binding information via memory retrieval. Specifically, these mechanisms might help bring previous representations into the present to support links between successive items across boundaries (Figure 5). Other work also suggests that spontaneous neural activity at boundaries may support the consolidation of recent associations in memory. In the following sections, we review exciting new data suggesting that the brain is not idle at the end of an episode. Rather, a host of neural processes replay and/or retrieve recent experiences in ways that either bridge consecutive events in memory or promote associative memory binding for features of the preceding event (Figure 5, panels 3 and 4).

2.1. Effortful retrieval of pre-boundary information

Human fMRI studies suggest that items from a just-experienced event may be actively retrieved or replayed just a few seconds after those items have passed. For example, it has been shown that demands on retrieval are increased when retrieving information that appeared before an event boundary (Swallow et al., 2011). The implication of this result is that the same kind of retrieval process isn’t needed when recovering items from the same context. This is consistent with other studies showing that even the slightest break, or boundary, between an encoding list and retrieval is associated with hippocampal activation (Öztekin et al., 2010; Öztekin et al., 2009). Hippocampal activity during the encoding of boundary items also relates to successful boundary recall when retrieval is cued by a pre-boundary item (DuBrow & Davachi, 2016). This finding suggests that, during sequence learning, hippocampal activity at event boundaries may signify the strategic retrieval of pre-boundary information and, thus be associated with a greater binding of pre-boundary to boundary representations.

Like the hippocampus, lateral prefrontal cortex (PFC) processes have also been shown to support memory integration across event boundaries (DuBrow & Davachi, 2016; Ezzyat & Davachi, 2011). Prior work suggests that lateral PFC may contain information about temporal context (Polyn & Kahana, 2008; Ranganath & Hsieh, 2016; Schapiro et al., 2013) and promote binding of item pairs across small gaps in time (Blumenfeld & Ranganath, 2006; Blumenfeld & Ranganath, 2007; Hales & Brewer, 2011; Hales et al., 2009; Qin et al., 2007). Patients with LPFC lesions also exhibit a specific impairment in temporal order memory in instances when top-down attentional control is required (Mangels, 1997). In this lesion study, patients with LPFC damage were only impaired on temporal order memory for lists of concrete nouns when they were instructed to intentionally learn the order of the words but not when they learned incidentally (i.e., when patients were instead instructed to make a size judgements about each word).

In other work, LPFC activation has been implicated in “refreshing,” or bringing to mind, just-encountered information (Johnson et al., 2005), especially when it is outside of the current or recent focus of attention (Öztekin et al., 2009). When viewed through the lens that event boundaries reduce the accessibility of recent information (Swallow et al., 2009; Zacks et al., 2007), these findings raise the possibility that engaging LPFC processes at event boundaries may contribute to the recovery of stimuli encountered just before a context shift (Danker et al., 2008), at least when retrieving or integrating that recent information is appropriate (e.g., consistent with one’s goals).

LPFC activity associated with temporal binding often occurs in parallel with increased hippocampal activation (Hales & Brewer, 2011; Hales et al., 2009). For instance, these brain regions co-activate when individuals retrieve the temporal order of recent items (Dudukovic & Wagner, 2007). Co-activation of LPFC and posterior hippocampus is also associated with successfully encoding and maintaining temporal position information in working memory (Roberts et al., 2017). Because LPFC and hippocampus are activated concurrently during temporal order working memory tasks, it may also be the case that they work in concert to support memory integration across short timescales via reactivation. Indeed, similar mPFC-hippocampal integration processes also emerge for novel sequences of stimuli (e.g., DuBrow & Davachi, 2016; Figure 3C). This suggests that mPFC-hippocampal interactions may rapidly extract the structure of ongoing sequences regardless of their familiarity.

Taken together, although sequential representations are more likely to become separated by event boundaries, there are also cognitive and neural mechanisms that can counter this process to ‘rescue’ memory integration. One robust strategy for integration involves implementing the goal of linking successive items using associative encoding strategies, such as forming a continuous narrative. Activity in the hippocampus and LPFC processes may support the immediate retrieval of pre-boundary information during sequence learning. At the same time, these regions may work together to hold ongoing representations in mind, thereby preserving memory integration despite changes in the external environment (Figure 5).

2.2. Post-event consolidation or replay processes may promote memory integration

A wealth of studies in rodents shows that, during spatial navigation tasks, hippocampal place cells rapidly replay recent experiences in the order in which they occurred (Carr et al., 2011; Foster & Wilson, 2006; Panoz-Brown et al., 2018). Such rapid neural replay has been hypothesized to be important for preserving memory for the sequential order of recent information, although more evidence supporting this hypothesis is needed (Ólafsdóttir et al., 2018). Identifying similar post-encoding replay and/or reactivation patterns in humans is an intense and active area of research (de Voogd et al., 2016; Gruber et al., 2016; Murty et al., 2016; Schlichting & Preston, 2014, 2016; Tambini et al., 2010; Tambini et al., 2016; Tompary et al., 2015).

Relevant to the discussion here, recent work suggests that post-encoding hippocampal activity and functional connectivity immediately following an event may enhance associative memory of that just-experienced information (Murty et al., 2016; Tambini & Davachi, 2013; Tambini et al., 2010). These neural patterns even emerge if measured in the few seconds following an item’s presentation (Cohen et al., 2015; Staresina et al., 2013). Of relevance to event boundaries, recent fMRI work has shown that a post-stimulus hippocampal offset signal predicts successful memory for a movie clip viewed immediately beforehand (Ben-Yakov & Dudai, 2011; Ben-Yakov et al., 2013; Ben-Yakov et al., 2014; Figure 4). Recent fMRI studies have identified a similar event-specific hippocampal offset signal during continuous movie viewing, suggesting that it is indeed driven by neural representations of events (Baldassano et al., 2016, Ben-Yakov & Henson, 2018).

Figure 4.

Post-event hippocampal and striatal activity relate to successful memory of just-experienced events. (A) At the offset of a naturalistic video clip, there is an increase in hippocampal and striatal activity that relates to better associative memory for details of that prior event (right panel). These post-event neural signals were not observed for recent scrambled videos and less so for forgotten information, suggesting that these mnemonic processes relate to the integration of meaningful episodic memory representations (adapted from Ben-Yakov and Dudai, 2011).

In one of these experiments, Baldassano et al. (2016) used a data-driven model to characterize shifts in stable cortical activity patterns representing meaningful events. Mirroring prior studies using discrete video clips, they identified a hippocampal signal that was time-locked to the shifts in cortical activity patterns. These cortical activity shifts also overlapped with time points in the video that a separate group of participants identified as being event boundaries. Critically, the magnitude of the hippocampal post-encoding response was correlated with the degree of subsequent cortical pattern reinstatement for that event during later free recall. While the magnitude of this signal was not specifically linked to later memory performance in this study, it was related to how long participants spent free recalling aspects of the preceding events. This suggests that the post-event hippocampal signal may have reflected the degree to which episodic details of the prior event were encoded into long-term memory. Whether this event-offset signal reflects an active rehearsal and (re)encoding of recent events or a more passive, offline consolidation process is unclear and warrants further investigation.

Importantly, these fMRI studies cannot speak directly to which information is represented by neural activity at event boundaries. A recent scalp EEG study, however, provided evidence that just-experienced episodic information may indeed be replayed at boundaries. In this study, scalp recordings revealed that event boundaries trigger activity that appears to reflect the reinstatement of just-experienced information (~200–800ms post-onset of event boundary; (Sols et al., 2017). Specifically, the similarity of spatiotemporal EEG patterns was more similar between the boundary item and the activity patterns seen during the prior event than it was when the same analysis was conducted on trials within an event. These results suggest that reactivation was retroactive and specific to the boundaries between events. Further, the magnitude of reactivation at boundaries was related to participants’ later associative memory across the boundary. These results therefore expand upon prior work examining how immediate post-event processes retroactively bind memories: whereas the previous fMRI studies show that average hippocampal activity at the end of an event relates to better associative memory for the prior event, Sols et al.’s data suggest that post-event memory reactivation also helps to maintain temporal memory integration across events.

Another important question concerns the directionality of neural replay. Research in rodents suggests that replay can occur in either a forward or backward manner (Ambrose et al., 2016), which may serve different functions for adaptive behavior. In humans, there are some initial indications from MEG that recently learned non-spatial sequences may be reinstated in a backwards manner (Kurth-Nelson et al., 2016). One interesting future direction would be to explore whether these bidirectional replay processes not only exist in humans but also support the proactive and retroactive temporal memory integration processes described here. It will also be important to characterize the factors, such as attention or encoding strategies, that can modulate the direction of neural replay and its influence on temporal learning.

2.3. Commonalities between mechanisms of memory integration at different timescales

Interestingly, the neural processes that contribute to memory integration appear to be strikingly similar between events learned across short and long timescales. This overlap, however, may only be evident under specific conditions. Across human fMRI studies, the hippocampal and prefrontal networks may be specifically engaged when it is necessary to bind contextually-specific, or highly detailed, representations of past events. At shorter timescales, recent event-specific information may be less accessible due to an intervening context shift (DuBrow & Davachi, 2013; Swallow et al., 2009). At longer timescales, the details of more remote episodic memories may become less accessible due to time-dependent memory decay and/or the transformation of those events into more generalized (Iess detailed) schema representations (Sekeres et al., 2018; Winocur & Moscovitch, 2011). Thus, more effortful retrieval processes that engage the hippocampus and lateral PFC may be important when it is desirable to link past occurrences to the present situation. Moreover, these retrograde memory integration effects may be necessary to resist temporal pattern separation processes typically induced by context shifts or the passage of time; for instance, hearing half of a lecture on a brain anatomy one day and then hearing the second half of the lecture the following day. Doing so may promote the formation of a unified memory representation that aids in comprehension and learning.

In summary, research suggests that hippocampal and lateral PFC processes help to recover and integrate context-specific representations at various timescales. In so doing, these memory retrieval processes may help to bridge episodes to maintain a sense of continuity and facilitate event comprehension across time. By contrast, forming and retrieving less detailed memories may not rely on hippocampal processes but rather on schematic, gist-like representations that are perhaps represented locally in mPFC. It is also important to consider the different contributions of hippocampal sub-regions to retrieval. For instance, coarser memory representations are also likely to engage anterior hippocampus and anterior hippocampal-vmPFC connectivity (Robin & Moscovitch, 2017).

In our view, a ‘schema’ typically signifies more of an extracted statistical rule based on prior knowledge and experiences, whereas ‘context’ typically refers to information concerning the more immediate environment. However, in the brain, both of these types of information appear to be represented in mPFC ensembles so long as the context is known and predictable. In that sense, temporal stability in a novel sequence might therefore act like a schema in terms of facilitating temporal binding, because it provides shared contextual information for linking successive items.

3. MECHANISMS FOR SEPARATING MEMORIES ACROSS TIME

Up until this point, we have discussed ways in which incoming sequential information becomes integrated into discrete event memory representations. To adaptively guide behavior, however, our memory systems must also be able to distinguish between repeated encounters with the same or similar perceptual or mental contexts. For example, after leaving work and walking to your company’s parking structure, the ability to locate your car requires distinguishing where you parked today from where you parked last week. Here, we discuss neural processes that support the separation of memory representations across time.

3.1. Learning and repetition may drive memory separation of discrete sequences or events

The work reviewed in the preceding sections highlight how temporal memory integration is facilitated by the temporal stability in the environment as well as our mental context, or internal thoughts. Both of these forms of context may be captured by neural measures of temporal stability. However, what happens to event representations at event boundaries? Findings from human fMRI studies show that hippocampal representations can also distinguish between adjacent and discrete events, particularly with learning or repetition. Using multivoxel pattern analyses, it has been demonstrated that hippocampal activity patterns after learning sequences/pairs of items become less similar between different sequences/pairs compared to within sequences/pairs, suggesting that representations of individual contexts (here, well-learned associates) become more distinct after learning (Chanales et al., 2017; Hsieh et al., 2014; Kalm et al., 2013). Over the course of learning, this differentiation in hippocampal representations may be especially robust if those discrete sequences share overlapping information (Chanales et al., 2017).

With repetitive exposure, hippocampal BOLD signal has been shown to significantly decrease at transition points (event boundaries) between temporally-clustered stimuli, along with lower hippocampal pattern similarity between items that were encountered on either side of this event boundary (Schapiro et al., 2016). Interestingly, this repetition-related decrease in hippocampal univariate activity has also been observed at the offset of a discrete event (see Section 2.2) (Ben-Yakov et al., 2014). By contrast, when learning novel sequences, hippocampal BOLD activation appears to increase at event boundaries in ways that promote associative memory binding (Ben-Yakov et al., 2014; DuBrow & Davachi, 2016). Thus, there may be a dynamic shift in hippocampal learning processes at boundaries as sequences become increasingly familiar. During novel encoding, hippocampal representations and activity perhaps represent an attempt to maintain a continuous representation of a temporally extended experience. By contrast, when it becomes clearer that boundaries distinguish meaningful events, hippocampal sequential representations might instead become more separated.

Another possibility is that task demands may determine the nature of hippocampal activity signatures at event boundaries. On the one hand, when the goal is to encode an entire list of information irrespective of boundaries, the hippocampus may attempt to maintain integration despite a contextual shift (e.g., Dubrow and Davachi, 2016). On the other hand, when the goal is to disambiguate overlapping sequences, hippocampal responses may reflect an attempt to separate those distinct memories. This hippocampal signal may be particularly apparent when the context shift within a sequence is subtle, therefore placing a higher demand on hippocampal pattern separation processes that can distinguish one event from the next.

3.2. Boundaries may enhance contextual drift, leading to memory separation

Shifting temporal context signals may play a role in facilitating temporal pattern separation at boundaries (DuBrow et al., 2017). The groundwork for this idea was first laid by theories positing that cognitive task switches or experiencing something novel – as would occur at an event boundary – can modulate the temporal context signal, such that sequence information on either side of that boundary is grouped into sub-clusters (integration) that also cluster away, or become separated, from each other during free recall (Polyn et al., 2009). In short, context switches simultaneously lead to the integration of temporally-clustered information and the separation of those emergent clusters, or events, in memory. Building on this, Horner et al. (2016) showed that increasing the rate of change in a time-varying context parameter at event boundaries could account for findings that crossing through a doorway, a shift in spatial context, impairs temporal order memory (Horner et al., 2016)(Figure 6).

Figure 6.

Context shifts, such as a scene image inserted between two faces, may accelerate the drift of a slowly evolving temporal context signal (red rectangle) that may reside in MTL structures and the PFC. In turn, sequential stimuli (faces) separated by a short lag become embedded in more distinct temporal contexts. This lack of contextual overlap relates to order memory impairments between two items (faces) that spanned a context shift (scene).

The modeling work conducted by Horner et al. demonstrated that increasing the drift rate-change parameter by a single value predicts impairments in temporal order memory (Horner et al., 2016). However, from a distinctiveness perspective, greater temporal drift between two items within a sequence could theoretically promote better recency discrimination. In particular, when comparing each item to the retrieval context, the more dissimilar the items are from each other, the easier it may be to select the one with the greater match to the retrieval context. Indeed, fMRI studies show that, across time, neural pattern change within regions thought to support event representations, including LPFC (Jenkins & Ranganath, 2010), hippocampus and mPFC (Jenkins & Ranganath, 2016), relate to better coarse order memory and recency discriminations, respectively. Whether temporal signal drift at boundaries helps or hurts temporal order memory therefore might depend on the encoding distance between the two items, how likely they were to be integrated in-sequence, and what retrieval processes are engaged (see DuBrow & Davachi, 2017).

The acceleration of temporal signal drift at event boundaries may also contribute to overestimations of time that has passed between two recently encountered items when they had appeared with an intervening event boundary (Ezzyat & Davachi, 2014). Indeed, recent fMRI work demonstrates that the hippocampal patterns of activity are sensitive to the temporal duration of events, even when duration isn’t explicitly attended (Thavabalasingam et al., 2018). Behavioral studies have shown that the amount and/or diversity of event boundary-types during encoding may amplify time dilation effects in memory (Faber & Gennari, 2015, 2017; Lositsky et al., 2016; Waldum & Sahakyan, 2013). Thus, the mnemonic consequences of context shifts may accumulate over time, resulting in participants overestimating the amount of time that had passed when learning a recent video or item sequence.

As potential evidence of this shifting temporal context signal, one recent fMRI showed greater changes in patterns of entorhinal cortex activity during encoding were associated with larger retrospective judgments of duration between two clips from a recent video (Lositsky et al., 2016). It is noteworthy that, along with recent electrical stimulation work in humans (Goyal et al., 2018), these data highlight the important role of extra-hippocampal regions, such as the entorhinal cortex, in representing temporal information, particularly in areas that feed information directly into hippocampus. Intriguingly, recent fMRI work in humans suggests that while hippocampal processes contribute to subjective mnemonic representations of time, lateral entorhinal cortex activity reflects the objective amount of time that has elapsed between events (Bellmund et al., 2018; also see Montchal et al., 2018). In a similar finding, evidence in rodents suggests that the lateral entorhinal cortex, in particular, is important for encoding temporal information across short and long timescales (seconds-to-hours; Tsao et al., 2018). In the coming years, we expect that these convergent findings will ignite great interest in the entorhinal cortex’s role in processing and organizing temporal information in memory.

The processes that could speed-up temporal signal drift at event boundaries are less known. One possibility is that context shifts rapidly modulate activity in subsets of neurons specialized to represent temporal context information. Emerging research in rodents has identified ensembles of hippocampal CA1 neurons, or “time cells,” that fire in a temporally organized manner for distinct sequences of odors (Eichenbaum, 2014; MacDonald et al., 2011; Mankin et al., 2012). An important feature of these neurons is that they do not appear to track the objective passage of time. Rather, hippocampal CA1 time cells “re-time” when the temporal structure of learning is violated, as might occur at event boundaries, with the recruited neuronal ensembles and firing patterns being altered from before. Potential parallels between rodent time cells and hippocampal activity in humans raise intriguing questions about the processes that modulate temporal memory representations. Exploring potential relationships between episodic memory organization and hippocampal time cell function will be an exciting venture for future neuroscience research.

4. MECHANISMS FOR ORGANIZING MEMORIES ACROSS LARGER GAPS IN TIME

Much research on human memory organization has focused on how context changes influence different aspects of episodic memory for a single learning episode (e.g., a sequence, video clip, narrative etc.), with encoding and memory retrieval occurring within a single experiment session. However, in the real world, episodic memories are formed and organized over a lifetime of experience. Recalling the episodic details of such vast amounts of information thereby requires mechanisms that can also link or distinguish events across broader timescales of experience.

Here, we refer to ‘long’ timescales as learning events that are not temporally contiguous but rather are encountered either hours or days apart. These distal events may be perceptually similar, such as occurring in the same spatial context, or may be perceptually distinct. In the following sections, we review empirical work suggesting that time-dependent hippocampal mechanisms help to determine whether temporally discontinuous events become integrated or separated in memory. We also highlight recent fMRI research in humans showing that hippocampal and mPFC representations of overlapping contextual events may become more similar over time, and how the reinstatement of a prior context may link remote and recent memories together in a meaningful way.

4.1. Integrating overlapping memories across large gaps in time

Although time itself may function as boundary separating events, there is evidence that temporally discontinuous events can still become associated if they occur within a couple of hours (Cai et al., 2016; Kastellakis et al., 2016; Rashid et al., 2016). Like the moment-to-moment integration of contextually related stimuli, these associative learning effects also appear to involve the hippocampus (Figure 7).

Figure 7.

Effects of temporal context and proximity on hippocampal pattern integration versus separation. Separate learning events that occur within a specific time window (e.g., minutes-to-hours) may become integrated in memory. During learning, a distinct subset of neurons represents a specific context/memory (left panel; yellow dots). Up to 5 hours later, those neurons remain residually active (dim yellow dots) and may overlap with a subsequent event active. This similar but new learning episode (middle panel; purple dots) may lead to the recruitment of these overlapping hippocampal neuronal ensembles, so that these two events become linked in behavior (middle panel; yellow/purple combination dots). Across very long delays (days), hippocampal ensembles may become more differentiated despite representing similar spatial or perceptual contexts, including revisiting the same space (right panel; blue dots).

In one experiment, rodents were exposed to three different spatial contexts across different time lags (Cai et al., 2016). Seven days after exposure to neutral context A, rodents were exposed to neutral context B (long gap), which was followed 5 hours later by a fear conditioning manipulation in context C (short gap), in which animals received an aversive footshock. Memory for this fear conditioning procedure was measured as the amount of time the rodent spent freezing in a given context. The results revealed mice froze in context C, as expected, and, surprisingly, that they also froze in context B despite shock never having occurred in context B. This transfer of fear memory was not observed for neutral context A, which had occurred much earlier. This generalization across contexts was mirrored by hippocampal CA1 activity, which showed greater overlap in the neural ensembles coding for contexts B and C (short gap) than for A and C (long gap), suggesting that temporal proximity facilitates the co-allocation of distinct events to a shared memory representation.

In a related finding, conditioning mice to associate two separate tones with shock recruited overlapping neuronal ensembles in the amygdala (Rashid et al., 2016). Behaviorally, extinguishing the shock association with the second tone led to a decrease in fear responses to the first tone, suggesting that these separate memories/experiences had become associated. The memory-linking effects reported in both studies only occurred when the two distinct learning experiences occurred relatively close in time within the same day (5 or 6 hours apart) but not when they were experienced farther apart in separate experiment sessions (one week), suggesting that there is a limited time window in which residual neuronal excitability can overlap and become associated with a subsequent event (Figure 7).

Few studies in humans have explored neural mechanisms that support the effects of temporal proximity on associative memory. Yet behavioral evidence is consistent with the findings in rodents, whereby temporal proximity can link different, but conceptually related, learning events. Using fMRI, Zeithamova and Preston (2017) demonstrated that memory integration occurred when overlapping pairs of stimuli (faces and houses paired with the same object) were learned 30 minutes apart but not when they were learned 24 hours apart (Zeithamova & Preston, 2017). However, these time-dependent memory effects in humans were primarily associated with differences in integration evidence across visual cortical regions and the whole brain rather than specifically within hippocampus: On average, hippocampal BOLD activation was related to inference memory success across both temporal conditions. These differences might be driven in part by differences in methodology, such as the amount of time between learning events (minutes vs. hours), the types of learning, and types of stimuli. Yet the specific factors that might lead to a hippocampal temporal proximity effect in humans are unclear. Thus, further work is needed to determine similarities and differences between humans and animals in the mechanisms that co-allocate different experiences to shared neural ensembles (Schlichting & Frankland, 2017).

Integrating conceptually overlapping recent and remote information may also be critical for event comprehension. For example, one fMRI study showed that retrieving information about the first half of a movie viewed 24 hours prior supported memory for movie content from the second half of a movie (Chen et al., 2016). This retrieval of remote and related memories engaged the hippocampus as well as midline brain regions, including mPFC. Thus, like hippocampally-mediated binding across context shifts at relatively short time-scales (see Section 2.4), hippocampal processes also facilitate the retrieval of remote event representations to integrate and comprehend new task-relevant inputs. These neural patterns of remote memory retrieval are strikingly similar to dynamic mPFC-hippocampus interactions shown to support memory integration for novel sequences experienced on shorter timescales (DuBrow & Davachi, 2016). Notably, however, this hippocampal functional connectivity pattern may differ when encoding new facts that are congruent with prior knowledge (van Kesteren et al., 2014), suggesting that this interregional communication may be more important for novel, single-shot episodic encoding processes.

4.2. Hippocampal ‘timestamps’ may separate overlapping memories

The findings reviewed thus far suggest that temporal representations can either function as scaffolding for memory integration to occur or, when context shifts occur, become altered in ways that promote memory separation. The key factor in determining whether memory integration or separation occurs appears to be temporal proximity and/or overlap, either in internal representations of temporal context signals or the passage of time itself: Different learning episodes can become integrated in memory either because they appeared relatively close by in time or because they share overlapping contexts or content (e.g., Chen et al., 2016). On the other hand, accelerating drift in this signal may create the illusion that more time has passed, leading to changes in how we remember the timing and duration of past events.

Research suggests that gradual changes in hippocampal ensemble activity may also provide a “timestamp” for discretizing events on the order of seconds-to-minutes (Manns et al., 2007), hours-to-days (Mankin et al., 2015; Mankin et al., 2012), and even days-to-weeks (Rubin et al., 2015; Ziv et al., 2013). For example, in one rodent study using time-lapse imaging of thousands of hippocampal CA1 neurons, Rubin et al. (2015) decoded neuronal activity patterns that were unique to separate days of learning within the same spatial context. In fact, hippocampal CA1 activity patterns were more correlated for distinct spatial contexts experienced on the same day compared to activity patterns observed for the same spatial context experienced on different days. This finding suggests that temporally proximal – albeit distinct - environments lead to a hippocampal context signature that relates more closely to the time they were experienced rather than to their unique perceptual identities. Such temporal pattern separation processes between exposures to the same environment likely function to reduce mnemonic interference by providing repeated experiences with unique hippocampal signatures.

Interestingly, one recent fMRI study in humans showed that memory for word-item associations bear more distinct mPFC representations at retrieval when those pairs were studied in multiple learning contexts (i.e., across a night of sleep) as opposed to one context (Ezzyat et al., 2018). This neural pattern was associated with less forgetting for the overnight pairs versus same-day pairs seven days later. During retrieval, the degree of mPFC pattern differentiation was also correlated with hippocampal-mPFC functional connectivity, suggesting that these mechanisms help reduce interference between overlapping memories. Again, this is reminiscent of goal-relevant and context-appropriate binding processes modulated by interactions between mPFC and hippocampus. Further, it suggests that mPFC processes in humans may be important for representing specific contextual information in long-term memory across long timescales.

Studies in rodents show that, in addition to CA1, neurons in other hippocampal sub-regions, including hippocampal CA2, CA3, and dentate gyrus (DG), also represent temporal information (Mankin et al., 2015; Rangel et al., 2014; Salz et al., 2016). In fact, whereas temporal information coding in CA1 may support integrating events from the same spatial contexts across time (Ziv et al., 2013), there is evidence that hippocampal CA2 population activity may reflect the processing of temporal information based on this region’s relative insensitivity to processing spatial or sensory contextual information (Mankin et al., 2015). For information learned across longer delays, such as several weeks, converging theoretical (Aimone et al., 2014; Aimone et al., 2006) and empirical work (Rangel et al., 2014) suggest that neurogenesis in hippocampal DG may facilitate pattern separation in memory. This line of work suggests that more recent information becomes represented in newly developing DG neurons, whereas older memories are represented in older populations of DG neurons. Thus, different DG neuron populations uniquely support the temporal separation of distal memories. In sum, these studies suggest that the contributions specific hippocampal sub-regional processes to memory integration or separation may depend on the timescale across which two experiences occurred.

5. CONTEXT SHIFTS ENHANCE MEMORY FOR ITEMS AND THEIR SURROUNDING SOURCE INFORMATION

Most of this review has focused on how contextual stability influences the temporal structure of memory. However, context shifts also appear to influence other non-temporal aspects of episodic memory, such as later recognition of individual items and their surrounding source information. This may be driven, in part, by an increase in attention. As mentioned previously, influential theories of event cognition propose that event boundaries trigger attentional processes that prioritize new, incoming sensory inputs (Reynolds et al., 2007; Zacks et al., 2007). In turn, these increases in attention are associated with better encoding of information encountered at boundaries. Findings supporting this framework include work showing that item recognition is better for objects that had appeared at event boundaries compared with objects that had appeared within an event (Gold et al., 2017; Sonne et al., 2017; Swallow et al., 2009).

In a similar manner to event boundaries, salient events, such as the appearance of goal-relevant target or hearing sudden tones, can lead to memory enhancements for concurrently presented images, even if those salient stimuli occur incidentally to the task at hand (Swallow & Jiang, 2010, 2014). Similar enhanced encoding effects have been linked to transient increases in pupil dilation (Hoffing & Seitz, 2015; Tona et al., 2016), a biomarker of arousal and increased attentional load (Kahneman & Beatty, 1966). Likewise, highly arousing emotional stimuli or contexts have been shown to elicit pupil dilation patterns that also predict better item memory (Clewett et al., 2018).

Explicitly manipulating attention can also help to boost memory for information present at event boundaries. In one study, cuing individuals to encode a target still-frame image within a movie clip enhanced memory for images that coincided with event boundaries (Gold et al., 2017). Interestingly, cueing individuals to attend to the middle of event segments also led to similar mnemonic benefits that were observed for more ‘natural’ event boundaries in the clips (Gold et al., 2017). Eye-tracking evidence further suggests that such memory enhancements depend on whether those items are explicitly attended at that moment, as indexed by fixations (Swallow et al., 2009). Together these findings corroborate the idea that, insofar as they are salient or goal-relevant enough to garner attention, event boundaries help to amplify the encoding of incoming information. Future research could examine how manipulating attention to items appearing near boundaries influences their encoding. If arousal and salience are mechanisms of episodic memory organization, it may be the case that event boundaries will either enhance or impair recognition memory as a function of the priority of that proximal information (e.g., Clewett et al., 2017; Mather et al., 2015; Mather & Sutherland, 2011).

What might be the adaptive significance of enhanced boundary-information processing? One possibility is that anchoring salient boundary information in memory provides an entry point for recalling specific episodic events. Supporting this view, participants tend to make non-serial “jumps” to boundary information during recall (DuBrow & Davachi, 2016; Heusser et al., 2018). Forward transitions during recall have also been shown to be greater from boundary items than from pre-boundary items from a recently seen sequence of images (DuBrow & Davachi, 2013; Heusser et al., 2018). Taken together, these findings suggest that boundary representations form particularly strong memories, thereby providing a strong cue for serial recall as well. Recent evidence indicates that items presented at boundaries are bound more effectively to their source information, such as their background color, which may provide a strong episodic “tag” for distinguishing specific events in memory (Heusser et al., 2018; Figure 8).

Figure 8.

Memory tradeoffs between source memory-binding and temporal order memory elicited by event boundaries. (A) In this behavioral study, participants studied sequences with items displayed on a background colors. ‘Events’ were defined as 6 successive items with the same colored background, with each event being followed by a color switch, or event boundary. Participants were instructed to rate the pleasantness of each item-color pairing when each appeared. Following a list of 36 items, memory was tested for the background color of each object as well as the order between two items that were always the same temporal distance apart. (B) Results revealed that context shifts had different effects on source memory and temporal order memory, with shifts enhancing source memory for boundary items but impairing temporal memory for item pairs that had an intervening color switch. C) A tertiary split was performed on reaction times for the color pleasantness ratings during encoding and then further broken down by source memory accuracy for those items. Faster reaction times to event boundary items were associated with better source memory for the background colors. This finding suggests that greater attention to boundary representations was related to strong source memory binding for those items (adapted from Heusser et al., 2018).

Extending these findings, it has been shown that event boundaries (transitions between colored backgrounds) enhance source memory for a neutral word’s background color (Siefke et al., under review). This memory effect, however, only occurred under conditions of high context stability; that is, instances when the background color changes occurred after the same color was presented for several items in regular intervals. By contrast, there was no enhancement in color source memory for color-word ‘boundary’ pairs that appeared after an irregular or completely randomized series of color transitions. Similarly, in the reward domain, high absolute prediction errors have been associated with enhanced item and source memory across both high- and low-variance contexts (Rouhani et al., 2018). Together these findings highlight the importance of context stability in driving source memory for boundary representations: to trigger saliency signals that enhance source binding - likely through arousal-induced activation of neuromodulatory systems (e.g., Mather et al., 2015; Zacks & Sargent, 2010), the nature of the event boundary must deviate significantly from the recent encoding context.

The specific contributions of the hippocampus to boundary-enhanced source memory binding remain unclear. Theoretical models posit an ongoing competition between hippocampal pattern completion and separation operations (Rolls, 2013). Thus, one possible account is that boundaries may trigger a switch from retrieval operations – processes that help to maintain ongoing integration – to encoding new information. Two distinct hippocampal modes involved in these processes have been associated with unique physiological and neuromodulatory states (Duncan & Schlichting, 2018; Hasselmo et al., 1995); therefore, the boundary-triggered increases in arousal may be a mechanism of toggling between hippocampal binding versus separation.

Based on these data, it is conceivable that event boundaries elicit memory tradeoffs between retrieval (binding memories across time) and encoding (item/novel contextual information) operations in the hippocampus to parse events in memory. This shift may be due, in large part, to a rapid reorienting of attention. Indeed, recent behavioral work shows that greater attention to perceptual boundary items, as indexed by faster reaction times during color-object pleasantness judgments, is associated with impaired temporal memory across those event boundaries (Heusser et al., 2018 Figure 8C); Figure 8). At the same time, an increased bias towards encoding new inputs may anchor boundary representations into long-term memory more effectively, perhaps forming a contextually rich bookmark for recalling a specific episode later on.

6. SUMMARY OF BRAIN MECHANISMS THAT TRANSCEND TIME IN MEMORY

Identifying how the brain extracts and represents an overarching structure from experience is fundamental to our understanding of episodic memory. But what defines an “episode” in episodic memory? Prominent models of event cognition propose that event boundaries trigger mechanisms that parse continuous sensory inputs into discrete events. New research has begun to identify how event segmentation influences the neural processes that support the long-term organization of memory for those events. In accordance with a wealth of human and animal research on the key role of hippocampal processes in associative memory, these findings suggest that both proactive and retroactive integration processes, largely implemented in PFC and hippocampus, can transcend time to bind information into meaningful memory representations.

According to recent human fMRI data, the ways in which different context inputs influence temporal integration and separation processes can be indexed by both regional changes in the magnitude of the BOLD signal as well as similarities, or stability, in multivoxel activation patterns across time. An array of cognitive and situational factors, including attention, goals, and prediction, can modulate the ebb and flow between mnemonic integration and separation processes as experiences unfold over time. In novel situations, regularities and change in the elements of experience, including perceptual inputs and internal goal states, provide a scaffolding for two related processes: chunking related elements of experience into coherent episodes (integration) and pushing contextually-distinct representations into separate memory representations.

The cognitive event-parsing process that is triggered by context shifts, however, is not passive. Rather, it can be bridged by various factors, including higher-order cognitive processes. Mounting evidence suggests that top-down encoding strategies preserve temporal order memory despite shifts in lower-level sensory changes, such as a shift in spatial or perceptual features of the environment. In these cases, activation of the hippocampus and LPFC is associated with the recovery of recent pre-boundary information, which helps preserve memory for temporal order. Exposure to familiar sequences of information may proactively engage hippocampal retrieval/completion processes that integrate contextually related sensory inputs within an appropriate memory representation. These memory-binding processes appear to occur at both short and long timescales during sequence encoding.

Temporal memory integration also appears to occur retroactively immediately following an event. Specifically, event boundaries may trigger the reactivation of just-experienced information in the hippocampus, striatum, and frontoparietal cortex in ways that retroactively bind relational information in memory. But data suggest that context changes don’t always lead to memory separation. Rather, these effects can be overcome when new information is relevant to recent experience or one’s current goals. An important avenue for future research will be to clarify how and when these proactive and retroactive hippocampal and cortical binding processes may relate to and/or differ from one another.

There are some indications that experience shapes internal representations of time in ways that affect episodic memory organization. Converging work in animals and humans suggest that slowly evolving patterns of neural ensemble activity in the hippocampus, MTL, and PFC do not simply drift passively over time but rather may be modulated by context shifts during learning. The “resetting” or rapid drift in temporal context representations at event boundaries may function to separate sequential representations into distinct events, thereby facilitating later order memory. While more research is needed in this area, hippocampal time cells are a candidate neural mechanism, based on their sensitivity to the temporal structure of learning.

The recruitment of different neuronal ensembles appears to have a strong bearing over whether memories become separated or integrated. Recent rodent and human work also show that information learned in two different spatial/sensory contexts may become integrated through overlapping contexts and neural ensembles, if they are experienced close enough in time or when integrating disparate events (e.g., two halves of a movie) into a unified memory representation (e.g., representation of entire movie) is desirable. On the other hand, the acquisition of more distinct hippocampal timestamps may contribute to temporal pattern separation processes that differentiate memories of repeated exposures to the similar spatial/perceptual context. Temporal context signals thereby inform the separation or integration of events depending on whether an individual’s goal is to build a coherent memory irrespective of when things are learned or to minimize memory interference between perceptually overlapping events.

Together, these burgeoning lines of research have led to a more holistic understanding of how discrete memories emerge from continuous experience. New technical and conceptual advances in this area also raise many interesting questions about the neural mechanisms supporting the long-term organization of episodic memory, including how time is represented in the hippocampus and MTL, and how top-down signals from the PFC shape these cognitive representations. Much of this research may be informed by animal models, which show remarkable synergy with multimodal neuroimaging findings in humans.

7. CONCLUDING REMARKS

Whether there are specific principles or guidelines that determine when memories become integrated or separated is still unclear. The stability and volatility of ongoing experiences appear to be key factors driving long-term memory organization. Yet the boundary conditions of context-appropriate integration require further testing. For instance, if no boundary was present, would incoming information continue to be allocated to the same memory indefinitely? Answering these questions is challenging, as there are many contextual features in the real world that inevitably change. For instance, cues in the environment, such as whether it is day or night, are likely robust boundaries in everyday life. In addition, internal affective states, including mood, levels of arousal or wakefulness, and hunger/thirst continuously fluctuate over time, providing additional cues for segmentation. There is also the important question of how different types of theoretical boundaries, including changes in one’s thoughts versus changes in the external world, interact with each other to influence later memory.

As we’ve discussed, goals modulate the effects of boundaries on memory integration, suggesting there is flexibility in how events are constructed and integrated across time. Given this, perhaps there are no fixed rules that govern how and when integrated events emerge from experience, which may be a key strength of the episodic memory system. The brain may organize experiences somewhat automatically by tracking the stability of experience. Broadly, this in turn may bias how the structure of discrete sequences are preserved in memory (Dubrow and Davachi, 2014) and shape how we retrieve autobiographical memories in a temporally organized manner (Brunec et al., 2015). At the same time, however, having the ability to adaptively structure one’s own memories, particularly across larger gaps in time, might support how we derive meaning from our own unique experiences and goals (e.g., Barry et al., 2018; Schacter et al., 2007). Addressing these processes and their underlying brain mechanisms will be instrumental for understanding how our daily lives become woven into an autobiographical history of events. Perhaps more importantly, this line of work might also reveal flexibility in neural processing of events, which may be essential for constructing memories that can guide adaptive behavior.