Temporal and spatial context in the mind and brain

Abstract

Theories of episodic memory have long hypothesized that recollection of a specific instance from one’s life is mediated by recovery of a neural state of spatiotemporal context. This paper reviews recent theoretical advances in formal models of spatiotemporal context and a growing body of neurophysiological evidence from human imaging studies and animal work that neural populations in the hippocampus and other brain regions support a representation of spatiotemporal context.

Introduction

Since its earliest conception, episodic memory has been hypothesized to be associated with the recovery of the spatiotemporal context in which a memory was formed [1]. For this to be the case, the brain must contain a representation of spatiotemporal context that codes information about the spatial location and temporal relationships between events. The hippocampal place code provides dramatic evidence for a representation of spatial context in the brain [2, 3, 4]. A representation of spatiotemporal context should also change gradually over time to enable the expression of temporal relationships, tapping into a longstanding tradition in mathematical psychology in which a gradually-changing state of context mediates associations between stimuli [5, 6, 7]. Later work agumented these earlier formal models of contextual drift by hypothesizing that episodic memory is associated with recovery of a gradually-changing state of temporal context, enabling a concise account of behavioral contiguity effects, especially in the free recall task [8, 9, 10]. Neurophysiological work has long shown that the firing rate of neural populations in the hippocampus and elsewhere change gradually over time even when experimentally-controlled variables are equated [11, 12, 13, 14].

This paper reviews recent progress towards an understanding of the behavioral and neural evidence for a representation of spatiotemporal context. There is by now overwhelming neurophysiological evidence for a representation of spatial context—spatial correlates of neural firing in the hippocampus and related brain regions—from a range of species (see [15, 16, 17] for recent reviews). Because it is less well-known, this paper will focus on empirical evidence for temporal aspects of a context representation based on recent work from both animal and human studies. In addition to empirical developments, there have also been recent theoretical developments in our understanding of spatial and temporal context and their relationship to one another.

Empirical status of retrieved temporal context models

Retrieved context models hypothesize that a gradually-changing state of temporal context is recovered when an episodic memory is retrieved. This provides a natural account of behavioral effects that show associations between items presented close together in time. The importance of this contiguity effect in human episodic memory has recently been challenged [18]. However, that position seems inconsistent with recent empirical findings demonstrating that essentially all normally-functioning individuals show a temporal contiguity effect [19] in free recall studies and that the temporal contiguity effect in free recall is correlated with remember vs know judgments in recognition memory [20], suggesting that the processes supporting contiguity are not limited only to recall tasks. Moreover, detailed free recall modeling has shown that this approach is sufficient to account for detailed properties of the spacing effect in free recall [21] and patterns of recall and recognition performance associated with normal aging [22].

Recent evidence for a gradually-changing state of temporal context in the brain

There is now overwhelming evidence that neural population vectors in the rodent brain change gradually over macroscopic periods of time ranging from minutes up to at least days, providing a potential temporal context signal for temporally mediated associations. In recent years, hippocampal region CA2 has been identified as possibly having a central role in temporal variation, in contrast to other hippocampal regions that show both spatial and temporal correlates [23]. Gradual neural drift has been observed in animal preparations with a range of imaging modalities, including optical recordings [24] and activity-dependent markers [25, 26]. Notably, these last two studies showed that markers of gradually-changing neural activity in the amygdala and hippocampus coincided with temporally-graded behavioral associations for fear conditioning, consistent with the hypothesis that temporally-graded associations are mediated by a gradually-changing state of temporal context in the hippocampus [14, 24].

In addition to animal work, human studies have also shown that neural processes measurable with fMRI change gradually over time and are affected by the history of recent stimuli and, in turn, affect behavioral performance in a range of tasks. The representational similarity in the entorhinal cortex and other brain regions between studied events predicts the judged time between those events [27]. Similarly, pattern dissimilarity in the hippocampus during study predicts successful performance in the judgment of recency task [28]. Similarly, the degree to which hippocampal patterns reflect the content of intervening items—the temporal context—predicts success in judging the time that has elapsed between pairs of stimuli [29, 30]. A study in which sequences of stimuli were presented showed that the multivoxel pattern changed gradually over time and contained information about recent stimuli [31]. Subsequent work suggests that this temporal context signal is present not only in the hippocampus, but also other brain regions throughout the “core recollection network” [32]. Another study showed repetition suppression to stimuli presented in repeated temporal contexts [33], minimizing any concerns that the analyses in [31] were contaminated by a slow hemodynamic response.

fMRI evidence from the free recall task confirms the observation that recently-experienced stimulus categories contribute to the information contained in the BOLD signal [34]. Moreover, this information about the preceding temporal context in the BOLD signal predicts transitions among words preceded by similar temporal contexts in free recall. Taken together, the animal and human work leave little doubt that the brain contains a representation of the recent past that comports with the basic properties required by retrieved temporal context models.

Is temporal context recovered in episodic retrieval?

The other main contention of retrieved temporal context models—that temporal context is recovered as part of episodic memory retrieval—has not yet been definitively observed. There is very strong evidence that episodic memory results in a recovery of brain states observed during study. It is also clear that this recovery can cause associations that bridge across different experiences with a stimulus, as predicted by retrieved context models of memory. It is also known that behavioral markers of contextual retrieval correlate with hippocampal activity [35]. However, it has not yet been definitively established that gradually-changing brain states are in fact recovered during episodic memory. That is, suppose that the brain state is a superposition of a part that changes, and another part that reflects currently available information. The available neurophysiological data collected in the last few years is consistent with the hypothesis that only the part that reflects currently-available stimuli is recovered. Earlier studies attempted to directly test the hypothesis that a gradually-changing temporal context was retrieved [36, 37, 38], but each of these studies have some limitations that prevent a definitive answer to this question.

Although it is not direct neural evidence for recovery of temporal context, a growing body of evidence demonstrates that the hippocampus and PFC participate in building transitive associations that bridge across temporally disjoint events, as predicted by retrieved temporal context models. After study of a-b and, much later, b-c, a transitive association is observed to the extent that a and c become associated to one another. Transitive associations are a natural prediction of retrieved temporal context models. If, during study of b-c, b recovers its previous temporal context, which includes information from a, then this provides a natural mechanism for associations between a and c. Recent fMRI evidence shows that transitive associations are observed for rewarding stimuli dependent on connections with the hippocampus [39], reviewed in [40]. Other evidence shows that the degree to which transitive a-c associations are formed between neutral stimuli is dependent on the degree of hippocampal-PFC connectivity [41] (reviewed in [42]). This adds to a body of work indicating that temporal context affects hippocampal representations [43]. MEG evidence suggests that the retrieval of transitive associations takes place after recovery of item information, suggesting an alignment with the time course of retrieval in recognition memory [44]. It should be noted that transitive associations are not a unique prediction of retrieved temporal context models [45, 46].

Although there has not yet been a definitive study, technology exists to measure the recovery of a neural temporal context. First, any neural measure that generates a multivariate response can in principle be used to estimate a neural jump back in time. Suppose that we measure the multivariate response during retrieval of a stimulus originally presented in a randomly-assembled study list that includes many other stimuli. If the remembered stimulus was originally presented at serial position i, then retrieved temporal context models predict that the pattern after retrieval should resemble the neural pattern during study of stimuli presented close in time to serial position i. This strategy has been pursued previously with human single unit and ECoG studies [37, 36, 38], but with empirically-ambiguous results. Modern techniques for reactivating tagged neural ensembles [47] provides another possible avenue for establishing a causal connection between retrieved temporal context and behavioral associations. It is known that ensembles change gradually over many days [24] and that temporally-modulated representations covary with temporally-graded associations [26]. There is no conceptual or technical obstacle to a study in which the neurons active in a particular environment associated with fear are tagged. If reactivation of that ensemble in environments learned close together in time results in temporally-graded fear generalization, this would establish a causal connection between retrieval of temporal context and temporally-graded behavioral contiguity effects.

Advances in the theory of temporal and spatial context

In parallel with empirical progress, recent theoretical developments have led to an integrative hypothesis for spatial and temporal context that unifies a wide range of behavioral and neural phenomena. A large body of behavioral and theoretical work has suggested that the representations supporting memory should be temporally scale-invariant, exhibiting similar sensitivity to temporal relationships over a range of time scales [55, 56, 57]. Inspired by the idea that memory ought to be scale-invariant, this theoretical approach develops a set of equations to describe a compressed representation of recent experience [58, 59]. Because the temporal history changes gradually from one moment to the next, this neural representation leads to similar predictions regarding temporally-graded associations. Moreover, because it contains information about the temporal history leading up to the present, this representation can also be used to account for phenomena that depend on explicit temporal information, including judgment of recency tasks and scale-invariant timing behavior [50]. Rather than merely changing gradually over time, this compressed timeline provides a record of what stimuli happened at what point in the past (Fig. 1A).

Figure 1.

Theory of scale-invariant temporal context and neurophysiological evidence. A. A method for constructing a scale-invariant timeline. Top: an input function through time. Middle: a set of units retains the Laplace transform of the input function up to that point. Different lines are units with different time constants, corresponding to different values of the (real) Laplace domain variable. Bottom: By approximating the inverse Laplace transform a set of cells constructs an estimate of the history of the input function up to that point. Three units corresponding to different parts of a compressed timeline are shown. These units respond a characteristic time after the input was presented. Because the timeline decreases in accuracy as it recedes into the past, the “time fields” of these cells grow wider and the number of cells with time fields decreases as the delay goes on. After [48]. B. Neurophysiological observation of time cells with the qualitative properties predicted by theoretical work. Left: Firing rate properties of neurons in rodent CA1. These cells fire sequentially during the delay of a memory experiment, with increasing width of the time fields as the sequence goes on. After [49]. Right: Width of time fields as a function of the center location in various brain regions in rat. The theoretical framework in [50] predicts a linear increase. Clockwise from top left, rodent hippocampus (different colors for CA1 vs CA3) [51], rodent medial entorhinal cortex (different colors show grid cells vs non-grid cells [52], medial prefrontal cortex [53], and striatum (different colors are different delay durations) [54].

At the neural level, such a timeline would consist of cells that fire conjunctively when a particular stimulus was experienced at a certain point in the past. In this way, a set of these cells would enable reconstruction of what happened when in the past. These predictions resemble the properties of hippocampal “time cells” that fire sequentially during a circumscribed period within a delay period [60, 49]. Consistent with the hypothesis that the compressed representation of the past is accessed in many different forms of memory, time cells are not only observed in the hippocampus [51], but also the entorhinal cortex [52], striatum [54, 61], and medial PFC [53, 62]. Notably, in all of these studies, time cells show a characteristic compression as qualitatively predicted by theory (Fig. 1B).

The mathematical specifics of the compressed representation [58, 59] endow it with numerous desirable theoretical properties. The compressed timeline, manifest as a set of units with firing properties like time cells, is generated from an intermediate representation. This intermediate representation stores at each moment the Laplace transform of the input history as a function of time. By modulating the equations governing the Laplace transform with velocity, one can construct the Laplace transform with respect to position. By providing different input functions and modulating the Laplace transform with various functions, one can generate a “particle zoo” of place cells observed in the hippocampal formation during spatial navigation tasks in rodents [63] (Fig. 2A). As such, the same mathematical framework can be used to construct a representation of temporal context, of spatial context, and of conjoint spatiotemporal context [65].

Figure 2.

Spatiotemporal context in the hippocampus. A. Theoretical generalization of scale-invariant temporal context to spatial context accounts for a variety of place cell phenomena. a,b. Simulated place cells in enclosures of various shapes. In a, the simulated place cell does not change its place field location relative to the northwest corner as the environment is stretched. In b, the simulated place cell does not change as the environment is deformed in the east-west direction, but stretches when the environment is deformed in the north-south direction. c. A temporally modulated place cell. The plots left to right show simulated place cell activity averaged over different time intervals. The place field comes into existence, then persists for a time, then disappears. After [63]. B. After learning object locations in a virtual environment neural pattern similarity was computed between pairs of tested images. Left: Voxels where neural pattern similarity between pairs of images was correlated with the distance between images, with the time between them, and both. Right: Pattern similarity across the hippocampal ROI for pairs of images based on the quantile of spatial and temporal distance between the learned locations. Retrieval of the memories result in neural representations that reflect the temporal and spatial distances in the virtual world. From [64].

This unified account of spatial context, temporal context and spatiotemporal context aligns well with recent fMRI studies virtual reality suggesting not only that similar regions participate in recovery of spatial and temporal context [66, 67, 64] (see also Fig. 2B). These findings have been extended to real world memory [68], where hippocampal ensembles capture not only the spatial distance between memories and the temporal distance between memories, but also the interaction of spatial and temporal distance. Rodent work demonstrating that entorhinal grid cells have time-cell-like properties in time cell experiments [52], suggesting that spatial and temporal context representations are supported in some cases by the same neurons. Moreover, because of the mathematical properties of the representation, it can be extended further to generate compressed representations of the future [69].

Conclusions

Recent years have shed light on retrieved temporal context models of episodic memory. There is now strong evidence from animal and human work that brain states change gradually over time reflecting recent stimuli and that these brain states predict behavior, as predicted by retrieved context models. Although the recovery of these gradually-changing states—a jump back in time—has not been definitively established, a growing body of evidence is consistent with predictions of retrieved temporal context models, especially with regard to transitive associations that bridge across episodes. In addition, theoretical developments have produced a richer and more detailed hypothesis about the nature of temporal context. These new models account for a much wider range of behavioral phenomena, provide a natural explanation of sequentially-activated time cells and provide a natural theoretical unification between temporal context, spatial context, and spatiotemporal context.