Predicting the Past, Remembering the Future

Abstract

Rational analyses of memory suggest that retrievability of past experience depends on its usefulness for predicting the future: memory is adapted to the temporal structure of the environment. Recent research has enriched this view by applying it to semantic memory and reinforcement learning. This paper describes how multiple forms of memory can be linked via common predictive principles, possibly subserved by a shared neural substrate in the hippocampus. Predictive principles offer an explanation for a wide range of behavioral and neural phenomena, including semantic fluency, temporal contiguity effects in episodic memory, and the topological properties of hippocampal place cells.

Introduction

Why remember the past? George Santayana famously remarked that “Those who cannot remember the past are condemned to repeat it” [1], and William Faulkner expressed a similar sentiment in Requiem for a Nun: “The past is never dead. It isn't even the past.” Patterns of recurrence have fascinated historians from Polybius to Arnold Toynbee [2], and some have tried to harvest lessons from these patterns to guide political decision making [3]. More mundanely, the daily lives of most organisms are structured by cycles of sleeping, eating and other routine activities. At shorter timescales, many events can be anticipated using sensory information from the recent past, giving organisms with memory an adaptive advantage. Thus, memory of the past, spanning vastly different timescales, is relevant for predicting and controlling the future [4, 5, 6, 7].

Most research on human memory has focused on understanding, at a descriptive level, how information from the past is stored and retrieved, without contemplating the usefulness of this information for future action. However, researchers have increasingly come to appreciate the fact that memory is organized around predictive design principles [8], evident in multiple forms of memory (semantic, episodic, short-term and procedural). Computational models have formalized these principles mathematically, drawing upon ideas from library science, search engine algorithms, probability theory, and reinforcement learning. Despite this diversity of approaches, we will see that several of them can be unified in terms of a single predictive representation, repurposed (with slight modifications) for the needs of different memory systems. Neural correlates of this predictive representation have been observed in the hippocampus, suggesting a functional explanation for the region's involvement in both retrospective and prospective cognition.

The rational analysis of memory

John Anderson's “rational analysis of memory” was the first theoretical attempt to explain the structure of memory in terms of beliefs about the future [9, 10]. Anderson conceptualized the problem facing memory as one of determining need probability: the likelihood that a particular piece of information will be needed in the future. On the assumption that past need predicts future need, memory can used to produce a forecast. Importantly, because need probability may change over time, this forecast should adapt to the statistics of past experience.

It is worth highlighting here the extent to which this view represents a significant departure from prevailing ideas about episodic memory. The standard view (see [11] for a summary) holds that there is a fundamental tension between memory for specific instances (episodic memory) and memory for statistical regularities (semantic memory). In contrast, Anderson's rational analysis posited that episodic memory is structured by statistical regularities. To formalize this idea, Anderson adapted a model of library borrowing, likening memories to books in a library system. The usage of each book is subject to fluctuations, and the task facing the library system is to track these usage statistics in order to anticipate the probability that a book will be needed in the near future. Anderson showed that this model could capture many basic properties of memory, such as spacing, recency, fan and word frequency effects.

Several authors have explored different assumptions about the statistical regularities governing episodic memory. For example, Gershman and colleagues [12] developed a Bayesian nonparametric model in which the environmental dynamics can switch between an unknown number of “modes.” This model makes the prediction (confirmed experimentally) that abrupt changes will result in the inference that a new mode has been encountered, effectively creating an event boundary that reduces interference between items on either side of the boundary. The importance of event boundaries in memory formation and retrieval has been highlighted by a number of other recent studies [13, 14, 15, 16, 17, 18].

Rational analysis has also been applied to semantic memory. In this case, the need probability is dictated by the long-run marginal probability of items, rather than the temporally specific need probability used in the analysis of episodic memory. Griffiths and colleagues [19] showed that the semantic need probability is precisely what is computed by Google's PageRank search algorithm [20]. They conceptualized semantic memory as a directed network (a kind of mental World Wide Web) over which semantic processing flows according to a random walk. If we let L̄_ij = L_ij/ Σ_k L_ik denote the normalized link matrix (where L_ij = 1 if there is a link from item i to item j, and 0 otherwise), then PageRank computes the vector of item ranks r according to the recursive definition r = L̄r.

We can understand the rank of item i as proportional to its marginal probability P(i) after running the random walk on the semantic network for a long time. Intuitively, this means that the proportion of time a person spends thinking about item i scales with its rank. It can also be interpreted as a probabilistic model of environmental structure: if the need for an item over time can be described by a random walk, then its rank reflects its long-run need probability. Griffiths and colleagues asked participants to generate words beginning with a particular letter, and showed that the number of times a word was retrieved (a measure of semantic fluency) was well-predicted by its rank. This suggests that human semantic memory is structured to make predictions of future need probability over long timescales.

A predictive substrate

Another way to derive the PageRank algorithm is to construct a predictive representation for each item i that encodes the expected discounted number of times item j will be needed on a random walk initiated at i:

$$M_{\mathit{\text{ij}}}=\mathbb{E}\left[\sum _{t=1}^{\infty }{\gamma }^t\mathbb{I}(s_t=j)|s_0=i\right],$$ (1)

where s_t denotes the item needed at time t, 𝕀(⋅) = 1 if its argument is true (0 otherwise), and γ is a discount factor controlling the effective time horizon over which item counts are accumulated. The expectation is taken over randomness in state transitions and rewards. One can interpret 1 − γ as the constant probability that the random walk will terminate at any given time. An item's rank can be computed from the predictive representation simply by summing the rows of M: r_j∝ Σ_i M_ij. In other words, the long-run probability of visiting an item is obtained by summing its cumulative expected need across the collection of random walks initiated at different items.

This way of deriving PageRank is intriguing because the matrix M has another life in the reinforcement learning literature, where it is known as the successor representation (SR) [21]. The central problem in reinforcement learning is the computation of value, the discounted cumulative reward expected in the future upon entering a state:

$$V_i=\mathbb{E}\left[\sum _{t=1}^{\infty }{\gamma }^{t}R(s_t)|s_0=i\right],$$ (2)

where R(s) is the immediate reward expected upon entering state s. The SR renders value computation a linear operation, due to the identity V_i = Σ_j M_ij R(j). It therefore offers a significant computational advantage over traditional model-based planning algorithms that typically scale super-linearly with the number of states [22].

Further insight can be gleaned by considering the long-run average reward. It can be shown that the long-run average reward is a reward-weighted version of the rank (and hence need probability) returned by the PageRank algorithm, revealing a deep and surprising formal connection between reinforcement learning, information retrieval, and the rational analysis of memory. It suggests a way in which reward associations can reweight need probabilities, in accordance with well-known motivational influences on memory [23, 24, 25].

Learning the successor representation with temporal context

The SR is defined as an expectation over infinitely long trajectories, raising the question of how it can be tractably computed. One possibility, borrowing ideas from reinforcement learning, is to directly update an estimate the SR (M̂) from state transitions. Specifically, the SR can be updated incrementally using a form of temporal difference learning [26]:

$$\mathrm{\Delta }\widehat{M}(i,j)=\alpha [\mathbb{I}(s_{t+1}=j)+\mathrm{\gamma }\widehat{M}(s_{t+1},j)-\widehat{M}(s_{t,j})]e_t(i),$$ (3)

where α is a learning rate and e_t(i) is an exponentially decaying memory trace of recent states that specifies the “eligibility” of state i for updating. The term in brackets represents a “prediction error” analogous to the reward prediction error thought to be conveyed by phasic dopamine signaling [27], but in this case the prediction error is defined on state expectations rather than on reward expectations.

The temporal difference update described above is nearly identical to an influential theory of episodic memory, the Temporal Context Model (TCM; [28, 29]). TCM was originally developed to explain the temporal structure of memory retrieval in free recall experiments, but has more recently been applied to many other phenomena, ranging from false memory [30, 31] to spacing and repetition effects [32]. According to TCM, items are bound in memory to a slowly drifting representation of temporal context (a recency-weighted average of previous items), and at test the temporal context acts as a retrieval cue, preferentially drawing items based on their strength of association. It can be shown (see [26] for details) that the temporal context representation corresponds to the eligibility trace e_t, and the matrix of item-context associations corresponds to M̂.

The connection between TCM and the SR suggests another way in which episodic memory can be cast in terms of predictive mechanisms. However, the temporal difference learning update is equivalent to the originally proposed Hebbian update rule [28] only when items are presented once per list. In this case, the second term in the temporal difference update is always zero, because no predictions for novel items have been formed yet. When items are repeated, the second term comes into play, altering the theory's predictions. An elegant study [33] drew out these implications, testing the prediction that the temporal difference update will produce a context repetition effect: repeating the temporal context of a particular item will strengthen memory for that item, even if the item itself was not repeated. This prediction follows from the fact that the second term in the temporal difference update will be positive whenever an item's temporal context recurs. The study found support for this prediction across several experiments, as well as a boundary condition, whereby the effect only occurs when the item is strongly predicted by the temporal context. This again is entirely consistent with the theory, since the second term in the update will be smaller when the predictive relationship between context and item is weak. The context repetition effect was also recently demonstrated in rats using an object recognition task [34].

The role of the hippocampus

The multi-faceted role of predictive representations in memory suggests that they may have a dedicated neural substrate. Several lines of evidence suggest that the hippocampus, although traditionally viewed as a repository for episodic memory and spatial representation, may in fact be organized around predictive principles [35, 36]. First, place cells in the hippocampus sweep ahead of an animal's current position when it is at a choice point [37]; these forward sweeps may arise from phase precession, the progressive shift in spike timing relative to the ongoing theta oscillation as an animal moves through a place field [38, 39, 4]. Second, when an animal repeatedly runs on a particular trajectory, place fields tend to expand opposite the direction of travel [40], consistent with the idea that earlier place cells learn to predict upcoming locations. Third, functional MRI experiments in humans have indicated that the hippocampus is sensitive to the predictability of upcoming stimuli [41, 42, 43, 44]. Fourth, the hippocampus is activated when humans engage in episodic future-thinking [45], and damage to the hippocampus appears to severely impair this ability [46] (though this finding is controversial; see [47]).

Recent theoretical work has formalized the predictive function of the hippocampus in terms of the SR [48]. According to this view, place cells do not actually encode spatial location; they encode expectations about future locations (though see [49] for evidence that some place cells have no predictive properties during immobility). In an open field, these future locations will tend to be in the vicinity of the animal's current location, yielding classical radially symmetric place fields. However, predictive fields will become distorted when obstacles are introduced or the environmental topology is altered (Figure 1)—just as observed experimentally [50, 51, 52]. Likewise, replay of place cell sequences appear to respect the environmental topology [53]. The topological sensitivity of the SR is a natural consequence of the fact that an animal's future trajectory is constrained by barriers to movement.

Figure 1. The successor representation defined over space.

(Left) Neurons tuned to different locations encode prediction about future locations. Shading corresponds to expected number of times a neuron's preferred location will be visited along a trajectory initiated at the animal's current location. (Right) Simulated predictive place fields for different environments.

Even when there aren't physical barriers to movement, the SR may still distort along certain paths, for example when rewards occur in reliable locations. This follows from the fact that an animal's policy tends to be reward-seeking. Consistent with this hypothesis, place fields tend to cluster around rewarded locations [54]. Moreover, a recent experiment found that forward sweeps preferentially visit rewarded locations [55].

Early during learning, the SR may resemble a relatively “pure” representation of space, since the predictive relationships between states have not yet been learned. The fact that some place cells show no predictive properties during early exposure to an environment [49] aligns well with the classic finding that contextual fear conditioning is more effective if the animal is first pre-exposed to the environment (an effect that is hippocampus-dependent [56]). According to the SR theory, predictive relationships between states allow the shock association to propagate to other states (i.e., locations within the environment), leading to a more robust fear memory.

It is important to note that the SR theory does not entirely dispense with some notion of purely spatial representation, as posited by the classical place cell literature. In fact, the SR, when defined over space, is predicated on such a representation, since this is the only way to index spatially distinct states. Likewise, place cells could also index local perceptual features [57], allowing the SR to be defined over this feature space.

Another point of contact between the classical place cell literature and the SR theory concerns the mechanisms of SR updating. The previous section described a temporal difference learning algorithm for updating the SR based on experienced trajectories. However, it is well-known that animals can rapidly change their behavior in response to structural changes in the environment without directly experiencing those changes, such as the detour behavior observed by Tolman [58]. These rapid behavioral changes seem to require an internal model or “cognitive map” that can be updated in the absence of direct experience. One way to accomplish this within the SR theory is to posit a simulation mechanism that can use an internal model to update the SR [59, 60], possibly implemented by forward sweeps of place cells [37].

One of the strengths of the SR theory is that it provides a framework for understanding the origin of non-spatial, relational representations in the hippocampus. For example, the hippocampus is involved in the computation of transitive inferences [61], and damage to the hippocampus impairs transitive inference [62]. Recently, Garvert and colleagues [63] used functional MRI to directly examine the SR account of relational representation. Subjects in their study were exposed to sequences of items generated by a random walk on a graph. Representational similarity in the hippocampus (as measured by adaptation of the neural response across random pairs of items) scaled inversely with distance on the graph, and quantitative analyses revealed that this similarity structure was best described by the SR compared to other graph-theoretic measures.

Predictive spiking neurons

How does the SR arise in biologically realistic neural circuits? Neurons must spike within a few tens of milliseconds of one another for their synapses to be strengthened, much shorter than the temporal horizon of predictive codes like the SR (on the order of seconds). Brea and colleagues [64] have developed a framework for resolving this puzzle, using a variant of traditional spike timing-dependent plasticity [65].

The basic idea is illustrated in Figure 2. When a synapse between two neurons is potentiated, the presynaptic neuron can cause the postsynaptic neuron to fire earlier. This in turn allows the postsynaptic neuron to enter into plasticity with other neurons that are spiking even earlier. Thus, the membrane potential of the postsynaptic neuron comes to reflect the anticipatory drive from presynaptic inputs; in effect, the postsynaptic neuron is anticipating its own future spiking. This is the same kind of bootstrapping that underlies the temporal difference learning algorithm described above.

Figure 2. Prospective coding by spiking neurons.

(A) A network of spiking neurons (colored triangles), with spike times indicated by vertical lines. The green neuron drives spiking of the purple neuron, opening a potentiation window (red box) during which other spiking neurons can strengthen their synaptic connections with the purple neuron. (B) As a consequence of plasticity, the purple neuron can now be driven by the blue neuron, shifting the potentiation window earlier and allowing the orange neuron to strengthen its connection with the purple neuron. The activity of the green neuron thus becomes predicted by neurons spiking earlier in time. Adapted from [64].

Brea and colleagues [64] showed that a form of spike timing-dependent plasticity can be used to implement predictive spiking. In particular, under some fairly general assumptions their plasticity rule enables the dendritic potential to directly encode the expected discounted future spike rate at the soma—i.e., one column of the SR matrix M. Predictive spiking offers an explanation for widespread anticipatory responses in the brain [66, 43, 67].

Conclusions

Starting from the principle that memory is designed to serve a predictive function, a rich web of theoretical insights can be derived. Semantic and episodic memory can be linked via a common predictive representation initially studied in the reinforcement learning literature, allowing us to contemplate the computational properties that these different forms of memory may have in common. These commonalities may derive from a shared neural substrate, as suggested by the widespread involvement of the hippocampus across domains [68].

Despite the appeal of a unifying neural substrate, this viewpoint is in tension with taxonomies of memory that stipulate dissociable systems in the brain [69]. For example, transformational consolidation theories hold that episodic memories stored initially in the hippocampus are gradually transformed into semantic memories stored in neocortex [70, 11]. This viewpoint is supported by evidence that hippocampal amnesics show spared long-term semantic memory [71]. However, more recent experiments have found semantic impairments in amnesia (see [72] for a review), and amnesics exhibit semantically impoverished episodic future-thinking [73]. Beyond semantic memory, there is now considerable evidence that the hippocampus (and episodic memory more generally) plays an important role in reinforcement learning as well [7]. Collectively, these observations lend credence to the idea that multiple forms of memory may draw upon a common predictive representation in the hippocampus.

Another tension derives from dissociations between different forms of memory that support behavioral control in reinforcement learning tasks. For example, stimulus-response strategies depend on sub-regions of the striatum but not on the hippocampus, whereas allocentric “place” strategies depend on the hippocampus but not the striatum [74]. These dissociations indicate that a predictive representation in the hippocampus can only be one part of a larger multi-system architecture, whose components interact both competitively and cooperatively [7].

Of special interest.

• [44] A functional MRI study demonstrating that the hippocampus is sensitive to long-range temporal clustering of states.

• [11] A theoretical review linking the neurocognitive mechanisms of episodic and semantic memory to machine learning algorithms.

• [4] A computational model of how hippocampal theta oscillations can be used to predict the future, by translating a representation of stimulus history forward in time.

• [63] Evidence from functional MRI adaptation that the similarity structure of hippocampal representations is sensitive to predictive relationships between stimuli.

Of outstanding interest.

• [33] An elegant study of recognition memory that directly supports a temporal difference learning update rule for the successor representation.

• [55] Electrophysiological evidence that place cells are sensitive to reward predictions.

• [64] A spiking neuron implementation of the successor representation.

• An updated rational analysis of memory using concepts from graph theory and reinforcement learning.

• A single predictive representation can unify multiple forms of memory.

• The hippocampus may be responsible for encoding this predictive representation.