Navigating for reward

Abstract

An organism’s survival can depend on its ability to recall and navigate to spatial locations associated with rewards, such as food or a home. Accumulating research has revealed that computations of reward and its prediction occur on multiple levels across a complex set of interacting brain regions, including those that support memory and navigation. Yet, how the brain coordinates the encoding, recall, and use of reward information to guide navigation remains incompletely understood. In this Review, we propose that the brain’s classical navigation centres — the hippocampus (HPC) and entorhinal cortex (EC) — are ideally suited to coordinate this larger network, by representing both physical and mental space as a series of states. These states may be linked to reward via neuromodulatory inputs to the HPC–EC system. Hippocampal outputs can then broadcast sequences of states to the rest of the brain to store reward associations or to facilitate decision-making, potentially engaging additional value signals downstream. This proposal is supported by recent advances in both experimental and theoretical neuroscience. By discussing the neural systems traditionally tied to navigation and reward at their intersection, we aim to offer an integrated framework for understanding navigation to reward as a fundamental feature of many cognitive processes.

Introduction

Strong recall of rewarding experiences is crucial for survival. To navigate to a remembered food source or safe home, the brain must search in memory to retrieve predictions about where reward is located, given environmental features and the animal’s past experience. At the same time, human reward memory can become pathological in mental illnesses such as drug addiction¹. For example, the spatial context of an initial drug experience can invoke relapsed drug use². Investigating how spatial experience becomes associated with reward has therefore been a traditional pursuit of the addiction field, often focusing on the midbrain dopaminergic system. Although understanding spatial reward memory is key to the treatment of addiction and other mental illnesses, it is also crucial for our basic knowledge of how the brain amplifies specific information for future use.

To localize reward within a given experience, the brain must create a neural map of the external environment³. One brain region thought to play a critical role in forming this neural map is the medial entorhinal cortex (MEC)^4,5. Neurons in the MEC encode variables required for the computation of an animal’s position and movement in a spatial reference frame, including: spatial position^4,6, head direction^7-9, movement speed¹⁰, relative proximity to objects¹¹ and environmental borders^12,13. Complementing the MEC neural code, neurons in the lateral entorhinal cortex (LEC) encode variables such as time¹⁴ and the presence or absence of objects^15,16. Among these physiologically defined ‘cell types’, grid cells [G] in the MEC seem particularly poised to support navigation, as they tile environments via periodic, hexagonally organized firing fields⁴. An animal’s position can be precisely encoded with only a handful of grid cells¹⁷, and an intact grid network is crucial for optimally performing path integration — the process of calculating direction and distance travelled based on perceived self-motion^18,19. Grid cells have been observed in species ranging from rodents to humans (for review, see ²⁰), suggesting that MEC neurons may provide an evolutionarily conserved coding scheme to support navigation and the creation of spatial memories.

The MEC and LEC are highly interconnected with the hippocampus (HPC). Hippocampal subregions CA3, CA1 and the dentate gyrus (DG) primarily receive input from the superficial layers of EC (layers II and III), whereas CA1 and subiculum send outputs primarily to the deep layers of EC (layers V and VI). The microcircuitry between layers of the EC completes the HPC–EC loop²¹. In the HPC, many neurons are maximally active at one or few specific spatial locations in an environment, earning them the name place cells [G] ²². Place cells form tightly organized sequences that represent specific trajectories through space during two main behavioural modes. During movement, spike timing is sequentially organized across place cells in theta sequences [G] relative to the hippocampal theta rhythm^23-25. During immobility, high-frequency oscillations known as sharp-wave ripples [G] ²⁶ contain bursts of sequential spikes that reactivate spatial trajectories in replay events [G] ^27-30. Multiple works have linked the activity of place cells^31-33, theta sequences^34-36 and replay events^37-41 to memory-dependent behaviours. Recent work has further demonstrated that optogenetically stimulating specific place cell populations representing either reward locations or trajectory starts evokes behaviours associated with those locations⁴². Together, these findings establish a causal role for hippocampal place cell activity in the execution of navigational behaviors.

In recent years, accumulating evidence has demonstrated that hippocampal and MEC activity generalizes to non-spatial tasks. HPC and MEC neurons encode sequential structure in time^43-49, providing a potential neural code for elapsed time. HPC and MEC neurons also fire at discrete points along progressions of sensory stimuli that change with the proximity to a goal (such as particular frequencies of sound), without requiring changes in physical position^50-52. Moreover, evidence from human imaging⁵³ and computational modelling^54,55 suggests that MEC and HPC coding schemes generalize to relationships between elements of abstract spaces. It remains unclear to which abstract spaces this generalization applies, as well as how individual neurons might compute such abstract codes. Nevertheless, the HPC–EC system is clearly engaged in cognitive function beyond spatial navigation, consistent with evidence that the HPC–EC system in humans encodes the order of events in episodic memory⁵⁶.

In this Review, we focus on spatial navigation as a model for the mnemonic function of the HPC–EC system in associating rewards with the locations and events that surround them. We advocate that the HPC–EC system is ideally situated to connect outcomes to a sequence of states [G] ⁵⁴ that discretize an experience, in which each state is an instance in physical or abstract space. We suggest that the HPC–EC system specifically encodes the order of these states as a general sequence of events in an episodic experience. We review several lines of experimental evidence that the HPC–EC is involved in transforming these event sequences into predictions and memories of reward at the physiological and representational levels. First, neuronal representations of reward are present in multiple forms within the HPC and EC. Second, neuromodulatory centres, including those of the dopaminergic system, directly innervate the HPC and EC, shaping local plasticity and representations of space and reward. Third, the unique laminar circuitry of the HPC, especially the recurrent excitatory connectivity in CA3, allows the HPC–EC network to rapidly generate sequences of neuronal firing that correspond to both remembered and hypothetical orders of events. These bursts of activity can quickly propagate across the brain in both task-engaged and resting states, broadcasting sequences of events to downstream structures for decision-making computations or memory formation. Although reward-related firing patterns and neuromodulatory innervation are common to many brain areas, their union with sequence generation is unique to the HPC–EC system. The assignment of reward value [G] to these sequences may occur locally or through associative firing in downstream targets such as the prefrontal cortex (PFC) or striatum. With these features in place, the HPC–EC system is ideally poised to store and retrieve reward-related signals at multiple levels throughout the brain.

Reward in the hippocampal formation

Single cell representations of reward in the hippocampus.

A challenge to understanding how the hippocampus represents reward is that reward can be represented in multiple ways. For example, reward signals in dopaminergic neurons often relate to reward consumption or to reward-predicting cues. In the hippocampus, however, neurons that fire selectively during reward consumption have been elusive. This is partly because the prevalence of SWRs is enhanced by reward⁵⁷ in both CA3^57,58 and CA1^59-61. As SWRs excite neurons across the hippocampal circuit and occur during times of reward consumption, hippocampal spiking specific to reward consumption is difficult to dissociate from spiking during SWRs. Moreover, spatial tasks often lack sensory cues that directly predict reward, or involve linearized environments in which the animal’s movement direction cannot be dissociated from reward-prediction. Nevertheless, hippocampal reward signals have been observed at various behavioral timepoints with respect to when an animal receives reward. For the purposes of this Review, we subdivide these behavioural timepoints as follows: goal approach, goal arrival, time at the goal location (which may include reward consumption) and signals of reward history following reward consumption (Fig. 1a).

Fig. 1 ∣. Modulations of hippocampal-entorhinal activity at reward-related behavioural timepoints.

a ∣ Timepoints and associated behaviours surrounding reward acquisition. The magenta star indicates the goal location throughout the figure. Warmer colors indicate higher firing rates except where noted. In example spike plots, grey points indicate positions of the animal; coloured points indicate spikes. b ∣ Example hippocampal cell firing pattern during goal-approach to the east reward well in a 2D environment⁶⁴. c ∣ Place-field clustering in CA1 near three goal locations (white dots). Left: Place maps for an example cell before learning (pre), at the end of learning and during a probe session (post)³². Right: density of population place field centres (scale indicates proportion of cells). Because this overrepresentation of goals is characterized as a change in the time-averaged hippocampal activity over the course of a session, its specificity to goal approach versus goal arrival is not clear. d ∣ Example CA1 or subiculum cells showing reward-specific firing (right) or place firing (left). Red lines indicate reward locations. ‘A’ and ‘B’ denote distinct virtual environments. Each plot shows mean calcium activity across trials⁸⁹. e ∣ Left: continuous T-maze task, in which a rodent must choose between left and right goals that have different probabilities of reward. Right: Example CA1 cell showing increased firing rate based on reward history at the right goal (R+) compared with unrewarded times (R−) and left goals (L−, L+). Top: Spike raster for each outcome. Middle: Total occupancy of each spatial bin. Bottom: Average firing rate for each outcome⁷¹. f ∣ Goal approach activity in an example medial entorhinal cortex grid cell. Firing patterns in a 2D environment and continuous T-maze are shown. The cell exhibited higher firing rate on the centre stem on right choice trials¹¹⁴. g ∣ Increase in grid cell firing rates near a hidden goal zone (red box) when food is delivered inside the zone (right) vs. during random foraging for scattered food (left)¹¹⁷. h ∣ Shifting grid cell fields toward three reward locations (black dots) (similar format to part c). Red circle highlights the field that moves the most across learning¹¹⁸. Part b adapted from ref. [64], with permission. Part c adapted from ref. [32], with permission. Part d adapted from ref. [89], with permission. Part e adapted from ref. [71], with permission. Part f adapted from ref. [114], with permission. Part g adapted from ref. [117], with permission. Part h adapted from ref. [118], with permission.

One of the earliest demonstrations of reward-related hippocampal coding was the description of ‘goal-approach cells’, which increased their firing rate when rats moved toward odour cue and reward ports during an odour-discrimination task⁶². Running toward a known goal location was subsequently shown to induce place-specific firing along paths to goals that was distinct from random foraging in the same environment⁶³. This goal-approach activity occurs irrespective of the direction from which the animal approaches the goal^63,64 (Fig. 1b) and persists temporarily even after goals are removed⁶⁴. However, it remains difficult to distinguish goal-approach related firing from prospective firing, the modulation of the in-field firing rate of a place cell according to the animal’s future route^65-67. Prospective firing is stronger in CA1 than CA3⁶⁸, and CA1 place cells that fire prospectively additionally migrate their fields toward reward locations across behavioural trials⁶⁹. This CA1 firing activity is further modulated by the motivational state of the animal⁷⁰, as well as the probabilistic value [G] ⁷¹ and novelty of the goal⁷². An approach signal is encoded more explicitly in hippocampal cells of bats, via a vectoral representation of direction and distance to goals during flight⁷³. Thus, signals of goal approach, as well as the predictive value of locations that precede reward, may be layered onto a representation of the animal’s intended destination.

One of the most robustly reported effects of reward on the hippocampal map occurs at times encompassing both goal approach and goal arrival: place fields often cluster near reward locations, resulting in an overrepresentation of those locations by the neural population^32,74 (Fig. 1c). Reward-related place field clustering appears specific to hippocampal subregion CA1 compared with CA3³² and is observed across different types of environments and different rewards, including food^32,75,76, water^77-80, intracranial stimulation of the medial forebrain bundle^81,82 and an escape opportunity from the water maze⁷⁴. Similar to goal-approach signals⁷¹, place field clustering is influenced by the probability with which reward is delivered at known goal locations, with large, unexpected rewards yielding a greater overrepresentation⁸³. Overrepresentation of goals requires learning, during which existing place fields shift toward reward locations^{32,72,79,80,84} and non-place cells are newly recruited to represent the reward site^84,85. After learning, place cells near the reward location are selectively stabilized^84,85, and certain place cells will respond to multiple goals as opposed to a single goal location⁸⁶. The learning-dependent increase in place field density near goals suggests that the hippocampal map retains a prediction of where reward will be located.

Notably, place field clustering related to goal arrival is not observed in tasks that dissociate the location of the goal from reward delivery. Instead, the goal location elicits an increase in firing outside the primary field of a place cell, during the delay between goal arrival and reward delivery^87,88. This out-of-field firing occurs in both CA1 and CA3⁸⁸ and is probably distinct from the firing that occurs during SWRs^28,29,57, as the rate of SWRs tends to be lowest during delays in which an animal waits for reward^59,60,86,87. One potential explanation for the absence of place field clustering in such tasks is the lack of a clear predictive spatial relationship between the goal and the reward location, as the reward pellet in these studies was released randomly into the environment^87,88. This hypothesis points to the certainty of the spatial location of the reward as an important driver in reorganizing place fields. Another possibility is that the reward itself acts as the primary trigger for hippocampal place cell reorganization, in which case place field clustering would not be observed in tasks with reward locations that randomly vary.

Evidence of firing specific to times of reward consumption at the goal location has been limited. Recently however, two-photon imaging uncovered a small population of hippocampal neurons in CA1 and subiculum that seem to be specialized for encoding reward⁸⁹. These cells fired selectively at rewards regardless of the reward’s spatial location or the environmental context, distinguishing them from traditional place cells, which shift their fields but remain context-specific (Fig. 1d). The reward-specific activity was not restricted to the period of reward consumption, but instead spanned the period from goal arrival to departure⁸⁹, dissociating the reward site activity from spatially specific firing during immobility⁹⁰ and probably also from firing that occurs during SWRs²⁶. However, when multiple reward sites are present, even highly reward-specific cells tend to fire for only one reward site, suggesting these cells signal a combination of reward and position rather than exclusively reward⁹¹.

Evidence of reward history signals following reward consumption has also been limited, and may be more prominent in downstream areas that receive hippocampal input⁶⁰. As a notable exception, hippocampal cells modulate their firing activity after probabilistic reward delivery and after departure from the reward site, according to the reward outcome (Fig. 1e) and its probabilistic value^71,92. As with place field reorganization³², this value-modulated firing is observed primarily in CA1 but not CA3⁹². Similar to firing rate changes during reward approach, this reward-history-dependent firing reflects the animal’s choice⁷¹. Together, this collection of work indicates that the hippocampus processes reward-related signals across multiple behavioural epochs that surround navigation to goals.

Representations of reward across hippocampal subcircuits.

How reward-related dynamics vary across hippocampal subregions and heterogeneous subcircuits⁹³ remains an active area of investigation. Within CA1, cells in the deep pyramidal sublayer shift and restabilize their fields during goal-directed learning, whereas superficial cells maintain their spatial selectivity⁷⁷. Subpopulations of inhibitory CA1 interneurons are likewise differentially engaged in goal-directed behaviours, reorganizing their activity to coordinate with newly learned pyramidal cell patterns⁹⁴. For example, interneurons expressing vasoactive intestinal polypeptide (VIP) show modulation near learned goal locations that is crucial for reward-related shifts in the fields of pyramidal cells⁷⁸. Local circuit dynamics are therefore likely to critically shape CA1 representations of reward.

In the DG, two subpopulations of excitatory cells known to exhibit distinct spatial coding properties^95-97 — granule cells and mossy cells — have recently been shown to also exhibit distinct properties related to reward. DG granule cells respond to reward-predicting olfactory stimuli⁹⁸ and are also required for the reward-dependent enhancement of SWR reactivation in CA3, particularly in a working memory task⁵⁸. Mossy cells expressing the dopamine D2 receptor in the DG hilus [G], the region between the granule cell layers, respond to food cues and can suppress food intake when active⁹⁹. Together, these findings suggest that receipt of reward engages much of the dorsal hippocampal network.

An open question remains as to what degree dorsal and ventral HPC act as distinct circuits in navigation and memory processes. Historically, the dorsal HPC (dHPC) has been proposed to primarily encode spatial details, whereas the ventral hippocampus (vHPC) has been considered a centre for emotion and valence processing. These ideas are reinforced by denser innervation of the vHPC by catecholaminergic inputs, as well as stronger anatomical outputs from the vHPC to regions implicated in reward processing, such as the PFC and nucleus accumbens (NAc; for review, see refs ^100,101). In addition, vHPC place cells show modulation of firing around reward locations more often than dHPC cells do^102,103, and manipulations of vHPC projections to the NAc can drive or suppress reward-seeking behaviours^104-106. However, the behavioral effects of dHPC or vHPC inactivation on reward memory are mixed^107,108, and recent evidence has suggested a strong role for the dHPC in processing reward information. For example, reward increases the rate of SWRs only in the dHPC⁶⁰, and the dHPC seems to engage reward-related activity patterns in the NAc^60,109-111 more so than the vHPC does⁶⁰. It is worth noting, however, that the vHPC is incredibly heterogeneous in its cell types and targeting of downstream structures^93,112. The possibility of vHPC subcircuits dedicated to rewarding or aversive aspects of experience^100,101 remains to be investigated further.

Single-cell representations of reward in the medial entorhinal cortex.

As both a primary input and output of the HPC, the medial entorhinal cortex (MEC) is uniquely poised to supply fundamental components of the hippocampal code and read out the transformation of these components. These functions of the MEC have been classically considered in the context of physical navigation. Very few studies, particularly in rodents, have investigated MEC coding with respect to reward or higher cognitive functions. However, recent work has revealed that the navigational codes of MEC neurons are flexible depending on movement state¹¹³ and can reflect future destinations and past route origins^66,114,115 (Fig. 1f), thus implying the flexibility to encode goal approach and reward history. Moreover, the firing of MEC grid cells does not explicitly require physical movement, allowing for goal-related firing during immobility. Grid cells in non-human primates respond to changes in visuospatial attention¹¹⁶, and both hippocampal place cells and MEC grid cells in rats respond at discrete points along a manipulable auditory tone axis⁵⁰. Importantly, these responses are absent to passive tone playback without reward, but show weak tuning to constant tones as long as reward is subsequently provided⁵⁰. Taken together, these findings indicate that engagement in a rewarded task substantially contributes to MEC firing patterns, even in circumstances outside spatial navigation.

Recent studies have advanced this understanding, demonstrating changes in MEC firing related to goal arrival during spatial memory tasks^117,118. When reward is delivered in an unmarked zone of an open field, grid cells increase the firing rates of their fields near the reward zone (Fig. 1g), and non-grid spatial cells change the locations where they are active, likewise yielding a population increase in firing near the reward location¹¹⁷. When reward is delivered in multiple remembered locations of a cheeseboard maze [G], grid cells instead shift their firing fields toward the reward locations¹¹⁸ (Fig. 1h). Although many factors may contribute to the differences between results of these studies, one possibility is that the increased stereotypy of behavioural trajectories in the cheeseboard maze yields a shift much like the shift in place fields toward reward³², whereas approach to an unmarked zone from multiple directions yields a greater number of spikes near reward without changing the overall spatial distribution of firing fields (for example, as in ref. ⁸⁸). In addition, the holes in the cheeseboard maze provide location cues that themselves seem to distort the grid pattern seen in open arenas¹¹⁸. Despite these differences, both patterns of modulation could serve to amplify representations of specific locations in spatial reward memories.

Reward coding in the dopamine system

The reward-related modulation of hippocampal and entorhinal neurons naturally raises the question of where this reward information originates. Among other neuromodulatory inputs¹¹⁹, the midbrain dopaminergic system is a clear candidate for supplying reward signals to the navigational system. Here, we briefly review the current understanding of the dopaminergic system (for reviews, see refs ^120-124) to provide context for how these reward computations might play out in reward-directed navigation and spatial memory.

Dopamine neurons of the ventral tegmental area (VTA) are well known to signal reward prediction error [G] (RPE)^125-128, defined as the difference between expected and received reward¹²⁹. RPE serves as a fundamental teaching signal in a type of reinforcement learning [G] known as temporal difference learning [G] (TD-RL)¹³⁰, which can model the shift in dopamine neuron firing from the reward to a given predictor over time (Fig. 2a). With each outcome, the value of the predictor — how much it is ‘worth’ — is updated by RPE to improve the accuracy of future performance^120,122. This predictor can be a sensory stimulus, choice, action, location or environmental context. In addition to RPE, VTA neurons encode different levels of confidence about the reward outcome^131,132, representing a distribution of possible expected rewards as a population¹³³. Moreover, dopamine neurons encode various task parameters and decision variables not immediately evident from classical conditioning tasks^134,135.

Fig. 2 ∣. Dopaminergic signalling and innervation of the hippocampus.

a ∣ Reward prediction error (RPE) signalling. Dopaminergic neurons of the ventral tegmental area (VTA), which typically maintain a tonic firing rate, fire phasically in response to unexpected reward (positive RPE). As the reward becomes more predictable over learning, firing decreases for reward and increases for the reward-predictive cue, scaling with the degree of expectation and the value predicted¹²⁸. After extended learning, firing is suppressed if the expected reward is omitted (negative RPE) and increased if reward is larger than expected¹²⁶. b ∣ Cartoon of value or motivation signalling in the nucleus accumbens (NAc), similar between dopamine concentration and VTA axon activity (putative time course based on refs ^138,140,143). The example task here involves a movement to initiate the trial, such as a nosepoke, followed by a reward-predictive cue just before reward delivery, such as a feeder click. Phasic and ramping signals before reward delivery scale with recent rate of reward, which approximates value and increases motivation to perform the task. Note that RPE signals layer on top of this value signal, but here the reward delivered is as expected. c ∣ Distribution of VTA and locus coeruleus (LC) axons in the hippocampus. Darker yellow shading indicates greater LC axon density in CA3. d ∣ Summarized effects of dopaminergic input inactivation or activation on four hippocampal place cells (coloured blobs). Left: LC or VTA axon inhibition (colours as in part c), or dopamine antagonism in hippocampus, destabilizes place fields in sequential exposures to the same square environment. Right: LC or VTA axon activation with optogenetics (shown as a blue light) promotes the shift of place fields toward a goal location (magenta star). SLM, stratum lacunosum moleculare, SO, stratum oriens; SR, stratum radiatum; Sub, subiculum.

RPE signals are also reflected in dopamine release in downstream areas such as the NAc¹³⁶, where neurons encode rewards and the value of stimuli and actions that lead to reward (for review see ref. ¹³⁷). Concentrations of dopamine in the NAc ramp up as animals get closer to reward in time and space, reflecting a reward expectation signal^138-140. Additionally, however, there is strong evidence for roles of dopamine in movement and motivation [G] to work for future rewards^121,123. Dopamine release increases at the onset of reward-seeking actions^141,142 and is tightly correlated with both the value of each action and the vigour with which those actions are executed^138,143 (Fig. 2b). These release dynamics seem to be dissociable from VTA spiking¹⁴³ (but see ref. ¹⁴⁰), perhaps owing to local regulation of release via receptors on dopaminergic axon terminals in the NAc (for review see ref. ¹⁴⁴). In the context of navigation, dopaminergic RPE signals could facilitate learning in response to changes in reward presence or location, whereas value signals could help invigorate performance of learned routes. As others have eloquently described, however, value (how much reward is expected in each state) and RPE (change in value between adjacent states) are difficult to distinguish^121,140. Further work remains to reconcile these distinctions across tasks, as the relative contribution of value-specific and RPE-specific computations to dopaminergic activity may vary greatly by task and subcircuit¹²¹.

Although substantial work has focused on the VTA, the VTA is not the only source of dopamine in the brain. In addition to the substantia nigra pars compacta¹²³, dopamine is co-released with noradrenaline from neurons of the locus coeruleus (LC)¹⁴⁵, a noradrenergic centre commonly implicated in arousal, salience detection and cognitive flexibility (for review see refs ^146,147). Similar to VTA dopamine neurons, LC neurons respond to unexpected reward, rapidly shift their firing to reward predictors^148,149 and show firing correlated with movement effort^150,151. LC neurons also respond selectively to relevant stimuli when the rules of a task change^148,150, facilitating behavioural adaptation¹⁴⁶.

Finally, dopamine is not the sole arbiter of reward information in the brain, nor is it the only neurotransmitter released from dopaminergic neurons¹⁵². Multiple other neuromodulators have roles in reward processing and memory¹¹⁹, including opioids¹⁵³, noradrenaline^146,147, serotonin^154-156 and acetylcholine¹⁵⁷. In the remainder of this Review, we focus on the role of dopamine in the HPC–EC system given the more extensive research on this subject, but note that other neuromodulators could be equally fundamental to spatial memory and deserve further study.

Dopamine in plasticity and learning

Dopamine is known to modulate hippocampal and entorhinal synaptic plasticity (Box 1), and infusion of dopamine receptor antagonists into the HPC prevents rapid learning of novel locations and contexts^158-160. Yet, the principal source of dopaminergic input remains unclear. Both the VTA and the LC innervate the HPC^161,162 and EC¹⁶³. Although little is known about how these inputs affect functions of the EC, both VTA and LC inputs have been shown to regulate hippocampal memory retention^158,159,164. VTA axons to the dHPC are sparse (compared with the dense innervation of the vHPC) and primarily target the CA1 and CA2 pyramidal layer and stratum oriens [G] ^161,165,166 (Fig. 2c). LC axon terminals are prominent in the dHPC and are uniformly distributed across the hippocampal laminae, most densely innervating CA3^{158,159,162,164} (Fig. 2c). Complementing VTA dopamine, dopamine co-released with noradrenaline from LC axons¹⁴⁵ provides a large fraction of hippocampal dopaminergic tone¹⁶⁴. Moreover, dopamine mediates the memory changes observed after LC manipulations, as these effects are blocked by dopamine receptor antagonists but not noradrenaline receptor antagonists^158,159,164. The relative contribution of VTA and LC dopamine to hippocampal plasticity is still unclear, and may vary based on cellular target or behavioural demand ¹⁶⁷.

Box 1: Hippocampal and entorhinal dopamine-mediated plasticity.

The hippocampus expresses both D1-type dopamine receptors (including D1 and D5 receptors) and D2-type receptors (including D2 and D4 receptors), with CA1 primarily expressing D1-type receptors²²⁶. In slice physiology experiments, dopamine and D1-type receptor agonists amplify long-term potentiation at CA3 to CA1 synapses^165,241-243 without increasing the excitability of CA1 neurons²⁴¹, suggesting that dopamine works together with glutamatergic inputs to augment plasticity. These plasticity effects depend on the temporal dynamics of dopamine transmission. Tonic activation of ventral tegmental area (VTA) inputs to the hippocampus depresses CA3–CA1 synapses by recruiting local interneurons¹⁶⁵. Phasic activation instead enhances CA3–CA1 excitation¹⁶⁵, suggesting that phasic reward prediction error (RPE)-like activity in the VTA may facilitate new associations in the hippocampus. Intriguingly, VTA input seems not to affect entorhinal cortex (EC) to CA1 synapses, even though dopamine depresses EC–CA1 synapses^165,241. These results indicate that dopaminergic afferents differentially affect distinct hippocampal pathways.

The effect of dopamine on synaptic transmission in the EC is both concentration-dependent and lamina-dependent. In the lateral EC, low concentrations of dopamine reduce excitability in layers III²⁴⁴ and V²⁴⁵ but increase excitation in layer II via D1-type receptors^246,247. High concentrations of dopamine increase excitability in layer V²⁴⁸ and reduce excitation in layer II via D2-type receptors^246,249. This suppression in layer II is hypothesized to reduce the strength of sensory inputs during times of high dopamine release, boosting the signal-to-noise ratio of the most relevant inputs or preventing competition with ongoing memory processes^244,249. In the medial EC, high dopamine concentrations reduce the excitability of principal cells in layers II²⁵⁰ and III^251,252, primarily through the D2-type receptor. The suppressive effects of dopamine may be facilitated in part by a dopamine-evoked excitation of layer III interneurons²⁵³, resulting in increased inhibition onto principal cells in both layer III²⁵⁴ and layer II²⁵³. Dopamine is therefore well positioned to moderate the recurrent activity between excitatory and inhibitory MEC neurons that is thought to give rise to grid cell firing^5,255,256. However, the functional consequences of EC dopamine for spatial navigation and memory remain unclear.

Both VTA and LC dopaminergic inputs to the HPC have been implicated in shaping and stabilizing spatial representations. Stability of the hippocampal map is facilitated by VTA axon stimulation¹⁶⁸ and reduced by inactivation of the VTA¹⁶⁶ and LC¹⁵⁹ as well as by dopamine receptor antagonists^31,169 (Fig. 2d). Complementing a role in stability, dopaminergic transmission is required to flexibly update the hippocampal map to reflect information most relevant to the current task¹⁶⁹. Blockade of hippocampal dopamine receptors during learning impairs memory of reward–location associations¹⁷⁰ and prevents animals from learning to find reward relative to a new set of sensory cues¹⁶⁹, whereas activation of VTA input enhances goal location memory¹⁶⁸. Supporting this memory function, dopaminergic input is involved in shifting place cell representations toward learned reward locations (Fig. 2d). Optogenetic activation of VTA axons can shift the firing of place cells toward the location of stimulation, whereas inhibition of these axons tends to shift place fields away⁷⁵. Recently, LC axon activity in the HPC was shown to signal upcoming reward when the reward location moved⁷⁹, in line with previous reports that LC neurons signal unexpected reward contingencies^148,171. Activating this LC projection can reorganize place fields around a fixed reward location, but does not cause reorganization at unrewarded locations or when the reward location is unpredictable, as with random foraging⁷⁹. Consistent with findings that place field reorganization can occur without VTA input¹⁶⁶, this suggests that dopaminergic input alone is not sufficient to drive the hippocampal representation. Together, these studies point to multiple sources of dopamine converging with spatial learning demands to shape hippocampal representations.

Hippocampal sequences in reward memory

How hippocampal activity is influenced by neuromodulation remains to be explored further. However, the HPC can generate sequences that support reward-driven navigation regardless of how reward is represented locally. Hippocampal network oscillations organize sequential activity on multiple timescales (for reviews, see refs ^172,173), including theta sequences during movement and replay events during immobility. Such sequential activity is thought to support the brain’s ability to store reward-related memories and retrieve them for decision-making.

Theta sequences.

As an animal moves through space, hippocampal cells fire at progressively earlier phases of the theta rhythm, a phenomenon called theta phase precession^23,24. Spikes from cells with overlapping place fields are nested into a theta sequence^25,174,175, such that the neural representation of space ‘sweeps’ from behind to ahead of the animal within a theta cycle (Fig. 3a). Theta sequences have been observed in spatial and non-spatial tasks⁵¹ and are thought to provide a mechanism for deliberation during navigation.

Fig. 3 ∣. Hippocampal theta sequences and replay.

a ∣ A rodent running on a linear environment engages theta sequences. An example theta trace (local field potential filtered at 5–11 Hz) is shown below the track. Place cells with overlapping fields spanning just behind to just ahead of the animal’s position spike sequentially within each theta cycle (spikes are shown as vertical ticks, theta cycles separated by dashed vertical lines). Early phases of theta (0 to pi radians) contain spikes corresponding to past and present, whereas late phases (pi to 2 pi) contain more spikes corresponding to future positions. b ∣ A ‘W-maze’ alternation task (for example as in refs ^178,187,188) illustrating right and left choices represented as single spikes of place cells (green and yellow fields) on alternating theta cycles¹⁷⁸. Note that spikes occur on the late phases of opposite theta cycles (same example theta trace as in part a). On the W-maze, the animal is rewarded for visiting the opposite side arm from the previously visited arm when coming from the centre. Thus theta alternation could act as a mode of deliberation, with retrieval of information relevant to future experience taking place in the second half of the cycle¹⁸². c ∣ In periods of immobility such as during food consumption, sequences of places cells replay during sharp-wave ripples (SWRs). The same example SWR (local field potential filtered at 150–250 Hz) is shown to illustrate both forward and reverse replay events. d ∣ In the same W-maze task shown in part b, a rodent exhibits forward replay of both alternate trajectories while immobile, before beginning a run. Separate replay events (same SWR used for illustration purposes) are shown, displaying replay of leftward and rightward place cell sequences, putatively allowing the animal to evaluate possible future outcomes¹⁸⁸.

Hippocampal theta yields two patterns before navigational decisions. First, when the animal pauses at a maze junction and looks side to side, theta sequences sweep ahead of the animal in one direction and then in the other¹⁷⁶, putatively helping the animal evaluate each path before making a decision. These sequences extend the future locations represented, known as the ‘look ahead distance’, predicting the chosen goal¹⁷⁷. Second, during movement before a maze junction, future choices are represented on alternating theta cycles¹⁷⁸ (Fig. 3b). Future-signalling spikes occur on late phases of theta, whereas spikes signalling the current¹⁷⁸ or past¹⁷⁹ location occur on the early phases. On the same theta time-scale, neural representations can also alternate between current and previous goal configurations in MEC¹¹⁸. MEC cells that fire on alternate theta cycles also tend to have distinct directional preferences¹⁸⁰, which could help represent bifurcating navigational choices. Moreover, an ability to look ahead along future paths to goals has been proposed for MEC grid cells¹⁸¹, which tend to represent locations just in front of the animal¹⁰. Together, these phenomena imply that the HPC–EC system has access to a snapshot of the present, the immediate past and a hypothetical future scenario, all within a time window of approximately 125 ms. Such compression of experience would allow rapid predictions about upcoming states dependent on recent experience¹⁸². An important avenue of future study is to understand how theta sequences that alternate between future choices engage neurons representing the value of each choice.

Sharp-wave ripples and replay.

During pauses in movement and during sleep, SWRs²⁶ encompass approximately 50–200 ms-long¹⁸³ bursts of hippocampal spiking, ‘replaying’ sequences of place cells that recapitulate paths taken through space (for reviews, see refs ^184,185). SWRs during immobility in awake behaviour have been causally linked to accurate memory retrieval and learning^37,38. Recent work further demonstrated that replay of a specific environment during sleep is required to subsequently recall goal locations in that environment³⁹.

The order and content of replay at different moments in goal-directed behaviour may depend on task demands. Replay events occur in both forward and reverse directions relative to the order of neuronal firing during the original experience^28,29 (Fig. 3c). Forward replay occurs more often before goal-directed trajectories and during pauses at decision points^{29,72,186,187}, suggesting a role in retrieving past experience to inform current decisions (Fig. 3d). Reverse replay occurs primarily following receipt of reward after the completion of a path^29,59,72 and when working memory is required⁷², suggesting a role in storing associations learned from recent experience. Consistent with a role in decision-making, greater amounts of replay of future alternatives predicts better performance¹⁸⁸ and the replayed sequence can predict specific routes taken to goals¹⁸⁶. However, pre-decision replay may instead reflect how often and how far in the past the replayed trajectory was rewarded, rather than reflect planning per se¹⁸⁹. Consistent with a role for memory maintenance, in tasks with divergent motivational demands (such as a choice between food and water), replay corresponds to the unchosen option¹⁹⁰. Collectively, these studies demonstrate a dynamic interplay between the requirements of goal-oriented tasks and the structure of hippocampal replay.

After receipt of reward, an increased rate of SWRs⁵⁷ in dHPC⁶⁰ coincides with reward-evoked dopamine release, which could cement associations between recently taken paths and their reward outcomes²⁸. Consistent with this hypothesis, larger reward sizes augment the rate of reverse replay⁵⁹, and reward enhances how closely forward replay sequences match the experienced sequence⁶¹. These studies suggest that replay aligns with a dopamine signal (Box 2) to link place cells along rewarded paths with the outcome. Replay has also been recently suggested to facilitate the inferred association of reward-predicting cues and outcomes¹⁹¹, suggesting that the ability of SWRs to link reward with preceding events generalizes to nonspatial tasks.

Box 2 ∣. Interactions between dopamine, plasticity and replay.

Replay events rely on plasticity during experience to accurately replicate past episodes²⁵⁷. The more that place cells overlap in their fields and fire together during movement, the more they reactivate in the correct order during subsequent sharp-wave ripples (SWRs)²⁵⁸. Accordingly, the shifting of place fields toward reward during learning requires plasticity and increases the co-firing of neurons near reward locations, allowing for cells representing reward locations to be reactivated more often during replay³². Place field clustering may therefore increase the granularity of spatial reward memories that are consolidated through SWRs. Dopamine is likely to play a substantial role in this plasticity, as stimulation of axons projecting from the ventral tegmental area to the hippocampus during learning supports the shifting of place fields⁷⁵ and enhances the fidelity of replayed place cell ensembles during sleep¹⁶⁸. In turn, the reward-enhanced reactivation of place field maps during sleep supports the subsequent expression of those maps and the animal’s memory of the task^32,168. Whether dopamine further strengthens hippocampal synapses during reactivation in sleep is unknown. However, this possibility is suggested by evidence that stimulation of the medial forebrain bundle coupled to place cell spikes during sleep (many of which occur during SWRs) causes a behavioural preference for the field of that place cell during subsequent exploration of the environment³³. Dopamine release at the time of wake replay may have an additional role in solidifying the reactivated place cell maps by strengthening their synaptic connections. Consistent with this possibility, SWRs during wake at reward locations are required for stabilizing place fields over learning²⁵⁹. The effects of dopamine on hippocampal plasticity are well suited to influence what information gets stored during experience and to increase the probability that this information will be consolidated into a stable representation that guides behaviour.

The characteristics of MEC replay are not as well understood as in the HPC. Coherent replay between place and grid cells has been sparsely observed in MEC layers V/VI¹⁹² (but see ref. ¹⁹³). By contrast, superficial MEC layers exhibit replay in both the forward and reverse directions independently of the HPC¹¹⁴. At the same time, there is evidence that coordinated activation of MEC layer III is required for extended SWRs in CA1¹⁹⁴, which have been shown to contain longer replay sequences¹⁸³. These findings suggest that MEC activity may help propagate hippocampal replay sequences over longer representational distances.

Extrahippocampal interactions

Hippocampal interactions with downstream structures enable the navigational code to be combined with additional behaviourally relevant information. In this section, we focus on studies using simultaneous recordings across brain regions to examine how sequences broadcast by the dHPC, where spatial specificity is highest¹⁰⁰, are coordinated with neural activity patterns related to goal-directed behaviour in downstream regions (for reviews, see refs ^195,196).

Neocortex.

Hippocampal SWRs activate widespread neocortical regions¹⁹⁷, including neurons in the primary visual and auditory cortices during sleep^198,199. These cortices may in turn provide an upstream input to SWRs^199,200, as sound-responsive auditory cells can bias the HPC to replay spatial information associated with a sound cue^199,201. Cortical cells encoding reward-predicting cues could therefore influence the HPC to reactivate cue representations with spatial sequences and reward outcomes, forming a complete memory of a rewarding experience.

Hippocampal sequences also strongly engage the PFC, a set of cortical regions implicated in decision-making²⁰². PFC cells show goal-related coding in spatial tasks^203-205, spiking near remembered goal locations even when goals are dissociated from reward delivery²⁰⁶. In addition, PFC cells encode both spatial information and behaviourally relevant similarities across spatial trajectories, such as junctions or endpoints^203-205,207. This activity can be thought of as representing discrete task states²⁰⁸ along trajectories to goals, which may be important for generalizing task knowledge across similar experiences. During periods of working memory preceding spatial choices, PFC cells align their spiking to the hippocampal theta rhythm^203,209,210, exhibiting theta phase precession²¹¹. This theta coordination is enhanced by dopamine²¹⁰, perhaps reflecting reward-predictive computations preceding choice points. Recent work supports this possibility: HPC and PFC theta sequences concurrently represent spatial trajectories and goals^212,213, with the goal location represented in the PFC predicting upcoming spatial choices²¹⁴. These behaviourally relevant PFC sequences reactivate during SWRs in both sleep and wake^205,215. Coordinated HPC–PFC replay during wake may help associate spatial locations to the more generalized states represented by the PFC and retrieve these associations for decision-making²⁰⁵. Consistent with this hypothesized function, cohesive replay across HPC–PFC ensembles predicts an animal’s upcoming or recently traversed path to a greater degree than hippocampal replay alone²¹⁶.

Subcortical structures

Both theta oscillations and SWRs have been shown to engage subcortical structures implicated in reward and value processing. In the VTA, reward-encoding neurons spike during SWRs in sleep, coordinated with hippocampal neurons representing reward location²¹⁷. These VTA neurons are preferentially reactivated during replay sequences that move away from the animal’s current position²¹⁷, consistent with the idea that replay propagates reward value information backwards across locations²¹⁸.

In the NAc, neurons that fire at reward sites during behaviour are likewise reactivated following hippocampal replay in sleep²¹⁹. During awake immobility, however, dHPC SWRs instead reactivate NAc neurons that encode relative distances along spatial paths to goals⁶⁰. Similar to neurons in the PFC (for example, see ref. ²⁰⁷), these NAc neurons putatively encode generalized states that may be couple to goal-directed actions, such as trajectory initiation. NAc neurons reactivated during SWRs additionally exhibit a reward history signal, firing more along spatial trajectories after the animal has received a reward⁶⁰. The firing of these neurons may be modulated by local dopamine release that tracks reward history¹⁴³. Many task-responsive NAc neurons also fire according to hippocampal theta phase^60,220,221 or show theta phase precession¹¹¹. This theta coordination of HPC–NAc ensembles is well suited to associate spatial and reward information across the circuit during spatial exploration. Rewarded contexts enhance this theta coupling^109,110, and HPC–NAc neuron pairs active together during a rewarded experience are reactivated together in post-experience replay^60,109,219. Upon re-exposure to a rewarded context, direct dorsal CA1 innervation of NAc neurons, especially fast-spiking interneurons, is needed to organize and reinstate the spiking of NAc ensembles associated with the reward memory¹¹⁰. These ensembles may be recruited at times of decision-making, as hippocampal theta sequences at choice points recruit the spiking of reward-related NAc neurons²²², potentially facilitating predictions of upcoming reward given each spatial choice.

Additional subcortical circuits are candidates for linking hippocampal sequences to reward information. SWRs engage neurons in the lateral septum²²³ and basolateral amygdala (BLA)²²⁴, preferentially recruiting a subpopulation of lateral septal neurons that respond to reward and reward-predicting cues²²³. Of note, the BLA, lateral septum and NAc each comprise a potential conduit that translates hippocampal inputs into outputs to the VTA²²⁵, facilitating a loop between VTA and the HPC–EC system²²⁶. How computations are transformed at each step of this loop remains to be explored. The collective evidence suggests that hippocampal sequences join representations of space to generalized representations of task states and reward outcomes.

A model for reward and navigation

Recent computational work provides a model for the intersection of spatial navigation and reward prediction. This work posits that the HPC–EC system encodes a ‘successor representation’ (SR)⁵⁴, which quantifies the extent to which the current state predicts that the animal will occupy other states in the environment, discounted by how far in the future the other states are^54,227,228 (Fig. 4a,b). The SR model is a generalized form of temporal difference reinforcement learning (TD-RL), in that it uses prediction errors about the occupancy of states to update transition probabilities between states, just as TD-RL uses errors in predicted reward. Under the SR model, the value function [G] learned in TD-RL is decomposed into the SR matrix of predictive states, multiplied by the reward expected in each state²²⁸ (Fig. 4c). This factorization allows changes in reward in any given state to easily propagate to the entire series of connected states. Further, the transition probabilities between states can be learned even in the absence of reward (such as exploring a spatial environment before receiving food in it )²²⁷. The SR framework can therefore model both the dopaminergic system and the HPC–EC navigational system.

Fig. 4 ∣. Hypothesized interactions between brain systems in navigating to reward.

a ∣ A sequence of hippocampal place fields interpreted as a sequence of 5 states (s₁–s₅) that discretize forward movement on a linear track, with expected reward in each state (r₁–r₅). b ∣ The successor representation (SR) matrix for the 5 states depicted in part a. Hypothetical transition probabilities arise from the assumption that the hippocampal representation is mostly unidirectional on the linear track (that is, states in this sequence predict past states with very low probability and future states with high probability that decays with increased distance). Purple arrows indicate the firing field for hippocampus (HPC) cell 1 (column 1) and its SR (row 1). c ∣ Left to right, first: The successor representation vector M(s_i,:) for all states given trajectories initiated in state i for i = [1:5] (rows of the SR matrix). Darker colours indicate higher predicted occupancy. Second: The firing rates of each hippocampal cell in 5 spatial bins (that is, the 5 states) derived from the columns of the SR matrix. Darker colours indicate higher firing rates. Third: Each hippocampal cell is hypothetically coupled with a reward function that provides the expected reward in each state, here shown as a ramp of dopamine release peaking at the reward location. This coupling could occur via dopaminergic innervation of the HPC, or via spike coupling of HPC cells with nucleus accumbens (NAc) neurons, for example, which receive ramping dopamine. Fourth: The SR and reward are multiplied to estimate the value function for each state (combined colours). d ∣ In this simplified hypothesis, dopaminergic and other neuromodulatory systems convey reward prediction information to the HPC–entorhinal cortex (EC) system, which helps assign these values to discrete states that compose an experience. ‘States’ here are synonymous with spatial representations of the HPC-EC. State representations are sent to downstream areas (yellow), which layer additional information onto these states, such as task requirements and sensory features. No reciprocal arrow is shown for the basal ganglia because there is no known direct return projection, but the basal ganglia (including the NAc) help to use state values for action invigoration. The HPC–EC, frontal cortices and basal ganglia each project back to the dopaminergic system directly or indirectly, putatively providing updates about predicted outcomes and value changes to individual states. Interactions in this network contribute to memory storage, decision-making and action generation.

The SR model is compatible with multiple experimental findings in describing the HPC–EC system^53,54. Hippocampal place cells are proposed to encode SR as a rate code, reaching their peak firing rate when the animal is physically located in the state that is best predicted by a given cell (Fig. 4c). The SR model accurately predicts a higher density of place fields near goals⁷⁴ and may explain prospective firing rate changes based on future destination^65,66, as states on the centre stem of an alternation task could be dissociable based on how much they predict states just past the junction on a given trial. The EC is proposed to encode a low-dimensional readout of the SR, representing the underlying correlation structure of the relationships between states^54,229. Importantly, the SR develops through learning: the more a state is occupied over experience, the more it is predicted²³⁰. This means that, in open environments where reward is randomly scattered, the structure of the SR is represented as an evenly spaced grid, because the animal does not repeatedly visit any particular trajectory (that is, any particular sets of transitions between states)²³⁰. If certain trajectories are navigated many times, for example when running between fixed goal locations, increased occupancy of states along the trajectory shifts grid fields closer to the goals^118,230. If certain states are occupied more but not via repeated trajectories, such as when navigating to a hidden goal in the open field, transition probabilities only increase locally and may be reflected as an increased firing rate of grid cell fields¹¹⁷.

SR theory is useful in conceptualizing how state transitions (that is, the order of episodic events) learned in the HPC–EC could be used to make value predictions, consistent with a role for the HPC in value-based decision making^231-234. Replay has been proposed to support the assignment of value to sequences of states²¹⁸. Under the ‘prioritized replay’ proposal, forward replays are prioritized at moments of decision-making to compute the value of states along upcoming possible routes, increasing the animal’s probability of making a correct choice²¹⁸. Theta sequences could also perform this function by serially sampling alternative trajectories and estimating the value of each one^196,235. Once reward is received, reverse replay is prioritized to propagate any positive RPE backwards to update the value of preceding states²¹⁸. Alternatively, estimating values and storing newly updated estimates may be simultaneous processes that occur in both types of replay^184,185.

How value gets assigned to each state in the SR at the neural level remains an open question. If hippocampal cells solely represent SRs, they would need to combine their firing with an external reward prediction signal to compute value. Dopaminergic release in the NAc and the spiking of reward-related NAc or VTA neurons coincident with HPC–EC sequences may link states to value representations across the neural circuit. In this scenario, sequences broadcast by the HPC would be evaluated by basal ganglia circuitry in the process of selecting an action to achieve the desired outcome^196,236. To apply learned state–reward predictions in similar contexts, the spiking of PFC neurons may help to generalize value assignments from individual spatial states to task states that share similar features^53,208,230. Alternatively, reward-related firing patterns in the HPC–EC suggest that value could also be computed locally, perhaps via dopaminergic modulation. For example, neuromodulatory inputs could potentiate the synapses between CA3 and CA1 neurons to amplify the CA1 spike rate for higher-valued sequences²³⁷. In either case, the HPC–EC serves as an interface in a network of brain structures to link individual successor states (place), value and task states to facilitate the learning and performance of goal-directed actions (Fig. 4d).

Conclusion

Here, we hypothesize that a key role of the HPC–EC system in rewarded navigation is to generate sequences of events to link the beginning of an experience with its outcome. We propose that neuromodulatory inputs may sculpt HPC–EC representations to either provide local value information or shift the distribution of states, ‘weighting’ rewarded regions of space more heavily than unrewarded regions. Subsequently, hippocampal sequences may broadcast these states in a compressed manner to downstream regions, to be linked with episodic details in memory formation, generalized across task knowledge or utilized for action generation.

Framing HPC–EC activity as encoding states allows the known navigational codes of this system to be applied more flexibly in non-spatial domains²³⁵. Yet, many open questions remain (Box 3). Moreover, reward is intrinsically difficult to disentangle from studies of HPC–EC physiology, as laboratory animals are unlikely to traverse spatial environments without receiving reward. This necessity of motivation for behaviour may help to explain why signals of reward and value seem redundant throughout the brain. In this way, reward is reminiscent of other signals critical for survival, such as thirst and movement, which modulate activity in nearly all brain areas^238-240. Despite these challenges to understanding reward, the tractable nature of navigational codes makes the HPC–EC circuit a candidate model system to learn how reward drives both adaptive and maladaptive memory-dependent behaviours.

Box 3 ∣. Open questions and future directions.

Does hippocampal reward coding drive reward-related changes in the medial entorhinal cortex (MEC), or vice versa? Why have reward coding in both structures? It remains unknown whether reward-related changes in MEC grid cell activity indeed reflect a low-dimensional readout of hippocampal reorganization around reward sites. To better understand what might be unique versus universal across the hippocampus (HPC) and MEC in terms of reward coding, future work could use simultaneous recordings across these areas or develop computational models in the context of complex, goal-oriented tasks.

How do other neuromodulators, such as noradrenaline, serotonin and acetylcholine, influence reward-related changes in the HPC and EC? Do these neuromodulators work in concert with dopamine or exert independent effects on hippocampal plasticity and reward coding? Are different neuromodulators acting in the entorhinal cortex (EC) versus in the HPC, or perhaps engaged at different stages of learning? With the development of new receptor-based fluorescent sensors for these neuromodulators²⁶⁰, some of these questions can now be addressed with optical imaging techniques.

Are there dedicated subcircuits for reward processing in the ventral (or dorsal) HPC, distinct from those dedicated to coding for aversive experiences or environmental context? In what circumstances are the dorsal or ventral HPC most engaged in processing rewarding experiences?

To what degree do task demands drive reward representations in the HPC or MEC? Do the firing patterns that we describe here truly reflect ‘reward’ per se, or do they instead reflect task engagement more broadly? Although some studies have tried to address this issue with manipulations of reward size and probability^{71,83,88,92,234}, additional clarity could be gained by perturbing outcome valence and value across a range of tasks.

Does the HPC compute a successor representation alone (state occupancy), or additionally compute its own value function? The assignment of reward value to hippocampal sequences may occur locally, by modifying the spike rate, timing or synaptic strength of hippocampal neurons in the sequence, or it may occur through associated firing in downstream targets such as the prefrontal cortex or striatum. Further experimental and computational work is needed to understand whether HPC–EC firing patterns compute value locally or merely reflect state occupancy.

Glossary:

Grid cell

An entorhinal cortical cell that fires in triangularly spaced fields that tile the whole environment.

Place cell

A hippocampal cell that fires maximally in one or few discrete regions of space (its ‘place field’).

Theta sequences

Sequential spikes of multiple place cells that together encode a trajectory through space, ordered by the theta phase of each spike. Theta sequences occur during times of high theta power, typically during movement.

Sharp-wave ripples

(SWRs). High-frequency oscillations (about 150–250 Hz) in the local field potential (LFP) coincident with a sharp, low-frequency deflection in the LFP. These events reflect the coincident activation of many hippocampal cells in a short time period (about 50–200 ms) and typically occur during immobility.

Replay events

Sequential spikes of multiple place cells that typically occur locked to SWRs during immobility, and that together encode a trajectory through space. In high-fidelity replay events, place cells in the sequence are reactivated according to the order in which they fired during a previous run.

State

A snapshot of a situation, discretizing a longer continuous process that comprises an experience. As an analogy, if this snapshot were taken by a camera, the duration of the state would be the exposure time and would vary depending on the situation (for example, how dark it is outside).

Value

How much an outcome, or state that predicts an outcome, is ‘worth’. This worth includes the amount and likelihood of reward predicted.

Probabilistic value

The probability that reward will be delivered given a certain choice. Even if a choice is correct according to the task, changing the probability of reward delivery can modulate the value of preceding states.

Cheeseboard maze

A spatial task in which rewards are hidden in a subset of holes or wells in the floor of an open arena. This task is used as a spatial memory paradigm because the animal has to remember which wells are rewarded based on their position in the environment, and the reward locations can change across sessions or days.

Reward prediction error

The difference between reward received and reward expected. Positive RPEs indicate larger rewards than expected (including reward when none was expected), whereas negative RPEs indicate smaller rewards than expected (including the absence of reward when it was expected).

Reinforcement learning

A set of computational theories, often used for machine learning, to describe how states and actions are assigned values that inform how an agent can receive maximal reward.

Temporal difference learning

A type of reinforcement learning in which values are updated by a reward prediction error between temporally adjacent states, such that states preceding the reward receive a ‘cached’ value prediction

Motivation

The impetus an agent feels to perform reward-seeking actions. Value is used to inform motivation and invigorate reward-seeking actions (make them faster and more efficient).

Value function

A function of adjacent states, or states paired with actions, that computes the expected future reward in each state.