Integrating hippocampus and striatum in decision-making

1 Summary

Learning and memory and navigation literatures emphasize interactions between multiple memory systems: a flexible, planning-based system and a rigid, cached-value system. This has profound implications for decision-making. Recent conceptualizations of flexi-ble decision-making employ prospection and projection arising from a network involving the hippocampus. Recent recordings from rodent hippocampus in decision-making situations have found transient forward-shifted representations. Evaluation of that prediction and subsequent action-selection likely occurs downstream (e.g. in orbitofrontal cortex, in ventral and dorsomedial striatum). Classically, striatum has been identified as a critical component of the less-flexible, incremental system. Current evidence, however, suggests that striatum is involved in both flexible and stimulus-response decision-making, with dorsolateral striatum involved in stimulus-response strategies and ventral and dorsomedial striatum involved in goal-directed strategies.

2 Introduction

Theoretical perspectives on decision-making processes have traditionally treated decision-making from a single system perspective. Within many models of reinforcement learning, decision-making is viewed as learning a mapping of situations (world-states) s to actions a that maximizes reward by calculating the expected value E(V(s, a)) [1].

In contrast, the learning and memory literature has emphasized the interaction of multiple memory systems [2–4]. In the memory literature, these differences are distinguished between declarative and procedural systems [3]. Declarative information is broadly accessible in a range of circumstances and based on a variety of retrieval cues [5] whereas procedural information is narrowly bound and accessible only in rigid and specific sequences [6]. In the navigation literature, these differences are distinguished between cognitive map and stimulus-response (route-based) systems [7, 8]. The cognitive map system confers animals with the ability to plan trajectories within their environment and flexibly integrate new information (such as novel stimuli and reward). Stimulus-response systems provide the basis for simpler, non-integrated navigation functions such as stimulus recognition and approach.

Although many computational models of these systems have been presented, confirmation of specific mechanisms within the neurophysiology has been limited [8, 9]. Even without specific mechanisms, there has been a convergence in the proposed anatomical substrates underlying each component. These theories generally distinguish between a flexible planning system critically dependent on intact hippocampal function and a more rigid, more efficient system critically dependent on intact striatal function [3, 4, 8, 10]. As we review below, recent data now suggest roles for hippocampus in self-projection in planning strategies and striatal roles for evaluation and action-selection in both planning and cache-based strategies.

3 The hippocampus and decision making

Originally proposed in contrast to stimulus-response approaches to learning, the cognitive map described learning in terms of ubiquitous observation rather than reward-driven association, and decision-making in terms of goals and expectations rather than drives and responses [11]. Animals were hypothesized to store associations between stimuli and to plan actions based on expectations arising from those associations. But without an available computational language, identifying the mechanisms underlying decision-making in the face of explicit expectations has been difficult [11–13].

Recent considerations of goal-directed decision-making have emphasized explicit search processes, which predict and evaluate potential future situations [14–16, recalling early artificial-intelligence theories [17]]. These hypotheses parallel recent considerations of active memory that have emphasized the functional potential for projection of the conceptual self beyond the current situation [5, 16, 18]. Behaviorally, active memory processing within expectation theory predicts that animals will pause at decision points as they mentally explore available possibilities.

A key experimental observation of expectation-based theory of decision-making was that animals paused at choice-points and oriented towards potential options. This behavior, termed vicarious trial and error (VTE), occurs during early learning following initial exposure to a task, but before automation [11, 19]. (See Fig. 2 and supplemental movie.) Animals often show a sudden, non-linear increase in performance [20], correlated with VTE [19]. Furthermore, VTE behavior is related to hippocampal integrity — rats with hippocampal lesions display reduced VTE and choice performance [21]. VTE behavior is also correlated with cytochrome oxidase activity in hippocampus on hippocampal-dependent tasks [22]. Within the expectation-based perspective on decision-making, VTE is hypothesized to represent the behavioral residual of an animal considering different options as it plans a course of action.

Fig 2. Vicarious trial and error.

On the first example trial, the rat looks left and then goes right. On the second example trial, the rat looks right and then goes left. On the third example trial, the rat looks right, starts left, but then goes right. On the fourth example trial, the rat looks left before starting the journey down the central track and then does not pause at the actual choice point, suggesting the moment of decision may have been made before the journey down the central track in this last example. A video of a real rat running these four examples is shown in the supplemental movie.

To be useful for decision-making, planning requires several component processes: representation of the current situation (a classification process), prediction of the outcome of an action (representation of a future situation), evaluation of that predicted situation, and action-selection. While hippocampal place cell activity provides the basis for representing the current position (situation), recent reviews by Buckner and Carroll [5] and Schacter et al. [18] have suggested that self-projection processes require hippocampal function.

Recent data that patients with hippocampal leions are impaired at imagining potential futures supports this hypothesis. In a study by Hassabis et al. [23], participants were asked to imagine and describe common scenes (e.g. a market or a beach). Despite similar reports of task difficulty, the descriptions of participants with hippocampal damage displayed reduced content and profound deficits in spatial coherence compared to controls. These findings suggest that the hippocampal system plays a fundamental role in coherently representing imagined potential future situations.

The strong spatial correlates of rodent hippocampal pyramidal cells on spatial tasks (place fields, [7, 8]), and the strong spatial effects of hippocampal manipulations in rodents [7, 8] led O’Keefe and Nadel to suggest that the hippocampus could provide a neural substrate for cognitive maps [7].¹ A number of studies have shown that changes in the place cell mapping are associated with errors in behavior, implying a functional use of the spatial representation [26–28]. How these maps are used, particularly for planning, remains an open question.

In our recent work on the fast dynamics of hippocampal representations, we found that spatial representations transiently shifted ahead of the animal at a ⊤ choice point [29]. Forward shifted hippocampal representations coherently moved ahead of the animal into one ⊤ arm and then the other (see Fig. 3). These sweeps tended to occur during VTE-like behaviors, when animals paused and looked around at the choice-point. And in much the same way that VTE is task- and experience-dependent [19, 21, 22], the behavior of forward shifted representations were task and experience dependent. In a task with stable reward positions, forward shifted representations first became biased toward one of the two options and then became truncated with further experience [29]. Forward sweeping representations provide a potential representation mechanism for the prediction component of goal-directed decision-making.

Fig 3. Forward-shifted neural representations at the choice point.

Spatial representations were decoded from the activity of simultaneously recorded neural ensembles within the CA3 region of the hippocampus. Each panel shows a sample of the decoded hippocampal spatial representation in a cued choice task [29]. Panels are arranged in 40msec intervals from left-to-right, then top-to-bottom. Representations are displayed as a probability distribution over space (red = high probability, blue = low probability) and the animal’s position is shown as a white ○. The representations closely tracked the rat’s position as the rat approached the choice point. As the rat paused at the choice point, hippocampal spatial representations moved forward of the animal into each arm (first to the left, then to the right). Forward-shifted neural representations provide a potential mechanism for consideration of future possibilities. Data is from another example from data set originally reported in [29].

Representations of future possibilities are not sufficient for action-selection. Decision processes also require mechanisms for evaluation of future expectations as well as mechanisms for flexible translation into behavior. Although other candidates exist, three structures have been suggested as key to evaluation and action-selection processes in the planning system: orbitofrontal cortex [30–32], ventral striatum [32–34], and dorsomedial striatum [35, 36]. Hippocampal activity influences ventral striatum firing [37, 38]. Anticipatory representations in oribitofrontal cortex disappear after hippocampal lesions [39]. During learning, hippocampal and dorsomedial local field potentials become transiently coherent at specific frequencies (theta, 7–10 Hz) [36]. Both ventral striatum [33, 40, 41] and orbitofrontal cortex [31, 42, 43] have been found to encode reward value and expectancy. This suggests that the hippocampus may only be providing the prediction component; evaluation of the value of that prediction and the making of the decision may happen downstream of the hippocampal prediction process.

4 The striatum and decision making

Classically within the decision-making, memory, and navigation literatures, the dorsal striatum has been identified as a critical component of incremental (procedural, route-based) stimulus-response learning, particularly in contrast to the more flexible (declarative, map-based) hippocampally-dependent learning system [4, 8, 35]. Current evidence, however, suggests that striatum is involved in both flexible (planning) and stimulus-response (habit) strategies.

A distinction can be drawn between the more anterior, dorsolateral and the more posterior, dorsomedial components of striatum [6, 35]. Dorsolateral striatum is a critical component of incremental (procedural, route-based) stimulus-response learning [35, 44]. This idea has received extensive experimental support from lesion [10, 45, 46], pharmacological [47], and recording studies [48–51]. For example, rodent recording studies during spatial tasks are consistent with such a habit learning role for dorsolateral striatum, in that firing patterns develop slowly [44, 48–50], but only under conditions in which the relationship is rewarded (N.C. Schmitzer-Torbert, PhD Thesis, University of Minnesota; A.M. Graybiel, Soc Neurosci Abstr, 2006).

In computational terms, this incremental-learning strategy is primarily captured by tem- poral difference reinforcement learning (TDRL) models, wherein a process assigns value E(V(s, a)) to taking an action a within a situation s [1]. A central characteristic of such models is that they select actions based purely on a scalar value associated with taking the action in the current situation. This means that such models cannot accommodate more flexible responses such as latent-learning, devaluation, extinction, or reversal [15, 52].

In contrast to the involvement of dorsolateral striatum in outcome-independent control, recent evidence indicates that dorsomedial striatum is involved in flexible goal-directed actions, including the map-based components of navigation tasks [53, 54] and the learning and performance of goal-directed actions of instrumental conditioning tasks [55–57].

As reviewed above, the expression of flexible goal-directed behavior requires at least two processes: access to the knowledge that a given action leads to a particular outcome, and an evaluation of the outcome that takes the organism’s current needs into account. These processes and their striatal underpinnings have been dissociated in instrumental conditioning experiments.

Knowledge of the relationship between action and outcome can be probed by observing rats’ responses to degradation of the contingency between them. When rats trained to lever-press for food X are given a session where X is now delivered independently of lever presses, they will reduce lever pressing on a subsequent extinction test [58]. Rats with dorsomedial striatum lesions [56, 59] or with NMDA-receptor antagonist infusions into dorsomedial striatum [57], however, were insensitive to such contingency degradation, suggesting that dorsomedial striatum is a key component in the processing of action-outcome relationships.

In contrast, the extant evidence suggests that ventral striatum, particularly nucleus accumbens core² is involved in the evaluation component [63, 64]. Recordings from accumbens core have found firing correlates of reward receipt [33, 65], as well as anticipation of future rewards [37, 41, 66]. How such representations are integrated with action selection [67] is still unknown, but it is clear that striatal contributions to decision-making are not restricted to incremental, inflexible habit learning.

5 Conclusion

At this point, it is an open question whether the striatum is performing similar computational operations on different inputs to accommodate different strategies or whether there are internal differences within striatum as well. It is unknown whether computational models of dorsolateral striatum (e.g. TDRL models) can be extended to accommodate other, more flexible strategies. The most detailed theories are the self-projection theories [5, 18] and the expectancy theories [11], most-recently computationally instantiated in the recent Daw et al. [15] and Hasselmo et al. [14, 16] models. These theories suggest that orbitofrontal cortex, dorsomedial striatum, and ventral striatum may be selecting actions based on evaluations of expectancies derived from self-projection information (i.e. non-local representations) projecting in from the hippocampus and frontal cortices. The recent data from Johnson and Redish [29] that hippocampal representations sweep down potential trajectories during flexible decision-making provides a potential instantiation of a search process. Consistent with a hippocampalstriatal role in early learning is recent local field potental data that hippocampal and dorsomedial striatal theta signals increase coherence during early learning [36]. However, it is still unknown how stored contingencies are integrated with the evaluation of outcome expectancy to produce actions.

Supplementary Material

01. Movie 1.

Vicarious trial and error entails behaviorally identifiably active processes. Video shows four example trials from a one-⊤ loop task. On the first example trial, the rat looks left and then goes right. On the second example trial, the rat looks right and then goes left. On the third example trial, the rat looks left, starts right, but then goes left. Of particular note is the fourth example trial in which the rat looks right before starting the journey down the central track and then does not pause at the actual choice point, suggesting the moment of decision may have been made before the journey down the central track in this last example.

On this task, reward was delivered at feeders on either the left or right return rails under an all-Left, all-Right, or alternating schedule. The schedule was selected at random each day and switched to a new schedule at a random time approximately halfway through the 30 minute session.

Trials are selected from a single session, and are in order, but are not consecutive.

Fig 1. Decision-making under formulations of cache-based stimulus-action (A) and expectation-based planning (B) strategies.

Both strategies require a situation-recognition component to produce a starting point S for predictions within the decision-making process. In cache-based models (A), decision-making entails selecting the action a with the maximum expected return E(V). This means that actions are judged only in terms of their cached expected return. In planning models (B), active memory processes allow exploration of potential future situations S₁, . . . , S₄. The outcome of each potential future E(S_i ) can then be compared to the animal’s current needs to determine the expected value E(V). Because planning systems include future situation predictions, it can remain flexible under conditions in which cache-systems remain rigid. However, because the planning system must serially search into possible futures, it will require processing time not required by the cache-system.