Ventral striatum: a critical look at models of learning and evaluation

Abstract

Extensive evidence implicates the ventral striatum in multiple distinct facets of action selection. Early work established a role in modulating ongoing behavior, as engaged by the energizing and directing influences of motivationally relevant cues and the willingness to expend effort in order to obtain reward. More recently, reinforcement learning models have suggested the notion of ventral striatum primarily as an evaluation step during learning, which serves as a critic to update a separate actor. Recent computational and experimental work may provide a resolution to the differences between these two theories through a careful parsing of behavior and the instrinsic heterogeneity that characterizes this complex structure.

Decision making – or action selection as it is historically referred to in the striatal literature – typically depends both on past experience (learning) and current motivational state. Early conceptualizations of ventral striatal function emphasized its role in mediating the latter: Mogenson et al. (1) envisioned the nucleus accumbens as a pathway from motivation to action, a notion congruent with decision-related proposals such as overcoming effort required to obtain reward (2), incentive salience (3), and mediating the impact of behaviorally relevant (4) or temporally unpredictable cues (5). These proposals share the view that the ventral striatum influences action selection at the time of decision, broadly taken to include making a choice between options as well as the initiation or interruption of behavior. In contrast, more recent suggestions that ventral striatum is part of the “critic” component of an “actor-critic” temporal-difference reinforcement learning (TDRL) implementation casts it as enabling learning, but not itself involved at the time of decision (6). New computational models based on online evaluation processes may provide a potential resolution to these differences, suggesting that the “critic” can play a role in certain decisions at the time of decision as well as a longer-term learning role.

Ventral striatum as a reinforcement learning critic

TDRL models have been remarkably successful in predicting decision-related neural activity based on internal model parameters inferred from behavioral fits in human, non-human primate and rodent studies (7, 8). TDRL models assume the world is divided up into distinct situations or “states”: these states can change when important events happen in the world (such as a lever is provided to the subject) or through actions taken by the decision-maker or agent (such as pressing that lever) (9–11). Certain states result in reward delivery, while other states merely reflect categorizations of the situation (12). Because state space representations are internal to the agent, they need not be restricted to categorizations of the environment, but may also include predictive, working memory, or action history components (13, 14). Overall, the agent aims to maximize its reward and minimize its punishment by learning from experience which actions to take in any given state. The essential feature of the critic in the actor-critic model is that it provides a training signal to the actor to learn which actions to take. The critic accomplishes this by maintaining a value function across states that reflects the expected future reward from that state. When a state transition happens, the critic reports the difference between observed and expected value (value prediction error), which can then be used to train a separate actor. This can be seen, for instance, in the classic recordings from Schultz and colleagues of dopaminergic neurons in the ventral tegmental area (VTA), which learn to signal a prediction error upon appearance of a reward-predictive stimulus, even though that stimulus is not in itself a primary reward (15). Thus, not only do internal variables derived from TDRL models appear to fit neural signals well, but the TDRL language of states, values, and actions provides a way to formulate explicit theories that deal with learning, decision-making, and reward.

An influential proposal about the biological implementation of TDRL has suggested the ventral striatum implements the critic, encoding the value (expected future reward) and (actual) reward information necessary for the calculation of prediction error signals (6, 11, 16). In support of this idea, ventral striatum is strongly coupled to the VTA, potentially allowing VTA access to value and reward signals from the ventral striatum to compute prediction errors, and vice versa. Three main experimental approaches have been brought to bear on testing for the presence of such signals in the ventral striatum: measuring cerebral blood flow with functional magnetic resonance imaging (fMRI), dopamine concentrations using fast-scan cyclic voltammetry (FSCV), and spiking and field potential activity using electrophysiological recording.

Although the generality of the prediction error interpretation of VTA neuron activity remains controversial (17, 18), voltammetry studies have found value-prediction error signals in the dopamine levels in ventral striatum during behavioral tasks (19, 20), as predicted from dopamine neuron activity in the VTA (15). fMRI studies reliably report reward prediction error signaling in the ventral striatum (21–23), thought in part, although not entirely, to arise from VTA input (24, 25). Thus, in accordance with the actor-critic model, ventral striatum appears to have access to prediction error signal inputs. Unit recording studies in rats have looked for, but not reliably found, prediction error coding in ventral striatal spiking activity (26, 27), suggesting that this signal is transformed by ventral striatal processing. The actor-critic model suggests this transformation should result in a value signal.

Different experimental settings have been used to identify potential value signals in the ventral striatum. One possible candidate is a population of anticipatory “ramping” neurons, which gradually increase their firing rate when approaching or waiting for reward delivery (28, 29). Because future rewards are discounted, this pattern is similar to that expected of a critic state value function (9). Khamassi et al. (28) explicitly attempted to fit the firing patterns of such ramping neurons with TDRL models. They found the relationship to be mixed, with many neurons ramping up to some, but not other, reward deliveries, unlike what the critic theory would predict. By assuming fragmented state spaces – essentially allowing the agent to be confused about the true state of the world – they could reproduce TDRL value functions similar to the data. This interpretation highlights an important issue in the application of TDRL models to behavior and neural data: we may not know the true state space used by the organism (12, 30).

Value-related signals can also be related to actions taken by the agent. Direct representation of “action values” are likely part of the actor, not the critic, but actions likely result in state transitions which would be reflected in the critic’s value representations. Internal (covert) perparatory state transitions may also be reflected in the critic value signal, potentially producing what appear to be pre-action signals in the critic. There is substantial fMRI evidence for state value representations in the ventral striatum (31–33), but the data from unit recording studies has been less consistent. Ito and Doya (27) applied a comprehensive analysis of ventral striatum neural correlates using TDRL models fit to behavior on a choice task where action/outcome value could be dissociated from actions. Although they found a statistically significant population of neurons encoding the value of the upcoming outcome, this signal did not have a clear time course around the time of the decision, and the percentage of neurons involved was small (<10%). Similarly, Kim et al. (34, 35) and Kimchi and Laubach (36) found that ventral striatal activity contained little information about upcoming behavioral choice. In contrast, Roesch et al. (26) found a population of ventral striatal neurons (also <10%) coding for the value and direction of chosen actions, before the action was initiated. Similar populations have been found in other aspects of the striatum (34–36).

Finally, reward-related cues or state transitions occurring in the environment unrelated to the agent’s actions have long been known to trigger ventral striatal activity (37, 38), including responses to reward delivery (39). Consistent with the actor-critic TDRL formulation, the development of firing to reward-predictive cues in ventral striatum depends on dopaminergic input (40). In further support of the theory, there is evidence that ventral striatal firing to reward-predictive cues (26) and rewards themselves (39) are modulated by value, although these have not systematically been distinguished from motivation or palatability (26, 39).

Thus, experimental recording and imaging studies such as the above face a number of challenges in relating theoretical concepts to the data. First, many of the signals of interest – including prediction errors and value signals – are often correlated, and so require specific experimental designs for disentanglement (e.g. by including appetitive and aversive reinforcers (41)). Second, ventral striatum forms an interconnected network with a number of limbic areas, including the orbitofrontal cortex and the amygdala, in which similar value and prediction error signals appear to be present (23, 42). Third, the same behavioral task may be accomplished by different decision-making systems, with radically different information processing needs, which would change the expected action-selectivity of neural firing (43, 44). Finally, the ventral striatum is a heterogenenous structure with numerous anatomical and functional dissociations between, for instance, the core and shell subregions, electrical and neuromodulatory input gradients, and complex receptor expression patterns (45, 46). We discuss recent progress related to the last two issues next.

Functional heterogeneity: defining ventral striatal processing units

The well-known anatomical and functional heterogeneity of the ventral striatum (45, 46) contributes to the diversity and fragmentation of recording data; see for instance (47, 48) for recent examples that map known differences between the core and shell subregions to neural activity. Such differences have not yet been systematically related to value signals in the ventral striatum. However, voltammetry studies on the dopamine input to the ventral striatum suggest that there will likely be important differences. For example, Aragona et al. (49) found that cocaine sensitization of the dopamine signal on drug-receipt appeared in ventral striatum shell, while the dopamine signal that developed to the preditive cue appeared in ventral striatal core. Even within these subregions, the dopamine signal may not be unitary (50).

Ventral striatum receives a number of convergent inputs, many of which overlap at the population and single neuron level (45, 51, 52) thought to define functional subunits (53). In a recent demonstration of this, van der Meer and Redish (29) found that anticipatory “ramp” cells showed theta phase precession relative to the hippocampal theta rhythm. This systematic spike timing is thought to be important for the rapid encoding of sequences in the hippocampus, suggesting its extension to value-related signals in the ventral striatum may implement associations between places and rewards. In support of this idea, the projection from hippocampus to ventral striatum (shell) is known to be important for place-reward learning (54) and reward-related cells in the ventral striatum are more likely to exhibit coherent off-line “replay” with the hippocampus (55). A different example comes from a recent set of studies documenting the properties of gamma oscillations in the ventral striatum. It was found that “low” (40–60 Hz) and “high” (70–100 Hz) gamma oscillations not only had distinct behavioral correlates, but were also associated with distinct populations of putative fast-spiking interneurons (56–58). Low and high gamma oscillations displayed a distinctive “switching” pattern, suggesting the possibility that different ventral striatal network states, driven by distinct FSI populations, may form transient functional connections with different inputs as indicated by coherence or synchrony across areas (59). Thus, simultaneous recording from multiple structures provides an opportunity not only for electrophysiological identification of likely ventral striatal targets of specific afferents, but also for an examination of the computations implemented by these projections.

Behavioral heterogeneity: same overt behavior, different underlying processing

Current reinforcement learning theories suggest that tasks can be solved by two distinct processing mechanisms (43, 60, 61) — in the “model-based” system, the agent is able to search through a representation of potential consequences of its actions and evaluate those representations online, during the decision-making process itself; in “model-free” systems such as the actor-critic, the agent makes decisions based only on the representation of the state of the world at the moment (which, as noted above, may be complex, e.g. including working memory). The notion of model-free and model-based controllers rests on a large body of evidence for dissociable learning and decision-making systems in the brain (62–64); a key insight from these studies is that what overtly appears to be a very similar behavior (such as pressing a lever for food) may actually depend on different neural substrates depending on, for instance, the amount of training on the task. Which system is in control is not immediately clear but may be revealed by judicious probe trials or detailed behavioral analysis.

A prominent example is sensitivity to reinforcer devaluation, which is interpreted as evidence that a “model-based” system, with knowledge about the outcomes of actions and dynamically evaluating them is in control of behavior (60, 64). Although there is evidence for the involvement of ventral striatum in devaluation experiments in lever-pressing tasks, the pattern of results is complex (65, 66). However, recent studies in the rat appear to be converging on a role for ventral striatum in mediating the effect of reward value (US) on responses to a predictive cue (CS) when the US is devalued (67, 68). Thus, in a given decision situation, the involvement of ventral striatum in model-based decision-making may depend on the extent to which Pavlovian relationships and responses are congruent with the instrumental behavior (69, 70).

Although to our knowledge neural activity in ventral striatum has not been recorded during reinforcer devaluation, several recent studies have found ventral striatal firing during behaviors suggestive of “model-based” control. Krause et al. (48) found that ventral striatal neurons fired with the initiation of self-initiated movements towards places conditioned to be preferred by morphine injection (71). Recently, Nicola (72) found a striking contrast where ventral striatum was involved on “flexible approach” trials where the rat’s starting location for reward approach differed from trial to trial and/or the rat may have been engaged in other activities (such as grooming), but not on “habitual approach” trials where the rat executed a stereotyped response. Similarly, van der Meer and Redish (73) found that ventral striatal cells that normally fired during consumatory phases of reward showed increased activity shortly before decisions as rats engaged in vicarious-trial-and-error behaviors at a choice point. This effect subsided as the behaviors became more automated, and did not appear in dorsal striatal recordings on the same task (44).

Thus, ventral striatum appears to be involved in aspects of both model-free and model-based or flexible behavior. The computations underlying these systems are thought to be quite different, but value signals play an important role in both: facilitating learning as the “critic” in actor-critic TDRL models, and evaluating predictions about the future in model-based systems. The idea that ventral striatum can be a critic over actual states (as required for TDRL learning) and internally generated or hypothetical states (as may be required during flexible behaviors; (13, 14, 74)) may help bring together the known involvement in both learning in the model-free case, and performance in the model-based case (75).