A universal role of the ventral striatum in reward-based learning: Evidence from human studies

Abstract

Reinforcement learning enables organisms to adjust their behavior in order to maximize rewards. Electrophysiological recordings of dopaminergic midbrain neurons have shown that they code the difference between actual and predicted rewards, i.e., the reward prediction error, in many species. This error signal is conveyed to both the striatum and cortical areas and is thought to play a central role in learning to optimize behavior. However, in human daily life rewards are diverse and often only indirect feedback is available. Here we explore the range of rewards that are processed by the dopaminergic system in human participants, and examine whether it is also involved in learning in the absence of explicit rewards. While results from electrophysiological recordings in humans are sparse, evidence linking dopaminergic activity to the metabolic signal recorded from the midbrain and striatum with functional magnetic resonance imaging (fMRI) is available. Results from fMRI studies suggest that the human ventral striatum (VS) receives valuation information for a diverse set of rewarding stimuli. These range from simple primary reinforcers such as juice rewards over abstract social rewards to internally generated signals on perceived correctness, suggesting that the VS is involved in learning from trial-and-error irrespective of the specific nature of provided rewards. In addition, we summarize evidence that the VS can also be implicated when learning from observing others, and in tasks that go beyond simple stimulus-action-outcome learning, indicating that the reward system is also recruited in more complex learning tasks.

1 Introduction

As any living organism, humans are faced with the need to make decisions about how to act in response to a plethora of environmental cues every day. Often, we encounter similar situations repeatedly, which enables us to use past experiences to predict future outcomes (Cohen, 2008). Learning from trial and error, or reinforcement learning, has been extensively studied in the last decades. Normative computational models have proven to be successful in explaining learning in terms of a reward prediction error, i.e., a mismatch between predicted and actual rewards. On a neural level, it has been shown that the dopaminergic midbrain neurons, with their massive projections to the ventral striatum (VS), represent this reward prediction error and play a central role in reward-based learning (for reviews see Delgado (2007); Niv and Montague (2008); O’Doherty, Hampton, and Kim (2007); Schultz (2007)).

However, in ecologically valid settings, decision problems vary hugely in complexity: organisms may have to decide which objects in the environment to categorize as nutrition, but also which partner to choose, or, in the human case, what career path to follow, and how to provide for retirement. These problems also vary in the rewards they provide: from simple primary and secondary reinforcers such as food, the option to reproduce, and money, to more abstract rewards such as love, social approval and (financial) stability. For many of these decisions, feedback from the environment is sparse and delayed (Hogarth, 2006). In addition, learning from errors is costly, and their commission is often avoidable. In these situations we have to additionally rely on other mechanisms than learning from trial and error, such as building abstract structural representations (Ribas-Fernandes et al., 2011; Diuk, Tsai, Wallis, Botvinick, & Niv, 2013), mental models of the environment (Daw, Gershman, Seymour, Dayan, & Dolan, 2011; Gläscher, Daw, Dayan, & O’Doherty, 2010) or learning from others (Burke, Tobler, Baddeley, & Schultz, 2010). Many species are able to learn from such indirect experience, i.e., by observing the outcome of others and imitating their actions (Chamley, 2004). Humans in particular have developed a complex form of communication, allowing them to pass on learned information by providing instructions (Li, Delgado, & Phelps, 2011).

The neural substrates of learning from abstract, incomplete or absent rewards are only currently being investigated. Humans are the ideal species to study these tasks: they are willing to work without explicit rewards after each trial, which facilitates the study of observational learning, and can be verbally instructed to examine the influence of prior knowledge. In addition, experimental methodology to study higher cognitive functions is well established, and a wealth of experimental paradigms, formal models and empirical behavioral data exists on learning in humans. Using functional magnetic resonance imaging (fMRI) it is possible to non-invasively study the brain activation of human participants performing complex learning and decision making tasks.

This review focuses on how central findings on the neural underpinnings of animal learning can be replicated and elaborated upon by examining different types of reward and more complex forms of learning in human participants. To this end, we present a summary of influential results in the animal literature, and discuss how activation data obtained using fMRI relates to more direct measures of neural firing as acquired in electrophysiological recordings, and specifically to dopaminergic activity. We then go on to summarize recent findings on the neural substrates of learning in humans, and argue that phylogenetically old pathways that mediate simple stimulus-response learning are recruited even in the absence of explicit rewards to solve complex decision making tasks.

2 Reinforcement learning and the dopaminergic system

In order to optimally behave in any given environment it is crucial to determine which actions result in rewarding events given a specific state of the environment. These rewarding events come in many different shapes and flavors: even primary rewards can be as diverse as the opportunity to mate or a drop of juice. Any agent striving to maximize positive outcomes can benefit from a system that unifies these diverse sensory inputs by encoding the motivational properties of stimuli, thereby providing a “retina of the reward system” (Schultz, 2007; Schultz, 2008).

2.1 The dopaminergic system: A system for reward and motivation

The seminal experiments of Olds and Milner (1954) were a first step towards identifying a system in the mammalian brain that is dedicated to processing motivational information. They described several brain sites in the rat where direct electrical stimulation acted as a reinforcer, inciting the animal to stimulate itself. Subsequent experiments showed that the tissue inducing the highest rates of self-stimulation is located in the medial forebrain bundle (MFB), which is connected to dopamine cell bodies in the ventral tegmental area (VTA) (Gallistel, Shizgal, & Yeomans, 1981), and that self-stimulation of the MFB is related to fluctuations in dopamine levels in one of the prominent targets of mesolimbic dopamine neurons, the nucleus accumbens (Garris, Kilpatrick, & Bunin, 1999; Hernandez et al., 2006). Gallistel et al. (1981) point out three central properties of behavior during self-stimulation: rats engage in self-stimulation with high vigor, i.e., a short latency and high intensity of responding. In addition, they pursue self-stimulation over other vital goals, and adjust their behavior flexibly to the magnitude of recent stimulation. These three properties can be viewed as hallmarks of motivation: motivation energizes the organism, directs its behavior towards a goal, and enables learning about outcomes (Gallistel et al., 1981).

2.2 The dopaminergic system in learning

Dopaminergic neurotransmission has been associated with a wide variety of functions, many of which are important for optimizing behavior, i.e., for maximizing reward and minimizing aversive outcomes. Different firing modes of dopaminergic neurons have been associated with the facilitation of a wide range of motor, cognitive, and motivational processes (Schultz, 2008; Bromberg-Martin, Matsumoto, & Hikosaka, 2010), with determining the strength and rate of responding (Niv, Daw, Joel, & Dayan, 2007), and with signaling different aspects of salience, including appetitive and aversive information (Schultz, 2007; Bromberg-Martin et al., 2010), novelty (Bunzeck & Düzel, 2006; Ljungberg, 1992; Wittmann, Bunzeck, Dolan, & Düzel, 2007), and contextual deviance (Zaehle et al., 2013). However, the majority of recent research has focused on one specific aspect of the firing of dopaminergic neurons: 75–80% of the cells convey a signal that is ideally suited to promote learning (Schultz, 2008). They respond with short-latency phasic bursts to unpredicted rewards and reward-predicting stimuli. Importantly, when rewards are fully predicted no response is observed, while the omission of predicted rewards leads to a decrease in baseline activity (Schultz, 2002)(see Fig. 1A). This behavior corresponds to a reward prediction error signal, i.e., to a coding of reward information as the difference between received and expected reward (Montague, Dayan, & Sejnowski, 1996; Schultz, Dayan, & Montague, 1997).

Figure 1. Brain responses to reward expectation and receipt.

A. In Macaque monkeys, spiking activity in dopaminergic midbrain neurons increases with the expected value (probability x magnitude) of the reward that is predicted by a conditioned stimulus, while the response during the receipt of the reward scales with the prediction error, i.e. the received minus the expected reward (Panel adapted from Tobler et al. (2005)). B. A similar activation pattern is observed when measuring fMRI activation in the ventral striatum (results only shown for the right VS; Panel adapted from Abler et al. (2006)). These responses are in line with the predictions of normative models of reinforcement learning (Montague et al., 1996; Schultz et al., 1997).

A reward prediction error signal has also been postulated as a teaching signal in normative theories of reinforcement learning (Bertsekas & Tsitsiklis, 1996; Sutton & Barto, 1990). These theories focus on providing normative accounts of how agents can optimize their behavior (Niv & Montague, 2008). A wide variety of reinforcement learning algorithms exists, however most of them share some core features: they predict that whenever faced with a decision, the agent calculates a value for each available option. In order to allow for random and exploratory behavior, these values are subsequently passed through a probabilistic function before the stimulus or action with the highest value is chosen. Whenever new information becomes available, e.g. in the form of an unpredicted reward or the omission of an expected reward, the values are updated using the reward prediction error multiplied by a learning rate (Cohen, 2008).

Such a mechanism for calculating the expected subjective value of future states of the environment and updating it using a simple prediction error signal would benefit any living organism: it provides a mechanism to make predictions based on the similarity between past experiences and the current state of the environment, enabling the organism to make informed decisions. Whenever these predictions are violated, they can be updated using a single scalar signal, without having to (re-)process all available sensory information (Schultz, 2008). A correspondence between the prediction error signals postulated by normative models and phasic dopaminergic firing was shown quantitatively (Bayer & Glimcher, 2005; Tobler, Fiorillo, & Schultz, 2005) and using several different paradigms, including blocking (Waelti, Dickinson, & Schultz, 2001) and conditioned inhibition (Tobler, Dickinson, & Schultz, 2003). This postulated role of short-latency firing of dopaminergic midbrain neurons in signaling prediction errors has been termed the reward prediction error hypothesis of dopamine (for extensive reviews see Schultz (2002); Schultz (2006)).

One potential mechanism by which phasic dopamine bursts could support learning is suggested by the observation that long-term potentiation in the striatum depends not only on strong pre- and postsynaptic activation, but also on dopamine release (this is sometimes referred to as the three-factor learning rule (Ashby & Ennis, 2006)). According to this model, the same dentritic spines of striatal medium spiny cells are contacted both by glutamatergic input from cortical neurons and dopaminergic input from midbrain neurons. Synaptic transmission is strengthened only on those neurons that simultaneously receive cortical input, coding some aspect of an event in the environment, and dopaminergic input, signaling via the reward prediction error that adjustments are necessary (Schultz, 2002). Hence, on the basis of contingent feedback, dopaminergic projections to the striatum are able to modify behavior in response to salient stimuli.

Indeed, using optogenetic activation of dopamine neurons, a causal role of dopaminergic signals in cue-reward learning has been established in rats (Steinberg et al., 2013). In humans, both striatal fMRI activations in healthy populations and impairments in feedback-based learning in patients with striatal dysfunctions were reported, indicating a critical role of cortico-striatal loops in feedback-based learning (Ashby & Ennis, 2006; Shohamy et al., 2004).

2.3 Outlook

The reward prediction error hypothesis of dopamine is not unchallenged (see e.g. (Berridge, 2007; Redgrave, Gurney, & Reynolds, 2008; Vitay, Fix, Beuth, Schroll, & Hamker, 2009)). However, it has inspired an extensive body of experimental research on reward-based learning, and models derived from it have proven to be able to predict behavior and neural activations across species and for a wide range of reinforcing stimuli and experimental paradigms. Even in their most simplistic form reinforcement learning models provide a parsimonious account of many behavioral phenomena and neural activation patterns. More extended models can account for complex phenomena observed in reward-based learning, such as the sensitivity to risk (Niv, Edlund, Dayan, & O’Doherty, 2012), or the ability to learn from rewards that were foregone (Fischer & Ullsperger, 2013; Lohrenz, McCabe, Camerer, & Montague, 2007), making these models an invaluable tool to derive quantitative trial-by-trial predictions about learning (Daw, 2011).

3 Examining the brain in humans using fMRI: the relationship between the BOLD signal, neuronal firing, and dopaminergic activity

In this review we present recent findings on learning-related activations in one of the most prominent targets of the dopaminergic midbrain neurons, the VS, in humans. While most experiments on dopaminergic activity in non-human animals are conducted using electrophysiological recordings, these are rarely available from human participants due to their highly invasive nature (exceptions are reported, e.g., in Münte et al. (2007); Cohen et al. (2009); Zaehle et al. (2013)). Functional magnetic resonance imaging (fMRI) can be used to safely visualize brain activation from the whole brain continuously while the participant is performing a task, providing both information about the network supporting task performance and how the involvement of that network changes over time. However, fMRI only provides an indirect measure of neuronal activity and no information about the involved neurotransmitter systems. In the present section we discuss how the fMRI signal relates to neuronal activation, and under which circumstances a relationship between ventral striatal fMRI activation and dopaminergic activity can be assumed.

3.1 Understanding the fMRI signal

In order to relate results from fMRI experiments to those obtained using electrophysiological recordings, it is crucial to understand the nature of the fMRI signal. However, despite the ever-growing popularity of fMRI to measure brain activation since its introduction in the early 1990s (Bandettini, Wong, Hinks, Tikofsky, & Hyde, 1992; Belliveau et al., 1991; Frahm, Bruhn, Merboldt, & Hänicke, 1992; Kwong, 1992; Ogawa, Lee, Kay, & Tank, 1990; Ogawa, 1992), the exact origin of the fMRI signal is still being investigated. As fMRI technology cannot provide direct information about the electrical activity of neurons, it has to rely on an indirect index based on the metabolic changes induced by neuronal activity. The BOLD (blood oxygen level dependent) contrast relies on the fact that hemoglobin, the oxygen-carrying molecule within red blood cells, has different magnetic properties depending on its oxygenation state, which allows it to act as an endogenous contrast agent. This contrast is assumed to reflect changes in oxygen consumption in combination with changes in cerebral blood flow and volume (Buxton, Uluda, Dubowitz, & Liu, 2004). These hemodynamic responses are supposed to be elicited by local increases in neural activity (Logothetis & Wandell, 2004; Logothetis, 2010); however, the coupling mechanisms between the BOLD signal and neural activity are complex and depend on factors like the link between the neuronal response and increased energy demands, the mechanisms that signal the energy demand, and the mechanisms responsible for triggering the subsequent hemodynamic response (Arthurs & Boniface, 2002; Logothetis, 2003).

One way to empirically investigate the origin of the BOLD contrast are simultaneous cortical electrophysiological and fMRI recordings. Results from such recordings suggest that the BOLD signal is slightly better explained by local field potentials than neuronal spiking activity (Goense & Logothetis, 2008; Kayser, Kim, Ugurbil, Kim, & König, 2004; Logothetis, Pauls, Augath, Trinath, & Oeltermann, 2001; Mukamel et al., 2005; Raichle & Mintun, 2006). Local field potentials are assumed to not be driven by the activation of output neurons of a given area, but rather by input to that area, as well as local interneuronal processing, which includes both inhibitory and excitatory activations (Logothetis, 2003; Logothetis & Wandell, 2004). In addition, neuromodulatory signals as carried by e.g. acetylcholine, serotonin, and dopamine can also influence the BOLD signal directly (Attwell & Iadecola, 2002; Logothetis, 2008).

The indirect nature of the BOLD-fMRI signal has to be kept in mind whenever fMRI activity is interpreted as neuronal activation. This is especially true when it is interpreted as reflecting the activity of specific neurotransmitter / neuromodulator systems, as is often done in studies of reward-based learning, which is intimately tied to the function of the dopaminergic system.

3.2 A potential link between BOLD response and dopaminergic activation

As detailed above, fMRI is not able to measure changes in dopamine release directly. However, there are indications that the BOLD response and dopaminergic neuron firing are correlated in midbrain areas containing a large proportion of dopaminergic neurons, i.e. the substantia nigra (SN) / ventral tegmental area (VTA), and in regions receiving massive projections from these areas, like the VS.

Injections of dopamine-releasing agents can increase the BOLD signal in the nucleus accumbens, and it has been suggested that the signal has the capacity to also capture signals induced by endogenous dopaminergic firing (Knutson & Gibbs, 2007). Consistent with this assumption, fMRI studies show activations at or near dopaminergic midbrain nuclei and the VS that correlate with both reward expectation and reward prediction errors, and thereby exhibit a striking similarity to the pattern of burst firing of dopamine neurons observed in electrophysiological recordings (D’Ardenne, McClure, Nystrom, & Cohen, 2008; Cohen, 2008; O’Doherty et al., 2007; Knutson, Taylor, & Kaufman, 2005; Niv & Montague, 2008) (see Fig. 1).

More direct evidence for the modulation of ventral striatal fMRI signals by dopamine was provided by Pessiglione, Seymour, Flandin, Dolan, and Frith (2006). In their instrumental learning paradigm, the administration of drugs that enhance or reduce dopaminergic function influenced the magnitude of BOLD-fMRI activation in the striatum in response to reward prediction errors (for similar results see van der Schaaf et al. (2012)). Subsequently, a quantitative role of dopamine in the fMRI signal was established by showing a correlation between reward-related dopamine release, as indexed by [¹¹C]raclopride PET, and fMRI activity in the SN / VTA and the VS across the same participants (Schott et al., 2008). An indication for the role of dopamine in the dorsal striatum was provided by a recent study by Schönberg et al. (2010). They showed that in patients with Parkinson’s disease, which is characterized by a loss of dopaminergic input to mainly the dorsal striatum, fMRI activation in response to reward prediction errors in the dorsal putamen is significantly decreased compared to responses observed in control participants.

Although the studies outlined above provide evidence for a link between dopaminergic neurotransmission and fMRI activation during reward-learning, some caveats apply. The reviewed results do not rule out that this link is mediated by non-dopaminergic processes, like glutamatergic costimulation. In addition, this evidence was only provided for the expectation and processing of (1) rewards in (2) the SN / VTA and striatum. As fMRI cannot isolate dopamine-related activity from the effects of other afferents, the BOLD signal and dopamine release might dissociate in other situations. For example, the receipt of aversive stimuli or the omission of expected rewards might result in increased inputs to inhibitory interneurons in the SN / VTA, and thereby elicit an increased fMRI signal in these areas (Düzel et al., 2009). Indeed, positive midbrain activations in response to negative feedback and positive errors in the prediction of punishment have been observed (A. R. Aron et al., 2004; Menon et al., 2007). In a recent meta-analysis of 142 fMRI studies on reward based learning, Liu, Hairston, Schrier, and Fan (2011) report that although responses to positive outcomes tend to be stronger than responses to negative outcomes, the VS is consistently activated by both positive and negative outcomes. Similarly, dopaminergic drugs affect learning from rewards and punishments differentially, while fMRI signals in the striatum in response to reward and punishment prediction errors are altered in the same direction (van der Schaaf et al., 2012). These observations underscore that, given the indirect nature of the fMRI signal, the direction of activations in response to positive and negative outcomes needs to be interpreted with care. Additionally, the reported results on the relationship between dopaminergic activity and the fMRI signal in response to reward prediction errors can also not be transferred to other target areas of the dopaminergic system, e.g. the orbitofrontal cortex or the medial prefrontal cortex, which are less physiologically homogenous and show different pharmacokinetics (Knutson & Gibbs, 2007).

3.3 Summary and conclusions

FMRI cannot assess the exact neuronal mechanisms underlying the studied tasks. The signal changes it measures are however a response to local changes in overall neural activity, and it allows measuring these signal changes with a moderately high spatiotemporal resolution – enabling event-related signal analyses -throughout the whole brain, including deep structures that are central to reinforcement learning like the dopaminergic midbrain nuclei and the ventral striatum. For these structures evidence exists that links fMRI activation during reward processing to dopamine (Düzel et al., 2009; Knutson & Gibbs, 2007; Pessiglione et al., 2006; Schott et al., 2008; van der Schaaf et al., 2012). In combination with its non-invasive nature, these findings make fMRI the best currently available tool to study the neural basis of reward-based learning in healthy human participants.

4 Ventral striatal activation in reward learning

Using fMRI, a wide range of rewards that have been shown to activate areas of the human dopaminergic system. However, when discussing the results of this body of research, two important limitations have to be considered.

Firstly, the reward prediction error hypothesis of dopamine postulates that dopaminergic midbrain neurons and their targets respond at two timepoints during a learning trial: (1) in response to the appearance of the reward predicting stimulus, and (2) in response to the receipt or omission of the expected reward. According to this hypothesis the first response is in proportion to the (subjective) value of the reward predicted by stimuli, while the second response reflects the difference between expected and received rewards. Correlates of these responses have been observed in both electrophysiological recordings of dopaminergic midbrain neurons (Matsumoto & Hikosaka, 2009; Tobler et al., 2005) and fMRI data in the striatum (Abler, Walter, Erk, Kammerer, & Spitzer, 2006; Haruno & Kawato, 2006; Cohen, 2007) (see Fig. 1). In line with these findings, in their meta-analysis Liu et al. (2011) report that the VS is activated during both the expectation and receipt of rewards, with potentially stronger activations in the latter phase. However, signals during reward expectation and receipt are highly correlated (Niv, 2009) and not addressed separately in many of the reported studies. For the purposes of the current overview we will therefore discuss both stages of reward-processing in parallel.

As a second limitation it has to be noted that although several fMRI studies report reward-related activity in the vicinity of the bodies of dopaminergic neurons in the SN/VTA (see e.g. Adcock, Thangavel, Whitfield-Gabrieli, Knutson, and Gabrieli (2006); D’Ardenne et al. (2008); Dreher, Kohn, and Berman (2006); O’Doherty, Deichmann, Critchley, and Dolan (2002); O’Doherty, Buchanan, Seymour, and Dolan (2006); Schott et al. (2007); Wittmann et al. (2005)), fMRI activations in response to reward are more consistently observed in the VS, one of the most prominent target areas of dopaminergic projections. Potential reasons for this observation include the nature of the fMRI BOLD signal (i.e., the signal might reflect the input to an area more than it’s output; see previous section) as well as several imaging difficulties that are specific to the midbrain (D’Ardenne et al., 2008). In the present section we therefore focus on recent findings on the different types of reinforcing stimuli that have been shown to evoke fMRI activations in the VS.

4.1 From primary to secondary reinforcers

Different types of reinforcers are traditionally categorized according to the way they have acquired their ability to strengthen the behavior they were consequent to. Primary or unconditioned reinforcers such as food, drink, and sex have a biological/evolutionary basis, and act as positive reinforcers without any antecedent learning. On the other hand, secondary or conditioned reinforcers acquire their ability to strengthen behavior by being associated with unconditioned reinforcers. The simplest example of a conditioned reinforcer is a previously neutral stimulus, such as a light, which is consistently paired with a primary reinforcer, such as juice, within the context of an experiment.

Although the categorization into primary and secondary reinforcers is a useful theoretical concept, in many settings the distinction between innate and learned responses is more gradual (see Fig. 2). For example, strictly speaking the flavor of most foods has to be considered a conditioned reinforcer which has acquired its value by being associated with the food’s nutritional postingestive effects (Sclafani, 1999). Additionally, during the lifetime of a person some stimuli are paired with a many other reinforcers, under a wide variety of conditions. They are therefore reinforcing in many contexts, and therefore have properties that are similar to those of primary reinforcers. These stimuli are sometimes referred to as generalized conditioned reinforcers, the most frequently cited examples of which are tokens such as money or school grades; however, many social stimuli such as attention, affection and approval, which consistently precede the receipt of primary reinforcers, have to be considered generalized conditioned reinforcers, too (Skinner, 1953). Partially conditioned stimuli fall in-between the conditioned/unconditioned categories. For example, the reinforcing effect of the ability to control the physical world around us through acquiring skills or information seems to be largely unconditioned, whereas the reinforcing effects of beauty in music or art have been postulated to be largely conditioned (Skinner, 1953).

Figure 2. Ventral striatal processing of a wide range of rewards.

The ventral striatum has been shown to respond to unexpected rewards from the whole spectrum of reinforcers, including largely unconditioned reinforcers such as food or drink, generalized conditioned reinforcers such as money, and stimuli that are associated with reward only in the context of the specific experiment.

Generalized conditioned reinforcers, and specifically monetary rewards, are administered in many fMRI studies of reward-based learning. Monetary rewards have several advantages, including that they are easy to administer and quantify, that they are likely not subject to devaluation over the course of the experiment, and that participants do not have to be liquid- or food-deprived for the experiment. Consequently, several studies and meta-analyses are available that compare the effects of monetary rewards to other classes of reinforcers. In the present section, we will discuss three frequently administered classes of reinforcers (primary, social and cognitive) and will assess their relationship with monetary rewards.

Primary rewards

In line with electrophysiological recordings from dopaminergic midbrain neurons, the VS is activated by primary rewards such as juice or glucose solution both during classical and instrumental conditioning (O’Doherty et al., 2002; Pagnoni, Zink, Montague, & Berns, 2002). In a direct comparison with monetary rewards, fMRI activation in the nucleus accumbens was observed to correlate more strongly with errors in the prediction of monetary than juice rewards (Valentin & O’Doherty, 2009). A recent meta-analysis of 206 fMRI studies on reward reports overlapping activation foci for primary and monetary rewards, however with more densely clustered foci in the nucleus accumbens for monetary rewards (Bartra, McGuire, & Kable, 2013), again suggesting stronger or more robust effects in response to monetary rewards. Similarly, in their meta-analysis of 87 studies with monetary, erotic and food rewards, Sescousse, Caldú, Segura, and Dreher (2013) report stronger activations for monetary rewards than for both primary rewards.

Taken together, these results might indicate that the nucleus accumbens is more involved in the processing of secondary rewards. However, Sescousse et al. (2013) also report two other important differences between studies administering primary vs. secondary rewards. While monetary rewards are often employed in studies that use probabilistic rewards and instrumental conditioning, primary rewards are often administered according a fully predicable schedule and in a passive observation setting. These differences might indicate that the VS is especially activated in response to surprising rewards, as postulated by the reward prediction error hypothesis, and in settings that require learning the optimal response to maximize rewards. Additionally, it has to be assumed that human participants are aware that after a short delay after the experiment they will have nearly unlimited access to primary rewards such as food, drink or sexually arousing images. Therefore, the actual subjective value of primary rewards offered in an experimental setting is unclear and might be lower than that of monetary rewards, indicating that the observed lower activations in the VS in response to primary than to monetary rewards might merely reflect a difference in subjective values.

Social rewards

Although they are sometimes treated as single class of reinforcers, social rewards span the whole continuum from unconditioned to conditioned reinforcers. While erotic images are often viewed as primary reinforcers, smiling faces and a gaze that is directed at the observer also carry information about approval and attention and therefore have properties of a conditioned reinforcer.

Studies investigating social rewards have consistently reported fMRI activation in the striatum or dopaminergic midbrain nuclei in response to as diverse stimuli as facial attractiveness in the opposite sex (Aharon et al., 2001; Cloutier, Heatherton, Whalen, & Kelley, 2008), gaze direction (Kampe, Frith, Dolan, & Frith, 2001), or images of romantic partners (Acevedo, Aron, Fisher, & Brown, 2012; A. Aron et al., 2005). In a direct comparison between monetary and social rewards, images of smiling faces engaged the VS at a similar locus as monetary rewards, but only at a lower threshold (Lin, Adolphs, & Rangel, 2012).

Interestingly, VS activation in response to social situations is not restricted to the processing of positive social cues. It also correlates with behavior in complex social tasks involving cooperation, norm-abiding and altruism. For example, the decision to donate to charitable organizations activates the VS (Harbaugh, Mayr, & Burghart, 2007; Moll, Krueger, & Zahn, 2006), especially when this donation is observed by others (Izuma, Saito, & Sadato, 2010). For a more complete coverage of the literature on social decision-making we refer to recent reviews by Rilling and Sanfey (2011) and Bhanji and Delgado (2013).

Cognitive feedback

Potentially because of both its effectiveness in guiding learning and the ease with which it is administered, cognitive feedback, or giving information about the correctness of an answer, is also often employed as a reinforcer in fMRI experiments with human participants. Given the importance of prediction errors for learning (Schönberg, Daw, Joel, & O’Doherty, 2007), it has also been suggested that the processing of information (Bromberg-Martin & Hikosaka, 2011; Niv & Chan, 2011) and cognitive feedback (A. R. Aron et al., 2004) also involve the reward circuitry. Cognitive feedback is often interpreted as a form of social approval and therefore a generalized conditioned reinforcer comparable to monetary rewards, but it also has a partially biological component of guiding the acquisition of a skill that can help the participant control the environment of the experiment. Psychologically, monetary rewards can be viewed as a form of extrinsic motivation and are known to undermine intrinsic motivation, which is not necessarily true for cognitive feedback (Murayama, Matsumoto, Izuma, & Matsumoto, 2010). We directly compared the neural correlates of monetary rewards with cognitive feedback using two parallel learning tasks, providing participants with either (1) information about the correctness of their decision, or (2) a monetary reward for every correct answer. While several dopaminergic projection sites were activated in both task versions, the nucleus accumbens as only structure was activated more strongly in response to cues predicting monetary reward compared to cues predicting cognitive feedback (see Fig. 3A). While this effect can be explained by the higher subjective value of monetary rewards, a closer examination of the nucleus accumbens activation indicated that it correlated with questionnaire items measuring intrinsic motivation in the cognitive feedback task, while the activation correlated with items measuring extrinsic motivation in the monetary reward task (Daniel & Pollmann, 2010). This suggests that reward-related processing in the ventral striatum is dependent on the motivational states induced by the specific type of reward. Indeed, when intrinsic motivation is undermined by providing extrinsic rewards, striatal activation to successful task performance is lower (Murayama et al., 2010).

Figure 3. fMRI activation in VS in response to different rewards.

A. With similar reward probabilities, the expectation of monetary reward (€ 0.2) leads to significantly higher activation in the VS than the expectation of cognitive feedback (a green circle signaling the correctness of the response) (Panel adapted from Daniel and Pollmann (2010)). B. After observational training in the complete absence of feedback, the VS responds to a prediction error on confidence, i.e. activation is higher when the participant is more confident than expected that his or her response is correct (Panel adapted from Daniel and Pollmann (2012)).

Summary and caveats

FMRI results show that the human VS responds at the same locus to the anticipation and receipt of a wide range of rewards, spanning the whole spectrum from unconditioned to conditioned reinforcers, including abstract concepts such as beauty and information about the correctness of an answer (for meta-analyses see Bartra et al. (2013); Clithero and Rangel (2013); Kühn and Gallinat (2012)). Yet, not all reinforcers are equivalent, and different types/strengths of reinforcers can induce different motivational states. In general, monetary reward seems to elicit the strongest activations across the reviewed categories. In addition, monetary rewards have also been shown to shift participants to an extrinsically motivated state, which alters both behavior and ventral striatal activation patterns (Murayama et al., 2010; Daniel & Pollmann, 2010). Further research is needed to determine whether these two effects are qualitatively inherent to monetary reinforcement, or whether they can be explained by the high subjective value of monetary rewards.

Given the reliable but varied response of the VS to a wide range of reinforcers, it is tempting to use VS activation strength as an index for the value of a reinforcer. However, as fMRI activation in the VS has also been shown to respond to non-rewarding events such as negative outcomes or saliency (Jensen et al., 2007; Menon et al., 2007; Litt, Plassmann, Shiv, & Rangel, 2011; Metereau & Dreher, 2013; Robinson, Overstreet, Charney, Vytal, & Grillon, 2013; Seymour et al., 2004; Zink, Pagnoni, Chappelow, Martin-Skurski, & Berns, 2006), this approach has to be interpreted with extreme care (Poldrack, 2006) and should always be accompanied by behavioral measures of reinforcer value.

The presented results indicate that the dopaminergic system plays a central role in processing a wide range of stimuli that can act as reinforcers in instrumental conditioning tasks. Recently the dopaminergic system has also been shown to be involved in learning outside the context of instrumental conditioning, e.g. when learning from observation. These results are discussed in the next section.

4.2 Indirect learning: learning from observation and instruction

When learning from a direct interaction with the environment, an agent experiences events and outcomes that are relevant to himself, and, in the case of instrumental learning, takes actions to influence them. However, this learning from trial-and-error can be costly: if one approach provides unsatisfactory results, a new course of action has to be initiated, and sometimes the opportunity to obtain reward or avoid punishment has been permanently lost. Making matters worse, in many environments immediate feedback is not available after each action, or is non-deterministic, so that the exploration behavior has to be repeated many times.

One way to reduce the cost of exploration is to gather information by observing how others act, and what the consequences of their actions are. While this indirect learning will not result in immediate rewards, it can help to learn about the structure of the environment and therefore inform future actions without having to directly engage in costly or dangerous behavior (Rendell et al., 2010). Observational learning has been shown in many species, ranging from penguins to monkeys (Chamley, 2004; Chang, Winecoff, & Platt, 2011; Biederman, Robertson, & Vanayan, 1986; Heyes & Dawson, 1990; Meunier, Monfardini, & Boussaoud, 2007; Myers, 1970; Range, Viranyi, & Huber, 2007; Subiaul, Cantlon, Holloway, & Terrace, 2004). In humans, learning from observing others, i.e. social learning, is considered one of the most central and basic forms of cognition (Bandura, 1977; Frith & Frith, 2012). In addition, as humans are capable of symbolic communication, decision tasks can also be mastered by following the instructions of other, more experienced individuals.

Behaviorally, it is known that observational learning can be less effective than trial-and-error learning for complex multidimensional tasks (Ashby & Maddox, 2005). However, it does reliably increase performance in a wide variety of tasks, including in tasks that are traditionally studied in the context of trial-and-error learning: in a two-armed bandit task observing the actions of a confederate leads to an improvement in the number of optimal choices; additionally observing the confederates’ outcomes increases the effect (Burke et al., 2010). Similarly, participants are able to utilize verbally instructed information on probabilities in a betting game (Li et al., 2011), and can learn to integrate information across different dimensions in a categorization task simply by observing stimulus-stimulus associations (Daniel & Pollmann, 2012). Recently it has been suggested that even behavioral automaticity can be induced by observational learning (Liljeholm, Molloy, & O’Doherty, 2012).

The neural networks that underlie observational learning are similar to those that support learning from direct experience (Dunne & O’Doherty, 2013), possibly reflecting an attempt to simulate the observed person’s mental states (Suzuki et al., 2012). However, mixed results have been reported on prediction-error related activity in the ventral striatum during indirect learning: while some studies show normal or slightly weakened responses to observational prediction errors (Bellebaum, Jokisch, Gizewski, Forsting, & Daum, 2012; Daniel & Pollmann, 2012; Kätsyri, Hari, Ravaja, & Nummenmaa, 2013), others show no (Cooper, Dunne, Furey, & O’Doherty, 2012; Suzuki et al., 2012) or inverted (Burke et al., 2010) prediction error related activity.

One possible explanation for these inconsistent results is that the VS responds to errors in reward prediction rather than to the errors themselves: activation in the VS was shown to increase when the agent receives unpredicted reward, irrespective of whether this is caused by erroneous or correct responses of another person (de Bruijn, de Lange, von Cramon, & Ullsperger, 2009). Therefore, it is possible that VS activation in response to positive errors of reward prediction is stronger if the observer is able to benefit from the rewards of the observed agent. In line with this argumentation, ventral striatal reward-related activation was reported when the observer’s rewards were coupled to the observed rewards of another person (Bellebaum et al., 2012; Kätsyri et al., 2013). The usefulness for future own actions of the observed information may also influence VS activation: no activation related to prediction-errors of an observed person has been shown when the task of the observed person is different from the observer’s task (Suzuki et al., 2012). In contrast, studies in which directly relevant information can be gathered via observation show VS responses even in the absence of rewards during observation (Daniel & Pollmann, 2012; Burke et al., 2010). The suggestion that the nucleus accumbens is only involved in observation whenever the observer is learning is also well in line with results showing that electrical stimulation of the nucleus accumbens in mice enhances the acquisition of behavior when learning from observation (Jurado-Parras, Gruart, & Delgado-García, 2012). The results of Li et al. (2011) provide a direct test for this account: fully reliable instructions on reward probabilities decreased VS responses to rewards, potentially because the reward carried no further useful information for learning. This decrease in responsiveness was paralleled by learning rates close to zero and increased negative functional connectivity of the VS with a dorsolateral prefrontal area. Other potential factors that might modulate the usefulness of observed information, and therefore VS implication, include social factors like the trustworthiness of the confederate, and his similarity to the agent himself (Delgado, Frank, & Phelps, 2005; Mobbs et al., 2009).

To investigate whether the VS is involved in task performance after information has been acquired observationally, we trained participants on a categorization task (Daniel & Pollmann, 2012). During training they passively observed category label-stimulus pairs; in this phase VS activation decreased with time, supporting the assumption that the striatum plays a role in encoding novelty (Bunzeck & Düzel, 2006) and stimulus-stimulus associations (den Ouden, Friston, Daw, McIntosh, & Stephan, 2009). In the subsequent retrieval phase, no rewards were obtainable, and no behavioral learning effects were observed. Nevertheless, activation in the VS followed a prediction error on confidence: it increased when the participant was more confident than expected based on the confidence in previous decisions, and decreased when the participant was less confident than expected (see Fig. 2B). This indicates that even in the absence of external feedback, activation in the VS tracks confidence in decisions, which is inherently related to the ability to successfully navigate the task environment to obtain potential rewards.

In summary, the VS is not unconditionally activated during observational learning by simulating the rewards of confederates; rather, activations in response to the confederate’s rewards are modulated by several factors such as implications for the observer himself (Daniel & Pollmann, 2012) or the similarity between the confederate and observer (Mobbs et al., 2009). In contrast, the VS is activated during indirect learning when task-relevant information can be acquired from observing the outcomes of others (Bellebaum et al., 2012; Burke et al., 2010), or even when observing task-relevant stimulus-stimulus associations (Daniel & Pollmann, 2012). During subsequent task performance, the VS might be involved in simulating or anticipating positive outcomes (Daniel & Pollmann, 2012), which are critical processes in supporting correct task performance and goal-attainment (Han, Huettel, Raposo, Adcock, & Dobbins, 2010). Additionally, VS activation in response to outcome information is attenuated when reliable instructed information is available (Li et al., 2011), further supporting the interpretation that the VS is mainly involved in processing task-relevant trial-by-trial information.

5 Summary and future directions

Traditionally, the dopaminergic system with its strong projections to the VS has been assumed to support the acquisition of new behaviors based on trial-by-trial rewards (Ashby & Maddox, 2005; Shohamy, Myers, Kalanithi, & Gluck, 2008). However, recent fMRI studies on human participants show that the VS is activated not only when learning from explicit rewards, but also from a wide range of stimuli that can be considered as feedback, including cognitive information (Daniel & Pollmann, 2010) or highly abstract concepts such as beauty (Salimpoor, Benovoy, Larcher, Dagher, & Zatorre, 2011). In addition, the VS is also implicated when observing other’s actions and other’s feedback, as well as in response to internally generated error signals, indicating that in human learning “reinforcement” has to be considered a very broad term.

Alternatively, the striatum may play a more general role in learning than just supporting the gradual acquisition of stimulus-response contingencies depending on reinforcement. Structures of the dopaminergic system, including the VS, have also been implicated in other forms of learning (Shohamy, 2011), including model-based learning (Daw et al., 2011), the formation and retrieval of episodic memories (Adcock et al., 2006; Han et al., 2010; Tricomi & Fiez, 2008) and complex social behaviors (Izuma et al., 2010; Phan, Sripada, Angstadt, & McCabe, 2010; Tricomi, Rangel, Camerer, & O’Doherty, 2010). Interestingly, recent results indicate that the VS might not only be involved in tracking errors in the prediction of the value of a reward (as would be expected if the VS were merely representing rewards in a common neural currency), but also in processing more complex surprises about the reward’s unique identity (McDannald, 2012).

The summarized results are very suggestive of a central role of the human dopaminergic system in a wide range of learning and decision making tasks. However, for these results to be conclusive, further research has to be conducted on the causal role and nature of the observed fMRI activations. As has been done for the gradual acquisition of stimulus-response associations based on explicit rewards (Ashby & Maddox, 2005; Shohamy et al., 2008), fMRI studies exploring the role of the VS in the complex, partially uniquely human domains summarized here have to be accompanied by PET, patient, and pharmacological studies. This will allow to confirm whether the observed activations in these broader domains are indeed caused by dopaminergic input to the VS, and whether VS involvement is essential for the acquisition of the reported behaviors (Shohamy, 2011). In addition, characterizing the tasks which recruit the striatum is not sufficient to gain a full understanding of the role of the dopaminergic system in learning; to achieve this goal it is also crucial to determine the actual process by which the striatum supports learning, and to explore its interactions with other structures like the hippocampus (Wimmer & Shohamy, 2012) or the orbitofrontal cortex (Klein-Flügge, 2013; Takahashi et al., 2011).

This future research will provide a holistic understanding of how reward-related and reward-independent processes interact to produce the rich behavior that allows us to function in the complex environment we face every day, and of the processes that are disrupted in diseases which impair this functioning.

Highlights.

• The ventral striatum processes reward expectation and reward prediction errors

• This pattern has been shown in many species and can be measured using fMRI

• In humans, simple reinforcers and abstract concepts elicit similar activations

• The ventral striatum is also activated when learning from observing others

• It therefore plays a universal role in processing reinforcement