Basal ganglia circuits for reward value-guided behavior

Abstract

The basal ganglia are equipped with inhibitory and disinhibitory mechanisms that enable to choose valuable objects and actions. Notably, a value can be determined flexibly by recent experience or stably by prolonged experience. Recent studies have revealed that the head and tail of the caudate nucleus selectively and differentially process flexible and stable values of visual objects. These signals are sent to the superior colliculus through different parts of the substantia nigra, so that the animal looks preferentially at high-valued objects, but in different manners. Relying on short-term value memories, the caudate head circuit allows gaze to move expectantly to recently valued objects. Relying on long-term value memories, the caudate tail circuit allows gaze to move automatically to previously valued objects. The basal ganglia also contain an equivalent parallel mechanism for action values. Such flexible-stable parallel mechanisms for object and action values create a highly adaptable system for decision making.

Introduction

The basal ganglia control behavior by disinhibiting desired actions and inhibiting undesired actions (Hikosaka et al 2000). This is possible because the basal ganglia contain serial and parallel inhibitory circuits (Fig. 1). A key circuit consists of serial inhibitory connections whose net effect is to remove inhibitions (i.e., disinhibition) on target areas. In parallel with this “direct pathway”, there are “indirect pathway” and “hyperdirect pathway”, whose net effects are to enhance inhibitions on target areas (Nambu et al 2002). These circuits together are capable of selecting desired actions.

Figure 1. Basal ganglia circuit controlling the initiation of saccadic eye movements.

Some neurons in the monkey caudate nucleus (CD) are excited by visual inputs which originate from the cerebral cortices and other areas. The CD neurons inhibit the tonic activity of substantia nigra pars reticulata (SNr) neurons through direct connections or enhance the tonic activity of SNr neurons through indirect connections. Since the SNr-connection to the superior colliculus (SC) is inhibitory, the direct signal from the CD leads to a disinhibition of SC neurons (as illustrated in the figure on right). On the other hand, the indirect signal from the CD leads to an enhanced inhibition of SC neurons. Arrows indicate excitatory connections (or effects). Lines with rectangles indicate inhibitory connections. Solid and hatched lines indicate direct and indirect connections, respectively.

How then is an action determined to be desired or undesired? Behavioral studies have shown that a fundamental determinant is reward value (Rangel et al 2008): Animals choose actions that will bring rewards and initiate them quickly (Takikawa et al 2002). This is, at least partly, caused by changes in activity of basal ganglia neurons: they often change their sensory or motor activity depending on the value of the expected reward (Hikosaka et al 2006).

It has been suggested that the reward-dependent plasticity of basal ganglia neurons is caused by inputs from dopamine neurons located in the substantia nigra pars compacta (SNc) and ventral tegmental area (Wickens et al 2003; Nakamura & Hikosaka, 2006). They typically encode “reward prediction error”: excitation when the new situation is better than expected, and inhibition when the situation is worse (Schultz 1998). The dopamine-guided plasticity would guide animals to choose actions that lead to better rewards (Schultz et al 1997). Eventually, the amount of reward obtained per choice would approach the maximum.

Yet, learning does not stop there. The chosen action, as it is repeated, becomes more accurate, quicker, stereotyped (Sakai et al 2003), and is eventually carried out automatically (Logan 1985). This is called “habit” or “skill” (Salmon & Butters 1995). This skill learning process guides animals to obtain maximum rewards “per unit time” (rather than “per choice”) (Hikosaka et al 2013). This is critical for survival because the chosen action can be shortened, say, from 10 min to 1 min after learning a skill (Newell & Rosenbloom 1981).

How can the brain embrace these two kinds of learning – choice and skill? Studies on procedural learning have suggested that the brain is equipped with parallel mechanisms, one for choice (initial learning) and the other for skill (late learning) (Hikosaka et al 1999). Underlying neural mechanisms are parallel cortico-basal ganglia and cortico-cerebellar circuits (Middleton & Strick 2000). These data lead to a more basic issue: How does the brain process two kinds of memories: short-term flexible memories and long-term stable memories? We address this question based on recent studies on the basal ganglia, particularly in relation to learning to choose good objects.

Flexible and stable values

Suppose you are in a grocery store trying to buy apples, and you find that there are six kinds of apples. Which would you choose? There are two extreme strategies. The first strategy is to rely on your long-term data: you choose one that you tried before and liked very much. This is easy, but may not be perfect. Your favorite apple might be overripe. You may have no experience with two of the six kinds, and they might be better than your favored one. The second strategy is to ignore the long-term data and rely on short-term data. You may take a sniff of them to estimate their ripeness or touch them to get some idea about their hardness. This short-term data may give you a better estimate of the best apple. But your onsite estimate may not be perfect. So, neither of the two strategies is perfect.

In everyday life we often face a similar kind of decision dilemma (Cohen et al 2007; Daw et al 2006; Evans 2008). Which strategy is better? The answer depends on the reliability of information you have gained. If the grocery store is the one you have been familiar with for ten years, the long-term data may be more reliable than the short-term data. If you are visiting a grocery store abroad, the short-term data may be more reliable. In many cases, however, both the long-term and short-term data are useful. Such interactions of information processing are formalized in Bayesian decision theory (Kording & Wolpert 2006; McNamara et al 2006) (Fig. 2). The scheme shown in Fig. 2 raises the possibility that the brain contains two parallel mechanisms which selectively process the long-term and short-term data. The present article is centered around this hypothesis.

Figure 2. General scheme of decision making.

A proper decision should be based on slowly accumulated and to-be-retained data (long-term data) as well as quickly acquired and to-be-erased data (short-term data). One interpretation, following Bayesian theory, may be that long-term data, short-term data, and decision correspond respectively to prior, likelihood, and posterior.

To test the hypothesis, we need to identify brain areas that process information selectively for the long-term or short-term data. This question turns out to be difficult since the three kinds of information co-vary. In particular, the short-term data affects the decision, which then affects the long-term data (Hikosaka et al 1999). Nonetheless, we came up with two experimental procedures in which the co-variation was minimized (Kim & Hikosaka 2013; Yamamoto et al 2013; Yasuda et al 2012) (Fig. 3).

Figure 3. Experimental procedures to selectively activate the flexible and stable value mechanisms.

(a) Flexible value procedure. Fractal objects change their values across blocks of trials. The values of objects are learned and tested in the same block. (b) Stable value procedure. Fractal objects stably retain their values during learning across days. The values of objects are then tested on separate days. During the test session, the objects are not associated with their stable values.

The first procedure is to let the short-term mechanism operate, while minimizing the operation of the long-term mechanism (Fig. 3a). Each object changed its associated reward value between high and low in blocks of trials, creating short-term data. The subject needs to keep track of the value change so that, if more than one object are presented, the subject can choose the object that has recently been high-valued. Due to the frequent reversals of the object values, the long-term mechanism should operate minimally. We call it a flexible value procedure, as the values of individual objects change flexibly.

The second procedure is to let the long-term mechanism express its signals, while minimizing the operation of the short-term mechanism (Fig. 3b). Each object is associated with a fixed value (i.e., high or low) during long-term training across several days, creating long-term data. Some time after that, the objects are presented, but reward is not given or given in a non-contingent manner. Hence, the short-term mechanism should not operate systematically. We call it a stable value procedure, as the values of individual objects remain stable.

Anatomical studies have suggested that the brain consists of parallel distributed networks (Goldman-Rakic 1988; Mishkin et al 1983). This feature is conspicuously shown for the connections between the cerebral cortex and the basal ganglia (Alexander et al 1986). Based on the experiments using the flexible and stable value procedures, we suggest that different regions of the caudate nucleus in the basal ganglia may serve the long-term and short-term mechanisms.

Differential value coding in caudate nucleus

Many neurons in the monkey basal ganglia respond to visual stimuli (Brown et al 1995; Caan et al 1984; Hikosaka & Wurtz 1983a; Joseph & Boussaoud 1985; Matsumura et al 1992; Sato & Hikosaka 2002). Visual signals are particularly prominent in the caudate nucleus (CD) throughout its rostro-caudal extent: head (CDh) - body (CDb) - tail (CDt) (Fig. 4a). Notably, however, the CD visual responses are often dependent on the behavioral context, especially memory and reward value (Hikosaka et al 1989; Kawagoe et al 1998).

Figure 4. Differential encoding of flexible and stable values in subregions of the caudate nucleus.

(a) Anatomy of the caudate nucleus and its subregions in the macaque monkey. (b) Average responses to the flexibly valued objects of neurons in the CDh, CDb, and CDt. Neuronal responses were averaged for the neurons” preferred values (magenta) and non-preferred values (black) using a cross-validation method. The yellow line indicates the difference between the preferred and non-preferred responses (mean ± SE). (c) Average responses to the stably valued objects in the three caudate subregions. (d) Proportions of flexible value coding neurons in the three caudate subregions. (e) Proportions of stable value coding neurons.

Using the flexible-stable value procedures (Fig. 3), these CD visual responses were categorized into two types (i.e., flexible vs. stable value coding) in a regionally distinct manner (Kim & Hikosaka 2013) (Fig. 4). In the flexible value procedure, CDh neurons in general showed clear value-based modulations, whereas CDt neurons, as a population, showed no value modulations (Fig. 4b). In contrast, in the stable value procedure, CDt neurons showed clear value modulations, whereas CDh neurons showed only weak value modulations (Fig. 4c). CDb neurons, as a population, showed both flexible and stable value modulations. The proportion of neurons showing significant value modulations were distributed in opposite gradients across the CD subregions (Fig. 4d and e).

To rephrase the data, CDh neurons overall showed value-differential responses in the flexible value procedure, but not the stable value procedure, suggesting that they serve as the short-term mechanism (Fig. 2). In contrast, CDt neurons showed value-differential responses in the stable value procedure, but not the flexible value procedure (Yamamoto et al 2013), suggesting that they serve as the long-term mechanism (Fig. 2).

Note that the CDb, which sits between the CDh and CDt, as a whole encodes both flexible and stable value (Kim & Hikosaka 2013). Then, the subregional differences in the CD may reflect a gradient of functionality, rather than dichotomy. A key factor may be learning speed: CDh neurons learn object values quickly across trials, whereas CDt neurons learn object values slowly across days; CDb neurons may have intermediate learning speeds. If so, the scheme in Fig. 2 may need to be revised: there may be more than two memory mechanisms with different learning speeds (Kording et al 2007).

These data suggest that different parts in the CD convey different kinds of information that could together guide decision making properly in the flexible as well as stable value condition. Obviously, decision making needs to be expressed as motor outputs. When you encounter a good (high-valued) object, you will orient your gaze by making a saccadic eye movement to the object and probably approach it (Awh et al 2012; Tatler et al 2011). This value-based orienting is universal among animals (Bromberg-Martin et al 2010) and crucial for survival (Hikosaka et al 2013). Indeed, gaze (or attention) is strongly attracted by objects with high reward values and this fundamental decision making is regulated by the CD, as shown in the following sections.

Oculomotor expression of object values

Primates (humans and monkeys) heavily rely on saccadic eye movements. In front of a crowed scene, our gaze jumps between objects with saccades (Henderson 2003; Land 2006; Sheinberg & Logothetis 2001), but the pattern of saccades varies depending on internal thoughts and goals (Yarbus 1967). When we perform everyday motor activities, our gaze jumps to an object of interest before our hand reaches out to it (Johansson et al 2001). Macaque monkeys behave similarly (Miyashita et al 1996). In our daily or professional routines, we can easily find objects relevant to our goals (Chun & Nakayama 2000; Rothkopf et al 2007). Similar skills develop while performing psychophysical tasks (Ahissar & Hochstein 1996; Karni & Sagi 1993; Seitz et al 2009; Shiffrin & Schneider 1977; Sigman & Gilbert 2000). A common feature among these behaviors is that particular objects (among many others) are assigned high values in each context. In human psychophysical experiments, when a visual object is associated with a reward, attention (Anderson et al 2011; Awh et al 2012; Chelazzi et al 2013; Kristjansson et al 2010) as well as gaze (Anderson & Yantis 2012; Theeuwes & Belopolsky 2012) is attracted if the object appears subsequently. Intrinsically valuable stimuli (e.g., faces) attract gaze with faster saccades (Xu-Wilson et al 2009).

The gaze of macaque monkeys is also strongly affected by reward values. Suppose the saccade to one position (i.e., left) is followed by a large reward, while the saccade to another position (i.e., right) is followed by a small reward. The positional value bias induces a consistent bias in saccades: quicker (i.e., shorter reaction times) and faster (i.e., larger peak velocity) to the high-valued than low-valued position (Takikawa et al 2002). If the position-reward contingency is reversed, the saccade bias reverses within several trials (Lauwereyns et al 2002). In the flexible value procedure shown in Fig. 3a, similarly, saccades are quicker and faster to the high-valued than low-valued object (Kim & Hikosaka 2013; Yamamoto et al 2013; Yasuda et al 2012). If the two objects are presented simultaneously, the monkey chooses the high-valued object. These experiments examined oculomotor decisions based on flexible values that represent the short-term data (Fig. 2).

In the real-life situation, however, decisions are sometimes better guided by the long-term data (belief, intuition, or skill) (Bechara et al 1997; Marewski et al 2010) which emerges through long-term learning (Crossman 1959; Karni & Sagi 1993; Newell & Rosenbloom 1981). Robust memories are created by associating particular objects or actions (among others) with a consistently high value (Shiffrin & Schneider 1977). Gaze or attention is highly sensitive to such consistent object-value association (Bichot & Schall 1999; Kyllingsbaek et al 2001; Peck et al 2009). This is shown more systematically using the stable value procedure (Fig. 3b). When stably valued objects (some high-valued, the others low-valued) were presented, the monkeys looked at high-valued objects preferentially (Kim & Hikosaka 2013; Yamamoto et al 2013; Yasuda et al 2012) (Fig. S1). There are several important features about the value-based gaze bias. First, the gaze bias occurred even though no reward was given after the free viewing. Second, the gaze bias required 4-5 daily learning sessions to fully develop. Third, the gaze bias remained significant even long after the learning had stopped (Yasuda et al 2012).

These findings indicate that the values of objects influence gaze as well as attention, regardless of whether the values are changing flexibly or have been established stably after prolonged experience. This conclusion, in relation to the scheme in Fig. 2 and the findings in Fig. 4, leads to the following hypothesis: the oculomotor decision is controlled both by the CDh which represents flexible object values and by the CDt which represents stable object values (Kim & Hikosaka 2013).

Oculomotor mechanisms in the basal ganglia

We have suggested that biased object values elicit differential responses in CD neurons, which then lead to differential oculomotor responses. Indeed, the CD has a strong access to oculomotor mechanisms. First, electrical stimulation of the CDh and CDb (Kitama et al 1991; Watanabe & Munoz 2011) as well as CDt (Yamamoto et al 2012) facilitates the initiation of saccades. CDh-CDb stimulation also suppresses saccades (Watanabe & Munoz 2010). Second, dopamine depletion in the visual-oculomotor region of the CDh-b in the monkey induces a severe hemineglect (Kato et al 1995; Miyashita et al 1995). Finally, neural circuitries originating from the CD to the oculomotor outputs have been shown anatomically and physiologically, as described below in detail.

In general, the basal ganglia control body movements by changing the level of inhibitions on target motor structures (Wichmann & DeLong 1996). This explains why dysfunctions of the basal ganglia often show up as a too strong inhibition (e.g., akinesia) or a lack of inhibition (e.g., chorea). This principle is clearly implemented in the oculomotor mechanism in the basal ganglia (Fig. 1). The direct control over saccade initiation is achieved by the GABAergic connection from the substantia nigra pars reticulata (SNr) to the superior colliculus (SC) (Hikosaka & Wurtz 1983b). SNr neurons fire continuously with high frequencies, thus tonically inhibiting saccadic neurons in the SC.

In normally behaving animals, the SNr-SC tonic inhibition is reduced occasionally and temporarily by the direct GABAergic inhibitory inputs from the striatum (mostly CD) (Hikosaka et al 1993) (Fig. 1), but in particular behavioral contexts (Hikosaka et al 2006; Hikosaka et al 2000). This causes a disinhibition of SC neurons, allowing a saccade to be generated. In addition to the direct pathway, the CD has indirect pathways to control the SNr, which are mediated by the external segment of the globus pallidus (GPe) and possibly the subthalamic nucleus (STN) (Alexander & Crutcher 1990). Since GPe neurons are GABAergic and inhibitory, the net effect through this indirect pathway is largely facilitatory on SNr neurons (Fig. 1) and hence suppressive on saccade initiation. Thus, the combination of the direct and indirect pathways seems ideal for selective choice: direct pathway for approach and indirect pathway for avoidance (Hikosaka et al 2000; Mink 1996).

Basal ganglia guide value-oriented gaze

The basal ganglia have two output nuclei, the internal segment of the globus pallidus (GPi) and the SNr, but only the SNr projects to the SC (Marín et al 1998). Moreover, only a sub-population of SNr neurons project to the SC (Beckstead & Frankfurter 1982; Francois et al 1984; Parent et al 1983). In other words, the oculomotor function of the basal ganglia is largely carried out by the particular SNr neurons projecting to the SC. In the macaque monkey, a majority of the SC-projecting SNr neurons are localized in the dorsolateral portion of the SNr (Beckstead & Frankfurter 1982; Francois et al 1984). Notably, the CDt projects focally to the dorsolateral SNr (Saint-Cyr et al 1990). Indeed, a majority of neurons in the dorsolateral SNr project to the SC and encode stable values of fractal objects similarly to CDt neurons (Yasuda et al 2012) (Fig. 5).

Figure 5. High capacity memory of stable values encoded by SNr neurons.

(a) The locations of the CDt and SNr(p) shown on a coronal Nissl-stained section. The CDt (red) has a direct inhibitory connection to the dorsolateral SNr(p) (yellow) which then inhibits presaccadic neurons in the SC. (b-top) The responses of an SC-projecting SNr(p) neuron to 120 well learned objects (c). It was inhibited by most high-valued objects (red) and excited by most low-valued objects (blue). (b-bottom) The average responses of 151 SNr(p) neurons to high-valued objects (red) and low-valued objects (red) which were chosen randomly from about 300 well learned objects.

The SC-projecting SNr neurons were inhibited by stably high-valued objects and excited by stably low-valued objects (Fig. 5b). The inhibition may be due to direct inputs from the CDt, while the excitation may be due to indirect inputs from the CDt through the GPe (i.e., disinhibition). The categorical responses of SNr neurons can be explained by the following scheme: CDt neurons coding positive values (i.e., stronger responses to high-valued objects, Fig. 4e, red) project to the SNr directly, while CDt neurons coding negative values (Fig. 4e, blue) project to the SNr through the GPe. This hypothesis remains to be tested.

Functionally, when a stably high-valued object appears, SNr neurons are largely inhibited, and therefore SC saccadic neurons are disinhibited, thus facilitating saccades to the object (Fig. 5a). When a stably low-valued object appears, SNr neurons are largely excited, and therefore SC saccadic neurons are further inhibited, thus suppressing saccades to the object. Indeed, the monkeys look at stably high-valued objects intensely, while avoiding low-valued objects (Kim & Hikosaka 2013; Yamamoto et al 2013; Yasuda et al 2012) (Fig. S1).

The above results suggest that the SNr is the mediator of stable value signals originating from the CDt. However, previous studies have shown that the SNr is also a major mediator of signals from the CDh and CDb (Hikosaka et al 1993). Then, why do SNr neurons encode stable values, not flexible values? Preliminary observation from our lab suggests that there are in fact flexible value-coding neurons in the SNr, but they are located more anterior-medial-ventrally, largely separate from the stable value-coding neurons (Yasuda & Hikosaka 2013). Thus, the anterior SNr, or SNr(a), would encode flexible values, and the posterior SNr, or SNr(p) would encode stable values. The segregation is consistent with anatomical (Saint-Cyr et al 1990; Smith & Parent 1986) and electrophysiological (Hikosaka et al 1993) data. If the result proves true, it would indicate that the flexible value (short-term) mechanism and the stable value (long-term) mechanism are segregated from the input (CD) to the output (SNr) of the basal ganglia (Fig. 6). One interpretation, following Bayesian scheme (Fig. 2), may be that the CDh-SNr(a)-SC circuit serves the likelihood mechanism while the CDt-SNr(p)-SC circuit serves the prior mechanism.

Figure 6. Parallel CD-SNr-SC circuits underlying value-based decision making.

(a) Anatomical scheme. The CDh receives inputs mainly from the frontal cortical areas (Yeterian and Pandya, 1991), while the CDt receives inputs mainly from the temporal cortical areas (Saint-Cyr et al., 1990). The CDh and CDt have equivalent downstream mechanisms (as shown in Figure 4), but use different neural circuits before reaching the SC. (b) Information processing along the parallel CD-SNr-SC circuits which emulates the general scheme of decision making in Figure 1. These circuits have contrasting characteristics in terms in terms of memory and motor output mechanisms.

Memories required for the stable value mechanism

To make a decision you can rely on the short-term data only, long-term data only, or both (Fig. 2). If you rely only on long-term data, you need to maintain the long-term memory about each of all objects, actions, and events you have ever experienced in your life. In other words, the stable value mechanism must have a high capacity memory which is retained for a very long time, as suggested previously (Chun & Nakayama 2000) (Fig. 6b). Indeed, these memory features are what characterize neurons in the CDt and the SNr(p), which we suggested to serve as the stable value mechanism (Fig. 6).

As exemplified in Fig. 5, SNr(p) neurons virtually categorized many visual objects (up to 300 so far) based on their stable values (high capacity memory). Some of the stably valued objects were removed from the learning schedule and after more than 100 days they were shown to the monkey. SNr(p) neurons still categorized them almost perfectly (long-term retention). CDt neurons seem to have similar memory features (Yamamoto et al 2013). As expected from the SNr(p) neurons” high capacity memories, the monkey preferentially looked at stably high-valued objects which were chosen from about the 300 objects, even after more than 100 day retentions (Yasuda et al 2012).

There are two additional, but notable, features. First, stable values are learned slowly (Fig. 6b): the value-differential responses of CDt and SNr(p) neurons as well as the monkey”s preferential looking developed gradually across 4-5 daily sessions (Kim & Hikosaka 2013; Yasuda et al 2012). Second, stable values are signaled quickly. CDt and SNr(p) neurons started signaling the stable value of an object about 100 ms after the object appeared, which allowed the monkey to look at or look away from the object (Kim & Hikosaka 2013; Yamamoto et al 2013; Yasuda et al 2012). It seems unlikely that such a quick response is based on conscious thought which is characterized by sequential processing and is hence time-consuming (Sternberg 1969; Treisman & Gelade 1980). It rather suggests that the stable value-based responses occur subconsciously and automatically (Fig. 6b). Indeed, the categorization or identification of visual objects that requires long-term learning is performed subconsciously (Chun & Jiang 1998; Shiffrin & Schneider 1977; Sigman & Gilbert 2000). Lastly, the quick and automatic response based on stable values is an essential skill to survive in the competitive world (Hikosaka et al 2013).

Is the stable value mechanism essential, and if so how? Let us think about two situations: 1) choose among novel objects, and 2) choose among familiar objects (Fig. S2) (Kim & Hikosaka 2013). On each trial, 2 objects are presented among a set of 8 objects, and the monkey has to choose one. When all were novel objects (Fig. S2a), the monkey gradually learned to choose the high-valued object, while CDh neurons gradually differentiated objects by their flexible values. But CDt neurons showed no differential responses even after 100 trials. Thus, only the flexible value mechanism is necessary in a novel context. When all objects were familiar (previously well learned) (Fig. S2b), from the first trial, the monkey chose the high-valued object, while CDt as well as SNr(p) neurons differentiated all the objects by their stable values. But CDh neurons initially showed no differential responses. Therefore, the stable value mechanism is essential when previously well-learned objects are unexpectedly presented.

Controlled vs. automatic saccades

The schemes in Fig. 6 suggest that the CDh guides saccades based on the flexible values of target objects, while the CDt guides saccades based on the stable values of target objects. This hypothesis was supported by inactivating each of the CDh and CDt by injecting a GABA_A agonist (muscimol) (Kim & Hikosaka 2013). During inactivation of the CDh, the flexible value information largely lost its ability to affect saccades (Fig. 7b center). During inactivation of the CDt, the stable value information lost its ability to affect saccades (Fig. 7c bottom). In contrast, saccades remained affected by the stable value information during the CDh inactivation (Fig. 7c center) and by the flexible value information during the CDt inactivation (Fig. 7b bottom).

Figure 7. Differential impairments of controlled and automatic saccades by CDh and CDt inactivations.

(a) Injection sites of muscimol in the caudate nucleus: CDh (top) and CDt (bottom). (b) Effects on the controlled saccades that were predictively influenced by reward feedback in the flexible value procedure (Figure 2a). The differences in the target acquisition time between high- and low-valued objects are plotted before and during inactivation (mean ± SE). Data are shown for CDh inactivation (center) and CDt inactivation (bottom). (c) Effects on the automatic saccades that occurred without reward feedback but were influenced by the stable value of the presented object (Figure 2b). The differences in the probability of automatic looking between high- and low-valued objects are plotted before and after inactivation (mean ± SE). Same format as in (b). The effects are shown only for contralateral saccades.

From the viewpoint of motor output, saccades have different natures depending on whether they are guided by the CDh or CDt. As partially discussed in the preceding section, the saccades guided by the CDh can be characterized as “controlled” (Fig. 6a), because they are sensitive to the immediate reward outcome (Fig. 7). In contrast, the saccades guided by the CDt can be characterized as “automatic” (Fig. 6a), because they occur even though no reward was given after the saccades (Fig. 7).

These results are largely consistent with previous studies suggesting that different parts of the basal ganglia support “goal-directed behavior” and “habits” (Redgrave et al 2010; Yin & Knowlton 2006). In particular, the role of the CDt in automatic saccades supports the seminal research by Mishkin and colleagues suggesting that the connection from the temporal cortex to the CDt serves visual habits (Fernandez-Ruiz et al 2001; Mishkin et al 1984).

Object value vs. action value

Goal-directed behavior consists of at least two stages: 1) find good objects, and 2) manipulate the good objects to reach a goal (i.e., reward) (Hikosaka et al 2013). Each stage is driven by value: object value and action value (Rudebeck et al 2008). We have discussed that object values can be flexible or stable, and these two kinds of object values are processed separately in the basal ganglia. Previous studies have suggested that a similar functional organization is present for action values (Hikosaka et al 1999). Neurons in the anterior striatum (mostly CDh) were more active when the monkey learned a new sequential action (i.e., when action value was flexible), while neurons in the posterior striatum (mostly putamen) tended to be more active when the monkey performed well learned sequences (i.e., when action value was stable) (Miyachi et al 2002). Inactivation of the anterior striatum disrupted new learning, while inactivation of the posterior striatum disrupted the well learned performance (Miyachi et al 1997).

These results together suggest that, for both object and action values, the anterior striatum serve as the flexible value mechanism, while the posterior striatum serve as the stable value mechanism (Fig. 8). A key factor underlying this difference would be the speed of learning and forgetting. Studies on human motor learning suggest that the combination of the quick and slow learning mechanisms make the whole motor learning robust, as these mechanisms respectively act as the flexible and stable value mechanisms (Kording et al 2007).

Figure 8. Hypothetical parallel mechanisms controlling behavior based on object values and actions values.

The first mechanism aims at finding good objects, while the second mechanism aims at manipulating the objects. These mechanisms together enable animals and humans to gain access to rewards efficiently. For both mechanisms, the anterior part of the basal ganglia processes flexible values and guides controlled behavior, while the posterior part of the basal ganglia processes stable values and guides automatic behavior. However, they may not share the same neural circuits within the basal ganglia. Other brain structures may also constitute the mechanisms, especially those connected with the basal ganglia including the cerebral cortex (not shown).

Basal ganglia dysfunctions

The results discussed above are relevant to the clinical observation that humans with basal ganglia dysfunctions show impairments in controlled behavior and/or automatic behavior (Brown et al 1997; Lieberman 2000). Impairments in cognitive tasks, such as set-shifting and value reversal, are observed in patients with Parkinson”s disease (Brown & Marsden 1990; Cools et al 1984; Kehagia et al 2010; Lees & Smith 1983), Huntington”s disease (Lawrence et al 1996), focal lesions of the basal ganglia (Bhatia & Marsden 1994; Caplan et al 1990). On the other hand, patients with Parkinson”s disease may lose the ability to perform behaviors automatically (Marsden 1982; Redgrave et al 2010). They may have difficulty in carrying out two motor actions simultaneously (Benecke et al 1986; Schwab et al 1954). They are also impaired in performing tasks that require visual skills (Ashby & Maddox 2005; Knowlton et al 1996). These results are difficult to interpret if we view that the basal ganglia have a single function. Instead, they may reflect parallel functions discussed in this article: different parts of the basal ganglia process flexible and stable values of objects as well as actions relying on short- and long-term memories and guide behavior in a controlled as well as automatic manner.

Conclusions

To make a good decision, we need to consider both recently acquired data and previously accumulated data. This is an important and demanding task for the brain, but underlying mechanisms are unclear. In this article we have shown recent evidence that the CDh and CDt process information selectively and separately for the recently acquired data (flexible values of visual objects) and for the previously acquired data (stable values of visual objects). Such a parallel mechanism is imperative because flexible value information must be erased quickly while stable value information must be retained for a long time. Indeed, the two kinds of information are processed in separate circuits downstream to the CDh and CDt, through different parts of the SNr, until they reach the SC, influencing the initiation of saccades. The CDh-flexible and CDt-stable circuits would work in a complementary manner: a proper estimate of object values can be provided only by the CDh-flexible circuit if the objects are novel, only by the CDt-stable circuit if the objects are familiar but have not been experienced for some time, or otherwise by a compromise between these circuits.

These findings raise many questions. First, do the flexible and stable value signals interact with each other, and if so how? According to the scheme described above, the SC is the only place where these value signals are integrated. This seems unlikely, however. Stable values would serve as the prior information from which any neurons involving decision making would benefit. This might be achieved by the outputs of SNr neurons to the thalamus and then to many cortical and subcortical areas (Lynch et al 1994; Middleton & Strick 1996; Tanibuchi et al 2009). Second, how do CDh and CDt circuits selectively and differentially acquire flexible and stable values? Somewhere along each circuit, information on each visual object must be modified by the associated reward values. Dopamine may play a key role in this process, since neurons along the CDh-CDt circuits receive massive inputs from midbrain dopamine neurons which carry reward values signals (Schultz 1998). How then can dopamine neurons support short-term learning in the CDh circuit and long-term learning in the CDt circuit? Or, do different neuromodulatory mechanisms work for the CDh-CDt circuits? These questions may guide us to outstanding issues in neuroscience.

Acronyms and Definitions list

CD

caudate nucleus

CDh

head of caudate nucleus

CDb

body of caudate nucleus

CDt

tail of caudate nucleus

SC

superior colliculus

SNr

substantia nigra pars reticulata

GPe

external segment of globus pallidus

STN

subthalamic nucleus

GPi

internal segment of globus pallidus

SNr(a)

anterior portion of substantia nigra pars reticulata

SNr(p)

posterior portion of substantia nigra pars reticulata

GABAergic

transmitting or secreting γ-aminobutyric acid

Disinhibition

Removal of a sustained inhibition, causing an increase in excitability

Muscimol

an alkaloid extracted from the poison mushroom Amanita muscaria that selectively stimulates GABA_A receptors