Parallel basal ganglia circuits for voluntary and automatic behaviour to reach rewards

The contributions of topographically distinct basal ganglia circuits to reward-oriented behaviours are unclear. Kim and Hikosaka propose that parallel circuits support flexible voluntary behaviours required to obtain rewards, and stable automatic behaviours involved in manipulation of reward objects. This distinction may inform the diagnosis and treatment of basal ganglia disorders.

The contributions of topographically distinct basal ganglia circuits to reward-oriented behaviours are unclear. Kim and Hikosaka propose that parallel circuits support flexible voluntary behaviours required to obtain rewards, and stable automatic behaviours involved in manipulation of reward objects. This distinction may inform the diagnosis and treatment of basal ganglia disorders.

Abstract

The basal ganglia control body movements, value processing and decision-making. Many studies have shown that the inputs and outputs of each basal ganglia structure are topographically organized, which suggests that the basal ganglia consist of separate circuits that serve distinct functions. A notable example is the circuits that originate from the rostral (head) and caudal (tail) regions of the caudate nucleus, both of which target the superior colliculus. These two caudate regions encode the reward values of visual objects differently: flexible (short-term) values by the caudate head and stable (long-term) values by the caudate tail. These value signals in the caudate guide the orienting of gaze differently: voluntary saccades by the caudate head circuit and automatic saccades by the caudate tail circuit. Moreover, separate groups of dopamine neurons innervate the caudate head and tail and may selectively guide the flexible and stable learning/memory in the caudate regions. Studies focusing on manual handling of objects also suggest that rostrocaudally separated circuits in the basal ganglia control the action differently. These results suggest that the basal ganglia contain parallel circuits for two steps of goal-directed behaviour: finding valuable objects and manipulating the valuable objects. These parallel circuits may underlie voluntary behaviour and automatic skills, enabling animals (including humans) to adapt to both volatile and stable environments. This understanding of the functions and mechanisms of the basal ganglia parallel circuits may inform the differential diagnosis and treatment of basal ganglia disorders.

Introduction

The basal ganglia control body movements. This is a long-standing concept that has been confirmed repeatedly by animal experiments and human movement disorders. Experimental lesions of various parts of the basal ganglia impair the initiation, execution, and inhibition of spontaneous and planned body movements (Kennard, 1944; Crossman, 1987). Pathological lesions of the human basal ganglia are associated with a variety of movement disorders, including involuntary movements (Denny-Brown, 1968; Albin et al., 1989; Bhatia and Marsden, 1994). A number of neurodegenerative diseases involve the basal ganglia. The most common among them is Parkinson’s disease, which is typically associated with tremor, bradykinesia (akinesia), and rigidity (Crossman, 1987; Jankovic, 2008). Overall, basal ganglia dysfunctions lead to a remarkable set of movement disorders, suggesting that the basal ganglia contain multiple mechanisms for controlling body movements.

However, animals and humans with basal ganglia dysfunctions show deficits that may not simply be classified as movement disorders. For example, animals with large lesions in the striatum may ignore a moving object or obsessively follow it (Denny-Brown, 1962). Patients with Parkinson’s disease may have difficulty in performing two movements simultaneously (Schwab et al., 1954) or in learning of probabilistic classification (Knowlton et al., 1996). Dopamine deficiency in the striatum leads to severe contralateral hemi-neglect in monkeys (Miyashita et al., 1995). Humans with restricted lesions in the caudate nucleus or the globus pallidus may be unable to initiate everyday behaviours spontaneously (Laplane and Baulac, 1984; Caplan et al., 1990). Patients with Parkinson’s disease may be less motivated in achieving goals and may also show symptoms of depression (Pluck and Brown, 2002).

These observations, as well as many others not described here, suggest that the basal ganglia are involved in many mental processes, including motor, sensory, learning, memory, cognitive, executive, decision-making, motivational, and emotional functions.

Such diverse functionality makes it challenging to identify the essential principles of the basal ganglia function. One strategy for approaching this complexity in a tractable manner is to focus on one aspect of behaviour in which the basal ganglia are deeply involved. To this end, we chose ‘reward-oriented behaviour’ which is likely to require all the mental processes listed above. By discussing reward-oriented behaviour in view of diverse mental processes, we can cover a wide range of the diverse functionality, and do so in a logically connected manner. Conversely, by discussing each mental process in relation to reward-oriented behaviour, we may be able to reveal new aspects of the mental process.

To approach this general goal, we first discuss general features of basal ganglia circuits and functions for reward-oriented behaviour, and then discuss a new feature of basal ganglia function mostly by focusing on our recent studies. The last section is devoted to basal ganglia dysfunctions that are related to reward-oriented behaviour.

Because we will focus on reward-oriented behaviour, especially in relation to our study, this article may not cover some of the important concepts of the basal ganglia and related literature. Instead, we explore the literature widely in relation to reward-oriented behaviour, some of which have been rarely discussed in basal ganglia research. This exploratory approach may raise more questions and therefore our discussions may often be speculative. By doing so, however, we hope this article will trigger discussions among readers, potentially leading to novel concepts of the basal ganglia.

Basal ganglia circuits for selection of behaviour

The basal ganglia play an important role in the selection of behaviour. This hypothesis is supported by their parallel organization of their component circuitry—direct, indirect, and hyperdirect pathways (Fig. 1A). We first discuss how these pathways may work.

Figure 1.

Basal ganglia circuits for reward-oriented learned behaviour. (A) Direct, indirect and hyperdirect pathways. The striatum receives inputs mainly from the cerebral cortex. D1R-expressing neurons in the striatum connect to SNr/GPi directly (direct pathway). D2R-expressing neurons connect to SNr/GPi indirectly through GPe and STN (indirect pathway). STN receives inputs directly from the cerebral cortex and send outputs to SNr/GPi (hyperdirect pathway). Dopaminergic neurons in SNc/VTA heavily innervate the striatum. (B) Two processes for skill behaviour. Finding a high-valued object among many objects requires object skill, and manipulating the high-valued object requires action skill. D1R = dopamine receptor D1; D2R = dopamine receptor D2.

Direct, indirect and hyperdirect pathways in basal ganglia

The brain contains many motor pattern generators, each of which is devoted to a particular body movement (Grillner et al., 1998) such as gaze orienting (Sparks, 2002), locomotion/posture (Takakusaki et al., 2004), vocalization (Hage and Jurgens, 2006), reaching/grasping (Kinoshita et al., 2012), and eating/drinking (Nakamura and Katakura, 1995). Triggered by particular sensory inputs or internal states, these mechanisms can work independently to generate adaptive movements (e.g. vestibulo-ocular reflex). However, the whole behaviour could become uncontrollable if these motor mechanisms are allowed to be active by simply following their own rules.

How then can the brain solve the uncontrollable situation? An efficient way would be to set up a mechanism to suppress all of the motor mechanisms. The basal ganglia appear to perform this function. Their final output neurons are all GABAergic and inhibitory, are highly active continuously (Hikosaka, 2007a), and are connected to these motor mechanisms (Takakusaki et al., 2004; Grillner et al., 2005). They are located in two structures: substantia nigra pars reticulata (SNr) and globus pallidus internal segment (GPi) (Hikosaka, 2007a) (Fig. 1A). Indeed, humans and monkeys with basal ganglia dysfunction often show involuntary movements (Crossman, 1987; DeLong and Wichmann, 2007), which may be caused by disruption of SNr/GPi-mediated inhibition (Hikosaka and Wurtz, 1985).

However, the SNr/GPi-induced inhibitions must be weakened in certain contexts. Otherwise, all movements would remain suppressed, which may be a cause of akinesia in patients with Parkinson’s disease (Wichmann and DeLong, 1996). The weakening occurs via GABAergic inhibitory connections from the striatum (direct pathway) (Chevalier and Deniau, 1990), and the net effect is a reduction of inhibition (i.e. disinhibition) (Hikosaka et al., 2000). In addition, SNr/GPi neurons receive indirect inputs from the striatum via the globus pallidus external segment (GPe) and possibly the subthalamic nucleus (STN) (indirect pathway) (Smith et al., 1998). As both striatal output neurons and GPe neurons are GABAergic and inhibitory, the net effect may be an enhancement of inhibition. The combination of the direct and indirect pathways can, theoretically, make a selection among a repertoire of body movements, which may be called ‘motor action’ (Mink, 1996; Hikosaka et al., 2000; Hikida et al., 2010; Kravitz et al., 2010).

The hyperdirect pathway (Nambu et al., 2002) appears to act as a prominent suppressor of ongoing body movements. It is mediated by the STN, which consists of glutamatergic excitatory neurons (unlike most neurons in the basal ganglia) (Robledo and Feger, 1990) and transmits signals quickly from the cerebral cortex to SNr/GPi (Nambu et al., 2000), thereby suppressing body movements. Its major function seems to be behavioural switching (Aron and Poldrack, 2006; Isoda and Hikosaka, 2008), in that it suppresses quick and automatic movements so that slow and voluntary movements can be initiated. Damage of STN thus leads to severe involuntary movements (hemiballismus) (Crossman et al., 1984).

Role of basal ganglia in reward-oriented behaviour

The evidence considered thus far indicates that the basal ganglia are equipped with machinery that is clearly suited to behavioural selection. However, a good mechanism requires a good motive, and a universal motive is reward (Dayan and Balleine, 2002). Indeed, activity of neurons in various parts of the basal ganglia is strongly modulated by reward, especially by the expectation of reward (Hikosaka et al., 1989a; Schultz et al., 1992; Bowman et al., 1996; Kawagoe et al., 1998; Lauwereyns et al., 2002; Sato and Hikosaka, 2002; Takikawa et al., 2002a). A general hypothesis is that the direct pathway mainly processes reward-predicting signals thereby facilitating reward-oriented movements, whereas the indirect pathway mainly processes non-reward-predicting signals thereby suppressing unrewarded movements. These two pathways together thus constitute a reward-oriented motor action (Frank, 2005; Hikosaka, 2007b; Hong and Hikosaka, 2011). Experiments using reward-biased behavioural tasks have yielded results that were consistent with this hypothesis (Nakamura and Hikosaka, 2006; Hikida et al., 2010).

Strong support for the role of the basal ganglia in reward-oriented behaviour derives from the prevalence of dopamine effects among many vertebrate species (Richfield et al., 1987). The striatum, in particular, receives dense projections from dopamine neurons located in the substantia nigra pars compacta (SNc), ventral tegmental area (VTA), and surrounding areas (Haber et al., 2000; Joel and Weiner, 2000; Ikemoto, 2007) (Fig. 1A). These dopamine neurons are sensitive to rewards and typically encoding reward prediction error (RPE), in that they are excited by increases in reward value and inhibited by decreases (Schultz, 1998). The reward-related signals of dopamine neurons may affect the activity and synaptic transmissions of striatal neurons (Nicola et al., 2000; Reynolds and Wickens, 2002; Calabresi et al., 2007; Surmeier et al., 2007; Kreitzer and Malenka, 2008). As individual striatal neurons encode sensorimotor, cognitive or emotional signals (Crutcher and DeLong, 1984; Nishino et al., 1984; Hikosaka et al., 1989b, c; Kimura, 1990; Kimura et al., 1992; Kermadi and Joseph, 1995), their outputs would be modified by the predicted change in reward outcome. This mechanism may underlie the reward-related activity changes in striatal neurons that have been repeatedly observed (Hikosaka et al., 1989a; Apicella et al., 1992; Schultz et al., 1992; Kawagoe et al., 1998).

Notably, dopamine neurons appear to differentially affect two groups of striatal neurons through different receptors: direct pathway neurons through D1 receptors and indirect pathway neurons through D2 receptors (Gerfen et al., 1990). As the effects of these receptors are mostly opposite (i.e. D1: facilitatory, D2: inhibitory) (West and Grace, 2002; Eyny and Horvitz, 2003; Mallet et al., 2006; Nakamura and Hikosaka, 2006; Shen et al., 2008), when reward is expected, the direct pathway neurons would be more active whereas the indirect pathway neurons would be less active (Hong and Hikosaka, 2011). These opposing actions of dopamine within the striatum are consistent with the general hypothesis described above. Accordingly, the loss of dopamine inputs to the striatum would then incapacitate the selection process controlled by the direct and indirect pathways, namely the execution of rewarded actions and suppression of unrewarded actions. This model may explain the pathological consequences of dopamine depletion in Parkinson’s disease, which include loss of control over various body movements including locomotion/posture (Morris, 2000; Jankovic, 2008), vocalization (Robbins et al., 1986), reaching/grasping (Bennett et al., 1995; Morris, 2000), manipulation (Fellows et al., 1998), eye movements (Chan et al., 2005), and eating/drinking (Robbins et al., 1986).

However, motor action by itself is not sufficient to obtain rewards. To obtain a reward, an animal must find the valuable object (e.g. ripe apple) before executing the action (e.g. reach and grasp) (Hikosaka et al., 2013) (Fig. 1B). Finding valuable objects requires sensory information associated with reward values, which may reflect common inputs of the basal ganglia from sensory cortical areas (Kemp and Powell, 1970; Saint-Cyr et al., 1990; Flaherty and Graybiel, 1991). We will focus on this issue in the later part of this article.

Anatomical segregation of basal ganglia

There are two steps to reach rewards: (i) find valuable objects using multiple sensory signals; and (ii) manipulate the objects using multiple motor signals. Furthermore, reward-oriented behaviour needs to be regulated by various factors such as attention, motivation, context, uncertainty, and assessment of risk (Dayan and Balleine, 2002; Doya, 2008; Gottlieb, 2012). All of these signals are relayed to the basal ganglia, primarily in the striatum, which could therefore process them either separately or integratively. A prominent source of these signals is the cerebral cortex, which may in turn give rise to the functional specialization within the striatum: limbic functions more medially versus sensorimotor functions more laterally (Parent, 1990; Brown et al., 1998; Haber et al., 2000). This medial-lateral functional topography is present in both rodents and primates (Yin and Knowlton, 2006; Balleine and O’Doherty, 2010).

The evolution of primates has resulted in a unique topographical organization along the rostral-caudal axis which may reflect further functional specialization: associative versus sensorimotor (Parent, 1990; Lehéricy et al., 2004a, b; Draganski et al., 2008). The rostral part of the putamen and caudate nucleus primarily serve associative (or cognitive) functions by receiving inputs mainly from the prefrontal cortical areas (Fig. 2A). In contrast, their caudal parts primarily serve sensorimotor functions, with the caudal putamen receiving inputs from the skeletal sensorimotor cortical areas (Lehéricy et al., 2004a). Particularly remarkable is the caudate nucleus of primates (monkeys and humans), which runs through the frontal and parietal lobes and finally reaches the temporal lobe. The shape of the caudate changes considerably along the rostral to caudal span of the structure as it progresses through three subregions including the head (caudate head), body (caudate body), and tail (caudate tail) (Fig. 2A). Following this topography, caudate head receives inputs mainly from the frontal cortex (Lehéricy et al., 2004b; Draganski et al., 2008; Haber and Knutson, 2010), whereas caudate tail receives inputs mainly from the temporal cortex (Kemp and Powell, 1970; Yeterian and Van Hoesen, 1978; Saint-Cyr et al., 1990).

Figure 2.

Basal ganglia mechanisms for learning motor actions. (A) Differential cortical inputs to the striatum. The rostral striatum receives inputs mainly from the associative cortical areas. The caudal striatum receives inputs mainly from the sensorimotor cortical areas. The striatum includes the caudate nucleus (grey) and putamen (PUT) (dark grey). Caudate head (CDh), caudate body (CDb), and caudate tail (CDt) belong to the rostral, intermediate, and caudal striatum, respectively. R, C, D, V = rostral, caudal, dorsal, ventral. (B) An example of action skill (tying shoe laces) that consists of a sequence of many movements. (C) Two stages in learning of sequential movements. In the early learning stage, the movements are based on the spatial coordinates of target objects, and therefore are not specific to the learned hand and are performed consciously. In the late learning stage, the movements are based on the motor coordinates of the performing hand, and therefore are specific to the learned hand and are performed subconsciously. (D) Contrasting two separate mechanisms of action skill learning. Voluntary performance, which is learned quickly but is not retained for a long time, is controlled by the rostral basal ganglia circuit together with the association cortex. Automatic performance, which is learned slowly but is retained for a long time, is controlled by the caudal basal ganglia circuit together with the sensorimotor cortex.

These results need not imply that multiple signals are processed separately in the whole basal ganglia. They may converge in the downstream circuits (Fig. 1A) because the downstream areas (i.e. SNr/GPi, GPe, STN) are smaller and contain fewer neurons than the striatum (Percheron et al., 1984; Flaherty and Graybiel, 1993). However, anatomical studies have suggested that each of these areas (e.g. SNr) receives topographically organized inputs from the striatum (François et al., 1994) and likewise sends topographic outputs (Parent, 1990; Hoover and Strick, 1999; Middleton and Strick, 2002). The topographical separation may be maintained through the loop circuit mediated by the thalamus (e.g. basal ganglia – thalamus – cortex – basal ganglia) (Alexander et al., 1986; Middleton and Strick, 1996). As do the targets of the basal ganglia, the cerebral cortex contains diverse circuits that control learned body movements (e.g. fine finger movements) in the motor cortex (Jackson et al., 2006) and learned behavioural control (e.g. context-dependent decision) in the prefrontal cortex (Mansouri et al., 2009; Rushworth et al., 2011). In addition, the basal ganglia send their outputs through SNr/GPi to several brainstem regions (Hikosaka et al., 2000) that contain specific neural circuits devoted to innate body movements (pattern generators) (Grillner, 2003), as described above.

These findings suggest that different parts of the basal ganglia work independently as parallel circuits to control different behaviours. However, this hypothesis faces a basic question: how do these parallel circuits coordinate with each other? This question is important because everyday behaviour is composed of multiple motor and mental activities and yet is directed toward beneficial outcomes. In the rest of this article, we will discuss how the multiple circuits in the basal ganglia contribute to two fundamental behaviours—sequential motor learning and visual object-value learning—both of which are commonly used in our daily life.

Learning of sequential motor action

Action skill and basal ganglia

In daily life, we often physically manipulate objects. When you go out, you may change your shirt, button the shirt, wear shoes, tie shoelaces, open the door, walk out, close the door, lock the door, and walk to your car. Such a daily routine consists of a sequence of many movements (Fig. 2B). These behaviours may be called ‘action skills’ (Hikosaka et al., 2013), which have been acquired through long-term learning (Ericsson and Lehmann, 1996) and can be performed automatically (without conscious attention) (Seger, 1994; Destrebecqz and Cleeremans, 2001). In fact, our daily routines consist of a variety of action skills. Without such automatic action skills, we would spend an enormous amount of time and energy before approaching a goal. Animals also naturally develop a variety of action skills after long-term learning (Helton, 2008).

Action skills are acquired by repeated practice, during which performance changes drastically (Fitts, 1964; Anderson, 1982). In the early phase, the motor performance is done consciously and slowly with many errors (i.e. failure to reach the final goal). In the late phase, the motor performance is done automatically and quickly with few errors. With long-term learning, the time to complete one action becomes shorter, often up to 1/10 of the original time (Crossman, 1959; Newell and Rosenbloom, 1981). Once an action skill is acquired, it is hardly abolished (Ammons et al., 1958; Hikosaka et al., 2002). Therefore, we can acquire (or have acquired) many action skills, all of which together allow us to carry out daily routines smoothly. This suggests that the brains of humans and animals have the capacity to acquire and store long-term memories that are expressed as a variety of action skills.

Many studies on humans, monkeys, and rodents have suggested that the basal ganglia contribute to action skill performance and its learning (Whishaw et al., 1986; Hikosaka et al., 2002; Packard and Knowlton, 2002; Graybiel, 2008). A series of studies on monkeys and humans suggested that different parts of the basal ganglia contribute to action skill learning and performance in different manners (Hikosaka et al., 1999). In these studies, the subjects learned to press buttons sequentially in a spatially fixed order across many days using the left or right hand (Hikosaka et al., 1995). After experience with new sequences, they eventually acquired a repertoire of many learned sequences, half of which were learned with the left hand and the other half with the right hand. Both monkeys and humans learned to perform each sequence more accurately and faster with similar time courses across daily learning sessions: first, their performance became accurate (mostly <5 days), after which it continued to be faster (>10 days).

Detailed behavioural tests suggested that two separate learning mechanisms contribute to this action skill learning. The first mechanism that was not specific to the effector (i.e. hand) used for learning, whereas the second mechanism was specific to the effector (Seger and Spiering, 2011; Abrahamse et al., 2013). In experiments using monkeys, changing the hand with which they performed the task disrupted the learned performance only after extensive training (Rand et al., 2000). Thus, a non-hand-specific mechanism learned the sequence quickly, mainly to achieve the accurate performance, whereas a hand-specific mechanism learned the sequence slowly, mainly to achieve the fast performance (Fig. 2C). Moreover, the fast performance was retained for a long time with no further practice (6–19 months), but only when the same hand was used (Hikosaka et al., 2002). Human subjects also retained the well-learned performance for a long time (16 months), while retaining its speed more than accuracy, and yet had little awareness about their previous experiences (unlike the recently learned sequence) (Hikosaka et al., 2002). Therefore, these two mechanisms may be characterized as voluntary and automatic learning mechanisms (Fig. 2C and D). The voluntary mechanism would rely on signals encoding the spatial positions of the target buttons and thus can guide performance whichever hand is used, whereas the automatic mechanism would rely on signals encoding the movement of the performing hand and thus can guide the performance of one particular hand (Nakahara et al., 2001).

Different roles of rostral and caudal basal ganglia in action skill learning

The involvement of the basal ganglia in action skill learning was tested in two ways: reversible inactivation and single unit recording. The performance in the early learning phase was impaired by the inactivation of the rostral striatum (caudate head and rostral putamen), whereas the performance in the late learning phase was impaired by the inactivation of the caudal striatum (intermediate and caudal putamen) (Miyachi et al., 1997). Correspondingly, neurons in the rostral striatum tended to be more active in the early learning phase, whereas neurons in the caudal striatum tended to be more active in the late learning phase (Miyachi et al., 2002). These results suggest that the rostral striatum contributes to the voluntary learning mechanism, whereas the caudal striatum contributes to the automatic learning mechanism (Fig. 2D).

It is known that the rostral striatum receives inputs mainly from the associative region of the cerebral cortex (Selemon and Goldman-rakic, 1985; Parent, 1990; Haber et al., 2006) and may send signals back to the association cortex through a loop circuit (SNr/GPi – thalamus – cortex) (Alexander et al., 1986). In contrast, the caudal striatum is connected mainly with the sensorimotor region of the cortex (Kunzle, 1975; Flaherty and Graybiel, 1991; Takada et al., 1998). The rostral striatum processes visuospatial, attentional, working memory, and reward signals (Jueptner et al., 1997b; Lewis et al., 2004; Ding and Gold, 2010), together with the association cortex (Haber et al., 2006). The automatic signals in the caudal striatum may be related to sensorimotor signals derived from joints and muscles (Crutcher and DeLong, 1984; Alexander and DeLong, 1985; Kimura, 1986). The emergence of the effector (i.e. hand) selectivity during learning may be related to the emergence of automaticity or implicitness in action skill, because explicit control would not be affected by which hand is used.

Consistent with the experiments using monkeys described above, human functional MRI studies have shown that different regions in the basal ganglia and other brain areas become active depending on learning phases (Lehéricy et al., 2005; Jankowski et al., 2009; de Wit et al., 2012; Wunderlich et al., 2012; Wymbs et al., 2012): the association cortex (dorsolateral/dorsomedial prefrontal, parietal cortices) together with the rostral striatum is active in the early phase (Jueptner et al., 1997a; Tricomi et al., 2004; Poldrack et al., 2005; Monchi et al., 2006; Jankowski et al., 2009), while the sensorimotor cortex (M1/premotor cortex) together with the caudal striatum is active in the late phase (Jueptner et al., 1997a; Jankowski et al., 2009; Tricomi et al., 2009) when the performance becomes implicit (Rauch et al., 1997).

The basal ganglia are also essential for action learning in rodents (Koralek et al., 2012) and birds (Charlesworth et al., 2012), which in both cases are assisted by dopamine inputs (Faure et al., 2005). Equivalent functional regions for motor learning are found in rodents: dorsomedial striatum for the early phase of learning and the dorsolateral striatum for the late phase of learning (Yin et al., 2009). This may reflect the fact that the primate brain has been extended rostrocaudally compared with the rodent brain. If so, the dorsomedial striatum in rodents may correspond to caudate head in monkeys, while the dorsolateral striatum may correspond to the caudal putamen (Balleine and O’Doherty, 2010). These two striatal regions are often associated with two kinds of behaviour: goal-directed behaviour and habit (Yin and Knowlton, 2006; de Wit et al., 2009). In daily life, however, habit is often goal-directed (Aarts and Dijksterhuis, 2000; Wood and Neal, 2007) and may be better characterized as ‘skill’.

Gaze orienting to valuable objects

Object skill

The learning of motor actions is critical for all animals because it increases the chance of survival (Hikosaka et al., 2013). However, such learned motor actions are often guided by learned cognitive processes, and here also the basal ganglia play an important role (Graybiel, 2008). In most cases a reward is associated with an object (e.g. walnut). To reach the walnut-associated reward, an animal has to go through two processes (Fig. 1B): (i) find a walnut; and (ii) crack the walnut. Cracking a walnut requires an action skill (Takechi et al., 2009). Finding a walnut also requires another kind of skill (Pyke et al., 1977; Kamil and Roitblat, 1985), which may be called ‘object skill’ (Hikosaka et al., 2013). For primates, the finding process heavily depends on visual information and its behavioural outcome is gaze orienting. Among many objects, one is chosen at a time and gaze is oriented to it (with a saccadic eye movement) (Yarbus et al., 1967; Henderson, 2003; Land, 2006; Tatler et al., 2011). Furthermore, each object needs to be evaluated before gaze is settled on the most valuable object.

The fact that the basal ganglia may be involved in object skill is hinted at by their influence on the superior colliculus (SC) whose major function is to initiate orienting responses (Ingle, 1973; Carman and Schneider, 1992). Comparative anatomical studies suggest that superior colliculus is a major target of the basal ganglia among vertebrate species (Marín et al., 1998). In rats, a prominent effect of unilateral lesions or dysfunctions (including dopamine deficiency) of the striatum is spontaneous turning of the body and head, mostly to the ipsilateral side (Pycock, 1980). Inactivation of SNr causes involuntary saccades to the contralateral side in monkeys (Hikosaka and Wurtz, 1985) and rats (Sakamoto and Hikosaka, 1989). In monkeys, local dopamine deficiency, which is caused by injection of MPTP in the caudate head or caudate body, leads to contralateral hemineglect of gaze and attention (Kato et al., 1995; Kori et al., 1995; Miyashita et al., 1995). Interestingly, superior colliculus is targeted by SNr, but not GPi (Beckstead and Frankfurter, 1982). These observations imply that the control of orienting response is a major and distinctive function of the basal ganglia.

Basal ganglia circuit for object skill

In the basal ganglia, the signal for gaze orienting (saccade) is first processed in the striatum including caudate nucleus and putamen (Hikosaka et al., 1989; Gerardin et al., 2003; Neggers et al., 2012; Phillips and Everling, 2012), possibly more dominantly in the caudate nucleus (CD) for monkeys (Hikosaka et al., 2000) and putamen for humans (Neggers et al., 2015). Studies in monkeys have mostly focused on the CD-SNr-SC circuit (Hikosaka et al., 2000). Before eye movements, a group of SNr neurons that project to superior colliculus pause in their tonic firing and thus cause a disinhibition of superior colliculus neurons (Hikosaka and Wurtz, 1983a). The cessation of SNr neuronal activity is mostly caused by phasic firing of caudate head/caudate body neurons, which project to SNr (Hikosaka et al., 1993). Importantly, saccades are heavily biased toward the spatial position where reward is expected (Hikosaka et al., 2006). When a saccade is followed by a reward in one direction and no reward in the other direction, its reaction time is shorter and its speed is higher for the reward-associated direction (Takikawa et al., 2002b). Many neurons in caudate head/caudate body, SNr, and superior colliculus respond more strongly to the reward-predicting visual cues (Kawagoe et al., 1998; Sato and Hikosaka, 2002; Ikeda and Hikosaka, 2003) or show a reward direction-selective anticipatory changes (Lauwereyns et al., 2002; Ikeda and Hikosaka, 2003). These behavioural and neuronal changes occur quickly after a couple of repetitions (Kawagoe et al., 1998). These results suggest that neurons in the CD-SNr-SC circuit contribute to reward-guided gaze orienting by changing their signals flexibly (Itoh et al., 2003). These data provide deeper understanding about how the basal ganglia circuits (Fig. 1A) contribute to reward-oriented behaviour. Specifically, the CD-SNr-SC circuit plays a key role in finding valuable positions.

Does the CD-SNr-SC circuit also contribute to finding valuable objects (i.e. object skill)? This was tested by associating multiple visual objects with different amounts of reward. In a first experiment using two visual objects (fractals) (e.g. A and B), the reward association was reversed after a block of 30–40 trials (i.e. A-large/B-small versus A-small/B-large) (Fig. 3B, left). Neurons responding to visual objects were found in the central part of caudate head. Most of them were sensitive to the immediate reward outcome, usually responding more strongly when the object was associated with a large reward (Fig. 3B, centre) (Kim and Hikosaka, 2013). As expected from the CD-SNr-SC circuit scheme, the monkey made a saccade more quickly when a large reward was expected.

Figure 3.

Basal ganglia mechanisms for finding valuable objects. (A) Caudate nucleus (CD) of the macaque monkey. (B and C) Neuronal responses in the rostral caudate nucleus (CDh) and the caudal caudate nucleus (CDt) in response to visual objects with flexibly changing values (B) and stably fixed values (C). Neuronal responses were averaged for the neurons’ preferred (magenta) and non-preferred (black) values. The yellow line indicates the difference between the preferred and non-preferred responses [mean (yellow) ± SE (black)]. To test the neuronal responses to stable object values (C), the monkey had experienced each object consistently with a big or small reward or a long time (D), but during the test no reward was delivered after the object presentation. (D) Time course of object value learning for caudate head neurons (centre) and caudate tail neurons (right). The monkey experienced stable object-reward associations for several days of sessions (left). Neuronal value discrimination is shown schematically during learning (‘L’) and memory retention (‘R’). Reproduced with permission from Kim and Hikosaka (2013).

The values assigned to these objects can be called ‘flexible values’. In everyday life, however, values of many objects do not change (e.g. favourite foods since childhood), which can be called ‘stable values’. To simulate this everyday life situation, many fractal objects were presented repeatedly across many days, and each object was consistently associated with a large or small reward (Fig. 3C, left). When these objects were subsequently presented with no reward outcome, the caudate head neurons showed much weaker responses with little value bias (Fig. 3C, centre). Apparently, the lack of reward outcome caused the attenuation of the caudate head neuron’s response. According to the CD-SNr-SC circuit scheme, this would lead to the attenuation of the monkey’s gaze bias.

Surprisingly, however, the monkey’s gaze orienting changed dramatically, but slowly (i.e. over 5 days): when multiple objects were presented simultaneously, the monkey’s gaze was strongly attracted to the stably high-valued objects, even though no reward was delivered (Fig. 4A and B) (Yasuda et al., 2012; Kim and Hikosaka, 2013; Yamamoto et al., 2013). Such automatic gaze orienting emerged as the monkey experienced many more fractals (up to 400), and then remained intact for a long time (>100 days) during which the monkey never saw the fractals (Yasuda et al., 2012).

Figure 4.

Stable object value coding in caudal basal ganglia circuit. (A) Fractal objects associated with stable reward values, half associated with a large reward (high-valued) and the other half associated with a small reward (low-valued). (B) Free-viewing procedure. Stably high- and low-valued objects were chosen randomly and presented simultaneously. The monkey’s gaze was strongly attracted to the previously learned high-valued objects (‘H’), even though no reward was given. (C) CDt–cdlSNr–SC circuit for object skill. Neurons in caudate tail (CDt) and cdlSNr (caudal-dorsal-lateral part of substantia nigra pars reticulata) encoded stable values of objects (in boxes). In response to high-valued objects (compared with low-valued objects), caudate tail neurons were more excited, cdlSNr neurons were more inhibited, and therefore superior colliculus (SC) neurons were more excited (disinhibited). This pattern of activity account for the facilitation of saccades to the high-valued objects. ITC = inferotemporal cortex; LGN = lateral geniculate nucleus; HP = hippocampus. Reproduced with permission from Yasuda et al. (2012) and Kim et al. (2014).

What causes the automatic gaze orienting with no reward outcome? One candidate is the caudate tail, which is a slender caudal extension of the caudate nucleus (Figs 2A and 3A) and is mostly unique to primates (Hjornevik et al., 2007). It receives inputs mainly from the inferior temporal cortex (Kemp and Powell, 1970; Yeterian and Van Hoesen, 1978; Saint-Cyr et al., 1990), which is specialized for processing of visual object information (Tanaka, 1996). Similarly to inferior temporal cortex neurons, a majority of neurons in monkey caudate tail respond to visual objects differentially (Caan et al., 1984; Brown et al., 1995; Yamamoto et al., 2012).

Importantly, caudate tail neurons responded differently from caudate head neurons in relation to object values. The visual responses of caudate tail neurons were not influenced by flexibly changing values (Fig. 3B, right), but were modulated by the stable value, usually responding more strongly to stably high-valued objects, even under conditions when no reward was expected (Kim and Hikosaka, 2013; Yamamoto et al., 2013) (Fig. 3C, right). The idea that caudate tail contributes to the automatic gaze orienting to stably high-valued objects was further supported by its robust oculomotor output (Fig. 4C).

Unlike inferior temporal cortex neurons, whose receptive fields usually include the fovea (Gross et al., 1969; Tanaka, 1996), many caudate tail neurons have eccentric receptive fields, mostly in the contralateral hemifield (Yamamoto et al., 2012). Furthermore, weak electrical stimulation of caudate tail induced saccades aimed at the nearby neurons’ receptive fields (Yamamoto et al., 2012). This effect was mediated by a cluster of neurons in the caudal-dorsal-lateral (cdl) part of SNr (Saint-Cyr et al., 1990; Kim et al., 2014; Yasuda and Hikosaka, 2015), many of which project to superior colliculus (Beckstead and Frankfurter, 1982; Francois et al., 1984). cdlSNr roughly corresponds to the region called ‘pars lateralis’, which is particularly prominent in primates (Francois et al., 1985) (Fig. 4C).

cdlSNr neurons showed categorical responses in that they were inhibited by stably high-valued objects and excited by stably low-valued objects (Fig. 4C), and did so for as many objects as the monkey experienced (e.g. 400) (Yasuda et al., 2012). They were not influenced by flexible values, similarly to caudate tail (CDt) neurons. Moreover, the categorical responses remained virtually unchanged for a long time, similarly to the automatic gaze orienting. Finally, many of the stable value-coding SNr neurons projected their axons to superior colliculus, as determined by antidromic activation (Yasuda et al., 2012).

These results suggest that the CDt-cdlSNr-SC circuit processes stable values, but not flexible values, of visual objects (Fig. 4C). This system evidently has a high capacity for long-term memory, and specifically encodes information about the association of visual objects and reward values. The object-value memory appears to be strengthened or purified along the caudate tail-cdlSNr connection. This mechanism would allow the animal to quickly find a valuable object among many others, without checking individual items. The ‘finding’ is expressed as gaze orienting through the cdlSNr-superior colliculus connection, so that the animal is ready to manipulate the valuable object. Such ‘object skill’ would be crucial for survival, and the responsible mechanism (CDt-cdlSNr-SC circuit) is prominently developed in primates (Hikosaka et al., 2013).

Automatic versus voluntary gaze orienting

However, the object skill mechanism has a flaw, namely slow learning (Fig. 3D, right). If you encounter a new object, or if the value of an object has changed, the CDt-cdlSNr-SC circuit is unable to judge their values. This weakness is compensated for by the caudate head (CDh) circuit, because neurons in the caudate head are sensitive to the immediate reward outcome and thus learn the values of new objects quickly (Fig. 3D, centre) (Kim and Hikosaka, 2013). Importantly, the downstream components are largely separate between caudate tail and caudate head circuits. The visually responsive part of the caudate head also has direct connections to SNr, but selectively targets the rostral-ventral-medial (rvm) part of SNr (Smith and Parent, 1986). This region roughly corresponds to the pars reticulata proper of the SNr, and is distinct from the pars lateralis that receives input from the caudate tail (Francois et al., 1985). Neurons in the rvmSNr are indeed sensitive to flexible values (Yasuda and Hikosaka, 2015), and some of them project to superior colliculus (Francois et al., 1984; Yasuda and Hikosaka, 2015). The caudate body, or intermediate region of the caudate nucleus, contains a mixture of stable value neurons and flexible value neurons (Kim and Hikosaka, 2013) and projects mainly to rvmSNr (Hikosaka et al., 1993).

These results suggest that the CD-SNr-SC circuit, as a whole, operates as a reward value-based gaze orienting mechanism, but it is roughly divided into two parallel circuits: CDt-cdlSNr-SC circuit for automatic finding of stably valuable objects and CDh-rvmSNr-SC circuit for voluntary finding of flexibly valuable objects. For more details, see above and Fig. 6.

Figure 6.

Parallel basal ganglia circuits for voluntary and automatic behaviour. Receiving inputs from different cortical areas, caudate head (CDh) and caudate tail (CDt) have separate downstream circuits, both targeting superior colliculus (SC). Both circuits process object–value memory and learning, but do so under different conditions. These two learning mechanisms lead to different outcomes, namely voluntary and automatic behaviour. Separate groups of dopamine neurons may contribute to the different kinds of memory processing. GPe(r) = rostral part of globus pallidus external segment; GPe(c) = caudal part of globus pallidus external segment.

Dopamine circuits for differential reward value coding

Functional heterogeneity of dopamine neurons

Many areas in the basal ganglia, especially the striatum, are heavily innervated by dopamine neurons (Richfield et al., 1987). It has repeatedly been shown that activity and synaptic transmission of striatal neurons are modulated by dopamine inputs (Nicola et al., 2000; Reynolds and Wickens, 2002; Wise, 2004; Surmeier et al., 2007), mostly through D1 or D2 receptors (Fig. 1). The finding that dopamine neurons encode RPE signals (Schultz, 1998) triggered a set of new theories on reward-based learning: the dopamine-RPE signal biases sensorimotor signals in the striatum stepwise until the gain of reward is maximized (Dayan and Balleine, 2002). According to this unified theory, dopamine neurons would transmit the unique signal (i.e. RPE) to all brain areas. However, earlier studies (Schultz and Romo, 1987) as well as more recent studies, showed heterogeneities among dopamine neurons (Bromberg-Martin et al., 2010). First, dopamine neurons in different regions of SNc and VTA receive inputs from and send outputs to different regions of the brain (Haber et al., 2000; Ikemoto, 2007). Second, different groups of dopamine neurons encode different signals. In particular, many dopamine neurons are excited by punishment or its predictor (Horvitz, 2000; Feenstra et al., 2001; Coizet et al., 2006; Brischoux et al., 2009; Matsumoto and Hikosaka, 2009; Lammel et al., 2011). Dopamine neurons in the dorsolateral part of the monkey SNc are excited by both reward-predicting and punishment-predicting stimuli (thus encoding ‘motivational salience’). This response pattern is clearly different from dopamine neurons in the ventromedial SNc, which are excited by reward-predicting stimuli but inhibited by punishment-predicting stimuli (thus encoding ‘motivational value’). RPE is clearly encoded by the value-coding dopamine neurons, but not the salience-coding dopamine neurons, partly due to differential inputs from the lateral habenula (Matsumoto and Hikosaka, 2007). These results raise the possibility that different groups of dopamine neurons serve heterogeneous functions through their connections from and to different basal ganglia circuits.

Separate groups of dopamine neurons projecting to caudate tail and caudate head

This hypothesis was tested for dopamine neurons projecting to caudate nucleus in monkeys, where neurons change their responses to visual objects depending on their reward values (Kim and Hikosaka, 2013). This neuronal learning might be guided by dopamine inputs, as all parts of monkey caudate nucleus are innervated by dopamine neurons (Richfield et al., 1987). However, the time course of the neuronal learning is different between caudate tail (i.e. slow) and caudate head (i.e. quick). The difference might be caused intrinsically (within caudate nucleus) or extrinsically (due to external inputs). Supporting the latter possibility (but not excluding the former possibility), a tracer study revealed that caudate tail and caudate head are innervated by different groups of dopamine neurons (Kim et al., 2014) (Fig. 5). Neurons in the cdl part of SNc projected to caudate tail, whereas neurons in the rvm part of SNc projected to caudate head (Fig. 5B and C); few neurons projected to both caudate tail and caudate head. These caudate tail and caudate head-projecting neurons were all dopaminergic (as far as examined). Intriguingly, caudate tail-projecting dopamine neurons in cdlSNc tend to have larger and less circular cell bodies than caudate head-projecting dopamine neurons in rvmSNc (Kim et al., 2014). These connections lead to an intriguing topographic relationship between the caudate nucleus-SNr connection and the SNc-caudate nucleus connection: caudate tail-projecting dopaminergic neurons and caudate tail-recipient GABAergic neurons are located adjacently (cdlSNc and cdlSNr); caudate head-projecting dopaminergic neurons and caudate head-recipient GABAergic neurons are located adjacently (rvmSNc and rvmSNr).

Figure 5.

Separate dopamine projections to caudate head and caudate tail. (A) Injection sites of retrograde tracers. Two different retrograde tracers, Diamidino yellow (DY) and cholera toxin subunit B (CTB), were injected in caudate head (CDh) and caudate tail (CDt) of the same monkey. (B and C) Retrogradely labelled neurons in the SN (B: sagittal view, C: coronal view). Yellow triangles: caudate head-projecting neurons; red triangles: caudate tail-projecting neurons in B. Scale bar = 1 mm. In C, coronal sections of SN are shown from the rostral to caudal levels. Scale bar = 2 mm. DA = dopamine. Reproduced with permission from Kim et al. (2014).

The topography of dopamine neurons described above may represent one of many mutual connections between dopamine neurons and the striatum in monkeys (Haber et al., 2000) and rodents (Ikemoto, 2007). In general, dopamine neurons located medially (in VTA/SNc) and laterally (in SNc) are connected with the medial and lateral parts of the striatum, respectively. The medial-lateral topography may reflect a functional specialization: goal-directed versus habitual (Yin and Knowlton, 2006). Primates have developed another functional topography in the rostral-caudal direction in the cerebral cortex as well as the striatum (Lehéricy et al., 2004b; Draganski et al., 2008), which may reflect another kind of functional specialization: associative versus sensorimotor (Parent, 1990; François et al., 1994) (Fig. 2). Anatomical studies have suggested that the rostral and caudal striatum receive inputs mainly from dopamine neurons in the medial and lateral parts of SNc, respectively (Szabo, 1980; Francois et al., 1984; Hedreen and DeLong, 1991). These findings raise the possibility that dopamine neurons in different subregions of SNc or VTA serve different functions by projecting to different subregions of the striatum, as observed in caudate tail- and caudate head-projecting dopamine neurons (Fig. 5).

Particularly relevant to this hypothesis is a finding from patients with Parkinson’s disease, in whom the degeneration of dopamine neurons occurs heterogeneously (Fearnley and Lees, 1991; Damier et al., 1999). A common finding is that the degeneration occurs earlier and dominantly in the lateral part of SNc (Goto et al., 1989; Rinne et al., 1989; Yamada et al., 1990; Fearnley and Lees, 1991; Gibb and Lees, 1991), although this tendency may not be universal (Pillon et al., 1986) partly due to the heterogeneity of Parkinson’s disease itself (Zetusky et al., 1985). According to the topographical schemes described above, patients with Parkinson’s disease would show impairments in sensorimotor functions more strongly than the emotional or cognitive impairments. This will be discussed more comprehensively below.

Other neuromodulators

Dopamine neurons are controlled by inputs from many brain areas (Haber and Knutson, 2010; Watabe-Uchida et al., 2012). Functionally significant among them are the input from the lateral habenula through the rostromedial tegmental nucleus (Hong et al., 2011; Barrot et al., 2012) and the input from the pedunculopontine nucleus (Scarnati et al., 1987; Hong and Hikosaka, 2014), which are organized topographically.

In addition to dopamine, there are other neuromodulators acting on basal ganglia circuits, including serotonin, acetylcholine, and norepinephrine (Doya, 2008). Both serotonin and dopamine neurons may encode reward values, but differently (Nakamura et al., 2008), thus influencing motivation and mood differently (Proulx et al., 2014). Acetylcholine is delivered both internally within the striatum (Lehmann and Langer, 1983) and externally from the pedunculopontine nucleus (Mena-Segovia et al., 2008), and may interact with dopamine (Di Chiara et al., 1994; Chuhma et al., 2014). Neuromodulators may work together with classical neurotransmitters (e.g. GABA, glutamate) as well as other neuromodulators (Charara et al., 1996). The interactions among the modulators and transmitters may be critical for the intricate function of the basal ganglia.

Parallel circuits for voluntary and automatic behaviour

Finding valuable objects is a critical step to obtaining rewards (Fig. 1B), and recent studies revealed that the basal ganglia play a crucial role in this process (Kim and Hikosaka, 2013; Hikosaka et al., 2014). To reach this goal, the basal ganglia are equipped with two parallel mechanisms (Fig. 6), each of which consists of the elementary circuits—direct pathway, indirect pathway, and dopamine input to striatum (Fig. 1A). These circuits share a common goal (i.e. gaze orienting) by targeting superior colliculus, but they process the values of visual objects differently: (i) stably as long-term memories; and (ii) flexibly as short-term memories. Before reaching superior colliculus, these signals are processed in the same basal ganglia nuclei (caudate nucleus and SNr), but in separate regions (caudate tail versus caudate head, cdlSNr versus rvmSNr). Both circuits receive dopamine inputs, but again from separate regions (cdlSNc versus rvmSNc). A main mediator of the indirect pathway, GPe, also seems separate: caudate tail projects to the caudal-ventral part of GPe (Saint-Cyr et al., 1990), whereas caudate head projects to the rostral-dorsal part of GPe (Cowan and Powell, 1966; Parent et al., 1984). The separation of the two mechanisms seems reasonable, because the flexible and stable values are often mutually conflicting (stability-flexibility dilemma) (Liljenström, 2003; Abraham and Robins, 2005; Anderson, 2007).

According to this scheme (Fig. 6), gaze orienting can rely more strongly either on the stable circuit or on the flexible circuit, which makes the system more adaptable. In a familiar and stable environment, you encounter many objects (e.g. foods), which you have previously seen many times and therefore can well predict their values (i.e. tastes) before experiencing them. In this scenario, the CDt-cdlSNr-SC circuit would inhibit or excite superior colliculus depending on the stably fixed values of the objects (Fig. 3C, right), but the CDh-rvmSNr-SC circuit would provide no biased signals (Fig. 3B, right). Gaze would then be oriented to the previously reward-associated object automatically (automatic saccade), because caudate tail neurons and cdlSNr neurons respond to the object even when there is no reward outcome.

Occasionally however, familiar objects change their values unpredictably (i.e. sweet to sour orange). Or, if you encounter novel objects, you cannot predict their values. In these cases, you need to pick some of the objects and check their values (e.g. by tasting them). Then, CDh-rvmSNr-SC circuit would inhibit or excite superior colliculus depending on the flexibly changing values of the objects (Fig. 3B, centre), but the CDt-cdlSNr-SC circuit would provide no biased signals (Fig. 3C, centre). Gaze would be oriented to the object only if it has recently been associated with reward (voluntary saccade), because caudate head neurons and rvmSNr neurons respond to the objects only when they are expected to be associated with reward.

Novel objects would gradually become familiar if you experience them repeatedly. For experiences that are associated with rewards, the flexible CDh-rvmSNr-SC circuit would be active initially, but activity within this circuit would gradually be replaced by the activation of the stable CDt-cdlSNr-SC circuit until ‘object skill’ is established. This shift of the active site in the basal ganglia is similar to what happens during motor learning that results in ‘action skill’ (Fig. 2). In both cases, the active site shifts from the rostral to caudal part of the basal ganglia. Also common to the action and object systems is cortical inputs: the flexible-early mechanism receives inputs mainly from the association cortex, whereas the stable-late mechanism receives inputs from the sensorimotor cortex (c.f. caudate tail mostly from the sensory cortex).

This model for separate parallel circuits mediating voluntary and automatic function (Figs 6 and 7A) is further supported by behavioural findings. Many kinds of motor learning occur equally with or without conscious control (Destrebecqz and Cleeremans, 2001; Maxwell et al., 2001). Conscious control could even worsen the performance, because the automatic learning occurs independently of the conscious control (Mazzoni and Krakauer, 2006). Attending to one’s own skilled performance often impairs that performance (Baumeister, 1984; Beilock and Carr, 2001). Performing another task simultaneously may even improve the skilled performance, potentially by directing conscious attention elsewhere (Beilock et al., 2002). These behavioural findings suggest that the automatic skilled process can operate independently of the voluntary conscious process, and moreover that independent operation is the optimal state of skilled performance. To this end, the automatic and voluntary mechanisms should be separated, as we propose (Fig. 7A).

Figure 7.

Interaction between voluntary and automatic behaviour—a hypothesis. (A) Voluntary and automatic mechanisms are separated between rostral and caudal basal ganglia circuits. Each mechanism controls behaviour in two steps: finding good objects; and manipulating the good objects, and is modulated by reward outcomes. Critically, the reward effect is strong but temporary for the voluntary mechanism and weak but cumulative for the automatic mechanism. (B) Separate versus interactive operations of voluntary and automatic mechanisms. These mechanisms may operate independently, with voluntary mechanism being guided consciously while multiple automatic mechanisms being guided subconsciously (left). Alternatively, they may interact with each other, with the automatic mechanisms consciously selected by the voluntary mechanism (right), or the voluntary mechanism triggered by automatic mechanisms (not shown).

However, the strict dichotomy of memory into short-term versus long-term systems may not be correct. The time course of memory might vary in a graded manner. If so, the scheme should be composed of multiple mechanisms with different memory time courses. Being located between caudate head and caudate tail, caudate body shows intermediate properties (i.e. between short-term and long-term) on average (Kim and Hikosaka, 2013). The dichotomy scheme may also not be completely correct if there are neuronal connections across the two mechanisms. This might occur through the indirect pathway (Yasuda and Hikosaka, 2015), but few data are available showing the exact connections through this polysynaptic pathway (i.e. through GPe and possibly STN). Such across-mechanism connections, if they exist, might provide a higher level of adaptability: utilize long-term memories in a flexible environment or short-term memories in a stable environment.

We so far have considered that the automatic and voluntary mechanisms aim at the same motor goal (e.g. gaze orienting). More generally, however, this may not be the case. For instance, when you cook, several automatic processes operate simultaneously (e.g. meat cutting, flour mixing, egg breaking, and pan tossing) while your active voluntary process aims at the final outcome of the cooked food (Fig. 7B). In this case, multiple automatic mechanisms and one voluntary mechanism operate in parallel, but aim at a higher level common goal that involves multiple motor and mental processes. Unlike the common motor goal scheme (Fig. 6), these multiple parallel mechanisms may not always operate together. Instead, different sets of the automatic mechanisms should be activated depending on different contexts, and this has been investigated using ‘set-shifting tasks’ (as described in the next section). This procedure requires functional interactions between the automatic and voluntary mechanisms: the voluntary mechanism would be responsible for the selection of the automatic mechanisms, whereas the automatic mechanisms would provide the voluntary mechanism with long-term memory-based information (which would be helpful for the selection) (Ericsson and Kintsch, 1995).

Behavioural deficits caused by basal ganglia dysfunctions

We have discussed how the parallel neural circuits in the basal ganglia are organized and how they operate to achieve beneficial behaviours. The discussion provides a good opportunity to examine and reinterpret the behavioural effects of basal ganglia disorders, as shown below. To this end, we mostly focus on Parkinson’s disease, for which a large amount of clinical and experimental data are available.

Deficits in action skill

The proposed scheme (Fig. 6) predicts that dysfunctions of the basal ganglia cause different behavioural impairments depending on which region is mainly affected: automatic behaviour by caudal dysfunctions versus voluntary behaviour by rostral dysfunctions (Fig. 7A). In Parkinson’s disease, cell degeneration tends to occur earlier and dominantly in dopamine neurons in the lateral part of SNc, which project mainly to the caudal striatum. As the caudal striatum, especially caudal putamen, primarily serves skeletal sensorimotor functions (Alexander and DeLong, 1985; Liles and Updyke, 1985; Kimura et al., 1996), patients with Parkinson’s disease often show skeletomotor dysfunctions, including akinesia, rigidity, and tremor (Crossman, 1987; Jankovic, 2008). Although the patients often have difficulty in initiating a movement (akinesia), the movement could be initiated in special contexts (Marsden, 1980; Glickstein and Stein, 1991), such as in response to abrupt sensory inputs (Schwab et al., 1959), planned sensory cues (Morris et al., 1996), or emotional events (Schwab and Zieper, 1965). These observations imply that the basic motor mechanisms located in the brainstem or spinal cord are largely intact in Parkinson’s disease.

Why then do patients with Parkinson’s disease struggle in execution of daily routines? Some hints have been suggested by behavioural tests. They have difficulty in generating ballistic actions accurately, which are characteristic of normal skilled movements (Flowers, 1976). Patients with Parkinson’s disease may perform a single movement fairly well, but often have difficulty in performing two movements simultaneously (Schwab et al., 1954; Benecke et al., 1986). For instance, the act of leaving the house could involve walking to the entrance while also being ready to grab the door knob, although these movements are usually performed automatically and subconsciously. The whole daily routine consists of so many automatic movements, some of which are performed simultaneously (Fig. 7B, also see section ‘Learning of sequential motor action’). These considerations suggest that, in Parkinson’s disease, the automatic processes do not work efficiently (Redgrave et al., 2010) and this deficit is caused by the dysfunction of the caudal basal ganglia (Fig. 7A).

It is now useful to consider how the automatic and voluntary processes would work together. Our hypothesis is illustrated in Fig. 7B. In daily routine, a single voluntary process and multiple automatic processes would work simultaneously and independently. Having seen an object of interest (e.g. the door knob), the voluntary process may be focused on how you drive to the store you are heading to, while automatic process A may prepare for grabbing the knob and automatic process B may generate walking. All of the automatic processes would work in a coordinated manner, due to coordinated long-term learning. In an extreme case of Parkinson’s disease, all of these processes may be no longer automatic, but instead need to be performed voluntarily with full attention. The voluntary process would then need to deal with all the multiple processes sequentially one at a time. Such a demand could be so high in patients with Parkinson’s disease (as suggested by functional MRI studies) (Wu and Hallett, 2005) that the main goal of the voluntary process (i.e. how to drive to the store) can scarcely be processed. The whole daily routine would thus be performed sequentially, and therefore slowed down (Koerts et al., 2011) and often with mental fatigue (Friedman et al., 2007).

Importantly, the automatic processes are acquired after long-term practice (which would be called ‘action skill’) (Hikosaka et al., 2013). Indeed, patients with Parkinson’s disease often have difficulty in reaching a normal level of automaticity. This was suggested by using various motor tasks (Doyon et al., 1997), including a serial reaction time task (Jackson et al., 1995; Stefanova et al., 2000) and a pursuit rotor task (Heindel et al., 1989). Even if patients with Parkinson’s disease do learn action skills, their skills are not retained for a long time, in contrast to patients with Alzheimer’s disease (Mochizuki-Kawai et al., 2004). These results suggest that the learning and retention mechanisms in the caudal basal ganglia are compromised in Parkinson’s disease, partly due to the lack of dopamine inputs to the caudal striatum (Redgrave et al., 2010).

In Huntington’s disease, on the other hand, cell loss occurs predominantly in the striatum (Vonsattel et al., 1985) and patients with Huntington’s disease show deficits in learning of action skills (Heindel et al., 1988; Gabrieli et al., 1997), especially learning of sequential movements (Knopman and Nissen, 1991; Willingham and Koroshetz, 1993; Willingham et al., 1997).

Deficits in object skill

Action skill is triggered by an object of interest (e.g. door knob) (i.e. stimulus-response theory) (de Wit et al., 2009). In other words, in order to initiate an automatic action, the relevant object must be found first. This is often demanding and time-consuming because we are surrounded by so many objects, as shown in visual search tasks (Treisman and Gelade, 1980). However, with repeated experiences in daily life, the object-finding task becomes easier and can be performed subconsciously (Shiffrin and Schneider, 1977; Chun and Jiang, 1998; Sigman and Gilbert, 2000). This learned ability can be called object skill (Hikosaka et al., 2013), as described above. Dysfunctions of the basal ganglia seem to disrupt object skill, in addition to action skill.

The basal ganglia are also involved in sensory processing. The striatum receives inputs from sensory cortices: somatosensory (Flaherty and Graybiel, 1991; Graziano and Gross, 1993), visual (Hikosaka et al., 1989c; Kimura, 1990; Graziano and Gross, 1993; Kermadi and Joseph, 1995; Ding and Hikosaka, 2006), and auditory (Hikosaka et al., 1989c). In primates, basal ganglia dysfunctions are sometimes associated with visual deficits (Bodis-Wollner, 1990; Lieb et al., 1999). Human patients with Parkinson’s disease often experience visual hallucinations (Diederich et al., 2009). Monkeys with local dopamine denervation of caudate head or caudate body (by local MPTP infusion) show strong contralateral hemi-neglect of gaze and attention (Kori et al., 1995; Miyashita et al., 1995). Electrical stimulation of caudate head and caudate tail impairs monkeys’ performance of a delayed visual discrimination task (Cohen, 1972). Patients with Parkinson’s disease occasionally have visual defects similar to hemi-neglect (Villardita et al., 1983). While scanning complex visual images, patients with Parkinson’s disease make fewer saccades with smaller amplitudes (Matsumoto et al., 2011; Archibald et al., 2013). They may identify fewer landmarks and traffic signs during driving (Uc et al., 2006), may not benefit from the repetition of a complex visual pattern (van Asselen et al., 2009), or may be slower in detecting visual images appearing out of noise (Meppelink et al., 2008). However, these deficits are different from loss of vision; they seem related to the visual memory-based behavioural control rather than visual perception, as shown below.

Amnesia is a loss of memory that is usually caused by dysfunctions of the hippocampus and adjacent structures (Mishkin, 1978). However, the loss of memory in amnesic patients is only partial, if memory is broadly defined as the information created and retained in the brain. In fact, amnesic patients can learn a variety of behaviours, although they may not consciously remember what they have done. For example, they can learn to perform the ‘mirror reading task’ (i.e. reading texts in which the orientation of letters is reversed) (Cohen and Squire, 1980) and the ‘weather forecast task’ (i.e. predicting the correct answer by viewing multiple pictures, each of which is associated with the answer probabilistically) (Knowlton et al., 1996), similarly to control subjects. These findings suggested that there is another kind of memory that is not controlled by the hippocampal area. A likely brain area responsible for this other kind of memory is the basal ganglia. This is supported by the findings that patients with Parkinson’s disease have difficulty in learning the mirror reading task (Roncacci et al., 1996; Yamadori et al., 1996) and the weather forecast task (Knowlton et al., 1996). These two kinds of memory are called declarative memory (mainly controlled by the hippocampal area) and procedural memory (mainly controlled by the basal ganglia).

Unlike declarative memory, procedural memory directly contributes to decision-making. Procedural memory tasks (e.g. mirror reading, weather forecast) typically require the transformation of visual signals to motor outputs. Of particular interest is a concurrent discrimination task developed by Mishkin and colleagues (Malamut et al., 1984) for experiments using monkeys. In this task, the monkeys learned which of two objects to choose in order to obtain a reward. The task involved learning multiple pairs of objects, which in normal monkeys occurred slowly across days. Importantly, the learning was impaired by caudate tail lesions (Fernandez-Ruiz et al., 2001), but not hippocampal lesions (Malamut et al., 1984). Equivalent results are found in human patients. Patients with hippocampal lesions are amnesic, but can normally learn the concurrent discrimination task (Bayley et al., 2005). In contrast, patients with Parkinson’s disease (but not control subjects) are impaired in the concurrent discrimination learning if they are unaware of the cue-reward relationships; both of them showed normal learning if they are aware of the relationships (Moody et al., 2010). These findings provide further evidence that declarative memory is processed by the hippocampal region and that procedural memory is processed in the basal ganglia. According to the recent study on monkeys (Hikosaka et al., 2013), the visually driven procedural memory (i.e. object skill) is processed by a specific mechanism (CDt-cdlSNr-SC circuit) in the basal ganglia, and this mechanism is likely impaired in Parkinson’s disease.

It has been reported that cell loss occurs first in caudate tail among striatal areas in Huntington’s disease (Vonsattel et al., 1985; Gomez-Tortosa et al., 2001). This raises the possibility that object skill is impaired in patients with Huntington’s disease. Indeed, patients with Huntington’s disease are severely impaired in the ‘weather forecast task’ (Knowlton et al., 1996). They are also slow in finding the target object in a visual search task (Lawrence et al., 2000). Possibly related to these findings, patients with Huntington’s disease show impairments in saccadic eye movements (Leigh et al., 1983; Lasker et al., 1987, 1988; Winograd-Gurvich et al., 2003).

Deficits in voluntary behaviour

The impairment of automatic behaviour in patients with Parkinson’s disease is consistent with the degeneration of dopamine neurons, primarily in the lateral SNc in Parkinson’s disease. However, there are signs that they may also be deficient in voluntary behaviours (Stern et al., 1983). Voluntary behaviour is flexible as it relies on short-term memory (Fig. 6), which is the key to many cognitive functions (e.g. cognitive flexibility) (Gathercole, 1999). Indeed, patients with Parkinson’s disease show deficits in visual short-term memory (especially working memory) (Zokaei et al., 2014) and in performing various cognitive tasks (Brown and Marsden, 1990). Working memory is often used to predict upcoming events, which allows one to prepare for the next action (Beauchamp et al., 2003; Lewis et al., 2004). Such predictive behaviour may be lost in patients with Parkinson’s disease (Flowers and Downing, 1978; Stern et al., 1983; Bronstein and Kennard, 1985), in which case they rely on sensory cues to trigger their behaviour (Cooke and Brown, 1979). These behavioural changes may be related to the dysfunctions of the rostral portions of the basal ganglia (Fig. 7A), which contain many neurons encoding the preparation of actions (Lauwereyns et al., 2002; Miyachi et al., 2002) or the prediction of events (particularly reward) (Hikosaka et al., 1989a).

Such a neuron-behaviour correlation is shown clearly using a memory-guided saccade task (Hikosaka and Wurtz, 1983b), in which a saccade is guided by spatial working memory. Some neurons in caudate head/caudate body (Hikosaka et al., 1989b) and SNr (Hikosaka and Wurtz, 1983b) change their activity with memory-guided saccades (but not visually guided saccades). Patients with Parkinson’s disease have more difficulty in making memory-guided saccades than visually-guided saccades (Crawford et al., 1989; Nakamura et al., 1994; Jackson et al., 1995; Chan et al., 2005; Terao et al., 2011). They are instead more likely distracted by visual stimuli (Terao et al., 2011). These results support the hypothesis that the rostral basal ganglia serve voluntary behaviour. They also suggest that the function of the rostral basal ganglia is also compromised in Parkinson’s disease (Fig. 7A).

Another critical role of cognitive flexibility based on short-term memory is new learning (Kehagia et al., 2010), in which the choice of behaviour must be modified flexibly when outcomes changes. Neurons in caudate head quickly encode value memory of objects at the onset of learning, but neurons in caudate tail do not (Fig. 3B) (Kim and Hikosaka, 2013). It is theorized that RPE encoded by dopamine neurons is used to repeat or avoid the behaviour (Schultz et al., 1997). As the basal ganglia nuclei are heavily innervated by dopamine neurons, RPE-related dopamine signals may help new learning through basal ganglia circuits by activating its rostral portion, particularly caudate head (Tricomi et al., 2004; Hikosaka et al., 2006; Wunderlich et al., 2012). The fact that patients with Parkinson’s disease are slow in new learning (Frith et al., 1986) may be due to the dysfunction of the rostral basal ganglia in Parkinson’s disease.

Patients with Parkinson’s disease are also deficient in switching and set shifting (Cools, 1980; Gauntlett-Gilbert et al., 1999; Cools et al., 2001), in which cognitive flexibility is critical (Cools et al., 2001). As discussed above, daily routine behaviour could be generated by simultaneous or sequential activation of multiple automatic processes (Fig. 7B). However, if the context changes unexpectedly, the automatic processes may need to be suppressed so that the voluntary process can switch behaviour. This kind of behavioural switching can be achieved by the activation of STN (Isoda and Hikosaka, 2008), which receives context change signals from the pre-supplementary motor area (pre-SMA) (Isoda and Hikosaka, 2007; Nachev et al., 2008; Chao et al., 2009) or other cortical areas (Aron and Poldrack, 2006). This mechanism may be deficient in Parkinson’s disease (Witt et al., 2004).

Switching as described above may reflect a conflict between automatic and voluntary processes, which has been tested using various tasks (e.g. pro- versus anti-saccade) (Cameron et al., 2010). However, there are different kinds of behavioural switching. Switching may occur between different voluntary processes, for example, in tasks in which each stimulus changes its value between two conditions (e.g. A-large/B-small versus A-small/B-large). This switching between two states for a given stimulus may be called intra-dimensional set-shifting (Owen et al., 1991). Alternatively, switching may occur between automatic processes and be guided by the voluntary process, for example in tasks in which each stimulus has two features (e.g. colour and shape) and the value of the stimulus is determined by its colour (Condition 1) or shape (Condition 2). The switching between the ‘colour’ and ‘shape’ conditions may be called extra-dimensional set-shifting (Owen et al., 1991).

Patients with Parkinson’s disease seem to have difficulty in any of these kinds of switching. However, it is unclear whether these deficits are caused by the dysfunction of the rostral basal ganglia. First, the responsible brain region might be the frontal cortex (Taylor et al., 1986), because Parkinson’s disease may affect the frontal cortex as well (Scatton et al., 1983) and dysfunctions of the frontal cortex are also associated with switching deficits (Owen et al., 1993). Second, dysfunctions of automatic processes (rather than voluntary process) could cause switching deficits if the switching involves an automatic process (Koerts et al., 2009; Redgrave et al., 2010).

Patients with Huntington’s disease also show a wide range of cognitive impairments including deficits in visuospatial memory, executive functions, and extra-dimensional set-shifting (Lawrence et al., 1996). These impairments may be related to deficits in inhibitory control mechanisms in the basal ganglia (Lawrence et al., 1998), which would be crucial for shifts of attention (Sprengelmeyer et al., 1995).

Future issues

We have proposed that the basal ganglia utilize short-term and long-term memories in separate circuits to control behaviour voluntarily as well as automatically, and that these functions are guided by separate groups of dopamine neurons. The voluntary and automatic circuits are separated within each of the basal ganglia, nuclei mainly in the rostral-caudal direction (Figs 6 and 7A). This model in turn raises many questions that may steer the direction of future research on the basal ganglia. We discuss some future issues below.

Time course of memory

The difference between the voluntary and automatic circuits could be explained by the time course of memory: quick learning and unlearning for the voluntary circuit, and slow learning and unlearning for the automatic circuit. What factors give rise to these different time courses? It has been hypothesized that reward-based learning is guided by dopamine inputs to the striatum that induce long-term plasticity of cortico-striatal synapses (Reynolds and Wickens, 2002). The fact that the voluntary and automatic circuits receive inputs from separate groups of dopamine neurons raises the possibility that the time course of the synaptic plasticity is determined uniquely by each group of dopamine neurons. However, it is not currently known whether the properties of synaptic plasticity in the striatum vary across different dopamine inputs. Moreover, synaptic plasticity has been investigated mostly in vitro, and consequently over time, scales too short to be comparable to the time course of memory formation in reward-seeking animals. Since learning under real world conditions can easily exceed 1 year for the automatic circuit that processes skill memory, this issue remains a challenging question for future research of memory.

Storage of memory

The striatal synaptic plasticity theory (above) implies that procedural memory is stored in the cortico-striatal synapses. To test this hypothesis, we need to identify the signal carried by individual neurons along the basal ganglia circuits. For the object skill learning, the signal carried by caudate tail neurons expressing object-value memory is further enhanced in the downstream structure—cdlSNr. This enhancement could be caused by the separation of caudate tail signals to the direct pathway and the GPe-mediated indirect pathway before converging to cdlSNr. It is thus possible that synaptic plasticity occurs in a dispersed manner across the caudate tail, GPe, and cdlSNr. The synaptic changes underlying memory may be more dispersed for action skill learning, because it involves the cerebellum and the motor cortex (in addition to the basal ganglia) through the cortex-basal ganglia and cortex-cerebellum loop circuits (Hikosaka et al., 1999). To summarize, how procedural memory is localized or distributed remains to be studied.

Interaction between voluntary and automatic processes

If the voluntary and automatic circuits operate independently, how can their signals be integrated to produce an appropriate behaviour? We raise three possibilities. First, these signals are integrated only in the common target region (e.g. superior colliculus): whichever signal is more robust is translated to the behaviour (if the two signals are incongruent). Second, these signals are integrated as a consequence of a behaviour: for example, if gaze is oriented automatically to a stably valuable object, the updated gaze may trigger voluntary processes or vice versa. Third, these signals are integrated by neuronal connections between the voluntary and automatic circuits, which is not shown in our scheme because clear evidence is lacking.

Funding

This research was supported by the Intramural Research Program at the National Institutes of Health, National Eye Institute.

Glossary

Abbreviations

cdl

caudal-dorsal-lateral

GPe

globus pallidus external segment

GPi

globus pallidus internal segment

SC

superior colliculus

SMA

supplementary motor area

SNr

substantia nigra pars reticulata

STN

subthalamic nucleus

SNc

substantia nigra pars compacta

VTA

ventral tegmental area

RPE

reward prediction error

rvm

rostral-ventral-medial