Dopamine, Effort, and Decision Making: Theoretical Comment on

Abstract

Forebrain dopamine (DA) systems are thought to be a critical component of the brain circuitry regulating behavioral activation, work output during instrumental behavior, and effort-related decision making. Tasks that offer animals choices between alternatives that require different degrees of effort can be used to assess effort-related choice behavior. Rats treated with DA antagonists, or with accumbens DA depletions, tend to show reduced selection of instrumental behaviors with high response requirements, and instead they choose to engage in food-seeking behaviors that involve less effort. The accompanying article by Bardgett et al. (2009) describes a novel effort-discounting task that involves the modification of a previously developed T-maze choice procedure (Salamone et al., 1994). Each arm of the maze contained different magnitudes of food reinforcement, and in order to obtain the higher magnitude reward, the rats had to climb a barrier in that arm of the maze. With training, rats were able to climb successively higher barriers to obtain the larger amount of food, and the choice between the high barrier arm and the no-barrier arm with the smaller reward served as a template for assessing the effects of dopaminergic drugs. D1 and D2 family antagonists, as well as the DA releasing agent amphetamine, were able to produce a bidirectional modulation of choice behavior, while drugs that act on D3 receptors were ineffective. These studies illustrate features of the neurochemical regulation of effort-related decision making, and may have implications for the understanding of both natural and pathological features of motivation.

At times, a consideration of basic cellular processes can shed light on scientific problems that involve behavior in whole organisms. One such consideration begins with the statement by the biochemist, Albert Lehninger, from several decades ago. According to Lehninger (1975), who was a pioneer in the field of bioenergetics, “the cell is a nonequilibrium open system, a machine for extracting free energy from the environment” (p. 8). This statement is important both for what it says, and for what it does not say. Not only does it efficiently characterize some general features of the biochemistry of cell energy, but it also instigates thinking about the relation between energy extraction and energy expenditure at the level of whole organisms. Animals extract energy from the environment by ingesting nutrients. However, the process of seeking and obtaining nutrients typically involves considerable work. Moreover, organisms must work to obtain access to other significant stimuli such as water, nesting material, mates, or offspring. An animal foraging in the wild, or performing an instrumental behavior in a laboratory, must expend energy in the form of motor activity to gain access to stimuli that are important for its survival and well being. Furthermore, in view of the complex nature of the internal and external environments, the vast array of significant stimuli, and the multitude of potential paths that can be used to obtain access to them, organisms must frequently make effort-related decisions based upon cost-benefit analyses. Of course, decisions based upon effort represent but one dimension of the entire spectrum of the decision making process. In addition to effort, organisms also make complex calculations based upon time, probability of occurrence of various events, and several factors related to reinforcement value, and must allocate their behavioral resources accordingly. Nevertheless, within the last few years there has been an intensive growth in research on effort-related processes, including effort-related decision making. The accompanying article by Bardgett, Depenbrock, Downs, Points, & Green (2009), which introduces a novel behavioral approach to this area and presents pharmacological data on the bidirectional modulation of effort-related choice, provides a useful piece of this emerging literature. The present commentary will attempt to place these recent findings from Bardgett et al. (2009) into the overall context of this expanding area of research.

If effort-related processes are seen in the context of motivational theory and research, then it is worthwhile to begin by stressing the relation between the exertion of effort and activational aspects of motivation. For many years, it has been recognized that there is a useful distinction to be made between directional and activational aspects of motivation (Cofer & Appley, 1964; Duffy, 1963; Salamone, 1988; Salamone, 1992; Salamone et al., 1991). Motivated behavior is directed toward or away from particular stimuli and the behavioral interactions with those stimuli. In addition, motivated behavior is characterized by high levels of activity, vigor, persistence, and work output. The idea that behavioral activation is a fundamental aspect of motivation, which is particularly relevant for understanding how organisms overcome work-related obstacles that separate them from significant stimuli, has been present in the psychology and ethology literatures for some time. Approaches as diverse as optimal foraging theory (Krebs, 1977) and economic models of operant conditioning (Bickel, Marsch, & Carroll, 2000; Lea, 1978) have emphasized that work-related constraints influence behavioral output. DA, particularly in the nucleus accumbens, is important for aspects of behavioral activation, exertion of effort, and effort-related choice behavior (Correa & Salamone, 2002; Robbins & Everitt, 2007; Salamone, Correa, Farrar, & Mingote, 2007; Salamone, Correa, Mingote, & Weber, 2003; Salamone, Cousins, & Snyder, 1997).

There have been several methodological developments in terms of how effort-based choice behavior is assessed. One relatively simple procedure that has been used is a concurrent lever pressing/chow feeding task (Salamone et al., 1991). With this task, rats are given a choice between lever pressing on a ratio schedule (usually FR5) for a preferred reward, or simply approaching and consuming a less preferred chow that is concurrently available in the chamber. This test is essentially a modified operant behavior procedure that allows for the assessment of how responses are allocated in relation to the two alternative sources of food, and it has undergone extensive behavioral and pharmacological validation (Salamone et al., 1991, 1997; Koch, Schmid, & Schnitzler, 2000; Sink et al., 2008). Another task that was developed was the T maze choice task (Cousins et al., 1996; Salamone et al., 1994). Rats are trained on a discrete-trial T maze procedure in which one of the choice arms has a high density of food reward (e.g., 4 × 45 mg operant pellets), whereas the other arm has a lower density (e.g., 2 pellets). Effort-related challenges can be imposed in this task by placing a vertical barrier in the arm that contains the high density of food reward; this presents an obstacle to the rat, as they can only obtain the high density of food reward by climbing the barrier, but they also have an alternative, which is the option of simply selecting the low cost/low reward arm. Variants of the T maze task also have undergone considerable behavioral validation and evaluation (Cousins et al., 1996; Salamone et al., 1994; van den Bos, van der Harst, Jonkman, Schilders, & Spruijt, 2006), and have been used by several laboratories to characterize the effects of brain lesions or drug manipulations (Denk et al., 2005; Salamone et al., 1994; Schweimer & Hauber, 2006; Walton, Bannerman, & Rushworth, 2002). More recently, an operant task that involves effort discounting was developed by Floresco, Tse, and Ghods-Sharifi (2008). Despite the differences between these tasks, they all have yielded similar pharmacological results in relation to the role of DA systems. Low doses of DA antagonists that block D1 or D2 family receptors, including haloperidol, cis-flupenthixol, SCH 23390, SCH39166, raclopride, and eticlopride, alter effort-related choice behavior, reducing selection of more effortful choices and biasing animals toward lower-cost alternatives (Cousins et al., 1994; Denk et al., 2005; Floresco et al., 2008; Koch et al., 2000; Salamone et al., 1991; Salamone et al., 2002; Sink et al., 2008). Floresco et al. (2008) reported that these effects of DA antagonism on effort-related choice behavior occurred independently of any effect of DA antagonism on delay discounting (for a further discussion of delay and effort as features of decision making, see Denk et al., 2005; Wakabayashi, Fields, & Nicola, 2004; Walton et al., 2006). Together with papers involving operant schedules that have different ratio requirements (Aberman & Salamone, 1999; Salamone, Wisniecki, Carlson, & Correa, 2001; Correa et al., 2002; Mingote, Weber, Ishiwari, Correa, & Salamone, 2005), these studies on choice behavior have served to focus attention on the role of DA systems in behavioral activation and effort-related processes (Salamone et al., 2007).

The accompanying article by Bardgett et al. (2009) provides an important extension of this previous work. They developed a discounting version of the T maze task described above, in which the amount of food in the high-density arm of the maze was diminished every time the rats selected that arm. Thus, they employed an “adjusting-amount” discounting variant of the T maze procedures, which allows researchers to determine an indifference point for each rat (Richards, Mitchell, De Wit, & Seiden, 1997). Using this procedure, Bardgett et al. (2009) showed that administration of both the D1 family antagonist SCH23390 and the D2 family antagonist haloperidol altered effort discounting, making it more likely that rats would choose the arm with the smaller reward. In contrast, neither stimulation nor blockade of D3 receptors had an effect on choice behavior. Furthermore, elevation of extracellular DA with administration of amphetamine blocked the effects of SCH23390 and haloperidol, and also biased rats toward choosing the high reward/high cost arm, which is consistent with previous studies using DA transporter knockout mice (Cagniard, Balsam, Brunner, & Zhuang, 2006). Together with other results, the findings reported by Bardgett et al. (2009) support the suggestion that, across a variety of conditions, DA systems act to regulate effort-related decision making in a bidirectional manner.

These findings form part of an emerging literature indicating that DA, particularly in nucleus accumbens, participates in the regulation of behavioral activation, work output, and effort-related choice behavior. Of course, accumbens DA is only one part of the broader circuitry involved in these functions (see Figure 1). Several brain areas, distributed across a number of interconnected forebrain regions, also participate in the exertion of effort in instrumental behavior, as well as effort-related decision making. These areas include prefrontal/anterior cingulate cortex, nucleus accumbens, ventral pallidum, and basolateral amygdala (Farrar et al., 2008; Floresco & Ghods-Sharifi, 2007; Hauber & Sommer, in press; Salamone et al., 1991, 1994, 2001, 2007; Schweimer & Hauber, 2006; Schweimer, Saft, & Hauber, 2005; Walton et al., 2002, 2003, 2006). Moreover, neurotransmitters other than DA also are involved. Considerable evidence implicates adenosine A_2A receptors in nucleus accumbens, as well as the GABAergic ventral striatopallidal pathway and GABA_A receptors in ventral pallidum, in effort-related functions (Farrar et al., 2007, 2008; Font et al., 2008; Mingote et al., 2008; Mott et al., in press; Worden et al., in press). Future research employing a variety of tasks, including the T maze discounting procedure developed by Bardgett et al. (2009), could be used to assess the specific contribution made by each of these brain areas.

Figure 1.

This is a schematic circuitry diagram showing some of the anatomical connections linking cortical/limbic/striatal structures that are involved in effort-related processes. Various behavioral findings related to this figure are referred to in the text. The projection patterns of distinct accumbens subregions (i.e., core and shell) subregions are not depicted. VP = ventral pallidum; DA = dopamine.

Even as researchers are making progress characterizing the brain circuitry and neurochemical interactions involved in regulating effort-related functions, new questions are emerging, and old questions are being reconsidered in a new light. One important consideration is the potential difference between the exertion of effort (i.e., the generation of instrumental behavior with relatively high work requirements) and effort-related decision making (i.e., making choices among various options with different work requirements and reward values). Furthermore, a related point is that different brain systems may participate in these functions in distinct ways. There is considerable evidence that accumbens DA depletions or antagonism act to reduce the exertion of effort, and this result would clearly alter the outcome of any effort-related evaluations and decision making. In addition, on some tasks, accumbens DA depletions and anterior cingulate cortex lesions produce similar effects on effort-related choice behavior. Nevertheless, it is not clear that nucleus accumbens DA and anterior cingulate cortex participate in the decision making process in exactly the same way. Nucleus accumbens DA depletions that affect effort-relate choice also reduce locomotor activity (Cousins et al., 1993) and lower response output on ratio schedules (Aberman & Salamone, 1998; Salamone et al., 2001), whereas other brain manipulations that affect effort-related choice (e.g., anterior cingulate cortex or nucleus accumbens cell body lesions) do not necessarily produce similar effects on these measures. Another important conceptual issue is the relation between motor control and motivation. Although it would be tempting to assume that motor control and motivation are completely distinct functions, a detailed examination of research and theory in psychology suggests that this is not the case. Time that there is considerable overlap between aspects of motor control and motivation (Salamone, 1992; Salamone & Correa, 2002; Salamone et al., 2007). Behavioral measures such as lever pressing speed, latency in a runway or maze, or locomotor activity, are certainly outcomes of a motor execution process, but they also are measures that frequently are used for their motivational significance. If a food deprived rat is running faster in a runway, it seems pointless to ask “Is that motor control or motivation?”, when it clearly is both. Activational aspects of motivation represent the cusp between overlapping components of motor and motivational processes, and mesolimbic DA appears to operate within this area of overlap. Research in this area only serves to emphasize the obvious nature of the relation between motor and motivational processes, rather than weaken it.

In view of the fact that drug seeking behavior can involve considerable work, it is useful to consider this research in the context of studies related to drug seeking and drug addiction. Indeed, over the last few years there has been a growing recognition of the importance of behavioral activation and effort-related processes in drug self-administration (Colby et al., 2003; Czachowski et al., 2002; Marinelli et al., 1998; Nadal et al., 2001; Vezina, 2002). Moreover, research related to behavioral activation and effort may offer clues as to the neurobiological basis of energy-related disorders in humans (Salamone et al., 2006). Symptoms such as psychomotor slowing, fatigue, and anergia are psychopathologies that include a disorders of behavioral activation in humans, and they represent fundamental aspects of depression and other psychiatric and neurological disorders (Capuron et al., 2007; Demyttenaere et al., 2005; Majer et al., 2008; Salamone et al., 2006, 2007; Stahl, 2002; Yurgelun-Todd et al., 2007). There is a striking similarity between the brain systems involved in effort-related processes in animals and those that are linked to energy-related symptoms in humans (Salamone et al., 2006, 2007). Thus, research in this area may promote our understanding of the neural mechanisms involved in clinical psychopathologies related to behavioral activation and effort.