Dopamine, Behavioral Economics, and Effort

Abstract

There are numerous problems with the hypothesis that brain dopamine (DA) systems, particularly in the nucleus accumbens, directly mediate the rewarding or primary motivational characteristics of natural stimuli such as food. Research and theory related to the functions of mesolimbic DA are undergoing a substantial conceptual restructuring, with the traditional emphasis on hedonia and primary reward yielding to other concepts and lines of inquiry. The present review is focused upon the involvement of nucleus accumbens DA in behavioral activation and effort-related processes. Viewed from the framework of behavioral economics, the effects of accumbens DA depletions and antagonism on food-reinforced behavior are highly dependent upon the work requirements of the instrumental task, and DA depleted rats are more sensitive to increases in response costs (i.e., ratio requirements). Moreover, interference with accumbens DA transmission exerts a powerful influence over effort-related choice behavior. Rats with accumbens DA depletions or antagonism reallocate their instrumental behavior away from food-reinforced tasks that have high response requirements, and instead these rats select a less-effortful type of food-seeking behavior. Nucleus accumbens DA and adenosine interact in the regulation of effort-related functions, and other brain structures (anterior cingulate cortex, amygdala, ventral pallidum) also are involved. Studies of the brain systems regulating effort-based processes may have implications for understanding drug abuse, as well as energy-related disorders such as psychomotor slowing, fatigue or anergia in depression and other neurological disorders.

Limitations of the Reward Hypothesis of Dopaminergic Function

The last several years have seen substantial theoretical developments related to the hypothesized behavioral functions of nucleus accumbens dopamine (DA). It has become evident to many investigators that there are conceptual limitations and empirical problems with the traditional DA hypothesis of “reward” (Baldo and Kelley, 2007; Barbano and Cador, 2007; Salamone et al., 1997, 2005, 2007; Salamone, in press). Even the use of the term “reward” itself often is problematic (Cannon and Bseikri, 2004; Salamone et al., 2005; Salamone, 2006; Sanchis-Segura and Spanagel, 2006; Yin et al., 2008). Researchers rarely define what they mean by “reward” when they are using it to describe a psychological process; some use it as though it were a synonym for “reinforcement”, or in reference to “appetite” or “primary motivation”, while still others employ it as a code word to mean “pleasure”. In some papers, the word “reward” seems to be used as a rather monolithic, all-encompassing term that refers to any and all aspects of appetitive learning, motivation and emotion, whether conditioned or unconditioned. Used in this way, the term reward is a rather blunt instrument. These problems are not merely semantic, as it is difficult to test a hypothesis which maintains that a neurotransmitter mediates such an ill-defined set of functions. It has been suggested that it is advantageous to maintain the distinction between the terms reward and reinforcement; with this usage, reinforcement refers more directly to instrumental learning mechanisms (Wise, 2004; Sanchis-Segura and Spanagel, 2006), while reward connotes the primary motivational and emotional effects of reinforcing stimuli (Everitt and Robbins, 2005; Salamone et al., 2005, 2007).

Against the backdrop of these conceptual and terminological issues, there is a tremendous weight of empirical evidence that has built up against the various iterations of the DA hypothesis of “reward”. It is somewhat ironic that the processes most directly linked to the use of the term reward (i.e., primary motivation, subjective pleasure) are the ones that have proven to be most problematic in terms of demonstrating the involvement of mesolimbic DA (Salamone et al., 2007). For example, low doses of DA antagonists and depletions of nucleus accumbens DA have been shown to produce effects that do not closely resemble extinction (Salamone, 1986; Salamone et al., 1995, 1997; Rick et al., 2006), pre-feeding (Salamone et al., 1991; Aberman and Salamone, 1999), or appetite suppressant drugs (Cousins et al., 1994; Salamone et al., 2002; Sink et al., 2008). Although it is well known that whole forebrain DA depletions can produce aphagia (i.e., lack of eating), it is DA depletions in the lateral or ventrolateral caudate/putamen, rather than the nucleus accumbens, which have most conclusively been linked to this effect (Ungerstedt, 1971; Dunnett and Iversen, 1982; Salamone et al., 1993a). It has been shown repeatedly that nucleus accumbens DA depletions or antagonism do not substantially impair appetite for food, or produce a general disruption of primary food motivation (Ungerstedt, 1971; Koob et al., 1978; Bakshi and Kelley, 1991; Salamone et al., 1993a). In DA deficient mice, restoration of DA production in caudate putamen, but not nucleus accumbens, was able to rescue feeding behavior (Szczypka et al., 2001). In summarizing their findings that injections of DA D₁ or D₂ family antagonists into either the core or the shell subregions of nucleus accumbens impaired locomotion and rearing, but did not suppress food intake, Baldo et al. (2002) stated that DA receptor blockade “did not abolish the primary motivation to eat” (p. 176).

Furthermore, the idea that nucleus accumbens DA mediates the pleasure associated with positive reinforcers has been strongly challenged (Berridge, 2007; Salamone et al., 2007; Berridge and Kringlebach, 2008). Interference with accumbens DA transmission does not impair appetitive taste reactivity for sucrose (Berridge, 2007; Berridge and Kringlebach, 2008). Several studies in humans have reported that DA antagonists did not blunt the subjective euphoria produced by drugs of abuse (Gawin, 1986; Brauer and De Wit, 1997; Haney et al., 2001; Nann-Vernotica et al., 2001; Wachtel et al., 2002). Moreover, the potential role of DA systems in instrumental behavior or learning is not limited to situations involving appetitive motivation. There is considerable evidence that striatal mechanisms in general, and mesolimbic DA in particular, also participate in aspects of aversive learning and aversive motivation (Salamone, 1994; Munro and Kokkinidis, 1997; Blazquez et al., 2002; Pezze and Feldon, 2004; Delgado et al., 2008; Faure et al., 2008; Martinez et al., 2008). Although imaging studies often are used to support the idea that nucleus accumbens mediates pleasure (e.g., Sarchiapone et al., 2006; Wacker et al., 2009), this appears to be oversimplified; indeed, research employing various imaging methods has demonstrated that the human nucleus accumbens also responds to stress, aversion and hyperarousal/irritability (Liberzon et al., 1999; Jensen et al., 2003; Pavic, 2003; Phan et al., 2004; Pruessner et al., 2004; Levita et al., 2009). Physiological and neurochemical studies in animals clearly indicate that DA neuron activity is not simply tied to the delivery of primary reinforcers or pleasurable stimuli. Rather, VTA neuron activity and DA release can be activated by a number of different appetitive and aversive conditions (McCullough and Salamone, 1992; McCullough et al., 1993; Guarraci and Kapp, 1999; Roitman et al., 2004; Young, 2004; Anstrom and Woodward, 2005; Broom and Yamamoto, 2005; Marinelli et al., 2005; Schultz, 2007a,b; Brischoux et al., 2009), with changes seen across varying time scales, including tonic, slow phasic and fast phasic signals (Salamone, 1996; Salamone et al., 2007; Schultz, 2007a,b; Salamone, in press; see also Lapish et al., 2007 for a discussion of various time scales associated with the postsynaptic effects of DA release and DA receptor stimulation).

Of course, one would not want to throw the baby out with the bathwater. It is apparent that mesolimbic DA participates in several complex functions related to aspects of instrumental behavior, learning and incentive motivation, and pavlovian/instrumental interactions (Wise, 2004; Everitt and Robbins, 2005; Kelley et al., 2005; Salamone et al., 2005, 2007; Berridge, 2007; Robbins and Everitt, 2007; Redgrave et al., 2008; Yin et al., 2008). The more difficult aspect of research and theory in this area is to ask – which specific aspects? Exploration of these diverse areas of dopaminergic function has become a rich and fruitful area of inquiry. Indeed, this literature is so extensive that a thorough review of the behavioral functions of nucleus accumbens DA is beyond the scope of the present article (see Salamone et al., 2007). For the purposes of this special issue, the present review will focus upon the role of nucleus accumbens DA in effort-related processes, with a special emphasis on effort-based choice behavior that depends upon cost/benefit analyses.

Behavioral Activation, Exertion of Effort, and Nucleus Accumbens DA

Even as the popularity of the DA hypothesis of reward was growing during the 1980s, it was becoming apparent that there were alternative conceptual frameworks available for organizing what was known about the behavioral functions of DA systems, particularly mesolimbic DA. Mogenson et al. (1980) suggested that nucleus accumbens acted as a functional interface between the limbic system and the motor system, facilitating the ability of information related to emotion and motivation to impinge upon the neural systems involved in the instigation of action. It had been emphasized for several decades that behavioral activation, i.e., the vigor, persistence and effort seen in the pursuit of motivational stimuli, and the heightened activity induced by conditioned stimuli that predict reinforcers, was a fundamental aspect of motivation (e.g., Cofer and Appley, 1964). Several investigators suggested that DA systems were involved in behavioral activation. DA antagonists or accumbens DA depletions were shown to suppress the activities such as excessive drinking, wheel running, and locomotion that are induced by scheduled presentation of food (Robbins and Koob, 1980; Wallace et al., 1983; Salamone, 1986, 1988). It also was reported that the effects of DA antagonists on reinforced behavior interacted powerfully with the kinetic requirements of the instrumental response. For example, doses of DA antagonists that suppressed reinforced lever pressing had minimal effects on reinforced nose poking behavior (Ettenberg et al., 1981; Mekarski, 1988). Although 0.1 mg/kg haloperidol severely reduced responding on a fixed ratio (FR) 20 schedule of lever pressing, a dose four times that size had no effect on the reinforced response of simply being in proximity to the food dish on a fixed interval 30 s schedule (Salamone, 1986). As this research was being reported, investigators began to employ economic concepts, such as exertion of effort and cost-benefit analyses, to describe the behavioral functions of accumbens DA. Neill and Justice (1981) hypothesized that injection of amphetamine into nucleus accumbens could be increasing the “willingness” of rats to exert effort to obtain a given level of reinforcement. In a contemporary review of the behavioral functions of DA systems (Salamone, 1987), it was noted that DA in nucleus accumbens could be involved in the “exertion of effort”, and it was suggested that future experiments could “offer animals choices between various reinforcers that are associated with operants of varying difficulty” (p. 602) so that researchers could determine if the allocation of behavioral resources could be biased toward or away from more or less effortful responses by administration of dopaminergic drugs.

This recognition of dopaminergic involvement in the exertion of effort, and effort-based choices related to cost benefit analyses, fit nicely with an emerging emphasis in the behavioral literature on work, response costs or constraints, and economic models of operant behavior. Several behavioral investigators have emphasized how response costs or constraints affect operant response output (Staddon, 1979; Kaufman, 1980; Kaufman et al., 1980; Foltin, 1991). Collier and colleagues studied how work requirements, such as the number of lever presses necessary for obtaining food, could serve as determinants of response output and affect consumption parameters (Collier and Jennings, 1969; Johnson and Collier, 1987). Economic models of operant behavior have emphasized how a number of factors, including not only reinforcement value, but also conditions related to the characteristics of the instrumental response, can determine behavioral output (Lea, 1978; Allison, 1981, 1993; Bickel et al., 2000). Hursh et al. (1988) suggested that, in terms of behavioral economics, the price of food reinforcement as a commodity is a cost/benefit ratio expressed as the effort expended per unit of food value consumed. Optimal foraging theory was proposed to account for the observation that the amount of effort or time expended to obtain motivational stimuli was an important determinant of foraging choice (Krebs, 1977), an idea that is still very influential in the ethology research today (e.g., Hengeveld et al., 2009).

Over the last two decades, several lines of evidence have converged to strengthen the original observation that the effects of interference with DA transmission interact powerfully with the work requirements of an instrumental task. One of the ways of controlling work requirements in an operant schedule is to vary the ratio requirement (i.e., the number of times the animal must press the lever to receive a unit of reinforcement). The effects of the DA antagonist haloperidol on food-reinforced behavior were shown to be dependent upon the particular ratio schedule that was used [i.e., FR1 vs. progressive ratio; Caul and Brindle (2001)]. Accumbens DA depletions also produce effects that interact powerfully with the ratio requirement of the schedule employed. Ishiwari et al. (2004) found that accumbens DA depletions that substantially impaired FR5 lever pressing had no significant effect on FR1 performance. Aberman and Salamone (1999) systematically studied a wide range of ratio schedules (FR1, 4, 16, and 64) to assess the effects of accumbens DA depletions. While FR1 performance was not affected by DA depletion, and FR4 responding was only transiently and mildly suppressed, the schedules with large ratio requirements (i.e., FR16 and FR64) were severely impaired (Figure 1A). In fact, DA depleted animals responding on the FR64 schedule showed significantly fewer responses than those responding on the FR16 schedule (Aberman and Salamone, 1999). This pattern indicates that accumbens DA depletions exacerbate an effect known as ratio strain. In untreated animals, the overall relation between ratio size and response output is inverted-U shaped. Up to a point, as ratio requirements get larger, animals adjust to this challenge by increasing response output. However, if the ratio requirement is high enough (i.e., if the cost is too high), the animal reaches the point at which additional responses being required actually tend to suppress responding. For normal rats, responding at levels of FR64, FR100 or higher, even if there is only one 45 mg food pellet being delivered, does not seem to be problematic. A completely different function is shown by rats with accumbens DA depletions, which are much more sensitive to the size of the ratio requirement. In behavioral economic terms, this pattern can be described as reflecting an increase in the elasticity of demand for food reinforcement (Salamone et al., 1997; Aberman and Salamone, 1999; see Figure 1B). The term elasticity is widely used in economics, but price elasticity of demand refers to the sensitivity of consumption to changes in price (Vuchinich and Heather, 2003). Thus, if the ratio requirement is analogous to the price of the commodity (in this case, reinforcement pellets), it appears that rats with accumbens DA depletions are more sensitive than control animals to the price of the food reinforcers. Of course, rats do not use currency to purchase operant pellets; rather, it has been argued that an operant procedure is more of a barter system, in which the rat trades its work (or reductions in leisure) for a commodity (Rachlin, 2003). Thus, another way of describing this effect of impaired DA transmission is to say that rats with accumbens DA depletions are more sensitive than control animals to work-related response costs, or that they are less likely to trade high levels of work for food. In another study (Salamone et al., 2001), the increased effects of accumbens DA depletions with increasing ratio requirements were observed when rats were tested across a broader range of ratio schedules as high as FR300, even when the overall relation between lever pressing and food delivered per lever press was kept constant (i.e., FR50, one pellet every 50 responses; FR100, two pellets every 100 responses; FR200, four pellets every 200 responses; FR300, six pellets every 300 responses; Salamone et al., 2001). Thus, both the magnitude and the organization of the ratio requirement appear to be critical determinants of the sensitivity of an operant schedule to the effects of accumbens DA depletions.

Figure 1.

This figure shows the effect of ratio requirement on the number of lever presses emitted and operant pellets consumed in rats with accumbens DA depletions compared to rats in the vehicle control group. Figure (A) is re-drawn based upon Aberman and Salamone (1999); these data are depicted in terms of number of responses, as in the original article. For (B), the data are represented as number of operant pellets consumed. Each data point shown is the mean value from each group at each ratio level. Although comparable levels of consumption in DA depleted and control groups were seen with the FR1 schedule, DA-depleted rats showed markedly reduced consumption relative to the control group at higher ratio levels.

In order to be sure that these results reflected the influence of ratio size, as opposed to other variables such as time, additional studies examined the effects of accumbens DA depletions on tandem schedules, in which a ratio requirement was attached to an interval requirement. In a conventional variable interval (VI) schedule, a time interval must elapse before the first response is reinforced, and the particular time interval varies around an average value. A tandem VI/FR schedule has an additional ratio requirement attached to the interval. For example, with a tandem VI 30 s/FR5 schedule, the animal is reinforced for the fifth response after the interval elapses, rather than the first. In this way, one can vary the ratio requirement of a schedule while keeping the programmed time intervals the same. Research employing tandem VI/FR schedules with varying combinations (e.g., VI 30 s/FR5, VI 60 s/FR10, VI 120 s/FR10) has yielded a consistent pattern; accumbens DA depletions do not impair overall response output in rats responding on the conventional VI schedules (i.e., those requiring only one response after the interval), but do substantially reduce responding on the corresponding VI schedule with the higher ratio requirement attached (Correa et al., 2002; Mingote et al., 2005). These results are consistent with research showing that accumbens DA antagonism did not impair performance on a progressive interval task (Wakabayashi et al., 2004), and suggest that interval requirements per se do not pose a severe constraint to rats with compromised DA transmission in nucleus accumbens. This serves to underscore the critical importance of ratio requirements as providing a work-related challenge to rats with accumbens DA depletions or antagonism.

In summarizing these results, Salamone and Correa (2002) stated that nucleus accumbens DA depletions appear to have two major effects: (1) they reduce the response-enhancing effects that moderate-size ratio requirements have on operant responding (i.e., the ascending limb of the function relating ratio requirement to response output), and (2) they enhance the response-suppressing effects that very large ratios have on operant responding (i.e., the descending limb of the function, enhancing ratio strain). Furthermore, finer grained analyses of detailed patterns of responding reveal more insights into the behavioral manifestations of accumbens DA depletions. Accumbens DA depletions produce a slight reduction in the local rate of responding, as indicated by the distribution of inter-response times (Salamone et al., 1993b, 1999; Mingote et al., 2005). In addition, they enhance pauses in responding (Salamone et al., 1993b; Mingote et al., 2005). The latter may indicate a fragmentation in the pattern of responding (Mingote et al., 2005), a reduction in the ability to sustain uninterrupted response output, or a lack of engagement in the task (Nicola, 2007). Recently, computational approaches have been used to analyze these effects of accumbens DA depletions on response rate (e.g., Niv et al., 2007; Phillips et al., 2007). This relation between response output and DA function has been interpreted to mean that DA release in nucleus accumbens could provide a window of opportunistic drive during which the threshold cost expenditure to obtain the reward is decreased (Phillips et al., 2007).

In discussing the effects of dopaminergic manipulations on ratio performance, it is useful to mention the term “reinforcement efficacy”, which is sometimes used to describe the effects of drug manipulations on progressive ratio performance. With progressive ratio schedules, the ratio requirement increases as successive ratios are completed, and the “break point” is said to occur at the point at which the animal essentially ceases to respond. One can operationally define reinforcement efficacy in terms of the break point in a progressive ratio schedule (and also by measuring ratio strain in rats responding across different FR schedules). The determination of reinforcement efficacy can be a very useful tool for characterizing some of the fundamental reinforcing actions of drugs that are self-administered, and for comparing self-administration behavior across different substances or classes of substances (e.g., Richardson and Roberts, 1996; Marinelli et al., 1998; Woolverton and Ranaldi, 2002). Used in this manner, reinforcement efficacy is essentially being employed as an empirical descriptor of a particular behavioral outcome. Nevertheless, given the terminological problems mentioned above, it is worth emphasizing that the term “reinforcement efficacy” should not be used simply as a replacement for “reward”, nor should progressive ratio breakpoints be viewed as necessarily providing some direct and unambiguous measure related to the subjective pleasure produced by the stimulus (Salamone, 2006). Changes in progressive ratio break points can reflect more than just changes in the appetitive motivational properties of a reinforcing stimulus (Richardson and Roberts, 1996; Hamill et al., 1999). For example, changing the kinetic requirements of the instrumental response (e.g., increasing the height of the lever) was shown to decrease progressive ratio break points (Skjoldager et al., 1993; Schmelzeis and Mittleman, 1996). Although some researchers have maintained that the break point provides a direct measure of the appetitive motivational characteristics of a stimulus, it is, as explicitly stated in a classic review by Stewart (1974), more directly a measure of how much work the organism will do in order to obtain that stimulus. Progressive ratio break points and measures of ratio strain are essentially outcomes that result from effort-related decision making processes. The animal is making a cost/benefit choice about whether or not to continue to respond, based partly on factors related to the reinforce itself, but also upon the work-related response costs and time constraints imposed by the ratio schedule. For these reasons, interpretations of the actions of drugs or lesions on progressive ratio break points should be done with caution, as should be the case for any individual task. A drug that alters the break point could do so for many different reasons; it may be affecting functions related to the processing of reward value, or alternatively it could be affecting exertion of effort, or decision making processes.

Response Allocation, Effort-Related Choice Behavior, and Nucleus Accumbens DA

The ability to exert effort, sustain work, overcome obstacles, and attain access to motivationally relevant stimuli is necessary for survival. But it is only part of the story. In a complex environment, which affords many opportunities for obtaining significant stimuli, and multiple paths for accessing them, organisms must make choices. The variables that need to be evaluated to make these decisions are complex and multidimensional, but among the most important are those involving cost/benefit assessments based upon effort and reinforcement value (Salamone and Correa, 2002; Salamone et al., 2003, 2005, 2007; van den Bos et al., 2006; Walton et al., 2006). Considerable evidence indicates that nucleus accumbens DA, along with other transmitters and structures, participates in the overall circuitry that regulates effort-based choice behavior (Salamone et al., 2003, 2005, 2007; Floresco et al., 2008a; Hauber and Sommer, 2009).

One of the procedures that has been used to assess the contribution of accumbens DA to response allocation and effort-related choice behavior is a task that offers rats the option of either lever pressing to obtain a relatively preferred food (e.g., Bioserve pellets; usually obtained on a FR5 schedule), or approaching and consuming a less preferred food (lab chow) that is concurrently available in the chamber. Well trained rats under baseline conditions typically get most of their food by lever pressing, and consume only small quantities of chow (Salamone et al., 1991). Low-to-moderate doses of DA antagonists, which block either D₁ or D₂ family receptor subtypes, produce a substantial alteration of response allocation in rats performing on this task. The DA antagonists cis-flupenthixol, haloperidol, raclopride, eticlopride, SCH 23390, SKF83566, and ecopipam all decreased lever pressing for food but substantially increased intake of the concurrently available chow (Salamone et al., 1991, 1996, 2002; Cousins et al., 1994; Sink et al., 2008; Worden et al., 2009). The use of this task for assessing effort-related choice behavior has been validated in many ways. For example, the low dose of haloperidol that produced the shift from lever pressing to chow intake (0.1 mg/kg) did not affect total food intake or alter preference between these two specific foods in free-feeding choice tests (Salamone et al., 1991). Although DA antagonists have been shown to reduce FR5 lever pressing and increase chow intake, appetite suppressants from different classes, including amphetamine (Cousins et al., 1994), fenfluramine (Salamone et al., 2002), and cannabinoid CB1 antagonists (Sink et al., 2008), failed to increase chow intake at doses that suppressed lever pressing. Similarly, pre-feeding to reduce food motivation was shown to suppress both lever pressing and chow intake (Salamone et al., 1991). Furthermore, attachment of higher ratio requirements (up to FR20) caused animals that were not drug treated to shift from lever pressing to chow intake (Salamone et al., 1997), indicating that this task is sensitive to work load. Together with other results, these findings demonstrate that interference with DA transmission does not simply reduce appetite, but does act to alter response allocation between alternative sources of food that can be obtained through different instrumental responses.

The shift from lever pressing to chow intake in rats performing on this task is associated with DA depletions in nucleus accumbens, but not the neostriatum. Although it has been suggested that caudate/putamen DA may have some types of motivational functions related to feeding (Palmiter, 2007), DA depletions in anteroventromedial neostriatum, which is dorsal to nucleus accumbens, had no behavioral effect, while ventrolateral neostriatal DA depletions produced severe motor impairments that merely decreased both lever pressing and feeding (Cousins et al., 1993). In contrast, decreases in lever pressing and increases in chow intake occur as a result of accumbens DA depletions, as well as intra-accumbens injections of D₁ or D₂ antagonists (Salamone et al., 1991; Cousins et al., 1993; Cousins and Salamone, 1994; Sokolowski and Salamone, 1998; Koch et al., 2000; Nowend et al., 2001). The shift from lever pressing to chow intake on this task has been shown to occur in rats if D₁ or D₂ family antagonist are injected into the medial core, lateral core, or dorsal shell subregions of the accumbens (Salamone et al., 1991; Nowend et al., 2001). Thus, although lever pressing is decreased by accumbens DA antagonism or depletions, the rats show a compensatory reallocation of behavior and select a new path to an alternative food source. Consistent with these effects observed in rats that have impaired DA transmission, DA transporter knockdown mice, which have enhanced DA transmission, show increased selection of lever pressing relative to chow intake when tested with this task (Cagniard et al., 2006).

Salamone et al. (1994) also developed a T-maze procedure in order to assess the effects of DA antagonists and accumbens DA depletions on effort-related decision making. With this procedure, the two choice arms of the maze can have different reinforcement densities (e.g., four vs. two food pellets, or four vs. zero food pellets), and under some conditions a 44-cm barrier can be placed in the arm with the higher density of food reinforcement to present an effort-related challenge to the rat. When no barrier is placed in the arm with the high reinforcement density, rats mostly choose that arm, and neither haloperidol nor accumbens DA depletion alters their response choice (Salamone, 1994). When the arm with the barrier contained four pellets, but the other arm contained no pellets, rats with accumbens DA depletions were very slow, but still managed to choose the high density arm, climb the barrier, and consume the pellets (Cousins et al., 1996). Yet accumbens DA depletions dramatically altered choice behavior when the high density arm (four pellets) had the barrier in position, and the arm without the barrier contained an alternative food source (two pellets). In this case, DA depletions or antagonism decreased choice for the high density arm, and increased choice for the low density arm (Salamone, 1994; Cousins et al., 1996; Denk et al., 2005; Mott et al., 2009). Like the operant concurrent choice task, the T-maze task for measuring effort-based choice behavior also has undergone considerable behavioral validation and evaluation (Salamone, 1994; Cousins et al., 1996; van den Bos et al., 2006; Correa et al., 2009). For example, in a recent T-maze choice study with mice, it was confirmed that haloperidol reduced choice of the arm with the barrier, and it also was demonstrated that haloperidol had no effect on choice when both arms had a barrier in place (Correa et al., 2009). Thus, dopaminergic manipulations did not alter the preference for the high density of food reward over the low density, and did not affect discrimination or memory processes related to arm preference. Over the last several years, variants of this task have been used by several laboratories to characterize the effects of brain lesions or drug manipulations (Salamone, 1994; Walton et al., 2003; Denk et al., 2005; Schweimer and Hauber, 2005; van den Bos et al., 2006; Bardgett et al., 2009; Hauber and Sommer, 2009; Mott et al., 2009). The results of the T-maze studies in rodents, together with the findings from the operant concurrent choice studies reviewed above, indicate that low doses of DA antagonists and accumbens DA depletions cause animals to reallocate their instrumental response selection based upon the response requirements of the task, and select lower effort alternatives for obtaining rewards (see reviews by Salamone et al., 2003, 2005, 2007; Floresco et al., 2008a).

Recent papers have used effort discounting procedures to study the effects of dopaminergic manipulations. Floresco et al. (2008b) investigated the effects of dopaminergic and glutamatergic drugs on both effort and delay discounting. The DA antagonist haloperidol altered effort discounting even when the effects of time delay were controlled for (Floresco et al., 2008b). A T-maze effort discounting task was recently developed (Bardgett et al., 2009), in which the amount of food in the high density arm of the maze was diminished each trial on which the rats selected that arm (i.e., an “adjusting-amount” discounting variant of the T-maze procedures, which allows for the determination an indifference point for each rat). Administration of both the D₁ family antagonist SCH23390 and the D₂ family antagonist haloperidol altered effort discounting, making it more likely that rats would choose the arm with the smaller reward. Increasing DA transmission by administration of amphetamine blocked the effects of SCH23390 and haloperidol, and also biased rats toward choosing the high reward/high cost arm, which is consistent with operant choice studies using DA transporter knockdown mice (Cagniard et al., 2006). Together with other results, the findings reported by Bardgett et al. (2009) and Floresco et al. (2008b) support the suggestion that, across a variety of conditions, DA transmission exerts a bidirectional influence over effort-related decision making.

Interactions between DA and Adenosine

As reviewed above, considerable research has demonstrated that DA antagonists and accumbens DA depletions affect behavioral activation, instrumental response output, response allocation, and effort-related processes (Salamone et al., 1991, 2007; Salamone and Correa, 2002; Phillips et al., 2007; Robbins and Everitt, 2007; Floresco et al., 2008a). Clearly, DA does not participate in effort-related processes in isolation, and for that reason it is important to review how other brain areas and neurotransmitters interact with dopaminergic mechanisms. Within the last few years, considerable emphasis has been placed upon interactions between DA and adenosine. Caffeine and other methylxanthines, which act as minor stimulants, are non-selective adenosine antagonists (Ferré et al., 2008). Recently, there has been a rapid growth of research on adenosine receptor neurochemistry and pharmacology, particularly concerning the A_2A subtype of adenosine receptor. DA-rich striatal areas, including both the caudate/putamen (neostriatum) and the nucleus accumbens, have a very high degree of adenosine A_2A receptor expression (Schiffmann et al., 1991; DeMet and Chicz-DeMet, 2002; Ferré et al., 2004). There is considerable evidence of a functional interaction between striatal DA D₂ and adenosine A_2A receptors (Fink et al., 1992; Ferré, 1997; Hillion et al., 2002; Fuxe et al., 2003). This interaction frequently has been studied in regard to neostriatal motor functions that are related to parkinsonism (Ferré et al., 1997, 2001; Hauber and Munkel, 1997; Svenningsson et al., 1999; Hauber et al., 2001; Wardas et al., 2001; Morelli and Pinna, 2002; Correa et al., 2004; Jenner, 2005; Pinna et al., 2005; Ishiwari et al., 2007; Salamone et al., 2008a,b). Several reports also have characterized aspects of adenosine A_2A receptor function related to cognitive processes (Takahashi et al., 2008), anxiety (Correa and Font, 2008), and motivation (Salamone et al., 2007; Mingote et al., 2008).

Adenosine A_2A receptors also are involved in aspects of behavioral activation and effort-related processes (Farrar et al., 2007; Font et al., 2008; Mingote et al., 2008; Mott et al., 2009; Worden et al., 2009). Injections of the adenosine A_2A agonist CGS 21680 directly into the accumbens can produce effects that resemble those of accumbens DA depletions or antagonism. Intra-accumbens injections of CGS 21680 were shown to reduce locomotor activity (Barraco et al., 1993). Local infusions of CGS 21680 into the accumbens reduced responding on a VI 60 s schedule with a FR10 requirement attached, but did not impair performance on a conventional VI 60 s schedule (Mingote et al., 2008); this pattern is similar to that previously shown with accumbens DA depletions (Mingote et al., 2005). In rats responding on the operant FR5/chow feeding concurrent choice procedure, injections of CGS 21680 into the accumbens decreased lever pressing and increased chow intake (Font et al., 2008), a pattern of effects similar to that produced by accumbens DA depletions and antagonism. Consistent with the observation that an adenosine A_2A agonist could produce actions similar to those resulting from DA depletion or blockade, it also has been reported the locomotor suppression induced by the DA antagonist haloperidol was reduced by injections of the adenosine A_2A antagonist MSX-3 into nucleus accumbens core, but not into the shell or the ventrolateral neostriatum (Ishiwari et al., 2007). Furthermore, it has been demonstrated that adenosine A_2A receptor antagonists can reverse the effects of DA D₂ antagonists on both the operant concurrent choice task (Farrar et al., 2007; Salamone et al., 2009; Worden et al., 2009) and the T-maze choice procedure (Correa et al., 2009; Mott et al., 2009). Recently, studies with intracranial injections revealed that systemic or intra-accumbens injections of the adenosine A_2A antagonist MSX-3 were able to block the effects of intra-accumbens injections of the D₂ antagonist eticlopride in rats responding on the operant concurrent choice task (Farrar, 2009, unpublished doctoral dissertation, University of Connecticut).

These studies afford an interesting opportunity to assess the overall interaction between DA and adenosine receptor subtypes. Adenosine A_2A receptor antagonists MSX-3 and KW 6002 reliably attenuate the effects of D₂ antagonists such as haloperidol and eticlopride in rats responding on the operant concurrent choice procedure (Farrar et al., 2007; Salamone et al., 2009; Worden et al., 2009). In contrast, MSX-3 was relatively ineffective at reducing the effects of the D₁ antagonist ecopipam (SCH 39166; Worden et al., 2009) on this task. Although the non-selective adenosine antagonist caffeine was able to partially reverse the effects of haloperidol on the concurrent choice task, DPCPX, which is highly selective for the adenosine A₁ receptor subtype, was ineffective (Salamone et al., 2009). Similar results were obtained with rats and mice responding on the T-maze barrier choice task. Although MSX-3 was able to reverse the effect of haloperidol on selection of the arm with the barrier, the A₁ antagonists DPCPX and CPT were not (Correa et al., 2009; Mott et al., 2009).

The results described above demonstrate that there is a relatively selective interaction between DA D₂ and adenosine A_2A receptor subtypes (Table 1). Based upon anatomical studies, it appears that this is likely to be due to the pattern of cellular localization of adenosine A₁ and A_2A receptors in striatal areas, including the nucleus accumbens (Ferré, 1997). Adenosine A_2A receptors are typically co-localized on striatal and accumbens enkephalin-positive medium spiny neurons with DA D₂ family receptors, and these receptors converge onto the same signal transduction pathways and show the capacity for forming heteromeric complexes (Fink et al., 1992; Ferré, 1997; Svenningsson et al., 1999; Hillion et al., 2002; Fuxe et al., 2003). Thus, adenosine A_2A receptor antagonists appear to be so effective in reversing the effort-related actions of D₂ antagonists because of direct interactions between DA D₂ and adenosine A_2A receptors located on the same neurons (Figure 2). On the other hand, DA D₁ receptors are more likely to be co-localized with adenosine A₁ receptors (Ferré, 1997), which could help to explain why it is more difficult for adenosine A₁ receptor antagonists to reverse the effects of D₂ receptor blockade. Interestingly, despite the fact that D₁ and A₁ receptors tend to be co-localized on the same neurons, the A₁ antagonists DPCPX and CPT were unable to reverse the effects of the D₁ antagonist ecopipam in rats responding on the concurrent choice operant procedure (Nunes et al., 2009). This suggests that A_2A antagonists exert an overall greater effect than A₁ antagonists on effort-related functions of nucleus accumbens.

Table 1.

Interactions between dopamine and adenosine receptor antagonists.

	Adenosine receptor antagonist
	Non-selective	A1	A2A
CONCURRENT FR5/FREE CHOW
D2 receptor antagonist	Reversal	No reversal	Reversal	Farrar et al. (2007), Salamone et al. (2009), Worden et al. (2009)
D1 receptor antagonist	–	No reversal	Partial reversal1	Worden et al. (2009)1, Nunes et al. (2009)2
T-MAZE WITH BARRIER
D2 receptor antagonist	Reversal	No reversal	Reversal	Mott et al. (2009), Pardo (2009)3

¹There was a mild increase in lever pressing in ecopipam-treated rats that received the A_2A antagonist MSX-3, but no reversal of the chow intake effect of the D₁ antagonist.

²Data from Nunes et al. (2009).

³Data from Pardo (2009), unpublished masters thesis, University of Jaume I.

Figure 2.

Anatomical diagram depicting the pattern of DA and adenosine receptor localization in nucleus accumbens. See text for details (see also Ferré, 1997; Hillion et al., 2002; Fuxe et al., 2003). mGP, medial globus pallidus; epn, entopeduncular nucleus; s. nigra, substantia nigra; VTA, ventral tegmental area.

Behavioral Theory and Analyses: Further Evaluation of Effort-Related Processes

Research on the brain mechanisms involved in effort-related processes may lead to new ways of thinking about behavioral analysis and theory in behavioral economics. One of the contributions that behavioral neuroscience can make to behavioral theory is to use manipulations (e.g., drugs, lesions) that dissociate complex behavioral processes into component parts (Salamone et al., 2007). In this regard, it is useful to consider that a given parameter that is generated from curve-fitting analyses, when viewed in terms of its biological characteristics, has many factors that contribute to it. A good example of this is the ED₅₀, which is used in pharmacology to provide a measure of the potency of a drug based upon dose-response analysis. Empirically, the ED₅₀ is the dose that produces an effect that is 50% of the maximal effect. Although the ED₅₀ is expressed as one number, that simplicity is deceptive because many biochemical factors contribute to it, including the affinity of a drug for a receptor, duration of action, drug metabolism, and penetration into the target tissue. A useful example of this principle from the behavioral neuroscience literature is the progressive ratio break point; as discussed above, this measure also has many factors that can contribute to it. Another case in which this point is important to consider is threshold measures used in intracranial self-stimulation studies. Such measures often are viewed as providing “rate-free” indices of reinforcement value, nevertheless, they are influenced by lever pressing ratio requirements as well as the electrical current level (Fouriezos et al., 1990).

Some research related to behavioral economics, reinforcer value, and the functions of DA systems has used response-reinforcement matching methods (e.g., Heyman and Monaghan, 1987; Aparicio, 2007). Matching equations have been employed to describe the results of studies with both conventional and concurrent VI schedules, and one of the parameters (R_e) can be used to represent reinforcement value (e.g., Herrnstein, 1974; see equation below for single-lever conventional VI schedules, in which B represents response rate, R represents reinforcement density, k is the constant for maximal responding, and R_e represents the reinforcement level that generates 50% of maximum responding).

B = k R/(R + R_e)

However, used in this way, R_e does not selectively represent only the reinforcement value of food per se; actually, it reflects the relative value of lever pressing for and consuming the food reinforcer compared to the reinforcing value of all other stimuli and responses available (Salamone et al., 1997). Several factors can contribute to this composite measure, which is one of the reasons why other matching equations have been developed that account for deviations from matching by allowing for estimates of reinforcer sensitivity, as well as response preference or bias (Baum, 1974; Williams, 1988; Aparicio, 2001). Clearly, a drug or lesion manipulation could yield apparent effects on “reinforcement value” that actually reflect changes in response bias (Salamone, 1987; Salamone et al., 1997).

For these reasons, it may be useful to think more deeply about how terms such as value are used in neuroeconomics research. The aggregate reinforcement value of an instrumental activity (e.g., lever pressing for and consuming food) should perhaps be viewed as a composite measure that includes both the reinforcing value of the reinforcer itself, plus any net value or costs associated with the instrumental response that is required to obtain the reinforcer. Viewed in this way, the effects of dopaminergic manipulations on effort-related choice behavior could be described in terms of actions upon the response costs associated with the particular reinforcer, rather than the reinforcing value of the food stimulus itself. Although the effects of haloperidol on bias may be minimal when two levers that are relatively similar are used (e.g., Aparicio, 2007), they may be much larger when very different responses are compared (e.g., lever pressing vs. sniffing; lever pressing vs. unrestricted access to food; barrier climbing vs. locomotion). Future research will determine if measures of bias based upon the matching equations, or some other type of mathematical formulation, would be the best way to capture these drug effects quantitatively.

Summary and Conclusions

In summary, DA and adenosine in the nucleus accumbens interact to regulate effort-related functions. Additional research has shown that a number of components of the cortico-striato-pallidal loop system also are involved (Walton et al., 2006; Floresco and Ghods-Sharifi, 2007; Farrar et al., 2008; Mingote et al., 2008; Hauber and Sommer, 2009). Disconnection studies have revealed that serial connections between basolateral amygdala, anterior cingulate cortex, nucleus accumbens, and ventral pallidum are involved in the exertion of effort and effort-related choice behavior (Floresco and Ghods-Sharifi, 2007; Farrar et al., 2008; Mingote et al., 2008; Hauber and Sommer, 2009). Within the last few years, there has been considerable progress in characterizing the functional anatomy underlying this important aspect of motivation and decision making. Several transmitters across multiple brain regions are involved in effort-related functions, and researchers are only beginning to piece together the complex puzzle of all the potential brain systems that are involved. Presently, the specific way in which each structure contributes to the overall function of the system is unclear. It is uncertain which brain areas are involved in the exertion of effort, or the perception of effort, vs. the actual decision making process itself. For example, it is possible that nucleus accumbens is involved in the actual decision making processes, but it also is possible that it is mainly involved in regulating energy output, or setting effort-related constraints or feedback that in turn influences decisions made at other levels in the system. If the latter is true, then it is possible that the decision making effects of drug or lesion manipulations of nucleus accumbens are an outcome reflecting the constraints that are set after compromised DA function in accumbens, rather than a direct effect upon decision making processes per se. Future research will be necessary to tease apart these distinct aspects of effort-related function.

In addition to providing insights into aspects of animal behavior and natural motivation, research on effort-related processes also has clinical implications. Within the last few years, there has been a greater emphasis upon effort-related functions involved in drug self-administration (e.g., Vezina et al., 2002). Drug seeking behavior in humans involves many psychological processes, including effort. Addicts will go to great lengths to obtain their preferred drug, overcoming numerous obstacles and constraints, both behavioral and economic. Furthermore, addiction is characterized not only by a re-organization of the preference structure of the person, but also by a dramatic change in the allocation of behavioral resources toward the addictive substance; there is a heightened emphasis upon drug seeking and drug taking, typically at the expense of other motivational activities. As well as being related to aspects of drug taking and addiction, research on behavioral activation and effort has implications for understanding the neural basis of psychiatric symptoms such as psychomotor slowing, anergia, fatigue and apathy, which are seen in depression as well as other psychiatric or neurological conditions (Salamone et al., 2006, 2007). These motivational symptoms, which can have devastating behavioral manifestations (Stahl, 2002; Demyttenaere et al., 2005), represent impairments in aspects of behavioral activation and effort that can lead to problems in the workplace, as well as limitations in terms of life function, interaction with the environment, and responsiveness to treatment. There is considerable overlap between the neural circuitry involved in effort-related functions in animals and the brain systems that have been implicated in psychomotor slowing and anergia in depression (Salamone et al., 2006). Thus, research on effort-related behavioral processes, and their neural regulation, could have substantial impact on clinical research related to addiction, depression, and other disorders.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.