The Behavioral Pharmacology of Effort-related Choice Behavior: Dopamine, Adenosine and Beyond

Abstract

For many years, it has been suggested that drugs that interfere with dopamine (DA) transmission alter the “rewarding” impact of primary reinforcers such as food. Research and theory related to the functions of mesolimbic DA are undergoing a substantial conceptual restructuring, with the traditional emphasis on hedonia and primary reward yielding to other concepts and lines of inquiry. The present review is focused upon the involvement of nucleus accumbens DA in effort-related choice behavior. Viewed from the framework of behavioral economics, the effects of accumbens DA depletions and antagonism on food-reinforced behavior are highly dependent upon the work requirements of the instrumental task, and DA-depleted rats show a heightened sensitivity to response costs, especially ratio requirements. Moreover, interference with accumbens DA transmission exerts a powerful influence over effort-related choice behavior. Rats with accumbens DA depletions or antagonism reallocate their instrumental behavior away from food-reinforced tasks that have high response requirements, and show increased selection of low reinforcement/low cost options. Nucleus accumbens DA and adenosine interact in the regulation of effort-related functions, and other brain structures (anterior cingulate cortex, amygdala, ventral pallidum) also are involved. Studies of the brain systems regulating effort-based processes may have implications for understanding drug abuse, as well as symptoms such as psychomotor slowing, fatigue or anergia in depression and other neurological disorders.

In order to survive, organisms must gain access to significant stimuli such as food, water, sex, and other conditions. The processes involved in such behavioral activities are varied and complex, and the brain mechanism related to these processes are a subject of considerable research activity. Instrumental learning processes involving reinforcement and punishment lead to the acquisition of behaviors that regulate the probability, proximity, and availability of significant stimuli. But even when such responses are already acquired, multiple factors contribute to the selection of particular instrumental behaviors in a given environmental context. For example, in a complex environment, organisms typically have access to multiple reinforcers, which can vary with regard to their quality, quantity, and temporal characteristics. In addition, distinct instrumental actions can be associated with particular reinforcers, and these actions can vary widely in topography and in terms of the quantitative features of the response requirements. Several areas of inquiry in behavioral science, including research on response–reinforcement matching, optimal foraging theory, and behavioral economics, have emerged in order to characterize the choice behavior observed in these complex environments (Allison, 1981, 1993; Aparicio, 2001, 2007; Baum, 1974; Hengeveld, van Langevelde, Groen, & de Knegt, 2009; Hursh, Raslear, Shurtleff, Bauman, & Simmons, 1988; Madden, Bickel, & Jacobs, 2000; Madden & Kalman, 2010; Salamone, 1987; Tustin, 1995; Vuchinich and Heather, 2003; Williams, 1988). This research has provided approaches for understanding how reinforcement value, as well as response requirements, influence the relative allocation of instrumental behavior across multiple options.

This perspectives article will provide an overview of recent research on the behavioral pharmacology of a specific aspect of these broader issues. One response-related factor that profoundly influences instrumental behavior is work-related response costs (Foltin 1991; Hursh et al., 1988; Kaufman 1980; Kaufman, Collier, Hill, & Collins, 1980; Madden et al., 2000; Salamone, 1986, 1987, 1992; Staddon 1979; Tustin, 1995). The present review will focus upon the effects of drugs and neurochemical manipulations that affect dopamine (DA) transmission, and how these effects interact with the response requirements, particularly ratio requirements, imposed upon food-reinforced instrumental behavior. In addition, the article will review the literature on the role of DA in effort-related choice behavior, with a particular emphasis upon DA in a brain area known as the nucleus accumbens. Finally, the interactions between nucleus accumbens DA and other neurotransmitters and brain areas will be discussed, and the broader relevance of these findings will be considered.

HYPOTHESIZED ACTIONS OF DA ANTAGONISTS: THE DECLINE AND FALL OF THE “REWARD” HYPOTHESIS OF DA FUNCTION

There have been substantial theoretical developments in the last few years related to the hypothesized behavioral functions of DA, particularly nucleus accumbens DA. In order to consider the involvement of DA in work-related aspects of instrumental response allocation, one should place these ideas into a historical context relative to other hypothesized functions of DA. A few decades ago, it became common in the behavioral neuroscience literature to label DA as a “reward” transmitter, which was said to produce feelings of subjective pleasure or motivational appetites that mediate or drive positive reinforcement phenomena. However, it has become evident to many investigators that there are conceptual limitations and empirical problems with the traditional DA hypothesis of “reward” (Baldo & Kelley, 2007; Barbano & Cador 2007; Salamone, Correa, Farrar, & Mingote, 2007; Salamone, Correa, Farrar, Nunes, & Collins, 2010; Salamone, Correa, Mingote, & Weber, 2005; Salamone, Cousins, & Snyder, 1997; Salamone, et al., 2009), not the least of which is the use of the term “reward” itself (Cannon & Bseikri 2004; Salamone 2006; Salamone et al. 2005; Sanchis-Segura & Spanagel, 2006; Yin, Ostlund, & Balleine, 2008). The term “reward” is rarely defined by researchers when they are using it to describe a behavioral process. Some use the term as though it were a synonym for “reinforcement”, while others use it in reference to “appetite” or “primary motivation”. Still others employ this term as a thinly veiled label for “pleasure”. In many cases, the word “reward” seems to be used as a rather monolithic, all-encompassing term that refers globally to all aspects of reinforcement learning, motivation and emotion, whether conditioned or unconditioned. If used in this manner, the term reward is so broad as to be practically meaningless. It should be evident that it is difficult to test a hypothesis which maintains that a neurotransmitter mediates such an ill-defined set of functions. Thus, it has been suggested that it is advantageous to maintain the distinction between the terms reward and reinforcement; with this usage, reinforcement refers more directly to instrumental learning mechanisms (Sanchis-Segura & Spanagel, 2006; Wise 2004), while reward tends to connote the primary motivational and emotional effects of reinforcing stimuli (Everitt & Robbins, 2005; Salamone et al., 2005, 2007).

In addition to these lexicographical and conceptual issues, there also is a substantial body of empirical evidence that has been accumulated in recent years, which fails to support the various forms of the DA hypothesis of “reward”. One ironic observation is that the processes most directly linked to the use of the term reward (i.e., subjective pleasure, primary motivation) are ones that have been demonstrated to be most problematic in terms of demonstrating the involvement of DA systems (Salamone et al., 2007). For example, the idea that nucleus accumbens DA mediates the subjectively reported pleasure associated with positive reinforcers has been strongly challenged (Berridge, 2007; Berridge & Kringlebach, 2008; Salamone et al., 2007). Interference with accumbens DA transmission does not impair appetitive taste reactivity for sucrose (Berridge, 2007; Berridge & Kringlebach, 2008), which is a frequently used behavioral marker of hedonic reactivity in rodents. Human studies have reported that DA antagonists failed to blunt the subjectively rated euphoria produced by drugs of abuse (Brauer & de Wit, 1997; Gawin, 1986; Haney, Ward, Foltin, & Fischman, 2001; Nann-Vernotica, Donny, Bigelow, & Walsh, 2001; Venugopalan et al., 2011; Wachtel, Ortengren, & de Wit, 2002).

Moreover, the potential role of DA systems in instrumental behavior or learning is not limited to situations involving positive reinforcement. There is considerable evidence that striatal mechanisms in general, and nucleus accumbens DA in particular, also participate in aspects of aversive learning, punishment, and responsiveness to aversive stimuli (Blazquez, Fujii, Kojima, & Graybiel, 2002; Delgado, Li, Schiller, & Phelps, 2008; Faure, Reynolds, Richard, & Berridge, 2008; Martinez, Oliveira, Macedo, Molina, & Brandao, 2008; Munro & Kokkinidis, 1997; Salamone, 1994). Although human imaging studies are used to support the idea that nucleus accumbens mediates subjective pleasure (e.g. Sarchiapone et al., 2006), the situation is much more complicated (Pizzagalli, 2010); indeed, research employing various imaging methods has demonstrated that the human nucleus accumbens also responds to stress, aversion and hyperarousal/irritability (Delgado et al., 2008; Delgado, Jou, & Phelps, 2011; Jensen et al., 2003; Levita et al., 2009; Liberzon et al., 1999; Pavic, 2003; Phan et al., 2004; Pruessner, Champagne, Meaney, & Dagher, 2004). Neurochemical and physiological studies in animals clearly indicate that DA neuron activity is not simply tied to the delivery of primary positive reinforcers. In studies involving food reinforcement in trained animals, increases in DA release were more strongly associated with the instrumental response, or conditioned stimuli signaling reinforcer availability, rather than reinforcement delivery (Roitman, Stuber, Phillips, Wightman, & Carelli, 2004; Segovia, Correa & Salamone, 2011; Sokolowski, Conlan, & Salamone, 1998). Moreover, DA neuron activity and DA release can be activated by a number of different aversive (e.g. footshock, tailshock, tail pinch, restraint stress, aversive conditioned stimuli, aversive drugs, social defeat stress) and appetitive conditions (Anstrom & Woodward 2005; Brischoux, Chakraborty, Brierley, & Ungless, 2009; Broom & Yamamoto 2005; Guarraci & Kapp 1999; Marinelli, Pascucci, Bernardi, Puglisi-Allegra, & Mercuri, 2005; McCullough & Salamone, 1992; McCullough, Sokolowski, & Salamone, 1993; Schultz 2007a, 2007b; Young, 2004). These neurochemical changes are seen across varying time horizons (including tonic, slow phasic and fast phasic changes; Hauber 2010; Roitman et al., 2004; Salamone 1996; Salamone et al. 2007; Schultz 2007a, 2007b; Segovia et al., 2011). Studies of learning indicate that DA systems in general and nucleus accumbens in particular are not only involved in learning related to reinforcement (e.g. Wise, 2004), but also are involved in learning related to punishment (Salamone et al., 2007; Schoenbaum & Setlow, 2003). Thus, it has been suggested that the term “instrumental learning” would be more broadly applicable than “reinforcement learning” for describing the hypothesized role of DA in learning processes (Salamone et al., 2007).

If DA antagonism is actually interfering with the fundamental characteristics of reinforcing stimuli, this prompts one to inquire as to what those characteristics are. Of course, reinforcement refers to behavioral contingencies that act to strengthen a particular behavior; positive reinforcement refers to a process by which a response is followed by the presentation of stimulus that typically is contingent upon that response, and these events are followed by an increase in the probability of the occurrence of that response in the future. However, it is worthwhile to consider what properties enable a stimulus to act as a reinforcer. As is often noted, Skinner did not frequently discuss the critical characteristics of stimuli that allow them to act as reinforcers. Nevertheless, Skinner did, on occasion, consider the role of motivational variables such as food deprivation in the process of reinforcement. For example, Skinner (1953) stated "Reinforcement thus brings behavior under the control of an appropriate deprivation. After we have conditioned a pigeon to stretch its neck by reinforcing with food, the variable which controls the neck-stretching is food deprivation” (p. 149). Many other investigators have offered their own perspectives on this issue, and it has been argued that there are some common characteristics that are evident across different research areas (Salamone & Correa, 2002). A large number of investigators who have written about the fundamental characteristics of reinforcing stimuli have arrived at the conclusion that stimuli that act as positive reinforcers tend to be relatively preferred, or to elicit approach behavior, and that these effects are a fundamental aspect of positive reinforcement. For example, Tapp (1969) stated “At the simplest level, reinforcers have the capacity to direct an organism's behavior. Those stimuli that are approached are regarded as positively reinforcing” (p. 173). Reinforcers have been described as a commodity that is in demand, or a stimulus that is being approached, self-administered, attained or preserved; they also have been described as activities that are preferred, deprived or in some way being regulated (Dickenson & Balleine, 1994; Hursh et al., 1988; Lea, 1978; Premack, 1959; Staddon & Ettinger, 1989; Timberlake, 1993; Tustin, 1995; see discussion of the “motivational corollary of the empirical law of effect” in Salamone & Correa, 2002). According to the behavioral economic analysis offered by Hursh (1993) “responding is regarded as a secondary dependent variable that is important because it is instrumental in controlling consumption” (p. 166).

For these reasons, it is important to note that low doses of DA antagonists that suppress food-reinforced instrumental behavior typically have been shown to leave behavior directed towards the acquisition and consumption of food (Salamone et al., 1991); these manipulations have little effect on food intake (Fibiger, Carter, & Phillips, 1976; Ikemoto & Panksepp, 1996; Rolls et al., 1974; Rusk & Cooper, 1994; Salamone et al., 1991), discrimination and preference based upon food reinforcement magnitude (Martin-Iverson, Wilke, & Fibiger, 1987; Salamone, Cousins, & Bucher, 1994), and simple approach responses reinforced by food delivery (Salamone 1986). Although it is well known that whole forebrain DA depletions can produce aphagia (i.e., lack of eating), it is DA depletions in sensorimotor and motor-related areas of the lateral or ventrolateral caudate/putamen that have been most conclusively linked to this effect, rather than the nucleus accumbens (Dunnett & Iversen 1982; Salamone, J.D., Mahan, K., & Rogers, S., 1993; Ungerstedt, 1971). In contrast, nucleus accumbens DA depletion and antagonism have been shown repeatedly not to substantially impair food intake (Bakshi & Kelley 1991; Baldo, Sadeghian, Basso, & Kelley, 2002; Kelley, Baldo, Pratt, & Will, 2005; Koob, Riley, Smith, & Robbins, 1978; Salamone, Mahon et al., 1993; Ungerstedt 1971). Moreover, the effects of DA antagonists or accumbens DA depletions on food-reinforced instrumental behavior do not closely resemble the effects of pre-feeding or appetite suppressant drugs (Aberman & Salamone, 1999; Salamone, Arizzi, Sandoval, Cervone, & Aberman, 2002; Salamone et al., 1991; Sink, Vemuri, Olszewska, Makriyannis, & Salamone, 2008). Thus, fundamental aspects of primary reinforcement and motivation to obtain access to the reinforcer remain intact after DA antagonism or accumbens DA depletions.

Although it has been suggested that the “reward-related” actions of low doses of DA antagonists or nucleus accumbens DA depletions should produce effects that closely resemble extinction (e.g. Beninger et al., 1987; Wise, Spindler, de Wit, & Gerberg, 1978), there are several problems with this hypothesis. Even though the within-session declines in responding induced by DA antagonists have been labeled “extinction”, similar effects are seen in the motor symptoms of parkinsonism. Haase & Janssen (1985) observed that the micrographia shown by patients with neuroleptic-induced parkinsonism is characterized by a progressive worsening within a writing session. They stated that “An increasing degree of narrowing of the writing from stanza to stanza is particularly characteristic, and in typical cases the area covered by the writing assumes the shape of an inverted pyramid” (p 43). These authors also reported that the intensity of finger tapping generally decreases within a session in patients with neuroleptic-induced parkinsonism (p. 234). Similarly, parkinsonian patients that are repeatedly compressing their hands show progressively diminishing motor output (Schwab, 1972). In rats, DA antagonists cause within-session increments in response duration (Liao & Fowler, 1990), and within session decrements in lick force (Das & Fowler, 1996) and locomotion (Pitts & Horvitz, 2000). Furthermore, repeated administration of DA antagonists to rats leads to context-specific sensitization of the catalepsy response across sessions (Amtage & Schmidt, 2003). In addition, several studies have directly compared the effects of DA antagonism and extinction, and have identified substantial differences between these conditions (Asin & Fibiger, 1984; Evenden & Robbins, 1983; Faustman & Fowler, 1981, 1982; Feldon & Winer, 1991; Gramling, Fowler, & Collins, 1984; Gramling, Fowler, & Tizzano, 1987; Rick, Horvitz, & Balsam, 2006; Salamone 1986; Salamone, Kurth, McCullough, & Sokolowski, 1995, Salamone, et al., 1997; Spivak & Amit, 1986; Willner, Chawala, Sampson, Sophokleous, & Muscat, 1988; Wirtschafter & Asin, 1985). For example, Evenden & Robbins showed that low doses of α-flupenthixol (0.33–0.66 mg/kg) that reduced response rate did not produce effects that resembled extinction in rats responding on a win–stay/lose–shift task. Rick et al. reported that extinction increased behavioral variability in rats trained on an instrumental task, while low doses of the D1 antagonist SCH 23390 or the D2 antagonist raclopride did not.

Another example from this literature is Salamone (1986), which reported that the effects of 0.1 mg/kg of the DA antagonist haloperidol differed substantially from the effects of extinction in rats responding on a fixed ratio (FR) 20 schedule of reinforcement. Under extinction, rats responded at higher rates at the beginning of the session than rats treated with haloperidol, indicating that haloperidol-treated rats did not show an “extinction burst” (see also Salamone et al., 2005, which showed that rats with accumbens DA depletions actually start out responding more slowly in the beginning of the session, in contrast to the effects of extinction). Moreover, rats exposed to extinction emitted proportionately more ratios that were faster than the previous baseline response rate when compared to haloperidol-treated animals (Salamone, 1986). An additional experiment showed that, in contrast to the effects of 0.1 mg/kg haloperidol on FR 20 responding, a dose four times that size had no effect on the reinforced response of simply being in proximity to the food dish on a fixed interval 30 sec schedule (Salamone, 1986). The lack of effect of DA antagonism on this simple food-reinforced response stands in marked contrast to the effect of extinction, which substantially suppressed the instrumental response. In this same experiment, schedule-induced locomotor activity also was recorded in parallel with the instrumental response of being in proximity to the food dish. As shown in the upper panel of Figure 1, 0.4 mg/kg haloperidol suppressed motor activity induced by scheduled presentation of food but, as shown in the lower panel, haloperidol did not affect the reinforced response. In combination with other studies, these results highlight several important features of the effects of DA antagonism. First, the effects of DA antagonism do not closely resemble the effects of extinction across a broad range of conditions (Salamone et al., 1997). Second, DA antagonism suppressed schedule-induced motor activity; behavioral studies have shown that scheduled delivery of reinforcers can have activating properties (Killeen, 1975; Killeen, Hanson, & Osborne, 1978), and considerable evidence indicates that DA antagonism and accumbens DA depletions can reduce schedule-induced activities (McCullough & Salamone, 1992; Robbins & Everitt, 2007; Robbins & Koob, 1980; Robbins, Roberts, & Koob, 1983; Salamone 1988; Wallace, Singer, Finlay, & Gibson, 1983). Finally, these results were consistent with the growing body of evidence indicating that the effects of DA antagonists on instrumental behavior interact powerfully with the instrumental response requirement, and that some reinforced behaviors are relatively unaffected by DA antagonism (Ettenberg et al., 1981; Mekarski, 1988).

Fig 1.

This figure is re-drawn based upon data from Salamone (1986). Rats were tested in a large activity chamber, and were reinforced with 45 mg food pellets on a FI-30 sec schedule for being on the floor panel in front of the food dish. Locomotor activity also was recorded, and this procedure allowed for measurement of both the reinforced response of being on the panel (mean ± SEM time on the panel), and schedule-induced activity (mean ± SEM number of activity counts across the 30 min session, divided into five 6-min periods). The effects of 0.4 mg/kg haloperidol were compared with vehicle injections, and with extinction. Figure 1A shows that haloperidol suppressed schedule-induced motor activity. Figure 1B demonstrates that extinction lowered the instrumental response of being on the panel, but haloperidol did not.

THE EFFECTS OF DA ANTAGONISM AND ACCUMBENS DA DEPLETION INTERACT WITH THE INSTRUMENTAL RESPONSE REQUIREMENTS

In parallel with the historical developments described above, from the 1970s to the 1990s, there was an emerging emphasis in the behavioral literature on effort, response costs or constraints, and economic models of operant behavior. Several investigators emphasized how response costs or constraints affected operant response output (Foltin 1991; Kaufman 1980; Kaufman et al. 1980; Staddon 1979; Tustin, 1995). Work requirements, such as the number of lever presses necessary for obtaining food, were shown to act as determinants of instrumental response output and also to affect food consumption (Collier & Jennings, 1969; Johnson & Collier 1987). Behavioral economic models stress how a number of factors, including not only reinforcement value, but also conditions related to the characteristics of the instrumental response, can determine behavioral output (Allison, 1981, 1993; Bickel, Marsch, & Carroll, 2000; Lea, 1978). Hursh et al. (1988) suggested that the price of food reinforcement as a commodity is a cost/benefit ratio expressed as the effort expended per unit of food value consumed.

Several lines of evidence have served to strengthen support for the hypothesis that the effects of interference with DA transmission interact powerfully with the instrumental response requirement. One of the ways of controlling work requirements in an operant schedule is to use various ratio schedules. Caul and Brindle (2001) observed that the effects of the DA antagonist haloperidol on food-reinforced behavior were dependent upon the ratio requirement, with a FR 1 schedule being less sensitive than a progressive ratio. One can deplete accumbens DA by local injections of a neurotoxic substance such as 6-hydroxydopamine, and several studies have used this approach. Aberman and Salamone (1999) employed a range of ratio schedules (FR 1, 4, 16 and 64) to assess the effects of accumbens DA depletions. While FR 1 performance was not affected by DA depletion (see also Ishiwari, Weber, Mingote, Correa, & Salamone, 2004), and FR 4 responding showed only a mild and transient suppression, the FR 16 and FR 64 schedules were much more impaired. This pattern indicated that accumbens DA depletions promoted the induction of ratio strain; that is, rats with accumbens DA depletions were much more sensitive to the size of the ratio requirement. This pattern can be described as reflecting an increase in the elasticity of demand for food reinforcement (Aberman & Salamone 1999; Salamone et al., 1997, 2009). If the ratio requirement is analogous to the price of the commodity (reinforcement pellets), it appears that rats with accumbens DA depletions are more sensitive than control animals to the price of the food reinforcers (Figure 2). Needless to say, rats do not use currency to purchase operant pellets. Instead, it has been suggested that an operant procedure is more of a barter system, in which the rat trades its work (or reductions in leisure) for a commodity (Rachlin, 2003; Tustin, 1995). Thus, rats with accumbens DA depletions are more sensitive than control animals to work-related response costs, and less likely to trade high levels of ratio output for food. In a subsequent experiment, Salamone, Wisniecki, Carlson, and Correa (2001) reported that increased sensitivity to larger ratio requirements in rats with accumbens DA depletions were observed when rats were tested across a broader range of ratio schedules (as high as FR300), even when the overall relation between lever pressing and food delivered per lever press was kept constant (i.e., a unit price of 50: FR 50 for one pellet; FR 100 for two pellets; FR 200 for four pellets; and FR 600 for six pellets). These results showed that both the magnitude and the organization of the ratio requirement appear to be critical determinants of the sensitivity of an operant schedule to the effects of accumbens DA depletions.

Fig 2.

This figure shows the effect of ratio requirement on the number of lever presses emitted and operant pellets consumed in rats with accumbens DA depletions compared to rats in the vehicle control group (based upon data from Aberman & Salamone, 1999). The data are represented as a demand curve, calculated from the mean number of reinforcement pellets consumed (presented on a log scale) as a function of ratio requirement. Although comparable levels of consumption in DA depleted and control groups were seen with the FR1 schedule, DA-depleted rats showed markedly reduced consumption relative to the control group at higher ratio levels.

Additional experiments examined the effects of accumbens DA depletions on tandem schedules, in which a ratio requirement was attached to an interval requirement. This was done in order to ensure that the results by Aberman and Salamone (1999) and Salamone et al. (2001) reflected the influence of ratio size, as opposed to other variables such as time. Research employing tandem variable-interval (VI)/FR schedules with varying combinations (e.g. VI 30 sec/FR5, VI 60 sec/FR10, VI 120 sec/FR10) has yielded a consistent pattern; accumbens DA depletions did not suppress overall response output in rats responding on the conventional VI schedules (i.e., those requiring only one response after the interval), but did substantially reduce responding on the corresponding VI schedule with the higher ratio requirement attached (Correa, Carlson, Wisniecki, & Salamone, 2002; Mingote, Weber, Ishiwari, Correa, & Salamone, 2005). These findings are consistent with research showing that accumbens DA antagonism did not impair performance on a progressive interval task (Wakabayashi, Fields, & Nicola, 2004), and that accumbens DA depletions did not affect delay discounting (Winstanley, Theobald, Dalley, & Robbins, 2005). In addition, the DA antagonist haloperidol has been shown to increase the number of reinforced responses in rats responding on a DRL 72-sec schedule (Paterson, Balci, Campbell, Olivier, & Hanania, 2010). These results suggest that interval requirements per se do not pose a severe constraint to rats with compromised DA transmission in nucleus accumbens. Over and above any effect of intermittence or time, ratio requirements provide a work-related challenge that is very disruptive to rats with accumbens DA depletions or antagonism.

In summary, nucleus accumbens DA depletions appear to have two major effects on ratio responding: 1) they reduce the response-enhancing effects that moderate-size ratio requirements have on operant responding (i.e., the ascending limb of the inverted u-shaped function relating ratio requirement to response output), and 2) they enhance the response-suppressing effects that very large ratios have on operant responding (i.e., the descending limb of the function; enhancement of ratio strain; Salamone & Correa 2002; Salamone et al., 2001, 2007, 2009). Another important factor to consider when discussing drug effects is that the baseline rate generated the schedule of reinforcement (Barrett & Bergman, 2008; Dews, 1976; McMillan & Katz, 2002). Although baseline response rate was not a critical factor for inducing ratio strain in the Salamone et al. (2001) experiment, reductions in response rate seen across several schedules of reinforcement (various fixed and progressive ratio, FI 30 sec, VI 30 sec, and tandem VI/ FR schedules) that are produced by accumbens DA depletions do appear to be related to baseline response rate. Across these schedules, there is a linear relation between baseline rate of responding under control conditions and the degree of suppression produced by accumbens DA depletions, with the deficit being greater for schedules that generate increased response rates (Figure 3). Furthermore, molecular behavioral analyses indicate that accumbens DA depletions produce a slight reduction in the local rate of responding, as indicated by the distribution of interresponse times (Mingote et al., 2005; Salamone, Kurth, McCullough, Sokolowski, & Cousins, 1993; Salamone, Aberman, Sokolowski, & Cousins, 1999), as well as an increase in pausing (Mingote et al., 2005; Salamone, Kurth, et al., 1993; see also Nicola, 2010). Computational approaches have been used to characterize these effects of accumbens DA depletions on response rate on ratio schedules (e.g. Niv, Daw, Joel, & Dayan, 2007; Phillips, Walton, & Jhou, 2007). Phillips et al. suggested that DA release in nucleus accumbens appears to provide a window of opportunistic drive during which the threshold cost expenditure to obtain the reward is decreased.

Fig 3.

Scatterplot showing relation between baseline or control rates of responding on various interval and ratio schedules of reinforcement versus the magnitude of the suppression of response rate produced by accumbens DA depletions (expressed as mean percent of control responding) in rats responding on that schedule. Solid black circles and regression line are from Salamone et al., (1999). Additional data points are added for the tandem VI30s/FR1 and VI30s/FR5 schedules from Correa et al. (2002; grey triangles), and the tandem VI60s/FR1, VI60s FR10, VI120s/FR1, and VI120s FR10 schedules used by Mingote et al. (2005; open circles).

In the context of this discussion of the effects of dopaminergic drugs on ratio performance, it is useful to consider the term “reinforcement efficacy”, which is sometimes used to describe the effects of drug manipulations on ratio performance. With progressive ratio schedules, the ratio requirement increases as successive ratios are completed, and the “break point” is said to occur at the point at which the animal stops responding. One can operationally define reinforcement efficacy in terms of the break point in a progressive ratio schedule, or by measuring ratio strain in rats responding across different FR schedules. The determination of reinforcement efficacy can be a very useful tool for characterizing the actions of drugs that are self-administered, and for comparing self-administration behavior across different substances or drug classes (e.g., Marinelli et al. 1998; Morgan, Brebner, Lynch, & Roberts, 2002; Ward, Morgan, & Roberts, 2005; Woolverton & Rinaldi, 2002). Nevertheless, given the terminological difficulties discussed above, it is useful to stress that the term “reinforcement efficacy” should not be used simply as a replacement for “reward”, and that progressive ratio breakpoints should not be viewed as necessarily providing some direct and unambiguous measure related to the subjective pleasure produced by the stimulus (Salamone, 2006; Salamone et al., 1997; 2009). Drug-induced changes in progressive ratio break points can reflect actions on several different behavioral and neurochemical processes (Arnold & Roberts, 1997; Bickel et al., 2000; Hamill, Trevitt, Nowend, Carlson, & Salamone, 1999; Killeen, 1995; Lack, Jones, & Roberts, 2008; Madden, Smethells, Ewan, & Hursh, 2007; Mobini, Chiang, Ho, Bradshaw, & Szabadi, 2000). For example, changing the response requirements by increasing the height of the lever decreased progressive ratio break points (Schmelzeis & Mittleman 1996; Skjoldager, Pierre, & Mittlman, 1993). Although some researchers have maintained that the break point provides a direct measure of the appetitive motivational characteristics of a stimulus, it is, as stated in a landmark review by Stewart (1975), more directly a measure of how much work the organism will do in order to obtain that stimulus. The animal is making a cost/benefit choice about whether or not to continue responding, based partly on factors related to the reinforcer itself, but also upon the work-related response costs and time constraints imposed by the ratio schedule. For these reasons, interpretations of the actions of drugs or lesions on progressive ratio break points should be done with caution, as should be the case for any individual task. A drug that alters the break point could do so for many different reasons. Mobini et al., (2000) analyzed the effects of several drugs on progressive ratio responding using the quantitative methods developed by Killeen (1994), who suggested that schedule performance is due to interactions between multiple variables (specific activation, coupling, and response time). Mobini et al. reported that haloperidol affected both the response time parameter, and also decreased the activation parameter, while clozapine increased the activation parameter. Recent studies have shown that the DA antagonist haloperidol can suppress food-reinforced progressive ratio responding, and lower break points, but nevertheless leave intact the consumption of a concurrently available but less preferred food source (Pardo et al., 2011; Randall, Pardo, et al., 2011). These actions of haloperidol on this task differed markedly from those produced by prefeeding and appetite suppressant drugs (Pardo et al., 2011; Randall, Pardo, et al., 2011).

DA ANTAGONISM AND NUCLEUS ACCUMBENS DA DEPLETIONS AFFECT THE RELATIVE ALLOCATION OF INSTRUMENTAL RESPONDING IN EFFORT-RELATED CHOICE TASKS

As noted above, animals must make choices in complex environments that present multiple opportunities for obtaining significant stimuli, and several paths for accessing them (Aparicio, 2001, 2007; Williams, 1988). The variables that influence these choices are complex and multidimensional, and they include not only reinforcement value, but also response-related factors. Among the most important are those factors involving cost/benefit interactions based upon effort and reinforcement value (Hursh et al., 1988; Neill & Justice, 1981; Salamone, 2010a; Salamone & Correa 2002; Salamone, Correa, Mingote, & Weber, 2003; Salamone et al., 2005, 2007; Van den Bos, van der Harst, Jonkman, Schilders, & Spruijt, 2006; Walton, Kennerley, Bannerman, Phillips, & Rushworth, 2006). Considerable evidence indicates that low systemic doses of DA antagonists, as well as local disruption of nucleus accumbens DA transmission, affect the relative allocation of behavior in animals responding on tasks that assess effort-based choice behavior (Floresco, St. Onge, Ghods-Sharifi, & Winstanley, 2008; Floresco, Tse, & Ghods-Sharifi, 2008b; Hauber & Sommer 2009; Salamone et al. 2003, 2005, 2007).

One task that has been used to assess the effects of dopaminergic manipulations on response allocation is a procedure that offers rats the option of lever pressing reinforced by delivery of a relatively preferred food (e.g. Bioserve pellets; usually obtained on a FR 5 schedule), or approaching and consuming a less preferred food (lab chow) that is concurrently available in the chamber (Salamone et al., 1991). Trained rats under baseline or control conditions get most of their food by lever pressing, and consume only small quantities of chow. Low-to-moderate doses of DA antagonists, which block either D₁ or D₂ family receptor subtypes (cis-flupenthixol, haloperidol, raclopride, eticlopride, SCH 23390, SKF83566, ecopipam), produce a substantial alteration of response allocation in rats performing on this task; they decrease food-reinforced lever pressing but substantially increase intake of the concurrently available chow (Cousins., Wei, & Salamone, 1994; Koch Schmid, & Scnhnitzler, 2000; Salamone et al., 2002; Salamone, Cousins, Maio, Champion, Turski, & Kovach, 1996; Salamone et al., 1991; Sink et al. 2008; Worden et al. 2009).

The use of this task for assessing effort-related choice behavior has been validated in many ways. Doses of DA antagonists that produce the shift from lever pressing to chow intake do not affect total food intake or alter preference between these two specific foods in free-feeding choice tests (Koch et al., 2000; Salamone et al., 1991). In contrast, appetite suppressants from different classes, including amphetamine (Cousins et al., 1994), fenfluramine (Salamone et al., 2002) and cannabinoid CB1 antagonists (Sink et al., 2008) failed to increase chow intake at doses that suppressed lever pressing. Similarly, prefeeding reduced both lever pressing and chow intake (Salamone et al., 1991). Furthermore, with higher ratio requirements (up to FR 20, or progressive ratios), animals that are not drug treated shift from lever pressing to chow intake (Pardo et al., 2011; Randall, Pardo, et al., 2011b; Salamone et al., 1997), indicating that this task is sensitive to work load. These results indicate that interference with DA transmission does not simply reduce food intake, but instead acts to alter response allocation between alternative sources of food that can be obtained through different instrumental responses.

The shift from lever pressing to chow intake in rats performing on this task is associated with DA depletions in nucleus accumbens; decreases in lever pressing and increases in chow intake occur as a result of accumbens DA depletions, as well as local injections of D₁ or D₂ family antagonists into either the core or shell subregions of nucleus accumbens (Cousins & Salamone 1994; Cousins, Sokolowski, & Salamone, 1993; Farrar et al., 2010; Koch et al. 2000; Nowend, Arizzi, Carlson, & Salamone, 2001; Salamone et al., 1991; Sokolowski & Salamone, 1998). Thus, although lever pressing is decreased by accumbens DA antagonism or depletions, these rats show a compensatory reallocation of behavior and select a new path to an alternative food source.

Salamone et al. (1994) also developed a T-maze procedure, in which the two choice arms of the maze led to different reinforcement densities (e.g., four vs. two food pellets, or four vs. zero); under some conditions, a barrier can be placed in the arm with the higher density of food reinforcement to present an effort-related challenge. When the high density arm has the barrier in place, and the arm without the barrier contains fewer reinforcers, DA depletions or antagonism decrease choice for the high density arm and increase selection of the low density arm with no barrier (Cousins, Atherton, Turner, & Salamone, 1996; Denk, Walton, Jennings, Sharp, Rushworth, & Bannerman, 2005; Mott et al., 2009; Pardo et al., submitted for publication; Salamone et al., 1994).

Like the operant concurrent choice task, this T-maze task also has undergone considerable behavioral validation and evaluation (Cousins et al., 1996; Pardo et al., submitted for publication; Salamone et al., 1994; van den Bos et al., 2006). For example, when there is no barrier in the maze, rats overwhelmingly prefer the high reinforcement density arm, and neither haloperidol nor accumbens DA depletion alters their response choice (Salamone et al., 1994). When the arm with the barrier contained four pellets, but the other arm contained no pellets, rats with accumbens DA depletions still managed to choose the high density arm, climb the barrier, and consume the pellets (Cousins et al., 1996). In a recent T-maze study with mice, while haloperidol reduced choice of the arm with the barrier, this drug had no effect on choice when both arms had a barrier in place (Pardo et al., submitted for publication). Thus, dopaminergic manipulations do not alter the preference for the high density of food reward over the low density, and did not affect discrimination, memory or instrumental learning processes related to arm preference. The results of the T-maze studies in rodents, together with the findings from the FR5/chow concurrent choice studies reviewed above, indicate that low doses of DA antagonists and accumbens DA depletions cause animals to reallocate their instrumental response selection based upon the response requirements of the task, and select lower cost alternatives for obtaining reinforcers (see reviews by Salamone et al., 2003, 2005, 2007; Floresco, St. Onge, et al., 2008).

Effort discounting procedures also have been employed to study the effects of dopaminergic manipulations. Floresco, Tse, et al. (2008) demonstrated that the DA antagonist haloperidol altered effort discounting even when the effects of time delay were controlled for (see Wade, de Wit, & Richards, 2000; and Koffarnus, Newman, Grundt, Rice, & Woods, 2011 for a discussion of the mixed findings in the literature on the effects of DA antagonists and delay discounting). Bardgett, Depenbrock, Downs, Points, & Green (2009) recently developed a T-maze effort-discounting task, in which the amount of food in the high-density arm of the maze was diminished each trial on which the rats selected that arm (i.e., an “adjusting-amount” discounting variant of the T-maze procedures, which allows for the determination of an indifference point for each rat). Effort discounting was altered by the D₁ family antagonist SCH23390 and the D₂ family antagonist haloperidol; these drugs made it more likely that rats would choose the low-reinforcement/low-cost arm. Increasing DA transmission by administration of amphetamine blocked the effects of SCH23390 and haloperidol, and also biased rats towards choosing the high-reinforcement/high-cost arm, which is consistent with operant choice studies using DA transporter knockdown mice (Cagniard, Balsam, Brunner, & Zhuang, 2006). Together with other results, the findings reported by Bardgett et al. and Floresco, Tse, et al. support the suggestion that, across a variety of conditions, DA transmission exerts a bidirectional influence over effort-related choice behavior.

DA INTERACTS WITH OTHER TRANSMITTERS TO INFLUENCE EFFORT-RELATED CHOICE BEHAVIOR

As reviewed above, DA antagonists and accumbens DA depletions affect instrumental response output, response allocation, and effort-related choice behavior. Obviously, no single brain area or neurotransmitter participates in a behavioral process in isolation to other structures or chemicals; for that reason it is important to review how other brain areas and neurotransmitters interact with dopaminergic mechanisms. Over the last several years, several laboratories have begun to characterize the role that multiple brain structures (e.g. amygdala, anterior cingulate cortex, ventral pallidum) and neurotransmitters (adenosine, GABA) play in effort-related choice behavior (Denk et al., 2005; Farrar et al., 2008; Floresco & Ghods-Sharifi, 2007; Floresco, St. Onge, et al., 2008; Hauber & Sommer, 2009; Mott et al. 2009; Pardo et al., submitted for publication; Schweimer & Hauber, 2006; van den Bos et al. 2006; Walton, Bannerman, Alterescu, & Rushworth, 2003; Walton, Bannerman, & Rushworth, 2002).

Within the last few years, considerable emphasis has been placed upon DA/adenosine interactions. Caffeine and other methylxanthines, which are nonselective adenosine antagonists, act as minor stimulants (Ferré et al., 2008; Randall, Nunez, et al., 2011). DA-rich brain areas, including the neostriatum and the nucleus accumbens, have a very high degree of adenosine A_2A receptor expression (DeMet & Chicz-DeMet, 2002; Ferré et al., 2004; Schiffman, Jacobs, & Vanderhaeghen, 1991). There is considerable evidence of cellular interactions between DA D₂ and adenosine A_2A receptors (Ferré, 1997; Fink et al., 1992; Fuxe et al., 2003; Hillion et al., 2002). This interaction frequently has been studied in regard to neostriatal motor functions related to parkinsonism (Correa et al. 2004; Ferré, Fredholm, Morelli, Popoli, & Fuxe, 1997; Ferré et al., 2001; Hauber & Munkel, 1997; Hauber, Neuscheler, Nagel, & Muller, 2001; Ishiwari et al., 2007; Morelli & Pinna, 2002; Pinna, Wardas, Simola, & Morelli, 2005; Salamone, Betz, et al. 2008; Salamone, Ishiwari, et al., 2008; Svenningsson, Le Moine, Fisone, & Fredholm, 1999; Wardas, Konieczny, & Lorenc-Koci, 2001). However, several reports also have characterized aspects of adenosine A_2A receptor function related to learning (Takahashi, Pamplona, & Prediger, 2008), anxiety (Correa & Font, 2008), and instrumental responding (Font et al., 2008; Mingote et al., 2008).

Drugs that act upon adenosine A_2A receptors profoundly affect instrumental response output and effort-related choice behavior (Farrar et al., 2007, 2010; Font et al., 2008; Mingote et al., 2008; Mott et al., 2009; Pardo et al., submitted for publication; Worden et al., 2009). Intra-accumbens injections of the adenosine A_2A agonist CGS 21680 reduced responding on a VI 60-sec schedule with a FR10 requirement attached, but did not impair performance on a conventional VI 60-sec schedule (Mingote et al., 2008), a pattern similar to that previously shown with accumbens DA depletions (Mingote et al., 2005). In rats responding on the FR5/chow concurrent choice procedure, injections of CGS 21680 into the accumbens decreased lever pressing and increased chow intake (Font et al.). These effects were site specific, because injections of CGS 21680 into a control site dorsal to the accumbens had no effect (Mingote et al., 2008; Font et al.).

It also has been demonstrated that adenosine A_2A receptor antagonists can reverse the effects of systemically administered DA D₂ antagonists in rats tested on the FR5/chow feeding concurrent choice task (Farrar et al., 2007; Nunes et al., 2010; Salamone et al., 2009; Worden et al., 2009). Moreover, systemic or intra-accumbens injections of the adenosine A_2A antagonist MSX-3 were able to block the effects of intra-accumbens injections of the D₂ antagonist eticlopride in rats responding on the FR5/chow concurrent choice task (Farrar et al., 2010). In studies using the T-maze barrier procedure, adenosine A_2A antagonists have been shown to reverse the effects of DA D₂ antagonism in rats (Mott et al., 2009) and mice (Pardo et al., submitted for publication). Furthermore, adenosine A_2A receptor knockout mice are resistant to the effects of haloperidol on selection of the high-reinforcement/high-cost arm of the T-maze (Pardo et al.).

The pattern of effects seen in these studies depends upon which specific receptor subtypes are being acted upon by the drugs being administered. Although the adenosine A_2A receptor antagonists MSX-3 and KW 6002 reliably and substantially attenuate the effects of D₂ antagonists such as haloperidol and eticlopride in rats responding on the FR5/chow concurrent choice procedure (Farrar et al., 2007; Nunes et al., 2010; Salamone et al., 2009; Worden et al., 2009), they produce only a mild reversal of the effects of the D₁ antagonist ecopipam (SCH 39166; Worden et al.; Nunes et al.). In addition, the highly selective adenosine A₁ receptor antagonist was completely ineffective at reversing the effects of DA D₁ or D₂ antagonism (Salamone et al., 2009; Nunes et al.). Similar results were obtained with rats and mice responding on the T-maze barrier choice task; while MSX-3 was able to reverse the effect of the D₂ antagonist haloperidol on selection of the high-reinforcement/high-cost arm, the A₁ antagonists DPCPX and CPT were not (Mott et al., 2009; Pardo et al., submitted for publication). These results indicate that there is a relatively selective interaction between drugs that act upon DA D₂ and adenosine A_2A receptor subtypes (see Table 1). Based upon anatomical studies, it appears that this is likely to be due to the pattern of cellular localization of adenosine A₁ and A_2A receptors in striatal areas, including the nucleus accumbens (Ferré, 1997; Fink et al., 1992; Fuxe et al., 2003; Hillion et al., 2002; Svenningsson et al., 1999). Adenosine A_2A receptors are typically co-localized on striatal and accumbens enkephalin-positive medium spiny neurons with DA D₂ family receptors, and both receptors converge onto the same intracellular signaling pathways. Thus, adenosine A_2A receptor antagonists may be so effective in reversing the actions of D₂ antagonists because of direct interactions between DA D₂ and adenosine A_2A receptors located on the same neurons (Farrar et al., 2010; Salamone et al., 2009, 2010).

Table 1.

Adenosine receptor antagonists.

SUMMARY AND CONCLUSIONS: IMPLICATIONS FOR BEHAVIOR ANALYSIS AND PSYCHOPATHOLOGY

In summary, there is general agreement that nucleus accumbens DA and related brain systems participate in many functions that are important for instrumental behavior, though the specifics of that involvement are still being characterized. One conceptual limitation in this area is that global constructs such as “reward”, “reinforcement”, “learning”, “motivation” and “motor control” are too general to serve as useful descriptors of the effects of DA antagonism or depletion. These constructs actually involve several distinct processes, many of which can be dissociated from each other by brain manipulations such as drugs or lesions that severely impair one process while leaving another largely intact (Berridge & Robinson, 2003; Salamone & Correa, 2002; Salamone et al., 2007). Based upon the evidence reviewed above, interference with DA transmission does not impair “reward” in any general sense, because interference with DA transmission impairs some features of instrumental behavior while leaving fundamental aspects of primary reinforcement or motivation basically intact (e.g., reinforcement of simple instrumental responses; consumption of the reinforcer).

Another important consideration is the degree of overlap between very broad constructs such as “motivation” and “motor function”. Although one could attempt to adhere to a strict dichotomy between the motivational versus the motor functions of nucleus accumbens DA, it is not conceptually necessary to do so. It has been argued that “motor control” and “motivation”, though somewhat distinct conceptually, overlap considerably in terms of some of the specific characteristics of behavior being described and the brain circuits involved (Salamone, 1987, 1992, 2010b; Salamone & Correa 2002; Salamone et al., 2003, 2005, 2007). Consistent with this line of thinking, it is reasonable to suggest that accumbens DA performs functions that represent areas of overlap between motor and motivational processes (Salamone, 1987, 2010b; Salamone et al., 2007). Such functions would include the types of behavioral activation and effort-related processes discussed above. Nucleus accumbens DA is important for enabling animals to engage in schedule-induced activities (McCullough & Salamone, 1992; Robbins & Everitt, 2007; Robbins & Koob, 1980; Robbins et al., 1983; Salamone 1988; Wallace et al., 1983), and to respond to the work-related challenges imposed by ratio schedules (Aberman & Salamone,1999; Correa et al. 2002; Mingote et al., 2005; Salamone et al., 2002, 2003, 2005; Salamone, Correa, Mingote, Weber, & Farrar, 2006) and barriers in mazes (Cousins et al., 1996; Salamone et al., 1994). Moreover, the suggested involvement of accumbens DA in behavioral activation and effort is related to the hypothesis that nucleus accumbens is important for facilitating responsiveness to the activating properties of Pavlovian conditioned stimuli (Day, Wheeler, Roitman, & Carelli, 2006; Di Ciano, Cardinal, Cowell, Little, & Everitt, 2001; Everitt et al., 1999; Everitt & Robbins, 2005; Parkinson et al., 2002; Robbins & Everitt, 2007; Salamone et al., 2007).

Thus, despite the fact that animals with impaired transmission of accumbens DA remain directed towards the acquisition and consumption of primary reinforcers, accumbens DA does appear to be particularly important for overcoming work-related challenges presented by instrumental behaviors with high response requirements. This represents one function of accumbens DA, but certainly not the only one. As emphasized in previous papers (e.g., Salamone et al., 2007), it is unlikely that accumbens DA performs only one function, and evidence in favor of the hypothesis that DA is involved in the exertion of effort or effort-related choice behavior is not incompatible with the hypothesized involvement of this system in instrumental learning (Baldo & Kelley, 2007; Beninger & Gerdjikov, 2004; Kelley et al., 2005; Segovia et al., 2011; Wise, 2004), aspects of incentive motivation (e.g. reinforcer “wanting”; Berridge 2007; Berridge & Robinson, 2003; Wyvell & Berridge, 2001) or Pavlovian-instrumental transfer (Everitt & Robbins, 2005).

A measure derived from observations of behavior, or a parameter that is generated from curve-fitting analyses, can have many factors that contribute to it and, as noted above, pharmacological research often can dissociate between these factors, because a drug can severely affect one while leaving another basically intact. A useful example of this principle is the progressive ratio break point, which, as discussed above, is influenced by several factors (Pardo et al., 2011; Randall, Pardo, et al., 2011b). Another case in which this point is highly relevant is the measurement of intracranial self-stimulation thresholds. Such measures often are viewed as providing “rate-free” indices of “reward”, or even “hedonia”, nevertheless, they are influenced by lever pressing ratio requirements as well as the electrical current level (Fouriezos, Bielajew, & Pagotto, 1990). Recent studies with intracranial self-stimulation thresholds indicate that dopaminergic modulation of self-stimulation thresholds does not affect reward value per se, but instead alters the tendency to pay response costs (Hernandez, Breton, Conover, & Shizgal, 2010). Response-reinforcement matching also has been used in some research related to behavioral economics, reinforcer value, and the functions of DA systems (e.g. Aparicio, 2007; Heyman, Monaghan, & Clody, 1987). Matching equations have been employed to describe the results of studies with VI schedules, and parameters from matching equations (e.g., R_o) have been used to represent reinforcement value (e.g., Herrnstein 1974; R_o has been referred to as the rate of reinforcement from other sources, and is inversely related to reinforcement value of the scheduled contingencies). As noted by Killeen (1995), empirically, R_o represents a “half-life constant” for the curve fitting formula. However, used in this way, R_o does not selectively represent the reinforcement value of food per se. At best, this measure reflects the relative value of the entire activity of lever pressing for and consuming the food reinforcer compared to the reinforcing value of all other stimuli and responses available (Salamone et al., 1997, 2009; Williams, 1988). Several factors can contribute to this composite measure, and a drug or lesion manipulation could yield apparent effects on “reinforcement value” that actually reflect changes in response-related factors (Salamone, 1987; Salamone et al., 1997, 2009). Moreover, matching equations have been developed that account for deviations from matching by allowing for estimates of response preference or bias (Aparicio, 2001; Baum, 1974; Williams, 1988), which also could be affected by drugs.

In view of these points, it is useful to consider how terms such as “value” are used in behavioral economics and neuroeconomics research. The aggregate reinforcement value of an instrumental activity (e.g., lever pressing for and consuming food) should probably be viewed as a composite measure that includes both the reinforcing value of the reinforcer itself, and also any net value or costs associated with the instrumental response that is required to obtain the reinforcer. Viewed in this manner, the effects of DA antagonists or depletions on effort-related choice behavior could be described in terms of actions upon the response costs associated with the particular instrumental response, rather than the reinforcing value of the reinforcing stimulus itself. Although the effects of haloperidol on bias may be minimal when two levers that are relatively similar are used (e.g., Aparicio, 2007), they may be much larger when substantially different responses are compared (e.g., lever pressing vs. nose poking or sniffing; lever pressing vs. unrestricted access to food; barrier climbing vs. locomotion to a location containing food).

In addition to providing insights into aspects of instrumental behavior seen in the laboratory, research on effort-related choice behavior also has clinical implications. Addiction is characterized by a reorganization of the preference structure of the person, dramatic changes in the allocation of behavioral resources towards the addictive substance (Heyman, 2009; Vezina et al., 2002), and inelasticity of demand (Heyman, 2000). Typically, there is a heightened tendency to engage in drug-reinforced instrumental behavior and drug consumption, often at the expense of other behavioral activities. Addicts will go to great lengths to obtain their preferred drug, overcoming numerous obstacles and constraints. Thus, drug-reinforced instrumental behavior in humans involves many processes, including exertion of effort. Recent evidence indicates that DA synthesis inhibition induced by precursor depletion resulted in a decrease in progressive ratio breakpoints reinforced by nicotine-containing cigarettes, despite the fact that this manipulation did not affect self-reported “euphoria” or “craving” (Venugopalan et al., 2011).

As well as being related to aspects of drug taking and addiction, research on effort-related choice behavior has implications for understanding the neural basis of psychopathological symptoms such as psychomotor slowing, anergia, fatigue and apathy, which are seen in depression as well as in other psychiatric or neurological conditions (Salamone et al., 2006, 2007, 2010). These symptoms, which can have devastating behavioral manifestations (Demyttenaere, De Fruyt, & Stahl, 2005; Stahl, 2002), essentially represent impairments in aspects of instrumental behavior, exertion of effort and effort-related choice, which can lead to difficulties in the workplace, as well as limitations in terms of life function, interaction with the environment, and responsiveness to treatment. Within the last few years, there has been increasing interest in behavioral activation therapy for the treatment of depression, which is used to increase activation systematically by employing graded exercises to increase the patient's access to reinforcement and identify processes that inhibit activation (Jacobson, Martell, & Dimidjian, 2001; Weinstock, Munroe, & Miller, 2011). Furthermore, there is considerable overlap between the neural circuitry involved in effort-related functions in animals and the brain systems that have been implicated in psychomotor slowing and anergia in depression (Salamone et al. 2006, 2007, 2009, 2010; Treadway & Zald, 2011). Thus, basic and clinical research on effort-related behavioral processes, and their neural regulation, could have substantial impact on clinical research related to addiction, depression, and other disorders.