Neural Substrates Underlying Effort, Time, and Risk-Based Decision Making in Motivated Behavior

Abstract

All mobile organisms rely on adaptive motivated behavior to overcome the challenges of living in an environment in which essential resources may be limited. A variety of influences ranging from an organism’s environment, experiential history, and physiological state all influence a cost-benefit analysis which allows motivation to energize behavior and direct it toward specific goals. Here we review the substantial amount of research aimed at discovering the interconnected neural circuits which allow organisms to carry-out the cost-benefit computations which allow them to behave in adaptive ways. We specifically focus on how the brain deals with different types of costs, including effort requirements, delays to reward and payoff riskiness. An examination of this broad literature highlights the importance of the extended neural circuits which enable organisms to make decisions about these different types of costs. This involves Cortical Structures, including the Anterior Cingulate Cortex (ACC), the Orbital Frontal Cortex (OFC), the Infralimbic Cortex (IL), and prelimbic Cortex (PL), as well as the Baso-Lateral Amygdala (BLA), the Nucleus Accumbens (Nacc), the Ventral Pallidal (VP), the Sub Thalamic Nucleus (STN) among others. Some regions are involved in multiple aspects of cost-benefit computations while the involvement of other regions is restricted to information relating to specific types of costs.

1. Introduction

1A. Historical/Background Information

Some of the earliest laboratory studies of motivated behavior led researchers to observe that most complex behavior tends to occur in bouts and that specific behaviors such as feeding or grooming can be characterized by their frequency, intensity, temporal distribution and direction towards or away from a particular stimulus. One of the prominent researchers of the day went so far as to say that identifying the factors responsible for the initiation and termination of these specific bouts of behavior would be the central problem for experimental psychologists to understand (Richter, 1927). Over the years there have been numerous theories of motivation put forth (Bolles & Moot, 1972; Hebb, 1955; Hull, 1943; Young, 1961), each of which has been influential in stimulating what has been a continuous stream of experiments and research on this topic. There exist excellent reviews of many of these theories and concepts (Berridge, 2004).

Almost a century later, researchers from numerous fields including psychology, psychiatry, and neurobiology are still actively studying goal-directed motivation, which is the name that has been given to the set of biological and psychological processes which guides behavior in pursuit of a goal. Research in this realm of behavioral neuroscience has come a long way toward understanding the wide array of factors which come together to modulate goal-directed action. Neurobiologists are uncovering the widely distributed collection of neural circuits which underlie the various aspects of goal-directed motivation. This has led to the identification of limbic and midbrain regions including the Ventral Tegmental Area (VTA), Nucleus Accumbens (NAcc), and Ventral Pallidum (VP) which appear to be critical for invigorating effortful behavior. Additionally, cortical regions such as the Anterior Cingulate Cortex (ACC) and medial Prefrontal Cortex (mPFC) are crucial for comparing costs and benefits which becomes important when one is faced with several potential response choices. In addition to the basic work being done in animal models, clinicians and psychiatrists using modern brain imaging methods have started to uncover some of the neurobiological correlates of impairments in goal-directed motivation commonly seen in many forms of psychopathology, including schizophrenia and depression. Currently, the unprecedented technical arsenal of neuroscience tools available to researchers makes it an extremely exciting and fruitful time to be studying a question which has captivated researchers for nearly a century.

1B. Motivation: Energizing and directing behavior toward specific goals

All mobile organisms are faced with the universal challenge of living in a world in which the resources needed for survival may be limited in number and unevenly dispersed throughout the environment. Obtaining essential resources often requires one to overcome obstacles which inherently contain many different kinds of costs to the organism. When seeking food, water, or potential mates, one might be faced with any number of these costs, including: a physical distance one must traverse, the height of an obstacle one must climb, the number of responses one must make, or the commitment of time one must invest. Goal-directed motivation represents the set of processes which allows an organism to weigh these costs against potential benefits of obtaining a goal. It has been recognized by researcher for a long time that motivation serves two important functions, as it provides both a directional influence on behavior and also has an activational or energizing effect as well as (Duffy, 1957; Hebb, 1955); and more recent work has started to describe the underlying neurobiological substrates of both the directional processes (Kim, Lee, & Jung, 2013; Kimchi & Laubach, 2009) as well as activational processes (Anaclet et al., 2009; Pfaff, Martin, & Faber, 2012) and. Whereas the directional component of motivation guides behavior toward a specific goal and away from competing actions (Dickinson & Balleine, 1994), the activational component of motivation provides the energy or vigor needed to overcome the physical costs standing between the animal and its goal. This activational influence on motivation is reflected in the likelihood of initiation, and the speed, vigor and persistence of an action (Floresco, 2015; Salamone, 1992; Salamone & Correa, 2002; Salamone, Correa, E. J. Nunes, P. A. Randall, & M. Pardo, 2012)

Directional Effects of Motivation

The most general way in which the concept of directional motivation is used is to say that animals pursue positive stimuli (e.g. food, water, sex, etc.) and avoid negative stimuli (e.g. painful conditions, predators, stress) (Salamone, Yohn, López-Cruz, San Miguel, & Correa, 2016). A more specific definition of the concept of directional motivation is the processes which cause animals to choose one specific class of behavior to engage in at a given time over all others (i.e. Feeding, Drinking, Mating, Aggressive Behavior, etc.). This concept proves useful in that it allows researchers to attempt to figure out the physiological and environmental variables which influence animals to engage in one class of behaviors over another (e.g. feeding as opposed to drinking). This usage helps to explain observations such as when animals choose to pursue food following a long period of food deprivation it is the directional influence of motivation which leads the animal to pursue food while forgoing pursuits of other behaviors. This is unsurprising as there are distinct neural circuits which control food seeking as opposed to something like thirst (Kelley, Baldo, Pratt, & Will, 2005; Oka, Ye, & Zuker, 2015). There has been an extensive amount of research aimed at understanding what circulating hormones and brain regions are responsible for directional motivational effects for feeding (Belgardt, Okamura, & Brüning, 2009), thirst (Johnson & Thunhorst, 1997), as well as sexual behavior (DAVIDSON, 1966), and other social behaviors (Hong, Kim, & Anderson, 2014; F. Wang, Kessels, & Hu, 2014). We point readers to recent reviews of this literature (Sternson, 2013), as an extensive discussion of these directional effects are beyond the scope of the present review. In the present review, we focus on situations in which subjects are food restricted and working for food rewards (i.e. experimentally manipulated to be directed towards food), and we examine how different types of costs a subject must overcome to obtain the food reward alters both activational aspects of behavior and the choice of what specific action to take to obtain reward.

Activational Motivation Effects

As animals are deprived of necessary resources their behavior changes in a number of ways: (1) there is often an increase in general locomotor activity, (b) an increase the likelihood of performing actions known to lead to that deprived resource, (c) and an increase in the speed, vigor, and the persistence of these goal directed actions. (Floresco, 2015; Salamone & Correa, 2002; Salamone et al., 2012; J. D. Salamone, 1992). These changes in behavior are thought to reflect changes in the activational or energizing effects of motivation. It is this activational or energizing influence of motivation which allows animals to overcome the costs standing between them and the goal for which they are working. In this review, we focus specifically on what is known about the neural substrates that influence how the costs of responding affect the activational aspects of motivated behavior. We also examine what is known about the neural machinery involved in processing information about different types of costs that enter into the cost-benefit computation that guides choices about how to allocate effort in situations in which there is more than one response option that could lead to the desired resource.

1C. Cost-Benefit computations underlying motivated behavior

How does motivation properly guide an organism through the environment to overcome obstacles and meet needs necessary for survival? Current theories suggest that animals incorporate information from many different levels and perform cost-benefit computations which allow for adaptive decision making. A typical laboratory experiment in which a rat has learned to press a lever for a food reward serves as an excellent example of how this might work. A fully sated rat will make a very small number of lever presses for food. The few lever presses it does make will be made slowly with many pauses in between presses, and the rat will spend a substantial amount of time engaging in other behaviors such as exploring the chamber and grooming itself. The same animal’s behavior will look very different when its access to food has been restricted. Both the number of lever presses made as well as the rate/vigor of those responses are highly correlated with the percent body weight loss induced by the food restriction (Collier, 1969; Collier & Levitsky, 1967; Marwine & Collier, 1971). In these two scenarios the cost of responding is constant (i.e. the same number of lever presses is required in both situations), but the benefit or value of the food differs greatly. The difference between the cost and the benefit of pressing in each particular condition determines the direction of behavior (lever pressing and not exploring or grooming/etc.) as well as the intensity or vigor (response rate of the lever presses) with which the behaviors are executed.

Research over that last 5 decades shows that there are many factors which influence the cost-benefit decision making processes. These factors include environment factors (such as local food availability, time of day, or temperature), an animal’s experiential history (whether it was trained on a continuous or intermittent schedule of reinforcement), and their physiology (circulating hormone levels) and internal biological clocks (e.g. location in a circadian rhythm (Antle & Silver, 2015). Figure 1 illustrates a conceptual model of how all of these factors might act in concert in a hierarchical manner to modulate goal-directed motivation by influencing the underlying cost-benefit decision making processes and provides examples of these different factors influencing motivation (Simpson & Balsam, 2016). As shown in this figure, this model posits that the physiological state of the organism, the environment, and past history/learning of the organism interact to influence the representation of costs and benefits that determine the specific types of behavior at any given time. Moreover, the information about the costs and benefits are compared in a cost-benefit computation which then influences the selection and vigor of behavior. We present figure 1 to suggest one possible model of how goal-directed motivation may work, and to provide a context in which to place this review. We do not attempt to state which brain regions are definitively involved in specific stages of the Cost/Benefit computation process, rather we examine an array of studies which focus on the cost input to this computation. In doing so we compare 3 different kinds of costs: Effort, Time, and Risk.., Webring together these three separate lines of investigation to identify both the overlapping and distinct neurobiological substrates for processing these costs.

Figure 1. Hypothetical Model of Factors Influencing Motivational Cost-Benefit Decision Making Processes.

Shows a hypothetical model of how motivation is influenced by physiological state, environment, and past history to modulate an underlying cost-benefit decision making computation which gives direction and vigor to goal-directed behavior

1D. Scope and Purpose of the Review

The purpose of this review is to summarize and synthesize a number of varying studies which examine different types of motivated behavior through the framework of motivated behavior as relying on a cost-benefit computation to give rise to both the direction and vigor of behavior. The direction and vigor of behavior represent the final behavioral output which one can measure, and a number of studies are reviewed which have been performed to understand the neural locations at which manipulations to the region impact either directional or activational aspects of behavior. Additionally, we give a primary focus to studies which have examined motivated behavior through various forms of cost-benefit decision making. In this review, we systematically focus on studies which have manipulated one of the factors which goes into the cost-benefit calculation: cost, as this represents one critical side of the cost-benefit computations that guide motivated behavior. We first provide a summary of the behavioral data that demonstrates animals’ ability to process information related to various types of costs. We then discuss the more recent work examining the neurobiology of the activational effects of motivation. We finish by reviewing an array of studies aimed at understanding the neurobiological underpinnings of cost-benefit decision making by specifically focusing on studies which employed manipulations of three types of response costs: (1) effort, (2) time, and (3) risk. In doing so we describe studies which have employed neural manipulations such as various types of lesions, as well as locally delivered pharmacological manipulations. While we also discuss a number of results from systemic pharmacology studies, we have limited this to results which further inform our understanding of the neural circuits underlying the different behavioral processes discussed in the review.

2. Evidence of animals processing and using information about the costs going into cost-benefit computations underlying motivation

Motivation activates and directs behavior allowing organisms to overcome response costs to obtain specific goals. The decision to continue exerting effort in pursuit of a goal while neglecting other available response options is thought to be influenced by an underlying cost-benefit decision making process. During this process, the organism is thought to use information and knowledge of the costs of the current situation and weighs them against the anticipated benefit the effort will ultimately result in. There is a rich history of studies from experimental psychology in which various specific parameters of cost and benefit are manipulated which generally show that animals can make adaptive decisions in the face of changing costs and benefits (Atalayer & Rowland, 2009; Collier & Johnson, 1997). Additionally, there is evidence that animals can process and use information related to different response costs, including: distance, number, time, height, force and vigor. While an extensive literature exists on animals cognition of distance (Gallistel, 1989), and their sensitivity to manipulations of force required in a lever press (Ettenberg, 1989; Fowler, 1999) here, we limit the discussion to number of responses, time, and vigor/rate of responding as they are the most commonly used manipulations of cost in the studies covered in this review.

2A. Number of Responses as a Cost

Several elegant experiments demonstrated that animals are aware of the number of responses they have made, and that they are not only able to process this information but can also dynamically use it to guide behavior. In these experiments subjects were trained to make lever presses to earn rewards. In the testing phase of these experiments, subjects made lever presses on fixed ratio schedules, but the rewards were delivered without being cued when the criterion was reached. In two variants on this procedure, rats then had to either switch from Lever A and make 1 response on lever B to check if they received a reward (Mechner, 1958), or simply make a head entry to the receptacle when they thought the reward would be present (Platt & Johnson, 1971). Rats were not only able to estimate the minimum number of responses they needed to emit before checking for the reward, but they were actually able to use this information to guide behavior as they were shown to be sensitive to the consequences of their errors in either direction (checking after too few or too many presses) and were able to adjust their estimates to either overestimate or underestimate when they had done enough depending on the contingencies of the given situation (Platt & Johnson, 1971), reviewed in (Gallistel & Gelman, 1992).

Given that subjects have an awareness of how many presses they have made since the beginning of a bout of responding, it is then perhaps unsurprising that rodents can use this information when given a choice between working on two levers paying off after different numbers of presses. When given a choice on two different levers with different press requirements (whether on a Fixed Ratio or Random Ratio) subjects will allocate their responding in a manner which matched the relative payoff between the two levers (McDowell, 2013).

2B. Time

Animals are also sensitive to time and the temporal distribution of events (Balsam, Drew, & Gallistel, 2010; Balsam & Gallistel, 2009). When rewards are delivered following a response occurring after a fixed duration of time, as in a fixed-interval schedule of reinforcement, animals are most likely to respond around the time that reward is expected (Dews, 1978). Increasing motivation levels by increasing the probability of reinforcement on any given trial increases how precisely animals estimate this interval (Roberts, 1981; Ward et al., 2009). Additionally, when asked to discriminate between durations many studies have shown that a 15–20% change in duration is easily discriminated (Gibbon et al., 1984). Thus it is not surprising that choice is allocated based on payoff rates (McDowell, 2013) or that the relative delay to reward has a strong influence on response selection (Evenden & Ryan, 1996). Since all action occurs in time it is worth noting that manipulations of response number, response duration or distance to obtain a goal generally also involve changes in the time to reach that outcome.

2C. Rate or Vigor

Rate or vigor of responding is modulated by motivational factors such as deprivation level and reward magnitude, e.g. response speed tends to increase as a function of reward magnitude, whereas it decreases with increasing delays to reward, reviewed in (Bitterman & Schoel, 1970). Rats are able to process information about how vigorously they are responding and can subsequently modulate their levels of vigor when the magnitude of reward is made dependent on response vigor. Rats taking longer to run down a runway when reward size is increased contingent on increasing latency to reach the goal box (Logan, 1966). Similar results have been observed with lever pressing. When reward is contingent on response speed the vigor of the action can be both raised (Girolami, Kahng, Hilker, & Girolami, 2009; Tanno, Silberberg, & Sakagami, 2012) and lowered (Pizzo, Kirkpatrick, & Blundell, 2009; Tanno & Silberberg, 2014).

3. Activational Components of Motivation

3A.1. Activational component of motivation can be observed through measures of response vigor/persistence

There are a number of different tasks which have allowed researcher to quantify changes in response vigor/persistence. Many of the tasks which have been used involve having animals make responses of a single type to obtain the goal (i.e. running down a runway, or responding on a single lever). The activational component of motivation is readily observed in runway tasks as animals run faster for a food reward as a function of the duration that they have been deprived of food, or as a function of the magnitude of food reward/concentration of sucrose awaiting them in the goal-box (Bitterman & Schoel, 1970; Bower & Trapold, 1959; Goodrich, 1960; Kintsch, 1962; Knarr & Collier, 1962). Similar results have been observed in rates of lever pressing (Collier, 1969; Collier & Levitsky, 1967; Marwine & Collier, 1971), and rates of licking for a varied sucrose concentrations (Beer & Trumble, 1965; Vogel, Mikulka, & Spear, 1968; Ward et al., 2012). Much of the subsequent work which has examined the neurobiology and pharmacology of activational components of motivation has been done using a lever pressing tasks in which response cost is manipulated by varying the numbers of responses required to produce a reward. One commonly used task is called the Progressive Ratio (PR) (Hodos, 1961). In a PR schedule of reinforcement, the required number of responses can either be increased within a single session from one reinforcer to the next (Hodos, 1961; Hodos & Kalman, 1963) or can be changed between sessions over days (Czachowski & Samson, 1999), with the former being the most widely used. In PR schedules, subjects make an increasing number of responses until eventually they reach a breakpoint, a point at which the number of lever presses is too high for the animal to continue making responses (Hodos, 1961; Hodos & Kalman, 1963). The breakpoint is directly related to deprivation level and incentive value/reward magnitude (Cheeta, Brooks, & Willner, 1995; Covarrubias & Aparicio, 2008; Ferguson & Paule, 1995; Hodos, 1961; Rickard, Body, Zhang, Bradshaw, & Szabadi, 2009; Skjoldager, Pierre, & Mittleman, 1993). Many variants of PR have been used, demonstrating that the breakpoint is influenced by both the absolute response requirement (Aberman & Salamone, 1999; Skjoldager et al., 1993) as well as the step size of the ratio increase (Covarrubias & Aparicio, 2008).

3A.2. The Challenge of dissociating activational motivation effects from locomotor effects

While PR schedules have been used extensively to study motivated behavior, use of this task alone has made it challenging to discern whether increases or decreases in breakpoints represent changes in activational motivation OR an increase in non-goal directed general activity. If an animal is more hyperactive and makes all types of motor responses more rapidly this may also lead them to make many more lever presses in a similar amount of time. Conversely, if a manipulation has caused locomotor slowing and an animal makes all types of motor responses more slowly this could lead to making fewer responses purely due to a motor deficit. Many of the drug treatments and genetic manipulations which have been shown to increase or decrease breakpoints in a PR schedule also lead to a corresponding increase or decrease in locomotor activity in an open field test (Aberman & Salamone, 1999; Antoniou, 2005; Cagniard, Balsam, Brunner, & Zhuang, 2006; Hall, Stanis, Avila, & Gulley, 2008; Kellendonk et al., 2006; Mayorga, Popke, Fogle, & Paule, 2000; Randall et al., 2012; Sanders, Hussain, Hen, & Zhuang, 2007; Simón et al., 2000; Simpson et al., 2011; Zhuang et al., 2001). This correlation between PR performance and locomotor activity points out the challenge of being able to distinguish activational motivation effects from locomotor effects when using just one measure of motivated behavior. This challenge as one investigator put it is making, “The distinction between motor deficits (wants to but cannot) and motivation deficits (can but does not want to)” (Wise, 2008). To this, we add the opposite problem of no change in motivation (wants the reward to the same degree), but an animal is in a general hyperactive state which leads to making all types of behaviors (those which are goal directed as well as those which are not) at a faster rate which may make the animal appear to want the reward more. While there is no perfect solution to this challenge to date, we attempted to address the issue by developing methods for studying motivated behavior by altering the type of work requirements making rate of initiation unrelated to the level of wanting the reward.

3A.3 A Strategy for Dissociating Changes in Non-Goal Specific Locomotor Output from Changes in Activational Motivation and the Willingness to Perform Goal-Directed Work

To address the challenge presented when trying to distinguish motivational changes from general locomotor changes in behavior our lab developed a novel task known as the progressive hold down (PHD) task (Bailey et al., 2015), which was specifically designed to make hyperactive motor behavior incompatible with increased willingness to work. In the classic PR task, subjects must make more lever presses in order to earn each subsequent reward (Fig 2A). Unlike the classic PR task where the increasing work requirement is an increasing number of responses, in the PHD task the increasing work requirement is the duration of time a subject is required to maintain the lever in the depressed position during a single lever press. Thus, subjects are required to make single lever holds (maintaining the lever in the depressed position) for increasing durations of time in the PHD task in order to keep earning rewards (Fig 2B). This task intentionally makes increased goal-directed action and increased general locomotor arousal incompatible with one another as hyperactive lever pressing will continually reset the duration of each rapidly emitted press.

Figure 2. Effects of Methamphetamine in a PR and PHD Task.

(A–B). Shows a schematic representation of the Progressive Ratio Task (A) and the Progressive Hold Down Task (B). The yellow bars represent lever presses in (A), and lever holds in (B). The red arrows signify rewards. (C). IP administration of 1.0mg/kg of methamphetamine leads to significant increases in lever pressing and breakpoint in a progressive ratio schedule of reinforcement. (D). IP administration of 1.0mg/kg of methamphetamine leads to significant increase in the number of hold attempts in a PHD task, but does not lead to a significant increase in the highest duration requirement completed (BP). (E). Data from a single subject treated with methamphetamine or vehicle in the PHD task demonstrates the increased number of responses which occur while on the drug, but the responses are inefficient and of shorted durations that required by the schedule. Data in (C – E) from Bailey et al., 2015.

In an examination of this novel method, we first tested the manipulations of food deprivation and reward magnitude to see how these variables influenced behavior in the PHD task. Hungry mice worked for more rewards and reached higher breakpoints before quitting. The increased breakpoints in this task meant that hungry subjects were making lever holds of substantially longer durations. In a similar manner, subject’s willingness to work for rewards and breakpoints increased as a function of reward magnitude when working for sucrose solutions of increasing concentration. The observation that increasing food deprivation levels and increasing reward magnitude led to increases in BP’s in both the classic PR schedules (Skjoldager et al., 1993), as well as the BP in our PHD task (Bailey et al., 2015), suggests that these manipulations are impacting some central motivational mechanism which makes animals more willing to work for rewards regardless of the specific modality of the work (i.e. pressing versus holding).

As a test of this strategy of examining behavior in both a classic lever pressing PR alongside the lever holding PHD, we tested subjects who had been treated with methamphetamine in both tasks. As shown in Figure 2C–E, Meth treated subjects made more lever presses and had a higher breakpoint in the classic PR task. The subjects tested following treatment with Meth in the PHD task, however, did not show increases in long duration goal-directed presses, but showed an increase in rapidly initiated short duration hold attempts which were ineffective in the PHD test (Bailey et al., 2015). Unlike manipulations such as food deprivation and increasing the reward magnitude, Meth only increased the BP in the classic PR. We interpret the increase in ineffective short duration responses in the PHD task as reflecting Meth’s ability to enhance hyperactive motor output (which is important for initiating repeated numbers of responses). We also interpret the results to mean that Meth is not acting on a central motivation mechanism which would increase willingness to perform any type of work as is the case when animals are hungry vs sated. Thus, the PR appears to be a good measure of arousal, but cannot by itself dissociate goal-directed action from arousal or increases in general motor activity. Additional experiments have recently shown that selective inactivation of the dopamine D2 receptor expressing neurons in the indirect pathway of the striatum results in a similar increase in arousal and activation in the PR and overall locomotor activity in an open field, but at the cost of decreased efficiency as a result of bursts of short duration rapid responses in the PHD (Carvalho Poyraz et al., 2016).

3B. Neurobiology of Activational Components of Motivation

There have been a large number of studies which have examined different brain regions and neurotransmitters involved in vigorous effortful responding in operant lever pressing tasks. Many of these have used the PR to assess vigor or persistence in responding (Table 1).

Table 1.

Regional Brain Manipulations that Modulate PR behavior

Brain Region	Manipulation	Method/Drug	Result	Reference
NAcc Core	DA lesion	6-OHDA	↓	Hamill et al., 1999
NAcc Core	DA lesion	Quinolinic acid	↓	Bezzina et al., 2008
NAcc Core	Local D1/D2 antagonism	SCH-23390 (D1) Eticlopride (D2)	↓	Bari et al., 2005
NAcc Shell	DA lesion	6-OHDA	No Effect	Sokolowski et al., 1998
NAcc Shell	Local D1/D2 antagonism	SCH-23390 (D1) Eticlopride (D2)	No Effect	Bari et al., 2005
VTA	D1 antagonism	SCH-23390 (D1)	↓	Sharf et al., 2005
VTA	D2 receptor KD	shRNA KD	↑	de Jong et al., 2015
VTA	Partial DA Lesion	6-OHDA	No Effect	Drui, Carnicella et al. 2013
VTA	Local GHS-R1A agonism	Ghrelin	↑	Skibicka et al., 2011
DMS	Lesion	Quinolinic acid	No Effect*	Eagle et al., 1999
DLS	Lesion	Quinolinic acid	No Effect*	Eagle et al., 1999
SNc	Partial DA Lesion	6-OHDA	↓	Drui, Carnicella et al. 2013
STN	Lesion	Ibotenic acid	↑	Baunez et al., 2002
STN	Lesion	Quinolinic acid	↑	Bezzina et al., 2008
Ventral Hipp	Lesion	NMDA	↑	Gourley et al., 2010
Ventral Hipp	Lesion	Ibotenic acid	↑	Chambers et al., 2002
Prelimbic	Lesion	NMDA	↓	Gourley et al., 2010
mOFC	Lesion	NMDA	↑	Gourley et al., 2010
mOFC	Local D1/D2 antagonism	SCH-23390 (D1) Sulpiride (D2)	↓	Cetin et al., 2004
ACC	Lesion	Quinolinic acid	No effect	Schweimer and Hauber 2005

^*Indicates observed motor effects

The NAcc and Mesolimbic dopamine

A wealth of evidence implicates mesolimbic dopamine pathway, which consists of the dopamine neurons located within the VTA which project to the NAcc, in behavioral activation and energy expenditure (J. D. Salamone, 1992; J. D. Salamone, M. Correa, E. J. Nunes, P. A. Randall, & M. Pardo, 2012). Specifically, reducing dopamine levels in the mesolimbic pathway suppresses general locomotor activity (Maldonado-Irizarry & Kelley, 1994; Wu, Brudzynski, & Mogenson, 1993), as well as novelty-induced locomotion (Baldo, Sadeghian, Basso, & Kelley, 2002; Michael S. Cousins, Sokolowski, & Salamone, 1993; Koob, Riley, Smith, & Robbins, 1978). The effects of dopamine antagonist within the NAcc also impacts goal-directed locomotion as intra-NAcc dopamine antagonists lead to both increased latency to run down a runway maze and reach a goal box containing food reward (slowing of reward approach) as well as reductions in spontaneous locomotion in the start box (Ikemoto & Panksepp, 1996). Moreover, the readily observed increases in numerous different types of activity which develop following the scheduled presentation of food (excessive drinking, voluntary wheel running, and locomotion) are all correlated with increases in mesolimbic dopamine signaling (Louise D. McCullough & Salamone, 1992), and NAcc dopamine depletions suppress these behaviors (Louise D. McCullough & Salamone, 1992; Robbins & Koob, 1980; Wallace, Singer, Finlay, & Gibson, 1983). These observations resulted in the development of a number of different genetic models which alter dopamine signaling. A dopamine transporter knockdown mouse (DAT KD) shows elevated open field activity (Cagniard et al., 2006), and cell type specific loss of D1/D2 receptors have been shown to induce hypoactivity or hyperactivity with numerous different manipulations of these cell types (Kreitzer & Berke, 2011).

In addition to the studies on the locomotor activating effects of the mesolimbic dopamine pathway, there has been a specific focus of the role of this pathway on motivated responding in tasks which offer a single response choice and provide a measure of vigor or behavioral activation. Dopamine neurons which project to the NAcc have been found to be important for effortful responding. Early observations indicated that when a rat was lever pressing for food on a fixed ratio -1 (FR-1), levels of dopamine and DOPAC increased within the NAcc (L. D. McCullough, Cousins, & Salamone, 1993). Subsequent studies showing lesions of dopamine neurons with 6-hydroxydopamine (6-OHDA) projecting to either the NAcc Core or NAcc Shell had little impact on a behavior in an FR-1 schedule, a schedule with a low effort requirement (Salamone & Correa, 2002; J. D. Salamone, Correa, Mingote, & Weber, 2005). Furthermore, disruption of dopamine in either the NAcc Core or NAcc Shell also had no impact in a VI-30 schedule, which requires subjects to wait an average of 30 seconds before making a reinforced press (Sokolowski & Salamone, 1998). However, when the response cost was increased to an FR-05 schedule disruption of dopamine signaling to the NAcc Core, was found to impair responding, but dopamine depletion in the NAcc Shell did not have any effect (Sokolowski & Salamone, 1998). Additionally, there was a correlation between the number of presses made and the amount of dopamine present in the NAcc Core, but not the shell (Sokolowski & Salamone, 1998).

Subsequent studies further explored the impact of dopamine depletions in the NAcc Core across several different fixed ratio schedules (FR-01, 05, 10, 16, 32). In these studies, NAcc Core dopamine depletions reduced the amount of responding, and this reduction was greater in the higher FR schedules (Aberman & Salamone, 1999). This schedule dependent decrease in responding differs from that seen following pre-feeding manipulations, as pre-feeding leads to reductions in responding across all schedules, not just the more demanding ones (Aberman & Salamone, 1999). Further studies tested the effects of NAcc dopamine depletion in a time constrained PR and showed that DA depletions decreased breakpoints at both a PR+1 and PR+5, with the impairments being more marked in the more demanding schedule (Hamill, Trevitt, Nowend, Carlson, & Salamone, 1999).

Taken together, these studies indicate that dopamine depletion appears to affect an animal’s willingness to expend effort to earn a reward. The recognition of the specific involvement of dopamine signaling within the NAcc Core spurred lots of research on the dopamine receptor subtypes important for effort expenditure in these tasks. Numerous studies demonstrated that dopamine D1 or D2 receptor antagonists reduce responding in a PR (Aberman, Ward, & Salamone, 1998; Caul & Brindle, 2001; Cheeta et al., 1995; Olarte-Sanchez, Valencia-Torres, Cassaday, Bradshaw, & Szabadi, 2013), whereas drugs which can increase synaptic dopamine levels, such as amphetamine, increase breakpoints (Bailey et al., 2015; Mayorga et al., 2000; Sommer et al., 2014). Local administration of either the D1 antagonist (SCH-23390) or D2 antagonist (eticlopride) into the NAcc Core decreased lever presses for food in a PR schedule, but neither drug had any impact when infused into the NAcc Shell (Bari & Pierce, 2005). Both the dopamine depletion and localized drug infusion studies suggest that the activating effects of mesolimbic dopamine signaling appear to be quite specific to the NAcc Core.

Greater understanding of the nature of the deficit induced by NAcc Core dopamine manipulations has been revealed by more careful examination of the within session data for tasks in which dopamine depletion or antagonisms has an impact on behavior. The impact of NAcc Core lesions in the FR5 task was primarily seen through slower responding, which resulted from longer inter-response-times (IRT’s) in the FR-05 schedule (Sokolowski & Salamone, 1998), and slower response rates and longer post reinforcement pauses in a PR schedule (Bezzina, Body, Cheung, Hampson, Bradshaw, et al., 2008). Nicola et al., 2010 conducted a detailed behavioral analysis of the effects of intra-NAcc dopamine antagonism on different types of behaviors which further elucidate the nature of the within session behavioral changes. In a task which cues rats to make either 1 lever press or 8 lever presses for a reward (cued FR1 and cued FR8), it was shown that dopamine D1 and D2 antagonists both impair subjects ability to earn rewards, and that the primary influence of the drugs is to increase latencies to begin lever pressing when the animals are currently in a non-responding state (Nicola, 2010). Additionally, the subjects are more likely to be engaged in non-task related behaviors, and the latency to make a lever press (i.e. reengage in task related behavior) is independent of the class of responses subjects are engaged in (immobile resting, random locomoting, or grooming), which suggests that dopamine disruption in the NAcc Core may be impacting motivation by disrupting the initiation of “flexible approach behavior”.

Ventral Tegmental Area

Much like the locomotor effects induced by blocking NAcc dopamine, more recent studies which have used DREADD (designer receptors exclusively activated by designer drugs) methods to inactivate VTA dopamine neurons showed that this lead to suppression of general locomotor activity (Marchant et al., 2016). Far fewer studies have looked at the influence of the VTA in PR responding to see how this area impacts arousal/vigor processes of motivation. It is known, however, that both dopamine D1 and D2 receptors are important for the VTA’s influence on motivated responding. In one study, it was found that localized infusions of the D1 receptor antagonist (SCH 23390) into the VTA lead to a decreased breakpoint in a PR (Sharf, Lee, & Ranaldi, 2005). In another study, reducing the expression of D2 receptors in the VTA via shRNA knockdown lead to increased breakpoints for food in a PR, but did not impact baseline locomotor activity, fixed ratio responding, or responding in extinction (de Jong et al., 2015). The observation that decreasing D2 receptor levels within the VTA enhances motivation is in line with the finding that food deprived rats have lower levels of D2 receptor expression in the VTA relative to ad lib fed rats (Skibicka et al., 2013).

There have also been a number of other receptors on neurons within the VTA which have been examined. While an extensive discussion of all of these is beyond the scope of this review we highlight the role of ghrelin in the VTA, as it’s effects on motivated responding appear to be directly modulated through NAcc dopamine signaling. Ghrelin is a circulating hormone which promotes both food intake as well as motivated responding for food. Studies have shown that both systemic injections of ghrelin or intra-VTA ghrelin enhance food responding in a PR (Naleid, Grace, Cummings, & Levine, 2005; Perello et al., 2010; Skibicka, Hansson, Alvarez-Crespo, Friberg, & Dickson, 2011; Skibicka, Shirazi, Hansson, & Dickson, 2012). This effect of ghrelin has been shown to act by modulating the VTA’s dopamine output to the NAcc. Lesion of VTA dopamine neurons via 6-OHDA, suppresses ghrelin’s ability to increase responding on a PR (Weinberg, Nicholson, & Currie, 2011). Moreover, pretreatment with either a D1 or D2 receptor antagonist in the NAcc blocks intra VTA ghrelin’s ability to increase BP in a PR for food rewards (Skibicka et al., 2013).

The Dorsal Striatum and Nigrostriatal Dopamine

Another dopaminergic pathway in the brain, known as the nigrostriatal pathway, consists of dopaminergic neurons in the Substantia Nigra (SN) and projects to the dorsal striatum. While the role of the NAcc and mesolimbic dopamine signaling in PR schedules has been extensively studied, there has been a smaller amount of work examining the dorsal striatum and nigrostriatal pathways. An early study lesioned cell bodies within the Dorsomedial (DMS) and Dorsolateral Striatum (DLS) via quinolinic acid observed that lesions to both regions failed to alter breakpoints in a PR schedule, but destruction of these regions did have some impact on other aspects of motor performance in the PR (Eagle, Humby, Dunnett, & Robbins, 1999). There was an increase in the number of preservative presses as well as the latency to get to the food hopper when a reward was delivered. Worth noting is that these lesions in the dorsal striatum destroyed cell bodies via quinolinic acid. We are not aware of any studies which examined dopamine specific depletion in the dorsal striatum via 6-OHDA as was done in the studies mentioned above that focused on the NAcc.

Investigators have also examined the influence of the Substantia Nigra in motivated behavior. One study looked at the effect of inactivating the Substantia Nigra pars reticulata (SNr) during FR-05 responding and found that infusions of the GABA_A antagonist bicuculline resulted in a dose-related decrease in lever pressing. Additionally, GABA levels within the region were higher during the lever pressing than during baseline periods before the operant responding (Correa, Mingote, Betz, Wisniecki, & Salamone, 2003). Another study looked at the Substantia Niagra pars Compacta (SNc) on motivated behavior. This study induced partial lesions to the SNc which didn’t disrupt overall locomotor behavior resulted in decreased lever pressing in a PR for sucrose rewards, but these same effects were not observed with partial lesions of the VTA (Guillaume Drui et al., 2013). Additional evidence for the role of the Nigrostriatal DA system in motivated behavior comes from a recent study using a novel operant joystick based task. Subjects were head fixed and required to move the joystick at a given rate to earn a reward. MitoPark mice, which have progressive loss of SN to DA dopamine neurons (Ekstrand et al., 2007), show impairments in this task, and optogenetic inhibition of the nigrostriatal pathway induced similar impairments such that subjects took longer to complete a criterion number of trials. Electrophysiological recording from the neurons in this region showed that DMS neurons appear to be both representing and controlling movement vigor in this task (Panigrahi et al., 2015). This is in line with another recent study which used a self-paced nose poking paradigm to demonstrate that overall reward payoff expectancy as well as response vigor appear to be represented in the DS (A. Y. Wang, Miura, & Uchida, 2013).

Genetically induced Dopamine Receptor Manipulations

There have been a number of genetically modified mouse lines which have allowed researchers to examine the role of specific dopamine receptors in different brain regions. These studies have examined the effects of alterations in the levels of expression of dopamine receptors.

It has been shown that developmental overexpression of the dopamine D2 receptor (D2ROE) within the striatum (Kellendonk et al., 2006) leads to an impairment in PR responding (Drew et al., 2007; Simpson et al., 2011; Ward et al., 2012). This genetic model is developmental and D2R overexpression continues into adulthood. In contrast, viral vector mediated manipulations were developed which allow D2 receptors to be expressed at much higher levels selectively in adulthood (Trifilieff et al., 2013). While viral over-expression of the D2 receptor in the NAcc led to increased PR responding, this same effect was not observed when the over-expression was in the dorsal striatum (Trifilieff et al., 2013). As well as the difference in D2R overexpression during development, another important difference between the two models is that the viral D2 receptor over-expression is not restricted to the MSN’s (as it is in the developmental D2R-OE model). The dopamine D3 receptor also appears to be involved in the activational aspects of motivation as a developmental genetic model of dopamine D3 receptor overexpression which is restricted to the striatum also showed decreases in lever press behavior in a PR (Simpson et al., 2014).

Other Brain Regions Modulating response Vigor in a Progressive Ratio

There have been a number of brain regions in addition to the striatum and midbrain which have been implicated in motivation (McGinty et al., 2011), and have been investigated to determine their contribution to responding in a PR schedule). We briefly describe several of these other areas that are also involved in other motivational processes to be discussed later in the review.

Ventral Pallidum

Another brain region which is thought to be involved in activational aspects of motivation is the Ventral Pallidum (VP), a region which receives GABAergic projections from the NAcc (Root, Melendez, Zaborszky, & Napier, 2015). While we are unware of any studies which have manipulated the VP and specifically looked at PR responding, a number of studies have established the role of the VP in the motivation to eat and drink as damage to this region leads to a failure to voluntarily consume food and water, reviewed in (Root et al., 2015; Smith, Tindell, Aldridge, & Berridge, 2009).

Cortical Structures

Lesions studies of the prefrontal cortex have demonstrated that different sub-regions of the PFC contribute to PR performance. One study lesioned several cortical structures and found dissociations between a number of these regions. Lesions to the pre-limbic cortex (PL) were shown to decrease breakpoints in a PR schedule, and the same was found with lesions of the lateral orbitofrontal cortex (lOFC). Lesions of the medial orbitofrontal cortex (mOFC) lead to increased responding and increased breakpoints in one study (Gourley, Lee, Howell, Pittenger, & Taylor, 2010), but not another (Kheramin et al., 2005). Studies in which dopamine antagonists are locally infused into the mOFC suggests that this region is indeed involved in modulating PR performance as infusions of either the D1-receptor antagonist (SCH23390) or the DA D2-receptor antagonist (sulpiride) lead to reductions in the breakpoint in a PR while leaving food preference and consumption unchanged (Cetin, Freudenberg, Fuchtemeier, & Koch, 2004). The ACC is a region which also receives dopaminergic projections from the VTA, and has reciprocal connections with the NAcc, however an experiment which lesioned the ACC did not find any effect of the lesions on the breakpoint in a PR (Judith Schweimer, Saft, & Hauber, 2005).

Hippocampus

The hippocampus is another region which has received a small amount of attention for its role in motivated behavior assessed with a PR schedule. Lesions to the ventral hippocampus, (an area which projects to the mOFC) was shown to increase BP in the PR. In another study, neonatal ventral hippocampal lesions were shown to increase the BP’s of rats when tested in a PR in adulthood (Chambers & Self, 2002).

Sub Thalamic Nucleus

The sub thalamic nucleus (STN) is a basal ganglia nucleus which sends glutamate projections most densely to the pallidal complex and the SNr, and less dense connections to the striatum and SNc (Parent & Hazrati, 1995). Rats with lesions to the STN showed higher breakpoints in a PR (Baunez, Amalric, & Robbins, 2002; Bezzina, Body, Cheung, Hampson, Bradshaw, et al., 2008)), and another study found that discrete lesions of the STN increased responding for liquid sucrose rewards in a PR, but greatly decreased the motivation of rats for cocaine (Baunez, Dias, Cador, & Amalric, 2005).

Summary

There are a number of different structures which regulate behavioral activation and locomotor output in goal-directed responding in a progressive ratio task (Fig 3A). The mesoaccumbal dopamine system (VTA and NAcc), the nigrostrial dopamine system (SN and DS), as well as the subthalamic nucleus, ventral hippocampus, and a number of prefrontal cortical regions (PL and mOFC), all modulate behavior in a PR schedule. It is also clear that the NAcc core, innervated by dopaminergic neurons from the VTA, plays an important role in the activational aspects of motivation (Bari & Pierce, 2005; Bezzina, Body, Cheung, Hampson, Deakin, et al., 2008; Hamill et al., 1999). The NAcc shell, however, does not appear to be important for this activational process (Bari & Pierce, 2005; Sokolowski & Salamone, 1998). Additionally, the SN, and DS also appear to be involved in some aspects of this activational component of motivated behavior (G. Drui et al., 2014). Specifically, recent experiments which monitor in vivo activity within the dorsal striatum in awake behaving animals performing motivated operant behavioral tasks have demonstrated that this region appears to be important for representing and modulating response vigor (Panigrahi et al., 2015; A. Y. Wang et al., 2013). Finally, the role of, various PFC structures (Cetin et al., 2004; Gourley et al., 2010), the hippocampus (Chambers & Self, 2002; Gourley et al., 2010), and the STN (Baunez et al., 2002; Bezzina, den Boon, et al., 2008) appear to contribute to the activational aspects of motivated responding.

Figure 3. Brain Regions Which Impact Performance on a Progressive Ratio and Effort Based Choice.

A. Shows brain regions which have been studied to examine their involvement in PR performance through lesion, inactivation, or localized drug infusion studies which have been shown to modulate PR behavior (Orange), have no effect on PR behavior (Grey), or have yet to be examined (White). Areas which have been shown to modulate PR behavior include: the VTA, the Ventral hippocampus, the SN pars reticulata and SN par compata, the NAcc Core, the OFC, and the PR/IL cortex. Areas which have been studies, but damage or inactivation had no impact on PR performance include: ACC, and NAcc Shell. All other areas have not been studied with a PR task: VP, DS. IL, Hipp

B. Shows brain regions which have been studied to examine their involvement in effort based choice performance through lesion, inactivation, or localized drug infusion studies which have been shown to modulate effort choice behavior (Green), have no effect (Grey), or have yet to be examined (White). Areas which have been shown to modulate effort based choice include: the VTA (Reference), NAcc Core, NAcc Shell, VP, ACC. Areas which have been studies, but damage or inactivation had no impact include: OFC, PL, IL. Areas which have yet to be studied include: DS, STN, SN, Hipp, vSyb

4. Neurobiology of Cost-Benefit Decision Making: Manipulations of Different Costs

Over the last several decades a substantial amount of progress has been made toward understanding of the neural circuits involved in various aspects of cost-benefit decision making (Table 2). Many of the studies involved tasks which require subjects to make a choice between different response options. In these paradigms, animals have been faced with alternatives associated with different costs - differences in the effort requirements, the time delay from response choice to reward delivery, and the probability of reward of each option. As will be described in more detail below, what has emerged as a result of this work is an increasing understanding of the key neural structures and neurotransmitters involved in different forms of cost-benefit decision making. Within this distributed neural circuitry, several neurotransmitters have been found to be involved in the modulation of different types of decision making. Below, we discuss the specific brain regions and neurotransmitters involved in the cost-benefit decision making process in studies that manipulate effort requirements, time delays, and/or probability of reward associated with different choice alternatives.

Table 2.

Studies of Effort Based Decision Making

Brain Region	Manipulation	Method	Behavioral Task	Cost Decision	Result	Reference
NAcc	Bilateral Lesion	↓ High reward choices	T-Arm Barrier Maze	Climb barrier or not	↓ High reward choices	Hauber, Sommer, 2009
NAcc Core	DA depletion	6-OHDA	Concurrent Lever Press/Free Chow Consumption	Lever Press or Not	↓Presses ↑ Chow Intake	Salamone et al., 1991
NAcc Core	DA depletion	6-OHDA	Concurrent Lever Press/Free Chow Consumption	Lever Press or Not	↓Presses ↑ Chow Intake	Sokolowski et al., 1998;
NAcc Core	Inactivation	Muscimol /Baclofen	Operant Effort-Discounting	# of Presses Required	↓ large reward preference	Ghods-Sharifi and Floresco 2010
NAcc Core	Adenosine A2A Agonist	CGS 21680	Concurrent Lever Press/Free Chow Consumption	Lever Press or Not	↓Presses ↑ Chow Intake	Font et., al. 2008
NAcc Shell	DA depletion	6-OHDA	Concurrent Lever Press/Free Chow Consumption	Lever Press or Not	No Effect	Sokolowski et al., 1998; Salamone et al., 1991
NAcc Shell	Inactivation	Muscimol /Baclofen	Operant Effort-Discounting	# of Presses Required	No Effect	Ghods-Sharifi and Floresco 2010
VP	Inactivation	Muscimol	Concurrent Lever Press/Free Chow Consumption	Lever Press or Not	↓Presses ↑ Chow Intake	Farrar et., al. 2008
ACC	Bilateral Lesion	Excitotoxic Lesions	T-Arm Barrier Maze	Climb barrier or not	↓ High reward choices	Walton et al., 2003
ACC	DA lesions	6-OHDA	T-Arm Barrier Maze	Climb barrier or not	↓ High reward choices	Schweimer and Hauber, 2005
ACC	DA lesions	6-OHDA	T-Arm Barrier Maze	Climb barrier or not	No Effect
ACC	Local D1- antagonist	SCH23390	T-Arm Barrier Maze	Climb barrier or not	↓ High reward choices	Schweimer and \Hauber, 2006
ACC	Bilateral Lesion	Quinolinic acid	Concurrent Lever Press/Free Chow Consumption	Lever Press or Not	No Effect	Schweimer and Hauber 2005
ACC	Bilateral Lesion	Excitotoxic	FR16 (4) vs FR 4 (2)	# of Presses Required	↓ High Effort Lever	Walton, Groves et al. 2009
BLA	Bilateral Inactivation	Bupivacaine	T-Arm Barrier Maze	Climb barrier or not	↓ High reward choices	Floresco et al., 2007
BLA	Inactivation	GABA agonist muscimo/baclofen	Operant Effort-Discounting	# of Presses Required	↓ large reward preference	Ghods-Sharifi et al., 2009
ACC-BLA	Contralateral Disconnection	Mucosimol/Baclofen	T-Arm Barrier Maze	Climb barrier or not	↓ High reward choices	Floresco et al., 2007
NAcc-VP	Contralateral Disconnection	VP: muscimol NAcc: CGS 21680	Concurrent Lever Press/Free Chow Consumption	Lever Press or Not	↓Presses ↑ Chow Intake	Mingote et al., 2008
NAcc - ACC	Contralateral disconnection	NAcc: Quinolinic acid ACC: Quinolinic acid	T-Arm Barrier Maze	Climb barrier or not	↓ High reward choices	Hauber, Sommer, 2009

4A. The Choice Between Two Effort Options

There have been a number of tasks which have been developed to study effort based decision making which give animals a choice between 2 effort alternatives (high vs low) for 2 different types or amount of reward (high vs low). The development of these tasks has been important because it allows researchers to determine whether the critical functioning of dopamine in the NAcc Core and associated circuits are involved in processes related to effort expenditure and not the result of a dopamine related motor effects which enhance or impair an animal’s capacity to make a particular response. Below, we provide a summary of the different tasks employed and the findings which each have allowed researchers to make.

4A1. Concurrent Lever Pressing/Chow Feeding Task

One behavioral task which was developed to study effort based decision making is an operant lever pressing task often referred to as either a Concurrent Lever Pressing/Chow Feeding task or Effort-Based Choice Task (EBCT) (J. D. Salamone, 1991). We will hereafter refer to the task as the EBCT. In the EBCT testing sessions, subjects make a choice between lever pressing on a given schedule to earn a preferred reward (i.e. sucrose pellets or evaporated milk) or consume a freely available, but less preferred home cage chow. Rats and mice will earn most of their food in the task by lever pressing for the preferred reward (J. D. Salamone, 1991). As the effort to earn the preferred reward is increased subjects choose to press the lever less frequently and consume more of the freely available chow. This task has been useful in assessing willingness to expend effort for a preferred reward because pre-feeding subjects or giving them appetite suppressants leads to decreases in both lever pressing and chow consumptions (Randall et al., 2014; Randall et al., 2012; J. D. Salamone, 1991; J. D. Salamone, Arizzi, Sandoval, Cervone, & Aberman, 2002; Sink, Vemuri, Olszewska, Makriyannis, & Salamone, 2008), whereas a reduction in willingness to work is reflected in less lever pressing and more consumption of the freely available choice.

NAcc Dopamine and Extended Circuitry

Dopamine signaling in the NAcc is important in the EBCT, as 6-OHDA lesions in the NAcc Core decreased the number of lever presses made in the EBCT and lead to an increase in chow consumption, whereas the same lesions in the NAcc shell had a smaller impact (J. D. Salamone, 1991; Sokolowski & Salamone, 1998). Subsequent work has shown that both dopamine D1 (SCH 23390, SKF83566, and Ecopipam) and D2 (Haloperidol, cis-flupenthixol, raclopride, eticlopride) receptor antagonists produce similar shifts from lever pressing to consuming the less preferred freely available chow when injected systemically, or directly into the NAcc Core or Shell (M. S. Cousins & Salamone, 1994; Farrar et al., 2010; Koch, Schmid, & Schnitzler, 2000; Nowend, Arizzi, Carlson, & Salamone, 2001; J. D. Salamone, 1991; J. D. Salamone et al., 2002; Sink et al., 2008; Worden et al., 2009).

A series of experiments subsequently demonstrated the importance of the connection between the NAcc Core and Ventral Pallidum (VP) in regulating effort-based choice. Injections of the GABA_A receptor agonist muscimol into the VP decreases lever pressing and increases consumption of the freely available chow in the same manner as NAcc Core dopamine depletion (Farrar et al., 2008). Retrograde tracers injected into the same region of the VP in which muscimol caused a shift in effort-based choice behavior confirmed that NAcc Core was an input to the VP. While the VP also received input from the DS, it was previously shown that dopamine depletion in this region did not impact lever pressing or chow consumption (Farrar et al., 2008). Further studies revealed that both systemic treatment and intra-NAcc infusions of the Adenosine 2A receptor (A2aR) agonist (CGS 2168) could produce a decrease in lever pressing and increase in free chow consumption (Font et al., 2008). The effect of the A2aR agonist drug occurs through the GABA-ergic pathway between the NAcc and VP as CSG 2168 leads to an increase in GABA release in the VP (Mingote et al., 2008). Finally, the importance of the NAcc-VP projection was demonstrated in a circuit disruption experiment in which CSG 2168 was infused unilaterally into the NAcc and muscimol infused into the contralateral VP and resulted in decrease in lever pressing and an increase in chow consumption (Mingote et al., 2008) though ipsilateral infusions did not.

Following up on the observation that the A2aR agonist CSG 2168 could reduce lever pressing and increase chow consumption, it was later found that systemic treatment with an A2aR antagonist rescued a D1/D2R antagonist induced decrease in the choice of an effortful response (J. D. Salamone & Correa, 2009). Moreover, this effect appears to be selective to the A2A receptor as opposed to the A1a receptor as both A2a selective antagonists and Caffeine (a nonspecific A2a and A1a antagonist) rescue the dopamine antagonist impairment, whereas an A1A selective antagonist does not (J. D. Salamone & Correa, 2009; Worden et al., 2009).

4A2. Operant Effort Discounting

[Abbreviations in this section: Low Effort/Low Reward (LR); High Effort/High Reward (HR)]

Another task used to assess effort-based choice is known as the Effort Discounting task (Floresco, Tse, & Ghods-Sharifi, 2008). In this task, subjects have the option to make lever press responses on a Low-Effort/Low-Reward lever (LR) (e.g.1 press leads to 2 pellets or a High-Effort/High-Reward lever (HR) (e.g. 5, 10, 20, or 40 presses leads to 3 pellets). The requirement on the HR lever is increased over the course of a session. In this paradigm, well trained rats tend to earn all of their rewards on the HR lever when the requirement is 5 presses, and make fewer of the higher effort lever choices as the cost requirement is increased throughout the session.

NAcc Dopamine, the Basolateral Amygdala, and the Anterior Cingulate Cortex

A number of systemic pharmacology studies have implicated dopamine’s involvement in the Effort-Discounting task, similar to that which is seen in the EBCT. The non-selective dopamine receptor antagonist flupenthixol reduced choice on the HR lever in an Effort-Discounting task (Floresco et al., 2008). Moreover, the D1 (SCH23390) and D2 (Eticlopride) receptor antagonists were also shown to decrease the number of choices of the HR lever (Jay G. Hosking, Floresco, & Winstanley, 2015; Randall et al., 2014; Randall et al., 2012). Using a variant of this procedure in which rats could lever press on a high effort lever (FR12) for high reward (4 pellets) vs a low effort lever (FR4) for low reward (2 pellets), it was shown that the D2 receptor antagonists (haloperidol), caused rats to shift to the low effort lever (Walton et al., 2009).

Whereas dopamine antagonists reliably cause subjects to shift to make more responses on the low effort/low reward lever, amphetamine was found to exert a bi-phasic dose dependent effect on effort choice. At low doses subjects made more responses on the high effort large reward lever, whereas at high doses subjects made fewer responses on the high effort large reward compared to vehicle treated subjects (Floresco et al., 2008). These effects of dopaminergic drugs appear to be mediated via the NAcc Core sub region as local blockade of GABA _A and _B receptors decreases the selection of the HR lever under both standard and equivalent delay conditions, whereas the same effect was not seen when the blockade occurred in the NAcc Shell (Ghods-Sharifi & Floresco, 2010). A control experiment demonstrated that the inactivation of the NAcc Core did not alter the preference for 4 vs 2 pellets when the press requirement for each is equivalent.

Interestingly, dopamine’s impact on effort appear to be specific to physical as compared to cognitive effort, as Eticlopride and SCH23390 decreased willingness to expend physical effort, but had no effect on cognitive effort in a novel rodent cognitive effort task which allowed subjects to choose between an easy and difficult discrimination for small vs larger rewards (Jay G. Hosking et al., 2015).

Basolateral Amygdala

Using the operant effort-discounting paradigm, it has also been shown that the BLA is also involved in effort based decision making. Infusions of the GABA_B agonist baclofen and the GABA_A agonist muscimol combined into the BLA increased effort discounting, reducing the preference for the HR lever, even in conditions in which the delays to reward delivery were equalized across response conditions (Ghods-Sharifi, Onge, & Floresco, 2009). Additional evidence of the BLA’s involvement in effortful behavior comes from a study by Simmons et al. (2009) which showed that bilateral inactivation of the BLA with muscimol reduced lever pressing on an FR15 schedule while leaving consumption of food in a separate free consumption test unchanged (Simmons & Neill, 2009). This study found that the connection between the BLA and the NAcc Core appears to be important as inactivation of the NAcc Core as well as a contralateral inactivation procedure of the BLA and NAcc Core reduced lever pressing and left food consumption unaltered in a separate chow consumption test (Simmons & Neill, 2009).

Anterior Cingulate Cortex

The ACC was lesioned and subjects were tested in the EBCT, but lesioning of ACC had no effect on either the number of lever presses made or the amount of chow consumed (J. Schweimer & Hauber, 2005). In a different experiment, when rats were given the choice to lever press on a high effort lever (FR12) for high reward (4 pellets) vs a low effort lever (FR4) for low reward (2 pellets) lesions to the ACC caused a shift in responding from the high effort lever to the low effort lever (Walton et al., 2009). Moreover, lesions to the ACC were also shown to decrease willingness to expend cognitive effort in the variant of the task which allowed subjects to choose between an easy and difficult discrimination for small vs larger rewards (J. G. Hosking, Cocker, & Winstanley, 2014).

4A3. T-Arm Barrier Maze

The effort based choice paradigms which have been discussed so far have both involved continuous availability of choices in which at least one alternative involved operant lever pressing. Consequently, there is a specific motor element to these tasks as subjects must be able to repeatedly initiate responses for the high effort option and the exact times at which the options are being compared is unknown. To evaluate cost-benefit decision making with a different motor response in a task which isolated the decision point to a single action, a task was developed known as the T-Arm barrier maze (J. D. Salamone, Cousins, & Bucher, 1994). In this task, subjects are required to navigate down a T Maze, and choose to go either to the right arm or left arm in order to obtain a reward. In this paradigm, there is a high reward and low reward arm (e.g. 2 pellets vs 4 pellets; respectively). In the absence of any barriers, subjects will choose the high reward arm almost all of the time. When there is a barrier placed in the high reward arm that requires additional effort to reach the reward, subjects will still choose this high reward arm about 80% of the time.

Nucleus Accumbens Dopamine

Much like the studies of effort based choice in the operant lever pressing paradigms, dopamine appears to be involved in the processes underlying effort based decision making in the T-Arm Barrier Maze as well. Dopamine depletion in the NAcc via 6-OHDA and dopamine receptor blockade via systemic treatment with dopamine D1 and D2 antagonists decrease the likelihood of choosing the high-effort/high-reward arm (Bardgett, Depenbrock, Downs, Points, & Green, 2009; M. S. Cousins, Atherton, Turner, & Salamone, 1996; Mott et al., 2009; J. D. Salamone et al., 1994). In contrast, increasing dopamine levels with systemic treatment of amphetamine increase the likelihood of choosing the high-effort/high-reward arm (Bardgett et al., 2009). This effect seems to be specific to the effort requirement of climbing over the barrier in the high reward arm and not due to a change in the relative value of the high and low rewards, as subjects will choose the high reward arm in the absence of any barrier (J. D. Salamone et al., 1994), and subjects will choose the high reward arm with the barrier present when the other arm contains no pellets (M. S. Cousins et al., 1996). Just as with the NAcc dopamine depletion and dopamine receptor antagonism impairment seen in the EBCT, the impairments caused in the T-arm barrier maze can be reversed by systemic administration of A2a antagonists (Mott et al., 2009). Again, as in the operant effort based choice task, this rescue is receptor subtype specific, as the A1a receptor does not rescue the impaired behavior (Mott et al., 2009).

While the NAcc dopamine depletion effect on the T-arm barrier maze demonstrates that dopamine in the NAcc influences the effort based decisions, systemic dopamine manipulations may be acting in multiple sites. The ACC receives dopaminergic projections from the VTA (Hoover & Vertes, 2007; Lindvall, Björklund, & Divac, 1978). Two initial studies which lesioned dopamine neurons within the ACC via 6-OHDA showed mixed results as impairments in choosing the high-effort/high-reward arm were observed following dopamine depletions in one study (J. Schweimer & Hauber, 2005), but did not impair the behavior in another (Walton, Croxson, Rushworth, & Bannerman, 2005). Support for the hypothesis that dopamine does act in the ACC for effort based decision making in the T-Arm Barrier Maze comes from finding that localized infusions of the D1-Antagonists into the ACC disrupt high-effort/high-reward arm choices, whereas D2-Antagonists administered here do not (J. Schweimer & Hauber, 2006). The studies of dopamine which implicate the ACC as being involved in effort-based decisions fits with a number of other studies examining the requirement of ACC function choice in the T-arm barrier maze described in the next section.

The Prefrontal Cortex and Basolateral Amygdala

One of the early studies examining the role of the prefrontal cortex in effort based decision making using the T-Arm Barrier Maze looked at the effects of a broad, non-region specific lesion of the medial-prefrontal cortex (mPFC), encompassing several sub-regions of the mPFC including: the Infra-Limbic Cortex (IL), the Prelimbic Cortex (PL), and the Anterior Cingulate Cortex (ACC). These non-specific lesions which damaged all 3 sub-regions impaired effort based decision making, as subjects chose the low effort-low reward arm a higher percentage of the time (Walton, Bannerman, & Rushworth, 2002). Subsequent work revealed a functional specialization within the sub-regions of the mPFC as lesions to the ACC alone were sufficient to produce the impaired effort based decision making behavior, whereas lesions to the both the IL and PL were no different from the sham lesioned control group (Walton, Bannerman, Alterescu, & Rushworth, 2003).

Later studies went on to show that while bilateral destruction of the ACC was sufficient to impair effort based decision making, this same impairment could also be produced by damaging brain regions which directly project to the ACC. It was shown that bilateral inactivation of the Basao-Lateral Amygdala (BLA) with Bupivacaine lead to impairments in choosing the high-effort/high-reward arm (Floresco & Ghods-Sharifi, 2007). In a similar manner, bilateral lesions of the NAcc Core impaired choices of the high-effort/high-reward arm (Hauber & Sommer, 2009). Importantly, however, the common denominator between these two studies appears to be the connection between these nuclei and the ACC. It was shown that a functional connection between the BLA and ACC is involved in cost benefit decision making in the T-arm barrier maze as bilateral inactivation of this circuit leads to impaired responding identical to bilateral inactivation of either the ACC or BLA alone (Floresco, 2007). Similarly, bilateral inactivation of the NAcc – ACC circuit also disrupts the choices of the high-effort/high-reward arm (Hauber & Sommer, 2009).

Convergent evidence for an important role for ACC in effort based decision making comes from in vivo electrophysiological recording studies in behaving animals. Hillman & Bilkey (2010) employed a spatial cost benefit decision making paradigm, similar to the T-arm Barrier Maze task. Rats could choose to navigate to earn 6 rewards vs 2 rewards depending on which arm they chose. When a barrier was present in front of the 6 pellet arm, a substantial portion of ACC neurons (63%) exhibited significantly higher firing for one goal trajectory versus the other; for 94% of these cells, higher firing was associated with the arm with a barrier and 6 pellets. In intersession and intra-session manipulations involving at least one barrier, ACC activity rapidly adapted to the changing conditions and was consistently biased toward the low effort option relative to the configuration. Interestingly, when no barrier was present and the only difference between the 2 arms was the reward magnitude, the high reward arm was chosen on 84% of trials and ACC activity was minimal and nonbiased (Hillman & Bilkey, 2010). Together, these observations demonstrated that the High effort/HR bias was not simply attributable to the larger reward, the barrier, or behavioral preference.

Summary

The results from these three different effort based choice tasks reveals some converging observations implicating certain brain regions which appear to be involved in making decisions about effortful choice across different types of tasks (Fig 3B). These include the NAcc Core, VP, BLA, and the ACC. The NAcc core appears to be critical for effortful choice behavior, as it is was shown to be involved in all three tasks. The GABAergic connection between the NAcc Core and the VP has been shown to be important in the EBCT (Farrar et al., 2008; Font et al., 2008; Mingote et al., 2008; John D. Salamone et al., 2015), but to our knowledge has yet to be examined in other effort based choice paradigms, but such studies may be fruitful for future investigators as the VP is thought to be implicated in motivational processes (McGinty et al., 2011).

Brain targets which have direct connections with the NAcc Core are also important in effort based decision making. Specifically, the ACC, as well as its connection with the NAcc appear to modulate effort based choice behavior, as lesions/inactivation of the ACC and disconnection of the ACC - NAcc decreases willingness to choose the high effort option in the operant effort discounting task and the T-arm barrier maze task (Hauber & Sommer, 2009). Worth consideration, however, is the notion that while the ACC is involved in some types of effort-based decision making, this structure may not be universally involved in all situations requiring effort based decision making as there are some tasks in which lesions of this structure do not impact effortful choice behavior (J. Schweimer & Hauber, 2005). Moreover, some studies report that the deficits observed in effort based choice following lesions to the ACC are only transient which may suggest that other brain regions can compensate for the loss of this region in these situations.

The BLA is another region which is connected to the NAcc Core and has a role in effort based choice processes. In both the operant effort discounting procedure and the T-arm barrier maze, lesions of the BLA decreased the willingness to choose the high effort option for larger rewards (Floresco & Ghods-Sharifi, 2007; Ghods-Sharifi et al., 2009). While we are unaware of any studies which have specifically examined the BLA in the EBCT, the observation that BLA inactivation decreases responding in an FR-16 (Simmons & Neill, 2009) suggests the BLA may be involved in this behavior as well.

4B. Choice between two reward delays

Yet another cost which one can incur in a decision making task is the cost of waiting for a reward. The investigators who have studied behavior by systematically manipulating delay to reward have conceptualized this paradigm as reflecting one’s level of impulsive choice (summarized in Table 3). The central idea behind such tasks is as follows: how do subjects choose between a small reward delivered immediately versus a larger reward delivered after some delay, as a function of increasing durations of delay. Of note, is that these tasks are all similarly controlling effort by having equal physical effort requirements to initiate delays to the next reward.

Table 3.

Brain Regions Which Modulate Delay Based Choice Behavior

Brain Region	Manipulation	Method/Drug	Result	Reference
mPFC	D1/D5 antagonism	SCH-23390 (D1)	↓ delayed choice	Loos et al., 2010
mPFC	D1/D5 agonism	SKF 38393	↓ delayed choice	Loos et al., 2010
mOFC	Bilateral Inactivation	Muscimol (GABA A) Baclofen (GABA B)	No effect	Stopper et al., 2014
OFC	Lesion	Quinolinic acid	↓ delayed choice	Mobini et al. 2002
OFC	Lesion		↑ delayed choice	Winstanley et al. 2004
OFC	Lesion	Quinolinic acid	↓ delayed choice	Rudebeck et al., 2006
OFC	Lesion	Muscimol (GABA A) Baclofen (GABA B)	Dependent on Cue/baseline behavior	Zeeb et al., 2010
OFC	DA Lesion	6-OHDA	↑ delayed choice	Kheramin et al. 2004
OFC	D2 antagonism - No cue	Eticlopride (D2)	No effect	Zeeb et al., 2010
OFC	D1 antagonism -No cue	SCH-23390 (D1)	No effect	Zeeb et al., 2010
OFC	D2 antagonism- Cue	Eticlopride (D2)	↓ delayed choice	Zeeb et al., 2010
OFC	D1 antagonism -Cue	SCH-23390 (D1)	↓ delayed choice	Zeeb et al., 2010
BLA	Bilateral Inactivation	Quinolinic acid	↓ delayed choice	Winstanley et al. 2004
STN	Lesion	Ibotenic acid	↑ delayed choice	Winstanley et al. 2005
STN	Lesion	Ibotenic acid	↑ delayed choice	Uslaner et al., 2006
STN	Lesion	Quinolinic acid	↓ delayed choice	Bezzina et al., 2009
NAcc	DA Lesion	6-OHDA	No effect	Winstanley et al., 2005

Operant Delay Discounting

There have been a few different variants of paradigms developed to study delay-discounting. In an operant delay-discounting task, subjects have a choice between pressing a lever which will deliver a small reward (2 pellets) immediately or pressing a different lever which will result in a larger reward (4 pellets) which is delivered after some delay. The delay is usually increased over the course of a session. Under baseline conditions, subjects will make more choices on the long delay/high reward when the delays are relatively short and decrease their percent of long delay choices as a function of the reward delay (Floresco et al., 2008).

A number of studies have examined the impact of systemic administration of dopaminergic drugs on delay-discounting. The results of treatment with amphetamine on delay based choice have been mixed, and appear to depend on a variety of procedural factors. On the one hand, there have been numerous studies which have shown that amphetamine increases the number of large reward- long delay choices, which is interpreted as a decrease in impulsive choice (Barbelivien, Billy, Lazarus, Kelche, & Majchrzak, 2008; Floresco et al., 2008; van Gaalen, van Koten, Schoffelmeer, & Vanderschuren, 2006; Wade, de Wit, & Richards, 2000; Winstanley, Dalley, Theobald, & Robbins, 2003; Winstanley, Theobald, Dalley, & Robbins, 2005). In these studies, amphetamine appears to increase subject’s indifferent point for delay, meaning subjects are willing to wait longer to get the larger reward (Wade et al., 2000). Amphetamine treatment also leads to decreases in choice latency and increases the number of trials subjects will complete. Treatment with methylphenidate produces the same effect on indifference point as amphetamine (van Gaalen et al., 2006). These effects are mediated by the increases in extracellular dopamine levels as the increased choice of the large-delayed reward were mimicked by the selective dopamine reuptake inhibitor GBR 12909 but not by the noradrenaline reuptake inhibitor desipramine (van Gaalen et al., 2006). As previously alluded to, however, there have also been some studies which have shown either no influence of amphetamine on delay based choice, or a decrease in preference for the delayed but larger reward (Koffarnus, Newman, Grundt, Rice, & Woods, 2011; Slezak & Anderson, 2009; Stanis, Marquez Avila, White, & Gulley, 2008; Tanno, Maguire, Henson, & France, 2014). The existence of these mixed results suggests that drugs like amphetamine do not uniformly increase preference for larger, long delay rewards in all situations.

A number of studies have found that dopamine antagonists increase the number of small reward immediate choices associated with more impulsive choice (Floresco et al., 2008; van Gaalen et al., 2006; Wade et al., 2000). Specifically, the non-selective dopamine receptor antagonist flupenthixol (25, 50, and 100 μg/kg) and the D2 antagonist raclopride (40, 80, and 120 μg/kg), both decreased subject’s indifference point of delays suggesting subjects treat a shorter delay as being equivalent to the immediate reward delivery option as compared to vehicle treated subjects (Wade et al., 2000). The D1R antagonist SCH 23390 (5, 10, and 20 μg/kg), however, did not affect the indifference point.

Furthermore, treatment with the adenosine A2aR agonist (clonidine) was also shown to increase the selection of the low reward-low delayed option, whereas the A1aR agonist phenylephrine did not affect behavior in the delay-discounting task (van Gaalen et al. 2006). Interestingly, this is a similar pattern of results observed in the EBCT – A2aR, but not A1aR agonists acting like D1/D2R antagonists.

It does not appear as though these dopamine drugs are acting within the NAcc, as dopamine depletions via intra-NAcc 6-OHDA injections, which decreases DA and NA levels by 70–75%, had no impact on delay-discounting behavior alone (Winstanley, Theobald, et al., 2005), although this did transiently potentiate the d-amphetamine-induced decrease in impulsive choice of large reward- delayed choices (Winstanley, Theobald, et al., 2005). Dopamine signaling to the OFC appears to be important as lesioning dopamine neurons with 6- OHDA led to an increased indifference point (Kheramin et al., 2004). Further studies showed that levels of the dopamine metabolite DOPAC increased in the OFC when animals were performing the delayed discounting task compared to a yoked control condition (Winstanley, Theobald, Dalley, Cardinal, & Robbins, 2006).

The prefrontal cortex: importance of the OFC

A number of studies have examined different prefrontal structures involvement in delay discounting. The results on the involvement of the OFC were initially mixed, as it was shown that bilateral lesions for the OFC led to a decrease in the number of choices of larger-delayed reward in one study (Mobini et al., 2002), but another study found that it increased the choices of the larger delayed rewards (Catharine A. Winstanley, David E. H. Theobald, Rudolf N. Cardinal, & Trevor W. Robbins, 2004). Subsequent studies went on to discover that the role of the OFC in delay-discounting appears to be dependent both, on whether the delays are cued or not, as well as the baseline levels of impulsivity shown in subjects, which explains the discrepancy in the results between Mobini et al,. (2002) and Winstanley et al., (2004). Inactivation of the OFC was shown to increase impulsive choice of the small reward-small delay lever when the delay was cued, but only in rats low in baseline impulsivity, whereas the same OFC lesion decreases the number of small reward – small delay choices in an un-cued condition, but only in highly impulsive rats (Zeeb, Floresco, & Winstanley, 2010).

Several lines of evidence implicate dopamine signaling within the prefrontal cortex in delay based decision making. Results from a gene expression study found a positive correlation between baseline levels of impulsive choice and the transcript levels of the dopamine D1 and D5 receptor as well as the D1 receptor interacting protein Calcyon (Loos et al., 2010). Moreover, local mPFC infusions of the D1/D5 receptor antagonist SCH 23390 and the D1/D5 partial agonist SKF 38393 increased impulsive choice, which supports the notion that endogenous receptor D1/D5 signaling in the mPFC is involved in making choices about delayed rewards (Loos et al., 2010). As was observed with studies employing lesions in the OFC, whether or not the delayed duration was cued or not also appears to be important for dopaminergic manipulations in the OFC. Specifically, intra-OFC infusions of D2 antagonist (eticlopride) and D1 antagonist (SCH23390) do not alter delayed discounting in a delay-discounting procedure which does not cue the beginning of the delay duration, but both D2 antagonist (eticlopride) and D1 antagonist (SCH23390) decrease choice of large rewards with long delays when the delay duration is cued at the beginning of the delay (Zeeb et al., 2010).

The Basolateral Amygdala

One study looked at the role of the BLA and found that inactivation of the BLA leads to an increased the number of small immediate reward choices (C. A. Winstanley, D. E. H. Theobald, R. N. Cardinal, & T. W. Robbins, 2004).

The Sub Thalamic Nucleus

There have been a small number of studies which have investigated the influence of the STN on delay based decision making. In one study, it was found that lesions to the STN decrease impulsive choice, leading to a higher percent of choices on the large reward lever with a long delay (Winstanley, Baunez, Theobald, & Robbins, 2005), a result which was replicated in another study (Uslaner & Robinson, 2006). In a third study which manipulated both the duration of the large delay as well as the small delay, this same pattern of results was not observed (Bezzina et al., 2009). One key difference between the studies which found that STN lesions lead to more choices of the large reward lever with long delay (Uslaner & Robinson, 2006; Winstanley, Baunez, et al., 2005) and the (Bezzina et al., 2009) was that the former studies produced the lesions after the subjects were already trained to baseline on the task, whereas the later induced the lesions before any training. This is likely important as the effect of STN lesions on delay discounting appear to be most marked immediately after the lesion is made, and animals seems to compensate in some way after more time passes, a pattern of results which fits with observations made by Uslaner et al., (2006).

T-Arm Delay Maze

Another paradigm which has been used to study decision making with delays to reward is the T-Arm maze with delay. In this task, subjects can choose to either go left or right to choose either a high or low reward. In the low reward arm subjects were able to get access to 1 pellet immediately. In the high reward arm subjects were enclosed into a gated waiting area for 15 seconds, after which the gate to the reward area opened and the subject had access to 15 food pellets. In a study using this T-Arm Delay maze version of delay discounting, both the ACC and OFC were lesioned. Subjects which received OFC lesions chose the low reward arm far more often than the sham control group. In the group with lesions to the ACC, delay based decision making was unaffected as subjects didn’t differ from the sham control group (Rudebeck, Walton, Smyth, Bannerman, & Rushworth, 2006). Another study investigated the involvement of the OFC, mPFC and BLA in delay based choice using a T-arm delay procedure. In this study, bilateral inactivation of the OFC did not impact delay based choice, whereas bilateral inactivation of both the mPFC and BLA both lead to a decreased preference for the larger delayed reward (Churchwell, Morris, Heurtelou, & Kesner, 2009). In this study, it was demonstrated that contralateral disconnection of the mPFC – BLA circuit also reduced preference for the larger but delayed reward.

Summary

There appear to be some consistent findings in the literature on the brain regions and pharmacological manipulations which influence delay based decision making across behavioral paradigms (Fig 4A). The brain region which has been most reliably implicated in delay based decision making is the OFC. Both studies using an operant delay discounting procedure and a T-Arm Delay task have found that lesions to the OFC decrease the number of choices of the large reward lever with long delays (Mobini et al., 2002; Rudebeck et al., 2006; Zeeb et al., 2010). In one study inactivation of OFC did not change delay based choice behavior (Churchwell et al., 2009), though both the mPFC and BLA were found to be involved in delay based choice as manipulations of these regions as well as the connection between them can alter delay based choice behavior. While there is a large amount of converging evidence implicating the OFC in delay based choice, a comprehensive understanding of its involvement and the involvement of other regions has yet to be fully worked out.

Figure 4. Brain Regions involved in Delay and Risky Choice.

A. Shows brain regions which have been studied to examine their involvement in delay based choice through lesions, inactivation, or localized drug infusions which have been shown to modulate delay based choice (Blue), have no effect on delay based choice (Grey), or have not yet been examine (White). Areas which influence delay based choice include: the PL, IL, OFC, NAcc Core, BLA, STN, and VTA. Areas which do not influence delay based choice include: the ACC, and NAcc Shell. Areas which have not been studied include: VP, DS, Hipp, SN, vSub.

B. Shows brain regions which have been studied to examine their involvement in risk based choice through lesions, inactivation, or localized drug infusions which have been shown to modulate delay based choice (Red), have no effect on delay based choice (Grey), or have not yet been examine (White). Areas which influence delay based choice include: the PL, NAcc Shell, BLA, and VTA. Areas which do not influence delay based choice include: the ACC, OFC, and NAcc Core. Areas which have not been studied include: IL, VP, DS, Hipp, STN, SN, vSub.

Systemic treatment with drugs acting on the dopamine system produce consistent patterns of results across a number of studies. Drugs which increase synaptic levels of dopamine, like amphetamine, lead to an increase in choices of the large reward lever with long delays, whereas antagonists of the D1 and D2 receptor both lead to increased choices of the low reward lever with no delay. Systemic treatment with drugs altering dopamine signaling are not likely acting within the NAcc to impact delay discounting, however, as NAcc dopamine depletion does not alter delay based decision making. Dopamine does appear to be acting within the OFC, however, as D1 and D2 antagonists decrease the number of choices of the large reward lever with long delays (Loos et al., 2010; Zeeb et al., 2010). Finally, lesions to the STN have been found to lead to an increase in this type of choice (Uslaner & Robinson, 2006; Winstanley, Baunez, et al., 2005).

4C. Choice between two probabilities

The studies of choice between two costs up until this point have all been studies which systematically manipulated the effort requirements of the tasks (e.g. amount of work or length of delay to reward) to see what brain regions and neurotransmitter systems are involved in guiding behavior in the face of different costs. Another type of cost that can influence the choice between two options is the likelihood of each option paying off. This type of task is thought to represent risky decision making as when one chooses an option with a lower probability of paying off, but with the possibility of obtaining a larger reward they are taking a risk. There have been a number of studies examining the role of different brain regions in this risk based decision making behavior (Summarized in Table 4).

Table 4.

Brain Regions Which Modulate Risk Based Choice

Brain Region	Manipulation	Method	Result	Reference
mPFC	Bilateral Inactivation	Muscimol (GABA A) Baclofen (GABA B)	Procedural-dependent effects	St Onge et al., 2010
mPFC	D1 antagonism	SCH-23390 (D1)	↓ Risky choice	St Onge et al., 2011
mPFC	D1 agonism	SKF81297 (D1)	No effect	St Onge et al., 2011
mPFC	D2 antagonism	Eticlopride (D2)	↑ Risky choice	St Onge et al., 2011
mPFC	D2 agonism	Quinpirole (D2)	↓ Risky choice	St Onge et al., 2011
ACC	Bilateral Inactivation	Muscimol (GABA A) Baclofen (GABA B)	No effect	St Onge et al., 2010
mOFC	Bilateral Inactivation	Muscimol (GABA A) Baclofen (GABA B)	↑ Risky choice	Stopper et al., 2014
OFC	Bilateral Inactivation	d-amphetamine () Flupenthixol (D1/D2)	No effect	St Onge et al., 2010
NAcc	Bilateral Inactivation	Muscimol (GABA A) Baclofen (GABA B)	↓ Risky choice	Stopper et al., 2011
NAcc	D1 agonism	SCH-23390 (D1)	↓ Risky choice	Stopper et al., 2013
NAcc	D1 agonism	SKF 81297 (D1)	↑ Risky choice (high prob) ↓ Risky choice (low prob)	Stopper et al., 2013
NAcc	D2 antagonism	Eticlopride (D2)	No effect	Stopper et al., 2013
NAcc	D2 agonism	Quinpirole (D2) Bromocriptine (D2)	No effect	Stopper et al., 2013
NAcc	D3 agonism	PD 128 907 (D3)	↓ Risky Choice	Stopper et al., 2013
NAcc Core	Bilateral Inactivation	Muscimol (GABA A) Baclofen (GABA B)	No effect	Stopper et al., 2011
NAcc Shell	Bilateral Inactivation	Muscimol (GABA A) Baclofen (GABA B)	↓ Risky choice	Stopper et al., 2011
BLA	Bilateral Inactivation	Muscimol (GABA A) Baclofen (GABA B)	↓ Risky choice	Ghods-Sharifi et al., 2009

4C.1 Operant Risk Discounting

A variant of the Effort-Discounting task was developed to study Risk-Discounting. In this paradigm, the response cost remains fixed at 1 response, but the probability that the response will be rewarded is systematically manipulated. These risk-discounting paradigms give subjects a choice between a high-reward (4 pellets) low probability lever or a low reward (2 pellet) high probability lever which pays off 100 percent of the time. The probability of the high-reward low probability lever is varied over the course of the session in ten trial blocks (i.e. 4 pellets delivered with probability of- 100, 50, 25, 12.5%), and can be performed with either decreasing or increasing probabilities as the session progresses. When both levers have equal payoff probabilities, rats will choose the high reward lever most of the time and these high response choices become less likely as a function of the decreasing probability.

Dopaminergic Influence on Risk Based Choice

Numerous studies have found that administration of amphetamine increases the preference for the large risky reward, whereas dopamine antagonists such as Flupenthixol decreased preference for the large risky option (St. Onge, Chiu, & Floresco, 2010). Interestingly, the effects of systemic amphetamine are seen through an increased preference for the large risky options, but this is only observed when the probability decreased over the session, whereas the preference is actually reduced when the probabilities start low and get larger throughout the session (St. Onge et al., 2010). The effects of amphetamine can be blocked or attenuated with either systemic D1R (SCH23390) or D2R (SKF81297) antagonists, and are therefore not mediated by specific receptor type (St Onge & Floresco, 2009). Additionally, blockade of D1 or D2 receptors alone induced risk aversion (St Onge & Floresco, 2009). Other studies have examined the involvement of the Dopamine D3 receptors and have found that the D3 antagonist (PD 128,907) reduced the number of choices on the large/risky lever, whereas the D3 antagonist (nafadotride) potentiated the amphetamine-induced risky choice, but didn’t alter risk-based choice when administered alone (St Onge & Floresco, 2009). Finally, blockade (L745) or stimulation (PD168) of D4 receptors did not alter behavior (St Onge & Floresco, 2009).

Having found that systemic dopamine manipulations impact risk based decision making, subsequent studies then went on to more specifically examine the role of the NAcc and NAcc dopamine on risky behavior. Inactivation of the entire NAcc with a mixture of the GABA_A and _B agonists mucsimol and baclofen, lead to a decreased preference for the high reward - risky option (Colin M. Stopper & Floresco, 2011). Moreover, the sub regions of the NAcc appear to be differentially involved in aspects of risk-based decision making, as inactivation of the NAcc Shell impacted the percent of choices on the high reward – risky lever, but did not have any impact on the latency to make the choice (C. M. Stopper, Khayambashi, Kelly, & Floresco, 2010). Inactivation of the NAcc Core, on the other hand, impacted the latency to respond, but did not affect the percentage of high reward – risky choices.

Studies examining the role of the NAcc in risky choice also looked at the impact of dopaminergic drugs infused directly into the NAcc. NAcc infusions of the D1 receptor antagonist (SCH 23390) found that this NAcc D1 blockade decreased preference for the large/uncertain rewards, which occurred because of an enhanced negative-feedback sensitivity - reflected in the increased tendency to choose the smaller but more certain option immediately after an unsuccessful attempt on the large reward-high risk lever (Colin M. Stopper, Khayambashi, & Floresco, 2013). In contrast, NAcc infusion of the D1 receptor agonist (SKF 81297) had the opposite effect, increasing choice of high risky option when the risky lever probability was high and decreased preference when the risky lever had lower probabilities, suggesting an increased sensitivity to the probability of payoff (Colin M. Stopper et al., 2013). In contrast to the bidirectional effects of the D1 receptors on risk-based decision making, neither D2 antagonists (eticlopride) nor agonists (quinpirole or bromocriptine) in the NAcc influenced risky choice (Colin M. Stopper et al., 2013). Finally, the D3-preffering agonist (PD 128 907) decreased risky choice and subjects were more likely to shift to the low risk lever after a successful high risk outcome (Colin M. Stopper et al., 2013).

In a manner similar to the results observed with Effort-Discounting in the T-arm Barrier maze, dopamine appears to be acting to influence risk based decision making not only within the NAcc but also within the prefrontal cortex. Studies which locally administered the D1 antagonist (SCH23390) into the medial PFC found a decreased preference for the large/risky option, whereas infusion of the D1 agonist (SKF81297) caused a slight, nonsignificant increased in preference for the large risky lever (St. Onge, Abhari, & Floresco, 2011). Dopamine D2 receptor antagonists (eticlopride) infused into the mPFC reduced risk discounting and increased the percent of risky choices, whereas D2 agonist (quinpirole) induced an impairment in risk based decision making, as subjects were less likely to choose the high risk option, showing a flattening of the discounting function overall (St. Onge et al., 2011).

Prefrontal Cortex and Basolateral Amygdala

Within the Pre Frontal Cortex, inactivation of the PL cortex of the mPFC through infusions of the GABA_A and _B agonists (muscimol and baclofen) increased risky choice when the probability on the risky lever was decreased over the course of the session, but this same inactivation lead to decreases in risky choice when the large/risky reward probability increases over a session (St. Onge & Floresco, 2009). Control experiments demonstrated that the results following PL cortex inactivation could not be explained by a more general disruptions in flexible behavior (reversal learning) or judgments about the relative value of probabilistic rewards (St. Onge & Floresco, 2009).

In contrast to the results of inactivation of the PL cortex, inactivation of the OFC through infusions of muscimol and baclofen increased response latencies, but did not have any effect on risky choice (St. Onge & Floresco, 2009). Further studies demonstrated that inactivation of the medial OFC increased risky choice on risk-discounting task in either ascending or descending probability conditions (Colin M. Stopper, Green, & Floresco, 2014). This increased risky choice was associated with enhancement in win-stay behavior as rats showed a tendency to choose the risky option again following a rewarded risky trial. In contrast to the PL cortex and OFC, inactivation of the ACC via infusions of muscimo/baclofen did not affect risky choice or latency to respond (St. Onge & Floresco, 2009).

Finally, infusions of the GABA_A and _B agonists (muscimol and/baclofen) into the BLA disrupted risk discounting, inducing a risk averse pattern of choice, as there were observed increases in response latencies as well as trial omissions, with these effects appearing most prominently in cases with the greatest amounts of uncertainty (Ghods-Sharifi et al., 2009). A set of studies was performed which assessed the role of the functional connection between the BLA and mPFC, as both of these areas were shown to alter risk based choice when bilaterally inactivated. It was shown that functional disconnection of the BLA- mPFC connection altered risk based choice behavior such that subjects chose the risky option more often, which resulted from decreased sensitivity to negative feedback following an unsuccessful risky choice (St Onge, Stopper, Zahm, & Floresco, 2012). Moreover, it is specifically the top-down pathway by which the mPFC – BLA connection is involved in risky choice, as specific disconnection of the mPFC to BLA circuit altered risk based choice, whereas inactivation of the BLA to mPFC connection did not (St Onge et al., 2012).

Summary

There have been a number of consistent findings from studies which have looked at choice behavior involving different probability of payoff (Fig 4B). Drugs which increase synaptic levels of dopamine such as amphetamine increase the number of choices of the risky large reward, dopamine antagonists decrease these choices leading to more selections of smaller more certain rewards (St Onge & Floresco, 2009). Inactivation of the NAcc Shell and the BLA both decrease the number of choices of the high reward lever /lower probabilities (Ghods-Sharifi et al., 2009; Colin M. Stopper & Floresco, 2011). The effects of inactivation of the PL cortex of the mPFC appear to be dependent on whether the probabilities increase or decrease over the course of the session (St Onge & Floresco, 2010b). Inactivation of this region decreases risky choice when the probabilities decrease over the course of the session, but increase risky choice when the probabilities increase over the course of the session. Finally, inactivation of the NAcc Core, the OFC, and the ACC do not specifically impact risk based choice (St Onge & Floresco, 2010b; Colin M. Stopper & Floresco, 2011).

5. Summary and Future Directions

We have summarized a wide range of studies which address the question: how does the brain process and use information related to different types of response costs underlying motivated behavior? We focused on cost manipulations of effort, time delays, and risk/probability and found that a number of brain regions seem to appear to be important in many different types of tasks, whereas others appear to be highly specific to some tasks but not others.

5A. Dopamine influences response vigor, as well as cost, delay, and risk based decision making

Systemic dopamine treatments influenced performance in all of the different types of tasks covered in the review. Systemic treatment with drugs which increased synaptic dopamine levels, like amphetamine, leads to increased activation in a PR schedule (Bailey et al., 2015; Mayorga et al., 2000; Sommer et al., 2014), increases in high effort/high reward choice (Bardgett et al., 2009; Floresco et al., 2008), increases in long delay/large reward choice (Barbelivien et al. 2008; de Wit et al. 2002; Floresco et al. 2008; Monterosso et al. 2007; van Gaalen et al. 2006; Wade et al. 2000), and increases in risky choice for large rewards (St. Onge, Chiu, & Floresco, 2010). Systemic treatment with dopamine antagonists drugs decreased these behaviors in a PR (Aberman & Salamone, 1999; Caul & Brindle, 2001; Cheeta et al., 1995; Olarte-Sanchez et al., 2013), effort choice tasks (Cousins & Salamone, 1994; Farrar et al., 2010; Koch, Schmid, & Schnitzler, 2000; Nowend, Arizzi, Carlson, & Salamone, 2001; Salamone, 1991; Salamone et al., 2002; Sink et al., 2008; Worden et al., 2009), delay choice tasks (Floresco et al. 2008; van Gaalen et al. 2006; Wade et al. 2000), and risk choice tasks (St. Onge, Chiu, & Floresco, 2010). The site of action of these dopaminergic drugs differs for different behavioral processes. The NAcc Core is the important site of action of dopaminergic drugs for modulating behavior in the PR task (). Both the NAcc Core and Shell are sites of action of dopamine antagonists in two different effort based choice tasks (EBCT and T-Arm barrier maze) (Sokolowski et al., 1998), but only the NAcc Core was important in an operant effort discounting task (Ghods-Sharifi and Floresco 2010). In tasks which require subjects to make choices about delays to reward, both the OFC and the mPFC (PL and IL) are sites where dopamine acts, whereas NAcc dopamine has no impact on delay based decision making (Winstanley et al., 2005). For risk based choice, both the IL and PL cortex as well as the NAcc appear to be sites of action of dopamine (St Onge et al., 2011), but only the NAcc shell appears to be modulating risk based decision making (Stopper et al., 2011).

5B. Neural Substrates involved in effort, delay, and risk costs

There appear to be a number of different brain regions which are involved in multiple types of response cost related behaviors, but there are also some regions in which there appears to be selectivity in the types of costs they are required for making decisions about (Fig 5).

Figure 5. Summary Brain Regions and Types of Costs.

Shows an overall summary of the brain regions which have been shown to modulate different aspects of motivated behavior and overcoming response costs through lesion studies, chemical inactivation, and pharmacological manipulations. (A) Shows brain regions which have been shown to be modulate vigor in a PR (orange) have no effect on PR (grey), and have not been studied (white). (B) Shows brain regions which have been shown to be involved in effort based choice behavior (green), have no effect on effort based choice (grey), and have not be studied (white). (C) Shows brain regions which have been shown to be involved in delay based choice behavior (blue), have no effect on delay based choice (grey), and have not been studied (white). (D) Shows brain regions which have been shown to be involved in risk based choice behavior (red), have no effect on risk based choice (grey), and have not been studied (white). (E) Provides an overall summary of the different types of behavioral tasks each of the different brain regions has been shown to be involved in.

ACC

The ACC is a region which appears to be specifically involved in decisions about effort based choice. Lesions to the ACC did not impact PR responding (Schweimer and Hauber 2005), which suggests the region by itself cannot influence activational aspects of motivation. This is supported by a study which found that lesioning the ACC did not alter locomotor activity in an open field (Li et al., 2012). In decision making situations involving manipulations of delay to reward and risk/probability, lesions to the ACC also do not seem to have any effect (Rudebeck et al., 2006; St Onge et al., 2010). On the other hand, the ACC is involved in effort based decisions as lesions to the ACC lead to more choices of low effort options (Walton et al., 2009; Walton et al., 2003). While the majority of the studies done on the ACC’s involvement in effort based choice utilized the T-arm barrier maze (which all found the region is required for normal decision making), one study using the EBCT found that lesions to the ACC did not have any effect (Schweimer and Hauber 2005), whereas a study employing a variant of the operant effort discounting (FR-14 for 4 pellets vs FR-4 for 2 pellets), found that lesions to the ACC lead to a decreased selection of the high effort lever (Walton, Groves et al. 2009). Together, these results suggest that the ACC plays a specific role in the computation of whether the value of an outcome offsets the effort needed to obtain it.

OFC

Lesions to the OFC were shown to increase subject’s BP in a PR task (Gourley et al., 2010), which suggests that the region may be modulating vigor via activational influences of motivation. In line with this idea is the observation that lesions to the OFC in rats leads to increased locomotor activity in an open field test (De Bruin et al., 1983). Worth noting is the fact that in a PR schedule two things are systemically increasing together: the number of responses required for the next reward and the time the required bout of work will require to obtaining that reward. Thus, both the effort required and delay to reward are simultaneously manipulated in this task. While there have been many studies done which have found the OFC is involved in delay based choice behavior there has only been one study done to our knowledge which has looked at effort based choice (Rudebeck et al., 2006).

Of the numerous studies which have been done examine the influence of the OFC on delay based choice, a consensus is that lesioning or inactivation of the OFC leads to altered delay based choices such that subjects prefer smaller more immediate rewards and are averse to delays (Mobini et al. 2002; Rudebeck et al., 2006; Zeeb et al., 2010). Whereas many of the early studies did not specifically distinguish between sub regions within the OFC, recently the important functional distinctions between the medial OFC (mOFC) and lateral OFC (lOFC) has been recognized. Specifically, lesions or inactivation of the lateral OFC induces an impairment in delay based choice (Rudebeck et al., 2006; Catharine A. Winstanley et al., 2004), whereas inactivation restricted to the mOFC does not (Colin M. Stopper et al., 2014).

Finally, while initial studies seemed to suggest that the OFC is not involved in decision making about risk or probabilities (St Onge et al., 2010), these studies did not specifically disentangle the contribution of the medial and lateral OFC. Specific inactivation of the medial OFC leads to alterations in risk based choice (Colin M. Stopper et al., 2014), whereas inactivation of the lateral OFC does not (St Onge & Floresco, 2010a). The apparent lack of an effect of the OFC in the effort based choice behaviors, along with the regional specification of involvement in delay and risk based choice seems to suggests that the sub regions of the OFC may play a specific role in processing certain types of information going into the computation of whether the benefit of a specific outcome is worth the cost of obtaining that outcome. Future studies will be needed to further understand the regional distinctions of this structure as well as whether differential input and output targets can be identified based on their connectivity within the sub regions of the OFC.

The IL and PL cortex

Many of the studies which have made lesions to the mPFC have targeted the IL and PL cortex. The PL cortex appears to be involved in activational influences of motivation as lesions to this region decrease BP’s in a PR (Gourley et al., 2010), though in the one study to examine this it is likely that the IL was also damaged based on the reported histology. While damage to the PL can enhance response vigor in a PR, neither the IL nor PL appear to be involved in effort based choice (Walton et al., 2002). The PL and IL both appear to be involved in delay based choice, as local dopamine antagonist infusions into this region can lead to decrease selection of larger rewards with longer delays (Loos et al., 2010). Finally, the PL cortex appears to be involved in risk based decision making (St Onge et al., 2010). Thus these regions of mPFC seem to play a role in assessing the costs of both delays and risks but not effort. It is possible that an assessment of delay mediates all these results as the average delay to risky outcomes is greater than the delay of less risky outcomes in the studies reviewed here.

NAcc Core

The NAcc Core appears to be involved in activational aspects of motivation. Cell body lesions to this region enhance BP’s in a PR, whereas dopamine depletion and dopamine antagonist drugs lead to reductions in BP’s (Hamill et al., 1999; Bezzina et al., 2008; Bari et al., 2005). The NAcc Core is also involved in effort based choice, as dopamine depletion and dopamine antagonist drugs in this region leads to reduction of choosing high effort options in all three of the discussed measures of effort choice: EBCT, operant effort discounting, and the T-arm barrier maze (Salamone et al., 1991; Ghods-Sharifi and Floresco 2010; Hauber, Sommer, 2009). The NAcc Core is also involved in delay based choice as lesions of the NAcc Core lead to increased selection of the smaller more immediate reward (Cardinal, Pennicott, Sugathapala, Robbins, & Everitt, 2001; Cardinal, Robbins, et al., 2003; Cardinal, Parkinson, et al., 2003). This is different from the effects of dopamine depletion to the region, however, as this has been shown to not have any effect on delay based choice (Winstanley, Theobald, Dalley, & Robbins, 2005), and intra NAcc Core D1 and D2 receptor antagonists do not impair the ability to wait for reward in a cued progressive delay procedure (Wakabayashi, Fields, & Nicola, 2004).

Excitotoxic lesions of the NAcc lead to a risk averse pattern of responding in risk based choice task (Cardinal & Howes, 2005), but this does not seem to be specific to the NAcc Core because inactivation of the NAcc Core alone doesn’t alter risk based choice (Stopper et al., 2011). Thus the NAcc Core appears to process information about the effort and risk costs of an alternative but not be involved in processing delay costs.

NAcc Shell

The NAcc Shell is distinct from the NAcc Core in a number of ways. First, whereas cell body lesions within the NAcc Shell can increase responding in a PR (), dopaminergic depletion within the shell and D1 and D2R antagonists within the shell do not increase response vigor in this task (Bari et al., 2005). Additionally, while all 3 of the effort based choice tasks found the NAcc Core to be important for effort based choice, only the EBCT found that DA depletion and D1 and D2 antagonists could alter this behavior (Sokolowski et al., 1998), although to a lesser extent that within the Core. A study which examined NAcc DA involvement in delay based choice found that intra-accumbal infusions of 6-OHDA did not have any impact on delay based choice (Winstanley et al., 2005), but did impact risk based choice as inactivation of this region decreased risky choice behavior (Stopper et al., 2011). In sum, it appears that the NAcc Shell may play a very similar computational role to that played by the core, but it is not directly responsible for activational effects of dopamine as measured in a PR and is involved in risky choice.

VP

To our knowledge, the effect of lesioning the VP has not been examined in PR tasks, delay choice tasks, or risk based choice tasks. There is however, evidence that the VP is involved in effort based choice tasks as inactivation of this region lead to a decrease in willingness to choose high effort options in the EBCT (Farrar et., al. 2008). Given the direct connection with the NAcc Core, it seems like the VP may likely be involved in other behaviors which the NAcc Core is involved in and future studies may benefit from examining this region more closely.

STN

The STN also appears to be involved in activational aspects of motivation as lesions to this region increase BP’s in a PR (Baunez et al., 2002; Bezzina et al., 2008), and it also appears to be involved in delay based choice as lesions to the STN alter delay based choice behavior as subjects choose more of the large rewards with long delays (Winstanley et al. 2005; Uslaner et al., 2006). We are unaware of studies which have examined the impact of lesioning the STN in either effort based or risk based choice. Interestingly, the pattern of results observed with STN lesions are highly similar to those of OFC lesions (increased BP, increased delay based choice).

BLA

The BLA is another brain region which appears to be involved in many aspects of behavior covered in the review, as it influences behavior effort based choice (Floresco et al., 2007; Ghods-Sharifi et al., 2009), delay based choice (Winstanley et al. 2004), and risk based choice (Ghods-Sharifi et al., 2009). While we could not a find a specific study which used a PR, one which used a FR-16 found that inactivation of the BLA decreased responding (Simmons & Neill, 2009). Thus we hypothesize that the BLA may be processing information about the net costs and benefits of different behavioral options.

VTA

As discussed in the earlier section on the effects of dopamine in the various behaviors, the VTA appears to be involved in all of the behaviors discussed: PR, effort choice, delay choice, and risk choice. This is due to the fact that the VTA sends dopaminergic neurons to other brain regions for which dopamine is important for modulating these behaviors: including NAcc, OFC, BLA, IL, PL, and the ACC. D1 antagonists injected directly into the VTA leads to a decrease in BP, whereas over expression of D2 receptors in this region leads to increases in BP (Sharf et al., 2005). That VTA is involved in activational aspects of motivation is well known, and studies have shown that the area can modulate general locomotor activity, as injections of the GABA_A antagonist picrotoxin into the VTA increase locomotor activity in an open field (Mogenson and Manchanda, 1979). This set of data reflect the key role that dopamine plays in modulating the widely distributed network involved in these cost-benefit computations.

Hippocampus

While lesions to the ventral hippocampus were shown increase BP’s in a PR task (Gourley et al., 2010; Chambers et al., 2002), we are unaware of any studies which have specifically looked at the role of the ventral hippocampus in effort, delay, or risk based choice procedures. Given the direct connection between the OFC and the ventral hippocampus, it may be interesting to see if delay based choice requires ventral hippocampal functioning.

Conclusion/Future directions

The studies covered in this review suggest that there is a widely distributed network engaged during motivated action. Some of that network seems to energize behavior while other structures in the network are involved in more specific computations about different kinds of costs. It also seems likely that this network is involved in computing different kinds of benefits as well. When examining the summary of studies which have been done in Figure 5E, it becomes apparent that there are a number of different brain areas which have yet to be studied for certain types of processes. It may be helpful in developing a more complete picture of this distributed circuit to more fully understand what each region does with relation to each other. We believe that to understand motivation the field must continue to dissecting this network and map it to specific behavioral functions. The arsenal of new techniques which exist for cell type specific manipulation of circuits with high levels of temporal control should aid this continued quest to better understand the neural circuits guiding and directing behavior to overcome obstacles in the environment.

Abbreviations

6-OHDA

6-hydroxydopamine

A2A

Adenosine 2A receptor

BP

Breakpoint

DA

Dopamine

D2R

Dopamine D2 receptor

D2R-OE

Dopamine D2 receptor over-expression

DAT

Dopamine transporter

DAT KD

Dopamine transporter knockdown

DREADD

Designer receptors exclusively activated by designer drugs

EBCT

Effort-Based Choice Task

FR

Fixed ratio

HR lever

High-Effort/High-Reward lever

LR lever

Low-Effort/Low-Reward lever

PHD

Progressive hold down

PR

Progressive Ratio

VI

Variable Interval

Brain Region Abbreviations

ACC

Anterior Cingulate Cortex

BLA

Baso-Lateral Amygdala

DLS

Dorsolateral Striatum

DMS

Dorsomedial

IL

Infra-Limbic Cortex

lOFC

Lateral orbitofrontal cortex

MD

Mediodorsal Thalamus

mOFC

Medial orbitofrontal cortex

mPFC

Medial Prefrontal Cortex

NAcc

Nucleus Accumbens

OFC

Orbital Frontal Cortex

PFC

Prefrontal Cortex

PL

Pre-limbic cortex

STN

Sub thalamic nucleus

SNc

Substantia Niagra pars Compact

SNr

Substantia Niagra pars reticulata

VP

Ventral Pallidum

VTA

Ventral Tegmental Area