Quantity versus quality: Convergent findings in effort-based choice tasks

Abstract

Organisms must frequently make cost-benefit decisions based on time, risk, and effort in choosing rewards to pursue. Various tasks have been developed to assess effort-based choice in rats, and experimenters have found largely similar results across tasks and brain regions. In this review, we focus primarily on the convergence of different effort-based choice tasks where quality or quantity of reward are manipulated. In the former, the rat is typically presented with the option to work for a preferred reward or select a less preferred, but freely-available reward. In such paradigms, the rewards are of different identities but are confirmed to differ qualitatively in value by a food preference task when both are freely-available. In the latter task type, rats are required to select between higher magnitude versus lower magnitudes of the same reward, but each with a similar effort requirement. We discuss the strengths/limitations of these paradigms, and describe brain regions that have been probed that result in converging or equivocal findings. Results are also reviewed with reference to a need for future work, and the broader impacts and implications of studies probing the mechanisms of effort.

Introduction

How are effort decisions experienced in daily life? It is a situation we have all encountered: it has been a long day in the lab, perhaps you are trying to get that last bit of data for the conference next week, or you have an impending grant deadline. All that brain power requires energy, so what is the next thought on the journey home? That it is time to eat and there are many options to choose from. On the one hand, you can choose to cook your favorite meal, but there is an associated cost of going to the grocery store, gathering all the ingredients, making the trek home, and finally preparing the meal. Of course, it is always possible to simply purchase fast food, but that option is not nearly as tasty as your favorite meal, albeit less effort to procure. Herein lies the problem: how do we choose between these options with varying effort requirements, and what are the underlying neural mechanisms? There are several rodent models designed to address this question. This review will focus on convergent findings in the mechanisms of effort-based choice as assessed by several different behavioral paradigms and a set of behavioral economic analyses. Although a large amount of work has been published in this field, we will focus on summarizing the similarities across different tasks measuring the same construct of effort.

Quality effort-based choice

Much like the hungry scientist, rats can be presented with a choice between working for a preferred option versus selecting a low effort (freely-available), but less preferred option. This finding was first published in 1991, when Salamone and colleagues (Salamone et al. 1991) reported that either haloperidol and 6-hydroxydopamine lesions decreased effort output on a fixed-ratio 5 schedule of reinforcement for the preferred Bioserv pellets and instead biased behavior toward the lower effort lab chow. Since then, similar qualitative effort-based choice procedures have been adopted, primarily by varying the schedule of reinforcement. Others have provided the preferred food type on progressive ratio (PR) schedules (Randall et al. 2012; Randall et al. 2014; Randall et al. 2015; Yohn et al. 2016; Hart and Izquierdo 2017; Hart et al. 2017; Thompson et al. 2017; Hart, Gerson, and Izquierdo 2018; Munster and Hauber 2017) or random ratio schedules with higher work requirements than FR5 (Trifilieff et al. 2013; Bailey et al. 2018).

In a typical protocol administered by our lab, rats are first given fixed-ratio (FR)-1 training where each lever press earns a single sucrose pellet. Rats are kept on this schedule until they earn at least 30 pellets within 30 minutes. Following this, rats are shifted to a PR schedule where the required number of presses for each pellet increases according to the formula:

$$n_i=5e^{\left(i/5\right)}-5$$

where is n_i equal to the number of presses required on the i^th ratio, rounded to the nearest whole number (Richardson and Roberts 1996), after 5 successive schedule completions. Rats are then tested on the PR schedule until they earn at least 30 pellets on any given day (~5 d). Upon meeting this criterion, a ceramic ramekin containing 18 g of lab chow is introduced (modified from (Randall et al. 2012)) during testing. Rats are then free to select between consuming freely-available but less preferred chow or lever pressing for preferred sucrose pellets. This phase is what we can refer to as quality effort-based choice.

Quantity effort-based choice

In a similar way that palatable high carbohydrate pellets are preferable to lab chow, more of a food option is preferable to less of the same food option to hungry humans and other animals. Subjects must exert more effort to get to more food in this case. The first such procedure made use of a T-maze apparatus where rats could choose between two arms of a maze (Figure 1). Selection of one arm was reinforced by two highly palatable pellets that rats could simply approach without difficulty. Behavior toward the other arm was reinforced by four pellets, where rats had to climb a tall barrier (Salamone, Cousins, and Bucher 1994). There are variations to this maze task. For example, in an earlier study from our lab we assessed such effortful choices between a high- and low-magnitude reward on a T-maze (Experiment 1, (Ostrander et al. 2011)) and effortful choices when a cue signaled changes in that reward magnitude (Experiment 2), requiring more flexibility than the average effort-choice paradigm. Unlike the quality effort-based choice tasks, in such quantity effort-based choice T-maze tasks rats are required to first learn about the reward values associated with each arm of the maze (discrimination training with free sampling phase). In this learning phase, one goal arm is baited with a high magnitude reward (HR) such as 2 cereal loops, and the other with a low magnitude reward (LR), such as a ½ cereal loop. The rat is allowed to sample freely from both arms at the beginning of testing. In such paradigms, HR and LR arm designations are counterbalanced among rats but remain constant for each rat for the duration of testing. Also for these T-maze paradigms, there are typically forced-choice trials, where HR and LR arms are blocked, ‘forcing’ the animal to experience the reward contingencies/values, which serve as reminders throughout testing. These forced-choice trials typically occur on no more than 2 trials within a session, and during all other trials, the rat is allowed to freely select either the HR or LR arm.

Figure 1. Quantity effort-based T-maze and sample data.

(Left) An example of an effortful choice T-maze apparatus containing a start arm and two goal arms, each baited with either a high reward, HR (2 cereal loops) or low reward, LR (½ cereal loop). Barriers are placed in the maze that rats must climb to obtain the HR. (Right) Mean ± SEM percent HR choices per lesion group as a function of increasing barrier heights, 15, 20, 25, and 30 cm. Lower scores indicate greater work aversion. Basolateral amygdala (BLA)-lesioned rats displayed significantly fewer HR choices compard to orbitofrontal cortex (OFC)-lesioned or SHAM-operated rats. Adapted from Ostrander et al. 2011.

Notably, T-maze paradigms often feature far fewer trials of choice behavior than other automated operant procedures, and thereby limit the application of different theoretical models which require richer datasets. Other automated quantity effort-based choice tasks have since been developed. A frequently used paradigm involves rats selecting between two different levers, each of which earns either a small magnitude (often 1 or 2) pellets or a large magnitude (4) of the same food type (Floresco, Tse, and Ghods-Sharifi 2008). While the low effort/low reward lever requires only a single press, the high effort/high reward lever requires either 5, 10, or 20 presses, presented often in discrete trial blocks, in ascending order.

To our knowledge, no study has been conducted examining the experience of effort-based choice across tasks (i.e. order effects: quality to quantity, or vice-versa), and to other reward-based behaviors (e.g. transfer: effort tasks to reinforcer devaluation tasks, or effort tasks to reversal learning tasks). For example, if a rat chooses the high effort option in one task, does this attribute select for certain characteristic that enhance or detract from performance on a subsequent paradigm? How does learning about value in a cost-benefit paradigm bias performance on reinforcement learning, in general? These questions have not yet been fully addressed.

Convergent findings

Ventral Striatum

Much of the motivating force behind developing effort-based choice tasks was to further understand the behavioral functions of dopamine in the ventral striatum (VS). While early theories posited that dopamine was the “reward molecule” (Wise et al. 1978; Wise, Spindler, and Legault 1978), this idea fell out of favor in light of the many other functions of dopamine, including aversively motivated behaviors (McCullough, Sokolowski, and Salamone 1993), and reward prediction error (RPE) signals that drive learning (Niv, Duff, and Dayan 2005; Gershman 2017). Salamone and colleagues found that ventral striatal dopamine D2 receptor blockade or dopamine depletion made rats less willing to exert high levels of effort for the preferred food in the first published example of a quality effort-based choice task. Additionally, rats’ consumption of the freely available less preferred alternative increased. This work challenged the idea that ventral striatal dopamine directly signaled the reinforcing properties of appetitive stimuli: rats were simply less willing to work for reward, but their behavior was still directed toward the acquisition and consumption of food. The early findings in the VS suggested interesting possibilities for nuance in effort-related behavior, and make it the ideal region to begin detailing where there have been convergent findings in different tasks.

As mentioned above, both systemic dopamine receptor blockade and ventral striatal dopamine depletion decreased choice of the high effort option (lever pressing on a fixed ratio 5 schedule for highly palatable pellets) and instead increased consumption of the freely available lab chow (Salamone et al. 1991). Ventral striatal effects have been found in numerous other effort tasks since then. Indeed, the first quality effort-based choice task was quickly followed by the first quantity effort-based choice task using a T-maze (Salamone, Cousins, and Bucher 1994). Similar to the fixed ratio 5/chow choice procedure, both haloperidol and ventral striatal dopamine depletion decreased choice of the high effort option; in this case climbing a 44 cm barrier to acquire twice as many pellets as the low effort option.

These original findings have been validated several times in both quality and quantity effort-based choice tasks. In the FR5/chow task, both D1 and D2 receptor blockade in nucleus accumbens core and shell decreased high effort lever pressing and instead biased animals toward low effort chow consumption (Nowend et al. 2001). These effects seem to be mediated primarily by the accumbens core specifically, as inactivations with smaller injection volumes of core, but not shell, produce similar decreases of choice of the high effort option in a quantity effort-based choice task where rats selected between high effort/high reward and low effort/low reward levers (Ghods-Sharifi and Floresco 2010), and dopamine depletions of accumbens shell were not sufficient to suppress reinforced lever pressing (Sokolowski and Salamone 1998). Interestingly, in the lever quantity effort-based choice task pioneered by Stan Floresco and colleagues (Floresco, Tse, and Ghods-Sharifi 2008), where both options can be equated in their temporal delay to reward, which is experimentally problematic in quality-based tasks, accumbens core inactivations have the same effect when the time delays are not equivalent. Thus, the effects of accumbens core manipulations are not due to delay intolerance and instead reflect a robust and consistent role of the VS, particularly accumbens core, in supporting effort-based choice across tasks.

Anterior cingulate cortex

The idea that VS is not the only part of the brain involved in cost-benefit analyses involving effort took approximately 10 years to gain popularity following publication of the first effort-based choice tasks. Much of this research focused on the anterior cingulate cortex (ACC) based on its anatomical connectivity with the striatum (Brog et al. 1993; Berendse, Galis-de Graaf, and Groenewegen 1992) and its role in motivated behavior generally (Paus 2001). The first work to probe ACC in effort adopted the aforementioned T-maze quantity effort-based choice task in rats with extensive lesions encompassing ACC, prelimbic, and infralimbic cortex (Walton, Bannerman, and Rushworth 2002), which was soon after followed with smaller lesions targeting ACC (Walton et al. 2003). In both of these cases, lesions decreased choice of the high effort/high reward option. Barrier climbing ability and reward magnitude discrimination were unaffected under conditions where both arms were impeded by a barrier (i.e. equal work).

These first two quantity-based findings were soon replicated and expanded upon by exposing ACC lesioned rats to a qualitative choice task where rats chose between lever pressing for sucrose pellets on a progressive ratio and consuming freely available lab chow (Schweimer and Hauber 2005). Interestingly, although the ACC quantitative choice task findings replicated those assessed via a T-maze task, Schweimer & Hauber (2005) found no group differences when the same animals were subjected to the progressive ratio qualitative choice task. This was pursued several years later, when ACC lesioned rats were tested in a similar progressive ratio qualitative choice task with a slightly modified progressive ratio schedule (Hart et al. 2017). Notably, our lab in this case did find group differences in the progressive ratio qualitative choice task, replicating the earlier T-maze findings. Thus, the potential discrepancy in these findings could be due to differences in the schedule of reinforcement, or perhaps prior experience in the T-maze attenuated any group differences that may have been subsequently observed on the progressive ratio choice task. Nevertheless, there is now a consistent line of evidence implicating ACC in choosing a higher effort option over a less effortful alternative, whether the high effort option is better in terms of quantity or quality.

Basolateral amygdala

Regions of the amygdala, particularly the basolateral complex (BLA) project to both the striatum (McDonald 1991) and frontal cortex (McDonald 1991), and it was not long until an experimental line of evidence probing BLA in effort-based choice was initiated. The first of which made use of a T-maze quantity-based choice task. Floresco and colleagues found that BLA inactivations, like ACC lesions and VS manipulations, decrease choice of the high effort option without affecting motoric ability or reward discrimination (Floresco and Ghods-Sharifi 2007). This effect was later replicated in a similar T-maze quantitative choice task with BLA lesions (Ostrander et al. 2011). Replication of the T-maze findings in a lever-based quantitative choice task confirmed that effects of BLA manipulations on effort-based choice were not due to delay discounting (Ghods-Sharifi, St Onge, and Floresco 2009). In the first test of BLA effects on the progressive ratio qualitative choice task (Hart and Izquierdo 2017), pharmacological inactivations via baclofen-muscimol produced similar effects to those seen in quantitative tasks, decreasing choice of the high effort option without affecting motoric ability or appetite.

Although not explicitly examining the role of each of the aforementioned regions alone, disconnection studies provide considerable insight into the mechanisms of effort-based choice. By unilaterally inactivating or lesioning one structure and doing the same to another structure in the contralateral hemisphere, one can disrupt serial communication between two structures, thus determining the role of this communication in effort-based choice. Consistent with the idea that VS, ACC, and BLA act as part of a circuit that regulates effort-related functions (Salamone et al. 2007), both disconnection of ACC and VS (Hauber and Sommer 2009) as well as BLA and ACC (Floresco and Ghods-Sharifi 2007) have similar effects as interfering with any of these regions alone. With modern neuroscience tools including optogenetics and designer receptors exclusively activated by designer drugs, more precise circuit dissection can be achieved. For example, dopamine neurons can be targeted based on genetic characteristics to determine their role in PR responding (Fischbach-Weiss, Reese, and Janak 2018), and specific terminals from one region can be silenced in an efferent target (Lichtenberg et al. 2017).

Other regions

For the sake of brevity, here we will group together other brain regions where the current literature regarding their role in effort-related decision-making is mixed, starting with dorsomedial striatum (DMS). DMS has a longstanding role in controlling movement vigor, often defined as velocity or latency (Wang, Miura, and Uchida 2013; Opris, Lebedev, and Nelson 2011; Bartholomew et al. 2016; Panigrahi et al. 2015). How exactly vigor relates to effort choice, however, is unclear. While vigor is certainly a component of effortful behavior, work output is also characterized by persistence and duration of activity (Salamone and Correa 2012). While DMS is unequivocally involved in vigor, its role in effort-related decision-making is less clear. For example, DMS dopamine depletion and related manipulations do not have any effects on effort-based choice as assessed by the FR5 qualitative choice task (Nunes et al. 2013; Font et al. 2008), and no effects of methamphetamine withdrawal on c-Fos immunoreactivity were found in DMS following testing in the PR effort-based choice qualitative task (Hart, Gerson, and Izquierdo 2018). Nevertheless, others have found that DMS dopamine receptor blockade does reduce random-ratio 15 reinforced lever pressing in a qualitative choice task (Bailey et al. 2018). Given that DMS dopamine depletion can alter inter-response times, a measure of vigor, without affecting total lever presses (Salamone et al. 1993), it is plausible that DMS is involved in the total amount of effort subjects will exert as well as the vigor with which effort is exerted, and the extent to which DMS is involved in these depends on the schedule of reinforcement and other task constraints.

The orbitofrontal cortex (OFC), while clearly involved in many behaviors related to reward-learning and decision-making (Izquierdo 2017), has a less clear role in effort. Both OFC D1 receptor blockade (Cetin et al. 2004) and optogenetic inhibition (Gardner et al. 2018) decreased PR responding. Others found that pharmacologic inhibition increased PR responding, while pharmacologic excitation decreased PR responding (Munster and Hauber 2017). Ostrander et al. (2011) reported no such quantity effort-based choice impairment (Figure 1), only learning impairments when animals were required to use cues to make choices. In short, it is unknown whether these seemingly divergent findings are due to different interference methods, stereotaxic targeting, schedules of reinforcement, or whether/how cues are present (Ostrander et al. 2011), so future experiments should clarify the role of OFC in effort as well as in effort-based choice.

While the results of subthalamic nucleus (STN) manipulations are not mixed, they are certainly within the purview of this review. The STN receives most input from the ventral pallidum (Canteras et al. 1990), a major output structure of the striatum. While all of the aforementioned lesion and inactivation studies showed that ACC, VS, and BLA manipulations decrease effort for a preferred reward, STN lesions increase effort (PR responding), at least for food (Baunez, Amalric, and Robbins 2002; Baunez et al. 2005). Interestingly, the same STN lesions that increase PR responding for food decrease PR responding for cocaine (Baunez et al. 2005). Why is it that STN exerts opposite control over natural versus drug reinforcers, does its role extend to effort choice behavior, and is this the case for other parts of the brain? Future experiments should shed light on these questions.

Comparing effect sizes: Quantity stronger than Quality

A direct comparison of identical manipulations (e.g. haloperidol and nucleus accumbens dopamine depletion) on effort-based choice involving quantity/T-maze climbing vs. quality/lever pressing would be very informative (Salamone et al. 1991; 1994). Yet there are several limitations in directly comparing quality and quantity procedures: i. To compare effect sizes one needs means and standard deviations of the different groups, and these values are often not reported in many of the older studies and presented only in the figures as means with respective standard error bars, ii. there are strain differences that may factor significantly into these comparisons, iii. there are differences in the ecological validity of the paradigms. For example, manual T-maze tasks are easier for rodents to learn than automated lever responding tasks, perhaps mimicking more naturalistic foraging behavior observed in the wild. Relatedly, it may be more important for an animal’s survival to distinguish more vs. less (i.e. relative quantities) than more nuanced qualitative differences. In support of this, it has also proposed that the ability to determine magnitude or represent numerosity is a subcortical process, highly evolutionarily conserved (Collins, Park, and Behrmann 2017), and iv. there may be differences in the neural interference methods (e.g. lesion, pharmacological inactivation) that could account for choice behavior.

To overcome some of these issues, we compared the results from two reports from our lab: Ostrander et al. (2011) and Hart & Izquierdo (2017). Both of these studies involved targeting of the BLA using identical stereotaxic coordinates in male Long-Evans rats obtained from the same vendor (Charles River Laboratories, Hollister CA). We reanalyzed high reward (HR) choice probabilities when the HR arm was impeded by the tallest (30 cm) barrier in sham-operated (SHAM) rats versus those with BLA lesions via ibotenic acid. We then compared this effect size with that obtained when rats selected between lever pressing for a sucrose pellet reward or consuming freely-available lab chow, following BLA baclofen/muscimol inactivation (or vehicle). When we computed Cohen’s d for each of these studies, we found that BLA lesions resulted in a much larger effect size in quantity/T-maze climbing effort-based choice (d=2.87) compared to BLA inactivation in quality-lever press responding effort-based choice (d=0.95). It is tempting to attribute this difference in effect size solely to the manipulation strength (lesion vs. inactivation), yet it is known that recovery of function often occurs after “permanent” lesions and chronic microinfusions can similarly produce damage to surrounding tissue (Otchy et al. 2015). The difference is likely not explained by the manipulation technique because ACC lesions (Hart et al. 2017) also result in an effect size of d = 1.39 on quality effort-based choice, similar to BLA inactivation. Indeed, of the different experiments we have conducted in our lab investigating effects on quality effort-based choice, none of the effect sizes match that of quantity effort-based choice: experimenter-administered methamphetamine, d=1.29 (Thompson et al., 2017); intravenous methamphetamine self-administration d=1.05 (Hart et al. 2018), see Figure 2. This lends support to the idea that the representation of reward magnitude is more salient than reward identity.

Figure 2. Comparison of different manipulations on the same quality effort-based paradigm in our lab.

(Top left) ACC lesions decreased lever pressing in the context of a competing, low effort reward. (Top right) BLA inactivations decreased high effort lever pressing in the context of choice. (Bottom left) Withdrawal from experimenter-administered methamphetamine decreased choice lever pressing. (Bottom right) Withdrawal from methamphetamine intravenous self-administration decreased choice lever pressing. Data are adapted from Hart et al. (2017), Hart & Izquierdo (2017), Thompson et al. (2017), and Hart et al. (2018).

Theoretical Frameworks

Psychologists have adopted concepts from economics to explain behavior. Namely, the concept of demand elasticity, that is, the demand for a good when only the price changes, has been applied to understanding fixed-ratio responding. For example, ventral striatal dopamine depletions have no effect on fixed-ratio 1 responding for food but reduce lever pressing to a greater extent as the work requirement increases to fixed-ratio 64 (Aberman and Salamone 1999). Thus, ventral striatal dopamine depletions increase demand elasticity and decrease “willingness to pay”.

Economic analyses may be useful for comparing cost-benefit decisions in humans and in experimental animals (Bentzley, Fender, and Aston-Jones 2013; Hursh 1980; Hursh and Silberberg 2008). Economic measures may have high translational value because estimates of cost sensitivity and other parameters can be used to roughly equate measures across different schedules of reinforcement, reinforcers, and species in different tasks. Using these economic models, demand curves are generated that allow us to estimate consumption at the lowest effort cost (Q₀), the rate of decline in consumption as a function of increasing effort cost (α), and an index of demand inelasticity derived from α, essential value (EV). These measures have been shown to predict addiction-like behaviors (Murphy et al. 2009; Petry 2001; Gray and MacKillop 2014; Galuska et al. 2011; Bentzley and Aston-Jones 2015), decreased food demand (Galuska et al. 2011), and they can dissociate unconstrained consumption (i.e. Q₀) from willingness to pay (i.e. α) under dopamine antagonism or depletion (Aberman and Salamone 1999; Salamone et al. 2016; Salamone et al. 2017). We recently fit an exponential model (Hursh and Silberberg 2008) to our effort-based choice data (Hart, Gerson, and Izquierdo 2018):

$${\text{Q = Q}}_0\ast {10}^{\text{k}\left({\text{e}}^{-{\text{αQ}}_0\text{C}}{}_{-1}\right)}$$

where Q = consumption at a given price C, or cost (FR value), Q₀ = demand intensity, consumption at lowest price, k = constant parameter reflecting the range of consumption values in log10 units, α = demand elasticity, or the derived demand parameter reflecting the rate of consumption decline associated with increasing price. Q₀ and α are calculated based on the number of pellets earned at each ratio. Because the number of pellets earned at each ratio in our task is restricted to five, differences in α would be due to animals reaching higher/lower ratios or earning more/fewer pellets at a given ratio. It is unclear, therefore, how this model captures more than the session average performance since it allows no value update (see below), though this criticism can apply to other computational models of cost-benefit choice, including delay discounting. Other models, described below, may better capture within-session sensitivity to cost or benefit. For these economic models, for each rat, we have also previously calculated Essential Value (EV) to quantify the ‘strength’ of the reinforcer (i.e. sucrose pellets) (Kearns et al. 2017), based on the value of the exponential rate constant. As indicated above, the value of α is the rate of consumption decline and k is the consumption range for each rat in the following:

$$\text{EV = 1/}\left(100*\text{α*}{k}^{1.5}\right)$$

All behavioral economic indices can then be correlated with brain activity data, and regression analyses conducted to predict indices from brain data, as in Hart et al. (2018). Some factors worthy of future consideration in the context of how behavioral economic indices fit effort paradigms are: i. how direct comparisons of quantity and quality may differentially impact behavioral economic indices, ii. how different reinforcers could differentially support performance because they are presented in an open or closed economy, and iii. how different food restriction protocols in the homecage environment may influence effort-based performance in the operant chamber. As mentioned previously, T-maze tasks generate impoverished datasets relative to lever pressing tasks, and are unlikey to yield the necessary number of trials to fit these exponential models to behavioral data.

Standard reinforcement learning algorithms may also be used to capture behavior on either quality or quantity-based effort choice paradigms, but to our knowledge they have not been applied to rat behavioral data such as those described in this review. A conventional temporal difference model (Sutton and Barto 1998; Sutton 1988) consisting of a prediction error term may be used, since prediction error has been shown to reflect marginal utility (Stauffer, Lak, and Schultz 2014). Such a model would allow estimation of a more dynamic, within-session value change in effort-based choice. This is not a viable solution because in typical effort choice tasks, the rat has already learned about the contingencies and can easily discriminate the options. However, the rat could still learn about the (diminishing) value of the preferred, more effortful option as the session goes on. Changes could be driven either by changes in subjective effort or by subjective benefit of the reward, and no model effectively dissociates these possibilities. It will be interesting to fit existing effort-choice datasets to this model in the future, and use the discount factor and/or the diminishing utility function as a variable to correlate with neural activity in the regions identified here.

Finally, it should be noted that in such effort-based paradigms the more effortful, higher-valued option is often associated with more variability or uncertainty than the lower-valued option. This can be either because the animal ‘gives up’ on one more often, or simply because the value is more dependent on satiety level on any given day. Therefore, it will useful to compare models used in explaining effects on expected uncertainty or known risk (Stolyarova and Izquierdo 2017) to these data as well.

Why effort is important

To be successful at foraging, organisms must evaluate the cost of reward (i.e., risk, delay, effort) and often appreciate this cost in relative terms (i.e. in comparison with other options that may be concurrently available). This ability is of interest to a variety of fields including psychology, neuroscience, behavioral ecology, and ethology (Dayan and Daw 2008). Effort, delay, and risk discounting paradigms generally do not access how animals gather information of the options in unknown or changing reward environments (i.e. learning) but instead test how well they maximize or exploit rewards in well-known conditions (i.e. performance). As mentioned above, one important exception is that the rat must learn about and adapt to the decreasing value of each option, if food rewards are involved. This would ostensibly make brain regions involved in updating behavior according to motivational state changes critical to this behavior. When we consider a hierarchy of costs, some are given priority consideration: for example, risk of punishment or predation is weighed more heavily than physical effort costs, delay costs, or risk of non-reward. Similarly, the opportunity to exploit a resource often comes at the expense of exploring another (Addicott et al. 2017), known as the explore-exploit tradeoff. This is likely not a factor in environments where resources are not depleting, and where reinforcement contingencies are known to the organism. Physical effort has underpinnings deeply embedded in the mesocorticolimbic system. Indeed, we have found similarities with inhibition of normal functioning of ACC (Hart et al. 2017), BLA (Hart and Izquierdo 2017), and chronic methamphetamine experience, whether experimenter- or self-administered (Thompson et al. 2017; Hart, Gerson, and Izquierdo 2018), in making rats ‘selectively lazy’ for the qualitatively preferred option (Figure 1). It would be interesting to determine how exposure to other drugs of abuse may influence performance on this task.

Though this review focuses on convergence in effort tasks, future work should determine whether this is the case for tasks where other aforementioned costs (time, risk) are involved. To our knowledge, paradigms for assessing delay costs (Bickel, Odum, and Madden 1999; Coffey et al. 2003; Floresco, Tse, and Ghods-Sharifi 2008; Mendez et al. 2010) and risk costs (Floresco and Whelan 2009; St Onge, Stopper, et al. 2012; St Onge, Ahn, et al. 2012; Larkin, Jenni, and Floresco 2016; Ghods-Sharifi, St Onge, and Floresco 2009) have exclusively employed rewards that vary in quantity, save a few exceptions where the larger reward had a higher probability of receiving shock, rather than non-reinforcement (Orsini et al. 2015; Orsini et al. 2017; Simon et al. 2009). Interestingly, a recent report showed D2 receptor stimulation results in decreased risky choice under risk of shock (Blaes et al. 2018), where others have found increased risky choice with D2 receptor stimulation under risk of non-reinforcement (St Onge and Floresco 2009). This suggests that risk of punishment versus non-reinforcement may have nonoverlapping mechanisms. Is this the case for other neural substrates that regulate risk-based decisions? Would testing mechanisms found in typical delay-discounting paradigms provide similar results in a quality-based delay discounting task? These remain empirical questions.

Overcoming effort costs to acquire food is necessary for survival in humans and other animals. Of course, other appetitive events, such as exercise, sex, or drugs are often impeded by effort costs, and behavior directed toward any one of these events may or may not be adaptive depending on several situational factors. For example, it might be maladaptive for an obese rat to pursue more food at the cost of other rewards. It may be similarly maladaptive for addicted individuals to pursue drugs. Growing evidence points to similar underlying mechanisms in effort for food as those in willingness to exercise (Correa et al. 2016; Lopez-Cruz et al. 2018) or to seek drugs (Kavanagh et al. 2015). Future experiments should test whether the circuitry that is known to regulate effort to acquire food similarly affects other rewards. This line of work is important because effort tasks have high translational value (Treadway et al. 2009; Wardle et al. 2011; Treadway and Zald 2011; Treadway et al. 2012): effort is implicated in many conditions, including drug withdrawal (Thompson et al. 2017; Hart, Gerson, and Izquierdo 2018), depression (Treadway et al. 2012), and schizophrenia (Gold et al. 2013; Treadway et al. 2015). Further, fatigue, which can be characterized as decreased effort, is one the most common symptoms in general medicine (Demyttenaere, De Fruyt, and Stahl 2005), and tasks assessing effort could be useful in modeling effort-related symptoms that occur in depression and other conditions (Nunes et al. 2013).

Psychological science and other fields are currently undergoing a major reproducibility crisis (Estimating the reproducibility of psychological science 2015). That several tasks, methods, and brain region manipulation have produced consistent and replicable findings bodes well for this line of work. Indeed, the existing literature characterizing the neural bases of effort opens the door for future research investigating how these regions act as a network involved in the regulation of deciding how much effort to exert not just for food, but for drugs, sex, aggression, and other reinforcers. Indeed, with increasingly powerful tools to record electrical activity of hundreds of neurons (Jun et al. 2017) and open source optical imaging methods (Cai et al. 2016) coupled with sophisticated computational approaches (Wikenheiser, Marrero-Garcia, and Schoenbaum 2017; Sadacca et al. 2018; Mashhoori et al. 2018), future research can reveal many functions of the mammalian brain in effort and beyond.

Highlights.

• Different behavioral paradigms have been used to assess effort-based choice in rats

• The rat selects between an effortful, preferred vs. a low effort, less preferred option

• Generally, the preferred option is better in quality or magnitude

• We discuss a variety of experimental and theoretical approaches related to effort

• We highlight areas for future work and the relevance to the human condition

Abbreviations

ACC

Anterior cingulate cortex

BLA

Basolateral amygdala

DMS

Dorsomedial striatum

OFC

Orbitofrontal cortex

STN

Subthalamic nucleus

VS

Ventromedial striatum