Dissociating the Effects of Dopamine D2 Receptors on Effort-Based vs Value-based Decision Making using a Novel Behavioral approach.

Abstract

Cost-benefit decision making is essential for organisms to adapt to their ever-changing environment. Most studies of cost-benefit decision-making involve choice conditions in which effort and value are varied simultaneously. This prevents identification of the aspects of cost-benefit decision-making that are affected by experimental manipulations. We developed operant assays to isolate the individual impacts of effort and value manipulations on cost-benefit decision making. In the Concurrent Effort Choice (CEC) task, mice choose between exerting two distinct types of effort: the number of responses and the duration of a response, to earn the same reward. By parametrically varying response cost, psychometric functions are obtained that reflect how the two types of effort scale against one another. Direct manipulations of effort shift the functions. Because reward value is held constant in this task, differences in scaling of the two response types must be related to the effort manipulations.

In the Concurrent Value Choice (CVC) task, mice make the same type of response to earn rewards of different value (e.g. pellets vs sucrose solutions). Here the effort required to earn one reward type is parametrically varied to obtain the psychometric function that scales the value of the two rewards into the number of responses subjects will pay to earn one reward over the other. Direct value manipulations shift these functions. We tested the effect of the dopamine D2 receptor antagonist, Haloperidol, on performance in the CEC and CVC assays and found that D2R signaling is important for Effort-based, but not Value-based decision-making.

Introduction

Humans (Croxson, Walton, O’Reilly, Behrens, & Rushworth, 2009) and other animals (Atalayer & Rowland, 2009; Collier & Johnson, 1997) use cost-benefit decision making to behave in adaptable and flexible ways in response to changing internal physiological states and environmental conditions. In order to effectively carry out cost-benefit analyses, organisms must be able to accurately estimate the effort/costs required to obtain a goal in the future, accurately estimate the value/benefit of the future goal to be obtained, and determine if the outcome is worth the cost of pursuit (Rangel & Hare, 2010; Simpson & Balsam, 2016).

Laboratory animals can readily process information related to various types of costs, including effort (McDowell, 2013; Mechner, 1958), time (Balsam, Drew, & Gallistel, 2010; Balsam & Gallistel, 2009; Dews, 1978), and opportunity costs/risk associated with the uncertainty of an action’s outcome (Winstanley & Floresco, 2016). When effort costs are directly manipulated (Bailey, Simpson, & Balsam, 2016) such as the number of responses (e.g. lever presses) required (Baron & Derenne, 2000; Covarrubias & Aparicio, 2008; Floresco, Tse, & Ghods-Sharifi, 2008; Hodos, 1961; P. R. Killeen, Posadas-Sanchez, Johansen, & Thrailkill, 2009), the height of a barrier subjects must climb (Hauber & Sommer, 2009; J D Salamone, Cousins, & Bucher, 1994; Schweimer & Hauber, 2005), and the force which must be exerted to press a lever (Fouriezos, Bielajew, & Pagotto, 1990; Fowler, Morgenstern, & Notterman, 1972; Ishiwari, Weber, Mingote, Correa, & Salamone, 2004; Notterman, 1965), goal directed actions are sensitive to these costs.

Similarly, animals demonstrate sensitivity to changes in expected reward value via manipulations of magnitude (Bayer & Glimcher, 2005) (i.e. number of pellets, volume of a sucrose solution), quality (e.g. the concentration of a sweet solution (Flaherty, Turovsky, & Krauss, 1994; Glendinning, Gresack, & Spector, 2002), levels of deprivation (Hart, Leung, & Balleine, 2014), payoff rates (Andrew & Harris, 2011), and by comparing qualitatively different outcomes (Flaherty & Mitchell, 1999). Expected reward value modulates how long subjects will continue to work for an outcome (Bailey et al., 2015; Covarrubias & Aparicio, 2008; Floresco, Tse, et al., 2008; Skjoldager, Pierre, & Mittleman, 1993) the rate/vigor of responding (Hutsell & Newland, 2013), as well as the choice of responses when alternative actions lead to rewards of different value (Baum, 1974; Gallistel et al., 2007).

These studies represent only a fraction of the body of work demonstrating sensitivity to changes in effort requirements and reward value (see (Bailey, Simpson, et al., 2016; Levy & Glimcher, 2012) for reviews). However, a major challenge in studying the neurobiology of cost-benefit decision making comes from not being able to readily measure how effortful an animal perceives an action to be independently of the value of the consequences of that behavior. For example, if we study motivation by measuring how vigorously an animal will work to obtain a reward and find that a particular drug or a particular kind of brain manipulation increases the vigor or persistence of goal-directed behavior it could do so for several reasons including augmenting the value of reward and/or by reducing the representation of effort that is needed to obtain the reward (see table 1). These procedures do not permit unambiguous inferences about which of these factors might have changed. Changes in behavioral output in such tasks could be the result of alterations in effort-related processes, value-related processes, or hedonic processes. Another often used method of studying cost-benefit decision making is to give animals an effort-related choice (Bailey, Williamson, et al., 2016; Floresco, St Onge, Ghods-Sharifi, & Winstanley, 2008; J D Salamone et al., 1994, 1991; Shafiei, Gray, Viau, & Floresco, 2012; Ward et al., 2012), in which subjects are provided with a choice between a High Effort/ Large Reward, and a Low Effort/ Small Reward option. This means that each choice the animal makes involves taking into account both the effort requirement as well as the reward magnitude each time they make a decision (see the first 3 tasks in Table 1). Consequently, if manipulations affect choice in these types of paradigms, it is not possible to know if the manipulation affected estimates of benefit, cost, or the cost-benefit calculation.

Table 1.

Effort and Value Choices Under Comparison in some commonly used Effort-Related Choice Tasks.

Task	Effort Options Available		Value Options Available		Effort-Choice Comparison?		Value-Choice Comparison?
	High	Low	High	Low	Change?	By	Change?	By
T-Arm Barrier Maze	Climb	Walk	4 pellets	2 pellets	Yes	Climb/ No Climb	Yes	# Pellets
Concurrent Lever Press/Free Chow Consumption	5 Lever Presses	0 Lever Presses	Pellet	Chow	Yes	Press/ No Press	Yes	Type of food
Operant Effort Discounting	2–20 Lever Presses	1 Lever Press	4 Pellets	1 Pellet	Yes	# of Presses	Yes	# of Pellets
Concurrent Effort Choice (CEC)	1, 5, 10, 20, 40, 80, or 160 Presses	1 Lever Hold	0.01cc Milk	0.01cc Milk	Yes	Lever Press/ Hold	No	-
Concurrent Value Choice (CVC)	5 Lever Presses	5 Lever Presses	0.01cc 20 % Sucrose	0.01cc 0.5% Sucrose	No	-	Yes	Sucrose concentration

This difficulty of interpreting studies of cost-benefit decision-making when animals are asked to trade off costs and rewards has recently been discussed (Walton & Bouret, 2019). A few prior studies have included conditions in which either cost or benefit was equal to a comparison condition (Cowen, Davis, & Nitz, 2012; Gan, Walton, & Phillips, 2010; Hillman & Bilkey, 2010). Such studies successfully identified factors (single unit activity or dopamine release) that correlated with cost, benefit or cost/benefit ratio. However, the assay designs are not ideal for identifying the specific processes disrupted by perturbations (drugs, mutations etc). Assays that result in psychometric functions are more sensitive for detecting behavioral changes than assays that involve only 2 or 3 different test conditions. Also, in each of the above tasks, the costs of time and effort were neither experimentally controlled nor analyzed. Trials of greater effort always took longer to complete. A more complex paradigm, known as the “Reward Mountain Model” (Arvanitogiannis & Shizgal, 2008; Breton, Mullett, Conover, & Shizgal, 2013) has been successfully used to disentangle the effects of pharmacological agents on work, reward and opportunity costs (Hernandez, Breton, Conover, & Shizgal, 2010; Trujillo-Pisanty et al., 2011). One drawback to this powerful model is that reinforcement is mediated by brain stimulation, which is optimal for reward precision and control, but is more technically complicated to implement and may lack some of the face validity offered by natural primary reinforcers.

In the present set of studies, we provide a detailed method for dissociating whether a change in goal-directed behavior is the result of changes in value and/or effort processing. We developed and characterized two behavioral assays to study cost-benefit decision making which we refer to as the Concurrent Effort Choice (CEC) task and the Concurrent Value Choice (CVC) task (see table 1). To measure effort-based decision making, the CEC task manipulates effort requirements while holding the value variable constant. In each trial, subjects choose between two qualitatively different response choices, each of which earns the same type and amount of reward. Thereby, differences in response choice, relates to the type and amount of effort required, not the outcome and we checked that time (delay to reward) did not explain differences in effort sensitivity. The Concurrent Value Choice (CVC) task is more complex, requiring the comparison of choice functions generated over several sessions. Each trial involves the subject choosing to work for one type of reward which is fixed in value within and across all sessions (a single food pellet), or for a different type of reward that has a fixed value within sessions, but varies in value across sessions (sucrose of different concentrations). The response cost of the fixed value reward is varied across sessions, while the reward that varies in value, always has the same response cost. This means that within any one session each trial involves a choice between different response costs for different types of reward. However, because over sessions, we systematically manipulate the cost of one type of reward (the pellet) and not the other (sucrose), we can determine how the subject values the different sucrose concentrations, relative to the variable-cost, fixed-value pellet. If an animal is sensitive to reward value, their choice function for pellets will be shifted by changes in sucrose concentration, i.e. a subject will prefer making 20 lever presses for a pellet, if the alternative option is making 5 presses for 5% sucrose, but not when the alternative is 5 presses for 20% sucrose. With these tasks, we obtain measures of how sensitive subjects are to different response costs and reward values independently, and how these factors impact their decision making. We first carry out a detailed analysis of behavior in the CEC and CVC tasks to validate the tasks by directly manipulating either effort and reward value. We then use both tasks to determine the impact of the dopamine D2 receptor antagonist haloperidol on cost-benefit decision-making. We show that haloperidol selectively alters effort-based and not value-based decision making.

Methods

Subjects

Experiments used male C57BL/6J mice (The Jackson Laboratory, Bar Harbor, ME, USA) which were 10 weeks old at the start of the experiments. All subjects were maintained at 85% of their ad libitum bodyweight in order to motivate them to work for food rewards in operant procedures. All experiments and animal care protocols were in accordance with the Columbia University and NYSPI Institutional Animal Care and Use Committees and Animal Welfare regulations. The number of subjects used is indicated in the description of each experiment below.

Apparatus

Experimental chambers (ENV-307w; Med Associates, St. Albans, VT) equipped with liquid dippers and pellet dispensers were used in the experiment. Unless otherwise noted, the apparatus was identical to that used by Drew and colleagues, (Drew et al., 2007). Two retractable levers were mounted on either side of a feeding trough, and a house light (Model 1820; Med Associates) located at the top of the chamber was used to illuminate the chamber during the sessions. The specific rewards used are detailed for each experiment.

Behavioral Procedures

For an outline of all behavioral procedures used for training and testing subjects see supplemental Figure S1. All programs used for behavioral training and testing (written in MEDState Notation) are available upon request.

Lever Press Training Procedures.

Subjects in the CEC experiments were trained to press levers for milk rewards using the procedure described by Drew and colleagues (Drew et al., 2007). Briefly, mice were first trained to consume the liquid milk reward from the feeding trough when a dipper containing 0.01cc of evaporated milk was presented. To do this, mice were first placed into the operant chamber with the dipper in the accessible, raised position. From the time mice first made a head entry into the feeding trough, the dipper remained raised for another 10 seconds and then was lowered. Following a variable inter-trial interval (ITI), (mean = 40s) a new trial began, which was identical to the first trial. These sessions lasted 30 minutes or until the mice consumed 20 rewards. On the next day, the dipper began in the lowered position, but was raised while at the same time a 0.5 second tone was played. The dipper then remained in the raised position for 8 seconds and then was lowered. A variable ITI interval (mean = 40s) was followed by an identical trial through the remainder of the session. This was done to teach the mice to quickly get to the feeding trough to consume the reward when it was presented. Sessions of this type lasted 30 minutes or until mice earned 30 rewards. Mice were exposed to sessions of this type until they earned 30 rewards on 2 consecutive days. Once all subjects reached this criterion they were then moved on to the training phase to learn to make lever presses for milk rewards.

In the second phase of basic lever press training mice were taught to press levers to earn the rewards. In these continuous reinforcement sessions (CRF), mice were rewarded with a 5 second presentation of a dipper following every lever press. At the beginning of the session the lever was extended into the chamber, remained extended for 2 rewards, and then retracted. After a variable ITI (mean = 40s), the lever was re-extended and remained so for the next two rewards. Sessions continued like this for 1 hour or until 40 rewards were earned. Mice were trained on CRF until they earned 40 rewards on 3 consecutive days, and were then trained on random ratio (RR) schedules. They were only moved on to a higher RR after they reached this same criterion. In RR schedule, subjects were required to make some variable number of lever presses in each trial, with the number required being drawn from an exponential distribution with a given mean. Subjects were tested in RR05, RR10, and RR20 schedules until they earned 40 rewards on 3 consecutive days. Once this criterion was reached, subjects were then moved on to the lever hold-down training phase of the experiment.

Lever Hold-Down Training Procedures.

Subjects in all experiments were exposed to two different “hold down” procedures (Bailey et al., 2015): the Variable Interval Hold Down (VIH) and the Progressive Hold Down (PHD). In both schedules, a required hold duration was assigned prior to the start of each trial. This was the duration of time the subject was required to hold the lever in the depressed position in order to receive a reward. Each individual trial in either schedules followed a similar procedure: at the start of each trial, the house light was illuminated and a lever was extended. As soon as the mouse depressed the lever, a timer began counting how long the lever was in the depressed position. This timer stopped and reset to 0.0 if the mouse ended the lever press before the required time was reached. If the lever was depressed for as long as the required duration, the trial ended, and the subject received a reward. A tone (2 s) sounded to signal the presentation of the dipper (5 s) and the house light was shut off until the start of the next trial.

Variable Interval Hold training.

Following initial lever press training (described above), subjects were then trained to make lever holds using the VIH task. At the beginning of each trial, the required hold duration was drawn randomly from a truncated exponential distribution. This hold requirement remained in place until the subject was reinforced for completing the trial, at which time the next trial’s required hold duration was randomly determined. During the first session, the distribution of required hold durations had a mean = 0.5 s; (min = .01 s; max = 2.44 s). When a mouse earned 40 rewards on three consecutive days, the required hold durations for the subsequent session were drawn from an exponential distribution with a higher mean (1 s, 2 s, 3 s, 4 s, 5 s, 8 s, 10 s). Thus, during the final session of VIH training, subjects were required to hold down the lever for intervals that averaged 10 s, but could be as long as 18.8 s.

Concurrent Effort Choice Task

All subjects were first trained to press a lever using the basic lever press training procedure for a rewards of evaporated milk (.01 ml) delivered by raising a dipper located inside the feeder trough for 5 seconds. Subjects were then exposed to increasing RR schedules on a single lever. The progression was as follows: 3 days on CRF, 3 days on RR05, 3days on RR10, and 3 days on RR20. Subjects were next trained to make lever holds on the opposite lever. This was done using the VIH training program. Subjects were exposed to 3 days of each of the following VIH programs in increasing order (0.5, 1.0, 2.0, 4.0, 6.0, 8.0, 10.0 sec).

Once subjects were trained to make presses on lever A and holds on lever B they were then moved to the CEC task (Figure 1). In this task, subjects could lever press on one lever, or lever hold on the opposite lever to obtain a reward. CEC test sessions consisted of two separate phases. First, there were 10 forced choice trials in which either the press lever or the hold lever was presented (each lever was presented 5 times) after a variable inter- trial interval. Forced trials were terminated if not completed within 3 minutes of the lever presentation. Such failed trials were not included in the analysis of latency to respond or completion times, e.g. the time taken to complete forced press trials was the average time from lever presentation until reward delivery for successful press trials.

Figure 1.

(A). Top, Subjects were trained to lever press in sessions where one lever came out and they had to make a required number of presses to earn milk rewards. Bottom, Shows how the schedules of reinforcement were increased over days (CRF, RR05, RR10, and RR20). (B). Top, Subjects were trained to lever hold in sessions where one lever came out and they had to hold the lever down for the required duration to earn milk rewards. Bottom, Shows how the schedules of reinforcement were increased over days (0.5s, 1s, 2s, 4s, 8s, 10s). (C). Schematic Representation of a CEC session. Sessions began with single lever trials in which the Press Lever (PL) and Hold Lever (HL) were each presented 5 times. Subjects were then presented with choice trials in which both levers were presented simultaneously for subjects to choose which lever to work on to obtain the milk reward. (D). The number of presses required on the PL varied over days (1, 5, 10, 20, 40, 80, or 160), while the hold requirement on the HL was either 5 (left) or 10 (right) seconds.

After 10 single lever forced choice trials were presented, mice were then presented with free choice trials. In this phase of the session, subjects were presented with both levers simultaneously and they were able to work on either lever to obtain the reward. There was no time limit in which the mice had to complete the press or hold requirement in the choice trials. Choice was determined by the response type on which the subject earned a reward. Animals were able to work on either lever until a trial was successfully completed, i.e. the continuous press time requirement was achieved on the hold lever or the number of lever presses was completed on the press lever. Each session was terminated when the mouse had successfully earned a total of 50 rewards (forced and choice trials combined), or 1 hour had elapsed.

In this experiment, we maintained the hold duration required at a constant value (5 seconds), and varied the press requirement over days. We used the following press requirements (1, 5, 10, 20, 40, 80, and 160). After running through this progression 2 times, we then changed the hold requirement to 10 seconds and ran through the same progression of press requirements. Data presented represent average values from the two runs for each hold requirement. We did not analyze the effect of “run”, i.e., first or second time through the sequence of press requirements, due to an occasional missing data point (only one run for a specific requirement for a few subject). However, the effect of press requirement appeared to be stable across runs. In support of this apparent stability, when mice were drug tested using a randomized Latin Squares design (see below) we observed clear pharmacological effects. A strong effect of run could have prevented this observation.

Haloperidol Treatment in the CEC Task.

The same cohort of mice used in the Baseline CEC task were used to assess the effects of Haloperidol. Mice were tested in the fixed 10 second hold condition at ratio requirements of 10, 20, 40, and 80. Subjects were injected with Vehicle, 0.1 mg/kg Haloperidol, and 0.2 mg/kg Haloperidol 45 minutes prior to behavioral testing in a randomized Latin Squares design across the different schedules such that each animal was tested on each schedule on a given dose twice. Haloperidol (Sigma-Aldrich, St. Louis, MO) was dissolved in 0.2% lactic acid solution which was used as a vehicle control.

Concurrent Value Choice Task

All subjects were first trained to press a lever on each side of the operant chamber using the basic lever press training procedure described above. The lever on one side of the chamber delivered a liquid sucrose solution, while the opposite lever delivered a 14mg sucrose pellet. Subjects were then exposed to increasing RR schedules on each lever for the different outcomes, receiving one session for a single outcome per day. The progression was as follows: 3 days on CRF, 3 days on RR05, 3days on RR10, and 3 days on RR20.

Once subjects were trained to make presses on lever A for liquid sucrose and lever presses on lever B for a sucrose pellet they were then moved to the CVC task (Figure 4). CVC test sessions consisted of two separate phases. First, there were 10 forced choice trials in which either the sucrose lever or the pellet lever was presented (each lever was presented 5 times) after a variable inter- trial interval. Forced trials were terminated if not completed within 3 minutes of the lever presentation. Such failed trials were not included in the analysis of latency to respond or completion times. After 10 single lever forced choice trials were presented, mice were then presented with free choice trials. In this phase of the session, subjects were presented with both levers and they were able to choose which one they worked on to obtain the reward. There was no time limit in which the mice had to complete the press or hold requirement in the choice trials. Choice was determined by the response type on which the subject earned a reward. Animals were able to work on either lever until a trial was successfully completed. Each session was terminated when the mouse had successfully earned a total of 30 rewards (from forced and choice trials combined), or 1 hour had elapsed.

Figure 4.

(A). Subjects were trained to lever press for pellets on one lever (Pellet Lever) and liquid sucrose on the opposite lever (Sucrose Lever) of the chamber. (B). Shows the schedules of reinforcement (CRF, RR05, RR10, and RR20) subjects were trained with to learn to work for pellets and sucrose. A single reward was available each day and was alternated every other day. (C). Schematic representation of the trial structure in the CVC task. Session begins with 10 single levers trials in which either the Sucrose Lever or Pellet Lever is presented. After the presentation of 10 single lever trials subjects were then given choice trials in which both levers were presented for subjects to choose which lever to work on to obtain reward. (D). Shows the daily requirements for the different outcomes. The number of presses required on the Pellet Lever varied over days (10, 20, 40, and 80), whereas the # of presses required on the Sucrose Lever was always 5 presses.

Sucrose Concentration Manipulation.

In this experiment, the cost of liquid sucrose was fixed at 5 presses, whereas the cost for a pellet varied over days (10, 20, 40, or 80 presses). Subjects were first tested in the CVC with 20% liquid sucrose and were exposed to each of the different pellet costs in ascending order 2 times. Following this, subjects were then tested in the CVC with a 5% sucrose solution and were exposed to each of the different pellet costs in ascending order 2 times. The average of these two rounds of 2 exposures to each of the different pellet costs was taken for each sucrose concentration condition.

Outcome Devaluation Manipulation.

In this experiment, the sucrose concentration was kept at 5% and the cost of a sucrose reward was fixed at 5 presses. The cost of a pellet varied over days (10, 20, 40, or 80 presses). Subjects were tested in 3 consecutive weeks in this experiment. In week 1 (Valued-1), subjects were tested on the CVC schedule as described above. In week 2 (Devalued), subjects received a 30 minute exposure to pellets prior to the testing session, and then were tested in the CVC task. In week 3 (valued-2), subjects were tested on the CVC without pre-session exposure to pellets.

Haloperidol Treatment in the CVC task.

The effects of Haloperidol in the CEC task were tested using the same mice used for baseline testing. Vehicle, or 0.1mg/kg Haloperidol was injected 45 minutes prior to behavioral testing using a randomized Latin Squares design across the different schedules such that each animal was tested on each schedule on a given dose twice.

Data Acquisition and Analysis

A combination of MEDState Notation (MSN, MED Associates) language and custom MATLAB (Mathworks, Natick MA) scripts were used to retrieve data from timestamped data sets. Graph Pad Prism Version 8 (GraphPad Software, La Jolla CA) was used to carry out both planned statistical analyses and post-hoc comparisons where appropriate. When datasets were complete we performed repeated measures One- or Two-way ANOVA, according to the data structure, and Sidak’s multiple comparisons test when appropriate. For datasets with any missing values, we fitted a mixed effects model. The a priori alpha level was set at p < 0.05. The statistical details of each experiment, including the n, the statistical test, and significance, are provided in the results section and the figure legends. The results of main effects and interactions are reported in the text and the level of significance provided in the figures, the level of significance of multiple comparisons post-hoc tests are provided only in the figures.

Results

Measuring Effort-related choice using the Concurrent Effort Choice (CEC) Task.

Mice were trained to make lever presses on the Press Lever (PL) following exposure to several different ratio schedules of reinforcement over days (Fig 1A), and were then trained to make lever holds on the Hold Lever (HL) following exposure to several different hold duration schedules of reinforcement over days (Fig 1B). All subjects successfully earned the maximum allowed rewards in all sessions and were tested in the CEC task. In this task, subjects get 10 single lever Forced Choice trials during which either the PL or the HL is presented. Subjects had to make the required number of presses or hold for the required amount of time to earn rewards in the forced trials. These trials served to teach the subjects the two different requirements on any given day, because the hold duration and number of lever presses required to earn rewards changed over days. After completing the 10 forced trials subjects are then presented with choice trials in which both levers were presented and the subject can choose which lever to work on to earn rewards (see Fig 1C for a schematic representation of a daily session). We first tested mice with a 5s hold duration and increasing press requirements over days (1, 5, 10, 20 40, 80, 160 presses), and exposed them to each press requirement twice in ascending order before repeating this procedure for a 10 second hold duration (see Fig 1D for schematic of schedule requirements over days).

Both Response Number and Hold Time Modulate Effort-Related Choice

To examine subject’s sensitivity to the different effort requirements of response number and response duration, we computed the proportion of hold choices subjects made during the choice trials, defined as a trial in which holding lead to reward (Fig 2A). The curve for the 10 second hold condition displayed a rightward shift relative to the 5 second hold condition. This indicates that subjects were willing to make more lever presses when the alternative was a 10 second hold compared to when the alternative was a 5 second hold. For both hold durations subjects became more likely to choose the hold option as the number of responses required to earn reward increased. We observed a main effect of press requirement (F_(6,90) = 163.0, p < .0001), a main effect of hold duration (F_(1,15) = 88.75, p < .0001), and a significant interaction (F_(6,90) = 3.53, p = .003). A Sidack’s multiple comparisons test revealed an effect of hold trial duration on the proportion of hold choices when the press requirement on the alternate lever was all but the highest press requirement (160 presses). To quantify the effect of the hold duration requirement on subject’s choice we computed each subject’s point of subjective equality (PSE) by extrapolating the number of presses at which the choice behavior would be equal to 0.5, which gave us an estimate of the number of presses a subject considered equally effortful to a given hold duration. Subject’s PSE was significantly affected by the hold duration condition (t ₍₁₅₎ = 5.14, p < .0001; Fig 2B). All subjects were sensitive to this effort manipulation as indicated by both individual subjects PSE’s (Fig 2C), as well as individual subject’s choice functions (Supplemental Fig S2).

Figure 2.

Performance in the Concurrent Effort Choice (CEC) task. (A) Proportion of hold choices in the CEC task in 2 different hold duration conditions. (B) Point of subjective equality (PSE) for subjects in the 2 different hold duration conditions. (C) PSE values for each individual subject in the 2 different hold conditions. (D) Latency to the first lever press in single lever press trials. (E) Latency to the first lever press in single lever hold trials (F). Press vigor = number of presses made per second during single lever press trials. (F) Hold efficiency = the hold requirement time/ time take to complete the requirement. All values represented are mean ± (SEM), except for (C), which depicts individual subject values. n = 16; The significance of main effects are depicted: **p < .01; ***p < .001; ****p < .0001.

Effort-Choice in the CEC Task is Not Exclusively Driven by the Cost of Time.

To determine if the animal’s choice is driven by reward delay, we analyzed the average interval between lever presentation and reward delivery for each animal, for each trial type (holding or pressing) at the press requirement closest to each individual’s PSE. If an animal’s choice is driven exclusively by reward delay, the time to complete holding trials and pressing trials that are equally preferred under the choice condition would be the same, i.e. the difference between the time to complete holding and pressing trials at the PSE would be zero. Supplemental Fig. S3 shows that the difference in time to earn reward for pressing and holding in both the 5s and 10s hold conditions are significantly different from zero (One sample t-test, Theoretical Mean = 0, Mean for 5s hold requirement = 10.59, p< 0.0001, Mean for 10s hold requirement = 6.68, p= 0.02. Therefore, reward delay does not exclusively drive choice in this task.

Effort Requirement Impacts Latency to Begin Work, Work Vigor and Efficiency.

The data from the choice trials demonstrated subject’s sensitivity to different effort requirements and their ability to adapt their behavior in response to changing costs. We also looked for other behavioral indices of subject’s sensitivity to different costs in addition to their choice behavior. We examined the temporal dynamics of behavior while subjects performed the two different types of work (pressing vs holding). Because subjects completed very different numbers of presses and holds during the choice trials we examined behavior in the 10 single lever forced trials at the beginning of each session, as most subjects completed similar numbers of such trials. Unsurprisingly, it took subjects a longer total time to complete trials (from lever presentation to trial completion) with larger press requirements which was reflected in a main effect of press requirement (F_{(6, 90)} = 198.2, p < .0001). We also found that increasing the hold duration required for the hold trials in the same session significantly reduced the time to complete press trials (F_{(1, 15)} = 14.46, p = .0017). This arose mainly from a significant press requirement by hold duration interaction (F_{(6, 90)} = 3.639, p = .0028). At the higher press ratios subjects completed trials faster when a 10s hold was in effect than when the 5s hold requirement was in effect (Data not shown). Also unsurprisingly, the total time to complete hold requirements increased for the longer hold duration, as we detected a main effect of hold duration (F _{(1, 15)} = 61.69, p < .0001). The total time to complete holds decreased in sessions with higher press requirements (F _{(6, 90)} = 2.789, p = .0156). No hold duration by press requirement interaction was detected (data not shown).

To determine if work requirement influences latency to start working, we measured the amount of time that elapsed from when the lever was extended to start the trial until the time the subject made the first press, whether the trial was successfully completed or not. In the forced press trials, the latency to begin working increased as a function of the ratio requirement (Fig 2D), as indicated by a main effect of the press requirement (F_{(6, 90)} = 21.39, p < .0001). The hold requirement being implemented in the same session had no impact on latencies in the press trials, (F_{(1, 15)} = 2.47, p = 0.14). In the forced hold trials, the latency to begin working increased as a function of the hold requirement (Fig 2E), as indicated by a main effect of the hold requirement (F_{(1, 15)} = 14.45, p= 0.002). The press requirement being implemented in the same session had no impact on latencies in the hold trials, (F_{(6, 90)} = 1.099, p = 0.37).

To determine if press requirement influenced work vigor, we measured the number of presses made per second in the period from the first press made to the last press made in successfully completed forced trials with press requirements from 5 to 80 presses as some mice failed to complete any forced trials with a press requirement of 160. There was a main effect of press requirement on vigor, (F_{(4, 60)} =15.97, p < 0.0001), (Fig 2F).When the press requirement was greater than 20, vigor declined with further increases in the press requirement, a possible effect of the decreased reward rate or fatigue. The hold requirement being implemented in the same session also impacted vigor during the press trials (F_{(1, 15)} = 15.98, p = 0.001), indicating that longer hold requirements increased the vigor of pressing. No press requirement X hold duration interaction was detected.

To determine if work requirement influences efficiency in performing the hold trials, we divided the hold duration requirement by the time from first lever hold to hold completion and subsequent reward delivery (i.e. a single hold attempt would equal an efficiency of 1.0, if multiple hold attempts were made to complete a trial, efficiency <1.0). Efficiency was negatively affected by increases in both hold duration (F_{(1, 15)} = 20.70, p = 0.0004, Fig 2G) and the press requirement implemented on the alternate lever in the same session, (F_{(6, 90)} = 4.69, p = 0.0003). No press requirement X hold duration interaction was detected.

The Dopamine D2R Antagonist Haloperidol Affects Choice Behavior in the Concurrent Effort Choice (CEC) task.

To investigate the role of dopamine signaling via D2 receptors on effort-based choice, we injected mice with haloperidol before testing in the CEC task with a fixed 10 second hold requirement and press requirements of 10, 20, 40 and 80 presses. The proportion of hold choices increased with haloperidol (Fig 3A). There was a main effect of drug on the proportion of rewards earned employing the hold lever (F _(2,30) = 7.71; p = 0.002), a main effect of ratio requirement (F _{(3, 45)} = 26.16; p < 0.0001), but no significant drug by ratio interaction (F _(6,90) = 0.2.096; p = 0.0614). A value for PSE could be calculated for 13 of the 16 treated subjects. A one-way repeated measures ANOVA of PSE values revealed a significant overall effect of treatment (F(2,24) = 11.21, p = 0.0004 ) (Fig 3B). A Sidak’s multiple comparisons test revealed differences between treatment with vehicle and each of the two doses of haloperidol (0.1 mg/kg p = 0.0011, 0.2 mg/kg p = 0.0006; Fig 3C.

Figure 3.

Haloperidol effects in the CEC task. (A). proportion of hold choices in the CEC task following treatment with 0.1mg/kg and 0.2mg/kg haloperdiol. (B). Group point of subjective equality (PSE) for subjects under each drug condition. (C) PSE values for each individual subject in each condition. (D). Latency to the first lever press in single lever press trials. (E). Time taken to complete the single lever press trials. (F). Latency to the first lever press in single lever hold trials. (G) Time taken to complete the single lever hold trials. All values represented are mean ± (SEM), except for (C), which depicts individual subject values. n = 16; The significance of main effects and post-hoc multiple comparisons are depicted: **p < .01; ***p < .001; ****p < .0001.

In addition to looking at the choice trials, we also examined the effects of haloperidol on the time taken to respond in the forced single lever trials at the beginning of the CEC sessions. In the press trials, Haloperidol increased the latency to begin working (Fig 3D) as a mixed effects analysis revealed a main effect of drug (F _{(2, 30)} = 29.70, p < 0.0001). There was also an expected main effect of ratio (F _{(3, 45)} = 10.63, p < 0.0001) but no drug x ratio interaction. To determine if Haloperidol also increased how long it took subjects to complete press trials once they were begun, we calculated Press Completion Times (time from first lever press to trial completion and subsequent immediate reward delivery). This analysis was restricted to vehicle and 0.1mg/kg Hal treatment because several mice did not complete any press trials requiring 40 or 80 presses when treated with 0.2mg/kg Hal., (Fig 3E). 0.1mg/kg Hal increased press completion times as there was a main effect of drug (F _{(1, 15)} = 56.51, p < 0.0001). There was also a main effect of ratio (F _{(3, 45)} = 60.11, p < 0.0001), and a drug x press requirement interaction (F _{(3, 38)} = 24.58, p = 0.0001). Haloperidol did not alter the timing of responses in hold trials in the same way as it did press trials. The latency to begin working on 10s single lever hold trials was increased by haloperidol (Fig 3F), as it was in press trials, main effect of drug (F _{(2, 30)} = 20.03, p < 0.0001). Hold latencies were also increased in sessions in which the press trials had higher press requirements, (F _{(3, 45)} = 7.96, p = 0.0002) and there was a drug x ratio interaction (F _(6,78) = 2.65, p = 0.022). However, Haloperidol did not increase hold completion times (time from first lever depression to hold trial completion and subsequent immediate reward delivery). Instead, there was a trend for Haloperidol treatment to decrease the time in which mice completed hold trials, (F _{(2, 28)} = 2.70, p = 0.084), (Fig 3G). The requirement on the press lever in the same session had no effect on work hold times (F _{(3, 42)} = 0.35, p = 0.68). These result show that haloperidol increases latency to start working on all trials regardless of the type of work. This increase in latency to initiate a lever press makes subjects less time efficient on trials that require multiple press initiations (increased work time for press trials) but has the potential to make them more time efficient on trials that require single press initiations with longer intervals in between presses (hold trials).

Given the increase in latency and completion times observed after treatment with haloperidol, we analyzed the number of forced trials that were failed due to incompletion within the 3 minute time limitation that was imposed on forced trials. There was an effect of drug treatment (F _{(2, 30)} = 82.44, p <0.0001), press requirement (F _{(3, 45)} = 28.6, p <0.0001), and a press requirement by treatment interaction (F _{(6, 90)} = 15.20, p <0.0001), supplemental Fig S4A. Choice trials were not time limited, but subjects could miss choice trial opportunities if the session timed out at one hour before they had earned the maximum number of rewards (50 forced and choice combined). The number of choice trials completed was decreased by haloperidol. There was an effect of treatment (F _{(2, 30)} = 94.7, p <0.0001), press requirement (F _{(3, 45)} = 10.08, p <0.0001), and a treatment by press requirement interaction (F _{(6, 90)} = 12.11, p <0.0001), supplemental Fig S4B. The number of failed forced trials and missed choice trials did not influence the values presented in figure 3. Choice functions are derived from the proportion of completed trials of a specific trial type (hold trials) and the latency and completion times reflect average values from completed trials only.

Measuring value-related choice using the Concurrent Value Choice (CVC) task.

Mice were trained to earn liquid sucrose solutions by pressing on one lever and to earn sucrose pellets by pressing on the opposite lever (Fig 4A). All subjects learned to press for these different outcomes through a series of increasingly demanding random ratio schedules (RR5, RR10, and RR20), (see Fig 4B). In the CVC choice trials, subjects were able to work for either liquid sucrose, which always required 5 lever presses, or they could work for a sucrose pellet which required 10, 20, 40, or 80 presses on consecutive days. At the start of each session, subjects were exposed to 10 single lever forced trials in which either the sucrose lever or the pellet lever was extended. This was done so subjects would learn the cost of the two options each day. Subjects were then given choice trials in which both of the levers were extended (Fig 4C). To determine the impact of reward value on subject’s choice behavior, the task was run under two reward value conditions, 05% and 20% sucrose solutions. The progression of different conditions over days is shown in Fig 4D.

Subjects are Sensitive to Sucrose Concentration in a Value Choice Task

To examine subject’s choice behavior, we computed the proportion of liquid sucrose choices during the 20 choice trials for 5 and 20% sucrose solution. Choice behavior was parametrically modulated by the response cost for pellets (task ratio); F _{(3, 45)} = 87.71, p < 0.0001) and this cost-choice function was shifted for the two different sucrose concentration (F _{(1, 15)} = 64.16, p < 0.0001) (fig 5A). There was also an interaction between sucrose concentration and pellet cost (F _{(3, 45)} = 9.04, p < 0.0001).

Figure 5.

Reward quality alters value-based choice in the CVC task. (A) proportion of hold choices in the CVC task in 2 different sucrose concentration conditions. (B) point of subjective equality (PSE) for subjects in the 2 different sucrose concentration conditions. (C) individual subject PSE’s. (D). Latency to first press in single lever sucrose trials. (E). Time to complete single lever sucrose trials. (F). Latency to first press in single lever pellet trials. (G). Time to complete single lever pellet trials. All values represented are mean ± (SEM), except for (C), which depicts individual subject values. n = 16; The significance of main effects and post-hoc multiple comparisons are depicted: * p < 0.05; **p < .01; ***p < .001; ****p < .0001.

The impact of sucrose concentration on choice behavior was also reflected in the PSE for each subject, which reflects the point at which subjects are choosing both the sucrose reward and the pellet reward with equal probability. Changing the sucrose concentration led to a significant difference in the PSE (Fig 5B) as the 5% sucrose PSE was significantly higher than the 20% sucrose PSE (t ₍₁₅₎ = 7.95, p < 0.0001), and 15 out of 16 individual subjects exhibited this change in PSE (Fig 5C) and in their choice functions (Supplemental Fig S5).

Sucrose Concentration Influences both Latency to Work and how Fast Subjects Work

We next determined if other aspects of behavior were modulated by the concentration of the available sucrose reward. Since subjects completed different numbers of requirements on the different levers during the choice trials we analyzed the first 10 single lever forced trials, as subjects completed equivalent numbers of these trial types. As for the CEC task data, we examined latency to begin responding, as well as the time taken to complete trials. Sucrose concentration influenced latency to begin pressing on the forced, single lever sucrose trials as there was a main effect of sucrose concentration on latency (F _{(1, 15)} = 4.66, p = 0.047,Fig 5D). There was also a main effect on latency of the response cost (presses) for the pellets in the alternate forced choice single lever trials within the same sessions (F _{(3, 45)} = 4.04, p = 0.013), and a pellet cost × sucrose concentration interaction (F _{(3, 45)} = 3.26, p = 0.0301). The sucrose concentration also impacted the time to complete forced single lever sucrose trials (Fig 5E), (F _{(1, 15)} = 11.5, p = 0.004), therefore, even though the sucrose only ever required 5 presses, subjects were slower to complete those 5 presses when it was for 5% sucrose compared to 20% sucrose.

We also analyzed latencies and completion times on the forced single lever pellet trials. There was a trend for latency to press to be increased with increased pellet cost (Presses), (F _{(3, 45)} = 2.51, p = 0.07, Fig 5F). The latency to press on the pellet trials was increased in session in which the sucrose lever yielded 20% compared to 5% sucrose, reflecting the subject’s relative motivation for the pellet lever in those sessions (F _{(1, 15)} = 5.20, p = 0.0377). There was also a sucrose concentration x pellet cost interaction ((F _{(3, 45)} = 2.95, p = 0.043). As expected, the time taken to complete pellet trials increased with pellet cost (presses), (F _{(3, 55)} = 77.99, p < 0.0001, Fig 6G). Pellet trial completion time was also increased in sessions in which the sucrose lever yielded the higher sucrose concentration (F _{(1, 15)} = 11.5, p = 0.004), and there was a pellet cost x sucrose concentration interaction (F _{(3, 45)} = 3.33, p = 0.028).

Figure 6.

Reward devaluation in the CVC task. (A) proportion of hold choices in the CVC task for the different devaluation conditions. (B) group point of subjective equality (PSE) in the different devaluation conditions. (C) PSE values for individual subjects in the different devaluation conditions. (D) Latency to first press on single lever pellet trials. (E) Time to complete single lever pellet trials. All values represented are mean ± (SEM), except for (C), which depicts individual subject values. n = 16; The significance of main effects and post-hoc multiple comparisons are depicted: * p < 0.05; **p < .01; ***p < .001; ****p < .0001.

Reward Devaluation Impacts Value Based Choice in the CVC Task

In the CVC task, the quality of reward (concentration of sucrose) alters choice behavior, latency to begin working and the vigor with which subjects work. To determine if behavior in the CVC task is sensitive to changes in the current value of reward, we asymmetrically devalued one of the response outcomes. In this manipulation, subjects experienced one week of normal testing in the CVC task (Valued-1), followed by a week in which the pellet reward was devalued by 30 minutes of free access to pellets prior to the session (Devalued), followed by a second normal week of testing in the CVC task (Valued-2). When the pellets were devalued, the proportion of sucrose lever choices dramatically increased (Fig 6A), and an ANOVA detected significant main effects of devaluation condition (F_{(2 30)} = 119.4, p < 0.0001) and pellet cost (F_{(3, 45)} = 107.9, p < 0.0001), and a pellet cost by devaluation interaction (F_{(6, 90)} = 15.49, p < 0.0001).

The PSE was also significantly impacted by the devaluation condition (F_{(2 30)} = 122.3, p < 0.0001; Fig 6B), indicating that subjects were willing to pay a smaller cost for pellets when they had been devalued. This was true for 100% of subjects, as indicated in fig 6C and by their individual overall choice functions presented in Supplemental Fig S6.

Reward Devaluation Modulates Latency and Vigor of Work

In addition to altering the choice function, devaluing the pellet also altered response times. The latency to begin working for pellets was increased in the devaluation condition (Fig 6D), as an ANOVA detected a main effect of devaluation (F _{(2, 30)} = 16.21, p < 0.0001). There was also a main effect of pellet cost (F _{(3, 45)} = 4.65, p = 0.0065), and a significant devaluation × pellet cost interaction (F _{(6, 90)} = 7.08, p < 0.0001). The time to complete pellet trials (Fig 6E) was increased by devaluation (F _{(2, 30)} = 8.405, p = 0.0013) and by increased pellet cost (F _{(3, 45)} = 70.25, p < 0.0001) but there was no value condition x pellet cost interaction.

The Dopamine D2R Antagonist Haloperidol Does Not Impair Value-Related Choice Behavior in the Concurrent Value Choice (CVC) Task.

Using the CEC task, we showed that D2R blockade with the selective antagonist Haloperidol altered effort-based decision making and effort-related response times. We next implemented the CVC task to determine if D2R blockade would also impact value-based choice and value-related response times. Because of the high number of failed and missed trials in the CEC task after treatment with 0.2mg/kg of haloperidol (Fig S4), we used only the lower dose of haloperidol in this experiment. Subjects were treated with either vehicle or 0.1mg/kg haloperidol and tested under 5% and 20% sucrose concentration conditions across the different pellet cost schedules and the data analyzed by 3-way ANOVA. Haloperidol led to a change in the overall proportion of sucrose choices made (F _(1,15) = 29.58, p<0.0001 Fig 7A). However, the effect of Haloperidol on Sucrose choice was not related to reward value because while there was a significant ratio by sucrose concentration interaction (F _{(3, 45)} 6.96, P=0.0006), there was not a significant ratio by drug interaction (F _{(3, 45)} = 1.87, P=0.15), a sucrose concentration by drug (F _{(1, 15)} = 0.36, P=0.56), or a ratio by sucrose concertation by drug interaction (F _{(3, 45)} = 1.61, P=0.2). A lack of effect of haloperidol on reward-based choice was also reflected in the point of subjective equality between treatments with vehicle and 0.1 mg/kg Haloperidol across 5% and 20% sucrose concentrations (Fig 7B). There was a significant main effect of sucrose concentration (F _(1,15) = 97.45; p < 0.0001), and a significant main effect of drug (F _(1,15) = 0.26.7; p < 0.0001), but no significant drug by sucrose concentration interaction (F _(1,15) = 0.15; p = 0.70; Fig 7B). While Haldol shifted mice to choose the sucrose option at lower response requirements for the pellets, the difference between 5% and 20% sucrose was identical with and without Haldol.

Figure 7.

Haloperidol effects in the CVC task. (A). Shows the mean ± (SEM) proportion of hold choices in the 5% and 20% sucrose solution conditions after treatment with either vehicle (left) or 0.1mg/kg Haloperidol (right). (B) point of subjective equality (PSE). (C) latency to first press in the single lever sucrose trials for each sucrose concentration condition. (D). Time to complete single lever sucrose trials for each sucrose concentration condition. (E) latency to first press in the single lever pellet trials (F) Time to complete single lever pellet trials. All values represented are mean ± (SEM). n = 16; To avoid crowding the multi-part dense figures, the significance of main effects and post-hoc multiple comparisons are provided only in the text for all components except (B): ****p < .0001.

In addition to looking at the choice trials, we also examined the effects of haloperidol on response times in the single lever trials. Haloperidol resulted in increased latencies to begin working in the sucrose trials (Fig 7C), but this was not due to a change in response to reward value because while there were main effects of drug (F _{(1, 15)} = 82.36, p < 0.0001), sucrose concentration (F _{(1, 15)} = 23.6, p = 0.0002) and ratio (F _{(3, 45)} = 18.07, p < 0.0001) but there was no drug by sucrose concentration (p = 0.79) or drug by sucrose concentration by pellet cost interactions (p = 0.81). Haloperidol did not impact the time it took to complete single lever sucrose trials, which always required only 5 presses (Fig 7D). There was a trend for a main effect of sucrose concentration (F _{(1, 15)} = 4.06, p = 0.062), and no main effects of drug (F _{(1, 15)} = 0.094, p = 0.76), or pellet cost on the press trials in the same session, (F _{(3, 45)} = 0.51, p = 0.68). There was no drug by sucrose concentration (p = 0.14) or drug by sucrose concentration by pellet cost interactions (p = 0.36).

Haloperidol also increased the latency to start working on the single lever pellet trials (Fig 7E), there were main effects of drug (F _{(1, 15)} = 58.56, p < 0.0001) and pellet cost (F _{(3, 45)} = 21.0, p < 0.0001), but no main effect of sucrose concentration (F _{(1, 15)} = 0.79, p = 0.39). Unlike for the sucrose work times, (which involved only 5 lever presses), Haloperidol increased how long it took subjects to complete the pellet trials (Fig 7F), especially in the ratios with higher press requirements. There was a main effect of pellet cost (F _{(3, 45)} = 146.5, p < 0.0001), and a main effect of drug (F _{(1, 15)} = 32.98, p < 0.0001) as well as a pellet cost by drug interaction (F _{(3, 45)} = 15.59, p < 0.0001). In summary, haloperidol treatment did not alter subject’s sensitivity to reward quality. Sucrose choice, the latency to begin work and the vigor of work on sucrose trials were all modulated by sucrose concentration after Haloperidol treatment, as they were after vehicle treatment. All of the effects of Haloperidol were related to the demands of responding, independent of the value of the reward. This finding is congruent with the Haloperidol affecting effort-based choice and the effort-based response timing in the CEC task.

Discussion

In order to perform cost-benefit computations animals must be able to (1) estimate the effort that will be required to obtain a particular goal, (2) estimate the value or benefit of the goal to be obtained, and (3) compare the anticipated effort against the estimated value (Rangel & Hare, 2010). Because we cannot directly measure a subject’s experience of how effortful a task was, or how rewarding an outcome was, we must rely on behavioral readouts to infer these things in non-verbal organisms. In the present set of experiments, we have developed and characterized two behavioral tasks known as the Concurrent Effort Choice (CEC) task, and the Concurrent Value Choice (CVC) task to isolate the independent effects of effort and value in cost-benefit decision making.

The Concurrent Effort Choice Task Measures Subject’s Sensitivity to Effort Changes Independent of Changes in Reward Value.

In the CEC task, subjects made the choice between lever pressing to earn a milk reward, or holding a lever down for a required duration of time to earn the identical milk reward. We previously demonstrated that effort manipulations of hold duration are sensitive to the same motivational factors as the more traditionally used effort manipulation of number of responses (Bailey et al., 2015). In the choice trials of the CEC task, the choice of which work requirement is chosen is sensitive to both the effort manipulation of number of responses and duration of hold time. When choosing between pressing and holding for either 5 or 10 seconds, all subjects choose to make lever presses when the requirement is low and switch over to holding as the press requirement increases. As the press requirement increases subject become more likely to choose to hold for rewards. This sigmoidal function indicates that subjects are sensitive to the changes in effort as the number of responses changes. The fact that the entire choice function shows a rightward shift when comparing choice between the 5 and 10 second hold requirement indicates that subjects are also sensitive to the effort manipulation of holding time. We further demonstrated this by showing that subjects point of subjective equality, or the number of presses for which they choose to hold and press with equal probability, shifts as the hold requirement is increased from 5 to 10 seconds.

Because subjects chose between pressing and holding, the CEC task can be used to detect differences in sensitivity to either type of work, as well as relative preference for these two types of work. Lever pressing involves the repeated initiation of actions whereas lever holding requires sustaining a single action. These two distinct forms of motor activity are both regulated by the basal ganglia where manipulations can bias for one type of behavioral output over the other. For example, the psychostimulant amphetamine, which increases striatal dopamine release, drives hyperlocomotion, increases action initiation and reduces response durations (Bailey et al., 2015). Systemic pharmacology studies suggest that amphetamine-induced hyperlocomotion could be mediated via either dopamine D1 or D2 receptors (O’Neill & Shaw, 1999). However, local antagonism of D2 receptors on the striatopallidal output neurons can decrease the likelihood of initiating responses and increase the duration of single responses in operant tasks, including lever presses and head pokes (Fowler & Liou, 1994; Nicola, 2010). Therefore, the CEC task could be used to determine potentially different roles that the striatopallidal and striatonigral circuits may play in behavioral control.

In addition to Choice, the CEC task reveals the impact of effort judgment through changes in the timing of responding

In addition to the differences in subject’s choice behavior in the CEC task we were also able to identify two more measures which were sensitive to both number and time manipulations of effort. First, we observed that the latency to begin working was modulated by the effort requirement, as this went up both for the manipulation of number of presses as well as for the hold duration. Increase in latency as a function of the work requirement therefore provides another measure which reflects subject’s sensitivity to different effort requirements, an observation which has been made in previous studies (Capehart, Eckerman, Guilkey, & Shull, 1980; P. Killeen, 1969).

Manipulations of effort also affected the vigor or efficiency with which subjects completed trials. The time required to complete a trial will necessarily increase with the number of presses or duration of lever hold required. However, we also found changes in vigor and/or the time to complete trials that depended on the effort requirement of the alternate response. In press trials, the rate of pressing was affected by the duration of the requirement on hold trials. In hold trials, the time to complete the trial was affected by the response requirement on press trials. Therefore, the effort needed to complete a given kind of trial relative to the effort needed to successfully complete other types of trials affects the vigor and efficiency of a subject’s behavior as well as the choice between behaviors that require differential effort.

The Concurrent Value Choice task measures subject’s sensitivity to value changes

In the CVC task, subjects choose to work for either a liquid sucrose solution or a sucrose pellet. By altering the reward value of these two outcomes, either through titrating the sucrose concentration of the liquid sucrose solution, or by devaluing the pellet outcome with a pre-feeding procedure, we are able to see how these changes in reward value impact subject’s choice behavior. In the choice trials of the CVC task, subjects were sensitive to both sucrose concentration manipulation changes (5 vs 20% sucrose), as well as devaluation. We capture this shift in the value of these outcomes by looking at the psychometric functions and computing the point of subjective equality, defined as the number of presses required for pellets at which subjects will equally choose the pellets or the sucrose. When the sucrose concentration is higher, or the alternate pellet reward has been devalued, the point of subjective equality is shifted to the left compared to its value when the sucrose concentration is lower, or the alternate reward is more valued.

Reward Value Also Affects the Temporal Dynamics of Responding.

As in the CEC task, manipulations that affected choice also affected the timing of responding. Subject’s worked at a faster rate for 20% sucrose as compared to 5% sucrose. They also began working for the 20% sucrose sooner than 5% sucrose. We also observed a similar sensitivity to reward value through the pellet devaluation manipulation, as subjects worked faster for pellets and had shorter latencies when they were valued, compared to when the pellet was devalued. These two manipulations (sucrose concentration and devaluation) both reflect that the rate and likelihood of starting to engage in goal-directed action are modulated by reward value, and that the CVC task can measure this, without the confound of differences in effort requirements for the different reward values. Importantly, these two measures of behavior directly demonstrate that subjects are able to use expectations about reward value to guide their behavior, working faster for the higher sucrose concentration. It is important to distinguish this from responding at a faster rate while engaged in the consumption of a reward. It has been clearly demonstrated that subjects will increase their rates of licking for higher sucrose concentrations when the liquids are readily available to subjects (Glendinning et al., 2002). While lick rates in such tasks show that reward magnitude can modulate rates of consumption, the data in the CVC task importantly demonstrates that reward quality (sucrose concentration), in a valued state, guides the vigor of goal-directed behavior in anticipation of reward.

Advantages of CEC and CVC Tasks over previously used Methods

As Winstanely & Floresco point out, paying close attention to small experimental details is necessary to fully appreciate exactly what a behavior task can really teach us (Winstanley & Floresco, 2016). The shift in choice of the press lever when press requirement changes demonstrates subjects are sensitive to effort manipulations of number of responses is similar to that observed in an operant effort discounting task (Floresco, Tse, et al., 2008). In the operant effort discounting task, subjects can make a large number of responses (2, 5, 10, or 20) for 4 pellets, or they can make a single lever press for 2 pellets. A manipulation which causes a change in choice behavior in this type of task could be due to due to changes in (1) estimate of effort (2, 5, 10, or 20 vs 1 press), (2) estimate of value or benefit of the goal (4 pellets vs 2 pellets), or (3) the comparison of the anticipated effort against the anticipated value. The same is true for effort-based choice procedures (J D Salamone et al., 1991) in which animals are given a choice between making an effortful response for a preferred reward versus a less effortful response for a less preferred reward. In the CEC task, reward value does not differ for the two different work options of lever pressing or lever holding within a session, and does not differ for the different amount of lever presses required across days. Therefore, change in the behavior in the CEC task, can only result from a change in (1) the estimate of effort (evaluating press number relative to hold duration), or (3) the comparison of the anticipated effort against the estimated value, but could not be explained by (2) estimate of value or benefit of the goal, as both options offer the same reward.

Similarly, the CVC task offers advantages over some of the previously used measures of value or reward sensitivity. One such method which is known as a Preference Assessment gives a subject free access to either a liquid sucrose solution or water (Muscat & Willner, 1989). This test uses a subject’s development of a preference for the liquid sucrose over water to reflect the capacity to experience the hedonic pleasure associated and reward sensitivity (Ward et al., 2015). While this method does reflect a subject’s ability to make choices based on hedonic reaction and post-ingestional consequences associated with the flavor, it does not explicitly require subjects to use the reward value estimations to make decisions about how much effort they are willing to expend to gain access to that reward. The CVC task requires subjects to use information about reward value to guide on-going, goal-directed responding at different effort requirements.

The CEC and CVC tasks can be used to measure multiple behavioral effects of perturbations on value-based and effort-based decision making independently.

Our studies also emphasize the point that the dynamics of responding are affected by changes in both effort and value. Latencies, vigor and efficiency are affected by both changes in value and changes in effort requirements. Here we demonstrate that the CVC and CEC tasks can be used to identify the specific aspect of cost-benefit decision making that is altered. We found that administration of the D2R antagonist haloperidol selectively altered effort-based decision making while leaving the value-based decision making intact.

We tested Haloperidol because it has previously been reported that haloperidol treatment biases rodents away from actions that require a high level of effort to earn high value rewards and shifts their choice toward low-effort actions that earn less valued rewards (Bardgett, Depenbrock, Downs, Points, & Green, 2009; Farrar et al., 2007; John D Salamone & Correa, 2009). We therefore sought to determine if this shift in preference was driven by changes in reaction to the effort requirement or the reward value. Our finding that haloperidol altered Effort-based, but not Value-based decision-making is highly consistent with a large body of work demonstrating that disruptions to dopamine transmission that alter effort-based choice do not disrupt primary food motivation, i.e. consumption behavior in conditions that don’t involve effort-choice. Doses of haloperidol or the DA depleting agent tetrabenazine that altered effort-based choice did not alter preference for or consumption of the different foods used in the operant choice tasks (Nunes, Randall, Podurgiel, Correa, & Salamone, 2013; Marta Pardo, López-Cruz, San Miguel, Salamone, & Correa, 2015; J D Salamone et al., 1991). Similarly, in rats and mice tested on T-maze barrier choice tasks, doses of DA antagonists or depleting agents that produced a low effort bias did not alter discrimination of reinforcer quantity in the different arms of the maze when there was no barrier, or barriers to both arms (M Pardo et al., 2012; J D Salamone et al., 1994; Yohn et al., 2015). Commonly, the present study and the work described above show that DA perturbations that alter effort based-choice are not due to changes in reaction to food reinforcement, including food choice.

In addition to providing congruence with earlier studies, our assays allowed us to make a distinction between the effect of D2 antagonism on work initiation versus work maintenance. Haloperidol increased the preference for making a single long duration response over making repeated short duration responses while having no impact on the preference for 20% over 5% liquid sucrose reward. Our results show that the impact of D2R manipulations on cost-benefit decision making is driven by effort -based rather than value-based factors, consistent with our recently published study involving genetic manipulation of D2 receptors (Filla & Bailey et al., 2018).

Haloperidol also increased the latency to begin working on lever presses and hold trials. This slowing of response initiation coupled with increases in the amount of time it took subjects to complete a given press requirement resulted in substantial increases in the total time to complete press trials after haloperidol administration. In contrast, haloperidol decreased the tendency to reinitiate (failed attempts) during hold trials resulting in a trend for a decrease in the amount of time taken to complete hold trials. These findings are consistent with previous experiments showing that D2R antagonism increases response duration, but not response number (Fowler & Liou, 1994; Nicola, 2010).

Conclusions and Future Directions

There is a large interest in understanding how the central nervous system processes information about effort and value, and how it is able to use this information to enable subjects to engage in cost-benefit decision making. Human neuroimaging studies are attempting to identify brain circuits involved in these processes in humans (Croxson et al., 2009; Prévost, Pessiglione, Météreau, Cléry-Melin, & Dreher, 2010), clinical researchers and psychiatrists are attempting to understand how symptoms from a number of different disorders may be manifesting as a result of impairments in decision making (Fervaha, Foussias, Agid, & Remington, 2013; Fervaha, Graff-Guerrero, et al., 2013; Gold, Waltz, & Frank, 2015), and basic neurobiology researchers are aiming to further elucidate and dissect the circuits involved in this type of information processing (Hillman & Bilkey, 2010; Wang, Shi, & Li, 2017). The commonality across all of these research endeavors is that one must ultimately rely on some type of behavioral readout to draw inferences about how a human participant or experimental model organism is processing information about costs and reward values and subsequently using this information in a cost-benefit decision-making computation. We developed the CEC and CVC tasks with the hope that using these or similar methods will ultimately aid in more precise investigation of the behavioral, neural, and pharmacological mechanisms which influence cost-benefit decision making. Here we applied our behavioral assays to a systemic pharmacological manipulation. The same approach could be applied to measure the effects of transgene driven brain-wide or circuit-specific manipulations, lesions, DREADDs, optogenetics etc., The approach can also be taken to measure the relationship between effort and reward anticipation on circuit, cellular and neurochemical activity using in vivo imaging, electrophysiological or electrochemical techniques. All such neuroscience techniques either depend upon, or greatly benefit from being able to test subjects repeatedly, before and after manipulations. This can be done with the CEC and CVC tasks. Knowledge gained in experimental organisms may help to better understand and more effectively treat human disorders for which similar cost-benefit decision making processes have gone awry.