Rats value time differently on equivalent foraging and delay-discounting tasks

Abstract

All organisms have to consider consequences that vary through time. Theories explaining how animals handle intertemporal choice include delay-discounting models, in which the value of future rewards is discounted by the delay until receipt, and foraging models, which predict that decision-makers maximize rate of reward. We measured the behavior of rats on a two-option delay-discounting task and a stay/go foraging task that were equivalent for rate of reward and physical demand. Despite the highly shared features of the tasks, rats were willing to wait much longer on the foraging task than on the delay-discounting task. Moreover, choice performance by rats was less optimal in terms of total reward received on the foraging task as compared to the delay-discounting task. We applied a suite of intertemporal choice models to the data, but found that we needed a novel model incorporating interactions of decision-making systems to successfully explain behavior. Our findings highlight the importance of factors that historically have been seen as irrelevant, and indicate the inadequacy of current general theories of intertemporal choice.

Introduction

All organisms face problems of intertemporal choice, and managing tradeoffs between immediate and future consequences is a critical consideration across nearly all domains. A woodpecker, for example, must allocate search time between trees when foraging, and an aspiring homeowner must balance saving for a house and investing in retirement. For humans, dysfunction of intertemporal choice is correlated with economic irrationalities (Plous, 1993) and psychiatric disorders, like obesity, depression, and addiction (Bickel, Jarmolowicz, Mueller, Koffarnus, & Gatchalian, 2010). Thus, intertemporal choice refers to a general class of decision-making problems, many of which have important consequences. Here, we tested whether models of intertemporal choice could account for such generality. To do so, we focused on behavior by rats on two intertemporal choice problems from separate traditions: Time management during foraging—a topic in behavioral ecology—and the two-choice delay-discounting paradigm typical of psychology, neuroscience, and economics. We operationalized these problems to be as similar as possible, yet subjects valued time very differently in the two tasks: Rats were willing to wait more than three times longer on the foraging task as compared to the delay-discounting task. Moreover, purportedly general models of intertemporal choice did not predict this difference.

Intertemporal choice is frequently conceptualized in terms of delay discounting—the long-standing and influential idea that the impact of future consequences on current choice is discounted, so that the value of a future reward decreases as a function of the delay until its receipt (Madden & Bickel, 2010). These models, such as the hyperbolic discounting model, are frequently excellent descriptions of how subjects choose between two mutually exclusive options, one of which is worth more but becomes available later in time than the other (Kirby & Maraković, 1995; Mazur & Biondi, 2009).

Foraging theory approaches intertemporal choice from the normative assumption that caloric intake is linked to evolutionary fitness, so organisms ought to maximize the rate of food intake during foraging (Stephens & Krebs, 1986). Foraging problems tend to involve choice between accepting or rejecting encountered prey, or staying or leaving a site—usually called a patch—that provides food. Foraging models have successfully accounted for a range of data, from choice by insects, birds, rodents, and primates, including humans (Stephens & Krebs, 1986; Nonancs, 2001; Carter, Pedersen, & McCullough, 2015), to behaviors that are fundamentally similar to foraging, such as search through memory or visual displays (Hills, Jones, & Todd, 2012; Wolfe, 2013).

Despite their shared focus, some of the canonical findings from delay discounting and foraging literatures are contradictory. For example, discounting models are almost exclusively concerned with the delay until receipt of reward (the pre-reinforcer delay), and the non-human animal literature clearly suggests that other timing components are essentially ignored (Mazur, Snyderman, & Coe, 1985). In contrast, foraging models consider every moment spent foraging, including travel time between food sources, to be determinants of choice, and the foraging literature generally finds this to be true (Stephens & Krebs, 1986; Nonacs, 2001). Additionally, delay-discounting experiments typically find that non-human animals are impatient, so that delaying a reward by a single second might cause an animal to treat that reward as if it has lost the majority of its value (Stephens, 2002). Such impatience has been characterized as impulsivity or as being irrational given that it can lead to less reward received overall (Logue, Smith, & Rachlin, 1985). However, many animals are clearly sensitive to the future—as in caching behavior (Dally, Emery, & Clayton, 2006)—and the fact that normative foraging models have had such success flies in the face of the conclusion that animals cannot effectively manage trade-offs between immediate and future gains.

Inspired by previous work that has compared economically-matched foraging and delay-discounting tasks (Stephens & Anderson, 2001), or that has tested whether behavior on delay-discounting tasks might generalize to foraging tasks (Blanchard & Hayden, 2015), we devised two tasks that allowed us to compare intertemporal choice by rats making delay-discounting and foraging decisions. These tasks were matched in terms of physical features (method of reward delivery, physical demand) and outcomes (reward amounts and delays). This design allowed us to test quantitative predictions from purportedly general models of intertemporal choice. The tasks are the Spatial Adjusting Delay-Discounting [DD] (Papale, Stott, Powell, Regier, & Redish, 2012) and the Patch task (Figure 1).

Figure 1.

(A) DD task flow, (B) maze, and (C) example sessions in terms of the adjusting delay as a function of lap for a single animal faced with different starting delays (C). (D) Patch task flow, (E) maze, and (D) example sessions for the same animal as in C.

Materials and Methods

The DD task (Figure 1A) entailed the rat running through a central track then turning left or right towards a feeder zone (Figure 1B). One side provided a smaller-sooner reward (two 45mg food pellets after one second), while the other side provided a larger-later reward (six 45mg food pellets after D seconds). The delay D to the larger-later reward was adjusted by the rat's choices—if the rat chose the smaller-sooner reward, D was decreased by one second (to a minimum of two seconds), while if the rat chose the larger-later reward, D was increased by one second (no maximum). In practice, rats titrated D to a preferred delay and then alternated between the two options (Papale et al., 2012). Two example sessions from the same animal but with different starting delays are shown in Figure 1C.

The Patch task (Figure 1D) was run on the same maze as the DD task, but with one-side blocked off (Figure 1E). When the rat arrived in the feeder zone, two food pellets were delivered after one second (equivalent to the smaller-sooner reward for DD). If the rat stayed for an additional D-1 seconds, four more food pellets were delivered (so choosing to stay in the Patch task was worth six pellets in D seconds, identically to choosing larger-later in DD) and D was increased by one second. If the rat left the feeder zone before D-1 seconds, no additional reward was delivered for that lap, and D was decreased by one second (to a minimum of two seconds). This task includes two defining features of patch foraging: A leave/stay-type choice and decreasing rate of reward as the animal stays in the patch (2 pellets in one second for the first delivery is greater than 4 pellets in D-1 seconds as long as D was greater than three seconds). Behavior from two example sessions is shown in Figure 1F.

The two tasks were run on the same physical maze and the return paths were either direct (the “Easy” travel condition, 157cm travel, requiring an average of 6 seconds to run) or contained switchbacks (the “Hard” travel condition, 320cm travel, requiring an average of 10 seconds to run). This manipulation was of primary interest because, as mentioned, travel time is a critical component of foraging but almost totally ignored in delay-discounting. Maze variations were achieved by adding or removing walls. See supplemental material for more details.

Eight Brown-Norway rats (Harlan), aged 8-10 months, were maintained on a 12-hour light-dark cycle. Subjects were food deprived to no less than 80% of their free-feeding weight. Water was freely available in the home cage. After training (supplemental material), Rats completed 48 experimental sessions. Each session was a unique combination of task (Patch or DD), travel (easy or hard), side of adjusting-delay option (right or left), and starting value for the adjusting-delay (2, 5, 10, 15, 30, or 45 seconds). Starting delay was held constant for at least four days while the other factors varied randomly. Sample size was determined by a power analysis on a pilot experiment (supplemental material). All procedures complied with National Institutes of Health guidelines for animal care and were approved by the Institutional Animal Care and Use Committee at the University of Minnesota.

Analyses were conducted in R (R Development Core Team). To test for the effects of the task and travel manipulations, we calculated a matched-session difference score for the outcome of interest (e.g., for the effect of task, the outcome variable from a Patch session was subtracted from the corresponding DD session that matched it for rat, travel, side, and starting delay). We estimated marginal posterior distributions for each difference using hierarchical Markov chain Monte Carlo (MCMC) methods (Kruschke, 2015). The resulting posterior distributions are distributions of credibility over candidate values for the true difference, where the most credible value is given as the mode of that distribution. The values over which 95% of credibility is spread represent the highest density interval (HDI), or the candidate values for which we should have the most confidence. When zero is outside of the 95% HDI, one can take this as evidence that the true value can safely be considered non-zero (Kruschke, 2015). We also applied matched-sample Wilcoxon signed-rank tests as an alternative to the Bayesian difference-based comparisons.

To determine whether some promising discounting or foraging models could explain potential differences in time preference between the two tasks, we applied several models using Matlab (Mathworks). We examined two foraging models (Stephens & Anderson, 2001): long-term rate, which holds that decision-makers maximize reward rate based on all trial time, and time-to-reward, which specifies that decision-makers maximize reward rate based on only the pre-reinforcer delay. We also fit the hyperbolic and exponential discounting models (Madden & Bickel, 2010), which model subjective value as either a hyperbolic or exponential function of pre-reinforcer delay with discount parameters of k and γ, respectively. In testing these models, we assumed that rats defined the value of the options based on either the hyperbolic or the exponential model and then picked the highest-value option. We also tested two hybrid models. The heuristic model (Blanchard, Pearson, & Hayden, 2013) is the long-term rate model but set so that all non-pre-reinforcer delays are multiplied by u, which takes a value between zero and one (when u is one or zero the heuristic model is equivalent to the long-term rate and time-to-reward models, respectively). This model follows from evidence that deviations from long-term-rate-maximizing behavior is due to underestimating non-pre-reinforcer delays. Second, we fit the TIMERR model (Namboodiri, et al., 2014), which holds that decision-makers track the rate of reward over an integration window of T seconds into the past (all reward received within the previous T seconds over T) and then only pick options that increase that rate. See supplemental materials.

Results

If rats have time-based preference, the adjusting delay will reflect it: When one option is preferred over the other, the adjusting delay will move up or down as the rat chooses either the adjusting option or the fixed option, respectively; however, when the value of the two options are equivalent, the animal will alternate between them, thereby holding the adjusting delay constant. Work using the DD task (Papale et al., 2012) and our current data (supplementary material) indicate that rats spend the first part of a session titrating the adjusting-delay before settling into an alternation pattern near the end (e.g., Figures 1C and 1F). Such a pattern strongly suggests that the animals are aware of the delays associated with the options and have specific temporal preferences. We therefore took the average adjusting-delay for the final 20 laps of a session as the indicator of the rats’ time preferences (Papale et al., 2012). We examined four matched-session differences in this metric: First, we calculated the difference for Patch – DD sessions (matched for rat, side, starting delay, and travel condition). Second and third, we calculated the difference for matched Hard – Easy travel for Patch and DD sessions separately. Fourth, we calculated the difference of the second and third differences.

The rats titrated to very different delays on the matched Patch and DD sessions (Figure 2). Rats titrated to an average delay of 8 seconds (IQR=4, 11) on the DD task, but to an average of 31 seconds (IQR=28, 33) on the Patch task (modal difference = 23 seconds [95% HDI: 22, 24], Cohen's d=4.45 [95% HDI: 3.8, 5.3], p=2.2e-16). Behavior in both DD and Patch tasks were affected by the travel manipulation (Figure 2). Increasing travel for the DD task changed the average final adjusted delay from 6 seconds (IQR=[3, 8]) to 10 seconds (IQR=[6, 12]), modal difference = 3.4 seconds (95% HDI: [2.4, 4.5]), p=1.5e-6, whereas increasing travel for the Patch task increased the average final adjusted delay from 29 seconds (IQR=[26, 32]) to 32 seconds (IQR= 29, 34]), modal difference =2.4 seconds (95% HDI: [1.6, 3.3]), p=1.8e-8. Importantly, the effect of travel was similar between the two tasks, modal difference of differences = −1.1 seconds (95% HDI: −2.6 to 0.4), p=0.13.

Figure 2.

(A) Boxplots showing the average final adjusted delay for all Patch and DD sessions and (B) Patch and DD sessions split by travel. (C) The effects for task, travel, and their interaction represented by marginal posterior distributions.

We examined whether rats’ success at acquiring food differed as a function of task (Figure 3). We applied the same matched-pair-difference analyses described above to the number of 45mg food pellets received per session and found that rats tended to get more food during DD sessions as compared to Patch sessions (Figure 3A): Rats received 289.3 pellets (IQR=[256, 328]) on average on the DD task and 260.4 pellets (IQR=[236, 284]) on average on the Patch task (modal difference = −28.9 [95% HDI=−33.3, 24.4], p<2.2e-16). The number of pellets received was also affected by the travel manipulation (Figure 3B): Increasing travel for the DD task changed the average number of pellets received from 323.1 (IQR=[298, 348]) to 254.4 (IQR=[228, 278]), modal difference for DD = −68.9 (95% HDI: [−75.6, −62.2]), p<2.2e-16, whereas increasing travel for the Patch task decreased the number of pellets received from 282.1 (IQR = [257.5, 300.5]) to 239.4 (IQR = [225.5, 256.5]), mode of the hard – easy difference for Patch = −42.2 (95% HDI: [−46.7, −37.8]), p < 2.2e-16. Moreover, the effect of the travel manipulation differed for the two tasks, such that the travel manipulation decreased the number of pellets more so during DD sessions as compared to during Patch sessions, modal difference of differences = 26.7 (95% HDI: 20, 33.3), p=1.01e-9.

Figure 3.

Boxplots showing the number of pellets received for all Patch and DD sessions (A) and Patch and DD sessions split by travel (B). The effects for task, travel, and their interaction are represented by marginal posterior distributions (C).

To test for possible explanations for the effect of task on time preference, we fit the data with seven models (supplemental materials): long-term rate, time-to-reward, hyperbolic discounting, exponential discounting, the heuristic model, TIMERR, and a novel added-value model. Theoretically, behavior by each rat ought to be described by a single parameter, so we first assessed fit in terms of mean squared error (MSE) after finding the best-fit parameter for each rat for each model. Notably, fit of the parameter-less long-term rate and time-to-reward models was comparable to that of the single-parameter models (Figure 4A; Table 1), presumably because a single parameter could not account for behavior in both tasks. Therefore, we allowed parameters to vary across sessions (Figures 4B and 4C; Table 1). Not surprisingly, the single-parameter models generally fit the data better than the parameter-less models; however, large variation in best-fit parameters existed (Figure 5).

Figure 4.

Model fit given as distributions of mean squared error across (A) rats or (B and C) sessions. Panel A displays fit when parameters vary between rats, whereas panels B and C show fit when parameters vary between sessions. We also calculated fit for Patch sessions based on (B) best-fit parameters from the matched DD sessions and (C) vice versa.

Table 1.

Median mean squared errors

Model			Parameters from DD		Parameters from Patch
		per rat	DD	Patch	DD	Patch
Parameter-less models	Long-term	0.47	0.54	0.35	0.54	0.35
	Time-to-reward	0.37	0.39	0.35	0.39	0.35
Single-parameter models	Hyperbolic	0.37	0.30	0.34	0.47	0.27
	Exponential	0.44	0.28	0.47	0.51	0.24
	Heuristic	0.47	0.26	0.65	0.51	0.30
	TIMERR	0.49	0.44	1.00	0.49	0.31
	Added-value	0.34	0.39	0.27	0.39	0.27

Note: The rate models are equivalent for Patch sessions. This is due to the design of the Patch task.

Figure 5.

Best-fit parameters for each model for each rat given for both tasks.

It follows from the logic of these purportedly general models that varying fitted parameters ought to explain behavioral variation on both tasks for a given pair of matched sessions. Instead, we found that no model accounted for behavior on both tasks with the same set of parameters: Fitting the models to Patch sessions using the best-fit parameters from DD sessions produced poor fits (Figure 4B; Table 1), as did fitting DD sessions with the best-fit parameters from Patch sessions (Figure 4C; Table 1). Thus, it would seem that none of the models were general enough to account for our intertemporal choice data.

Foraging animals frequently err on the side of over-staying in patches (Nonacs, 2001), and rats’ behavior has been modeled by adding an “aversion-to-leaving” parameter to a standard foraging model (Wikenheiser, et al., 2013). Similarly, we conceptualized impending, cued rewards as having some amount of added value (Redish, Schultheiss, & Carter, in press; Loewenstein, 1987). We modified the time-to-reward model so that the value for the stay option in Patch was the sum of its objective value and a fitted parameter, ɑ—a model we refer to as the “added-value model”. Critically, we assumed the best-fit parameter from this model to be the same between the matched Patch and DD sessions, although no value was added to either option in the DD task. By definition, then, this model showed zero task-based variation in best-fit parameters (Figure 5). This model fit the data well for both tasks and because of its assumed asymmetry, it was the only model to maintain fit when explaining the behavior during sessions on one task with best-fit parameters from the other (Figure 4; Table 1).

Discussion

We have identified a large effect on intertemporal choice—rats value time completely differently when faced with foraging decisions as opposed to those typical of delay-discounting (Figure 2). Our results suggest that current conceptualizations of intertemporal choice are inadequate descriptions of this broad class of behavior. Moreover, our data were best described by a novel model that allowed for the stay option in the Patch task to take on additional value. As an alternative to our added-value interpretation, one might suggest an “added-cost” model where subjects were more aware of the cost of travel (e.g., increased time or effort) on the Patch task than on the DD task. In many ways, this model would make identical predictions to the added-value model; however, if added-cost led to increased staying on the Patch task, then one would predict an interaction between task and travel: Assessment of cost ought to scale with the travel requirement, so if the difference between preference in Patch and DD was due to an added-cost parameter, then the travel manipulation should have increased the average final adjusting delays more for Patch than for DD. This was clearly not the case (Figures 2B and 2C).

Based on work in multiple-decision-making-systems theory (Rangel, Camerer, & Montague, 2008; van der Meer, Kurth-Nelson, & Redish, 2012; Redish, Jensen, & Johnson, 2009; Redish, 2013), we propose that the added value in our model represents the influence of a Pavlovian valuation system (Redish et al., in press). We use the term “Pavlovian” to describe a system based on situation-recognition and associative processes that “release” species-specific behaviors in the presence of immediate cues, such as freezing at the sign of a predator. This use of the term “Pavlovian” is different from the classical definition—that consequences are not dependent on subjects actions (Bouton, 2007)—but it describes Pavlov's (1927) observation of salivation elicited by an association between a bell and food. This system is thought to track the value of states (Dayan, Niv, Seymour, & Daw, 2006) or to compute incentive salience (Berridge, 2012). For example, in our Patch task, the presence of stimuli that predict food would increase the Pavlovian valuation of waiting in the feeder zone (i.e., that “state” is assigned more value because of its association with reward). Such a process would increase choices to stay and thereby drive up the adjusting delay on the Patch task, but would have no effect on the DD task where choice occurs in a state devoid of salient, immediately clear food cues.

The Pavlovian-valuation explanation may also explain the fact that previous work has tended to find that deviations from optimality by patch-foraging animals is due to over-staying (Nonacs, 2001). However, not all studies of foraging find over-staying (Bhatt & Wasserman, 1987), suggesting that our Pavlovian valuation explanation may be moderated by species, experimental set-up (operant chamber vs. maze), or both. Similar points may explain the difference between our finding that rats receive less food on the Patch task as compared to the DD task and previous findings that behavior by birds and humans is more optimal (i.e., better described by long-term rate maximization) during foraging as compared to making DD-like choices (Stephens & Anderson, 2001; Carter et al., 2015). Although we found that the long-term rate model fit the data less well for DD sessions than for Patch sessions (Figure 4)—the hallmark result in the aforementioned previous work—these models were designed to explain behavior on fundamentally different procedures from ours (e.g., programmed rather than subject-controlled inter-trial intervals, fixed rather than adjusting delays). Although the specifics of the current experiment make interpreting further comparisons difficult, one possible explanation for the differences between our results and previous ones is that our design enhanced the influence of a Pavlovian system on willingness-to-wait, and as a result of spending more time waiting for food, subjects received less food on the Patch task than on the DD task.

Our primary findings have implications for the study of decision making more broadly. For example, it may be possible to increase patience during choice in real-world intertemporal choice (e.g., decisions about retirement investments) by reframing the problem so that it shares features of foraging. Likewise, in cases where further time investment is undesirable, options could be reframed to be more similar to the DD task. Moreover, the current task may function as an animal model of the endowment effect (Kahnemann, Knetsch, & Thaler, 1990; Morewedge & Giblin, 2015) if the Pavlovian value that we propose is affecting foraging also contributes to the value of owned items more than purchasable ones, presumably because cues associated with owned items are more salient. However, the clearest implication of our results is that current models of intertemporal choice are not sufficiently general to explain both foraging and delay-discounting behavior.