A framework for understanding and advancing intertemporal choice research using rodent models

Abstract

Intertemporal choices are common and consequential to private and public life. Thus, there is considerable interest in understanding the neural basis of intertemporal decision making. In this minireview, we briefly describe conceptual and psychological perspectives on intertemporal choice and then provide a comprehensive evaluation of the neural structures and signals that comprise the underlying cortico-limbic-striatal circuit. Even though great advances have been made, our understanding of the neurobiology of intertemporal choice is still in its infancy because of the complex and dynamic nature of this form of decision making. We close by briefly discussing recommendations for the future study of intertemporal choice research.

1.0 Introduction

Among the countless behaviors studied by neuroscientists, value-based decision making engenders great fascination because of the ease with which people can think of examples from their own lives-routine decisions like selecting what/when to eat as well as more profound decisions like choosing whether to pursue an advanced degree or determining the right time to invest in a home. Such examples highlight the fact that many of the value-based decisions humans and other animals regularly face are intertemporal choices, decisions between options available at different times.

Individuals across species respond to intertemporal choices by delay discounting, a phenomenon by which their subjective valuation of reward declines with a delay (Ainslie, 1975; Mazur, 1997). Not only do individuals prefer an immediately available reward over a delayed reward when given the choice between equally-sized reward options, but their preference for an objectively larger reward is also reduced when its receipt is delayed relative to the smaller reward. Even though delay discounting is the norm, the rate at which individuals discount future reward varies greatly, with high rates of discounting correlating with a variety of psychiatric and behavioral conditions such as psychiatric diseases and behavioral disorders (Koffarnus, Jarmolowicz, Mueller, & Bickel, 2013), college GPA (Kirby, Winston, & Santiesteban, 2005), texting while driving (Hayashi, Russo, & Wirth, 2015), environmental investment (Hardisty & Weber, 2009), social policy (Weatherly, Plumm, & Derenne, 2011).

Because intertemporal choices are common and consequential to private and public life, there is great interest in enumerating the processes that underpin normal and pathological intertemporal choices. While human studies are useful for identifying neural circuits and psychological processes involved in intertemporal decisions, rodent studies are able to extend those findings by disentangling the contributions of specific brain structures and systems. Thus, the aim of this minireview is to present a framework for understanding and advancing intertemporal choice research using rodent models. We begin by briefly describing conceptual and psychological perspectives on intertemporal choice. Then, we summarize rodent studies in order to review and evaluate the neural contributions of brain structures and systems. Finally, we close by highlighting open questions and making recommendations for future study.

2.0 Measuring intertemporal choice

Even though it is widely recognized that the goal of value-based decision making is to maximize value, it is difficult to predict an individual’s intertemporal choice because valuation is idiosyncratic and driven by delay discounting rate. Thus, to study intertemporal decision making, researchers have developed laboratory tasks to measure individual delay discounting rates. The tasks, often called delay discounting tasks, record subjects’ preferences during a series of choices between small rewards available after little/no delay (usually immediately) and larger rewards preceded by a range of delays. During the course of these tasks, human subjects are asked about their preferences during novel choices between food or money rewards paired with long delays (weeks, months, or years), and rodent subjects are well-trained to learn and express preferences for choices between food rewards associated with delays on the order of seconds. [In the rodent tasks, choices are typically presented with fixed delays in a prescribed order, i.e. increasing from 0 seconds, but there are versions with mixed delays and versions that allow rodents’ choices to determine the order.]. From these tasks, subjects’ choice behavior (% choice of large, delayed reward) is used to calculate delay discounting rates and identify decision making tendencies (Figure 1A). Individuals with higher rates than the group mean are identified as ‘impulsive’, whereas individuals with lower rates are identified as ‘patient’. Here, we use the term impulsive to refer only to impulsive choices and not impulsive actions.

Figure 1.

A. Example delay discounting curves. Such curves are generated using the choice behavior from delay discounting tasks, and delay discounting rates and decision tendencies are derived from them. The thick black line in the center represents a typical discounting curve, while the thin lines represent a decreased discounting/ ‘patient’ curve (top) and an increased discounting/ ‘impulsive’ curve (bottom). B. A schematic of the five behavioral and computational subprocesses that comprise the intertemporal decision making process (adapted from Rangel et al. (2008)). First, an individual represents the decision problem. Next, they assign each option a value through a valuation process. Third, they compare the computed values and select the action associated with the greatest value (action selection). Fourth, they compare the experienced value against that which they expected (outcome evaluation); and finally, they update their decision representations, valuations, and choices through learning, memory, and motivational mechanisms. (Of note is the fact that even though it is easy to conceive of subjective valuation and action selection as distinct processes, it is difficult to disentangle the two because a basic assumption of decision theory is that the chosen option is by definition the most valuable option.). In addition to the behavioral subprocess labels, the schematic is also populated with the psychological variables that have been identified in human and rodent studies. Only those that have been identified in both species are bolded.

Despite the differences in task structure, rodent delay discounting tasks are designed to measure the same process as human tasks. Although some have argued rodent tasks are not appropriate models for extending human research (see Hayden, 2015 and Killeen, 2011 for critical analyses), the similarities in results between species has justified their continued use. Not only do measures of human and rodent delay discounting rates both remain stable over time if measured under the same circumstances (Anokhin, Golosheykin, Grant, & Heath, 2011; Kirby, 2009; J. McClure, Podos, & Richardson, 2014), but there is significant overlap between psychological and neural processes that have been identified using these tasks in both species (see fourth and fifth sections).

3.0 Conceptual basis of intertemporal choice

In light of how intertemporal decision making is measured, it is tempting to view choice as a single, standalone behavior. But, it is more accurate to view it as a complex, multistep behavioral process. The conceptual model which captures this best divides the decision making process into five subprocesses that occur every time an animal encounters an intertemporal choice: decision representation, subjective valuation, action selection, outcome evaluation, and learning/updating (Rangel, Camerer, & Montague, 2008; Figure 1B). According to the model, animals must first represent the decision problem by determining the number and features (actions, delays, reward size) of the available options. Then, they integrate the feature information (reward and delay) to assign each option a subjective value, and next, they use that valuation to choose and perform the action(s) associated with the most valuable option. Finally, they compare the experienced value against their expected value and then update their decision representations, valuations, and choices based and what has or has not changed externally (actions, cues, delays, rewards, etc) or internally (hunger, thirst, affect, etc.). By outlining the entire decision making process, the model expands our conception of intertemporal choice beyond just the choice itself to include an interconnected web of behavioral subprocesses.

Even though the subprocesses were initially proposed to highlight testable computational variables for neurobiological experiments, they also implicate psychological variables to be tested in behavioral experiments. Specifically, the fact that each subprocess is likely linked to numerous psychological processes (including learning, memory, perception, and motivation-related processes) raises the possibility that some or many of those processes are critical for intertemporal choice.

4.0 Psychological basis of intertemporal choice

To empirically determine which psychological variables support intertemporal choice, experimenters have taken two approaches. The first is to identify scenarios and manipulations that change (increase or decrease) delay discounting rates; and the second is to identify correlations between psychological measures and delay discounting rates. To date, those approaches have been little applied to rodents and very frequently applied to humans, such that many psychological variables have been identified in human subjects (for excellent reviews, see Koffarnus et al., 2013; Lempert & Phelps, 2016; Peters & Büchel, 2011) and very few in rodent subjects. However, to hew to the focus of this minireview, in this section we discuss the three psychological variables that have been studied in both human and rodent subjects: timing, affect, and working memory. Not only do these examples illustrate how psychological variables can influence choice, but they also corroborate our claim that current rodent tasks can accurately model psychological aspects of human intertemporal choice.

First, timing or temporal perception has proven critical for human and rodent intertemporal choice. In both species, the more imprecise an individual’s timing ability, the higher their delay discounting rates (Baumann & Odum, 2012; Marshall, Smith, & Kirkpatrick, 2014; McClure et al., 2014). Consistent with that, when human subjects were directly trained to time more precisely, their discounting rates were lowered (Peters & Büchel, 2011). Such observations indicate that improper timing of the delays during the decision representation subprocess will result in a higher discount rate when integrated with reward during the valuation subprocess. This may also explain why many of the same human disorders associated with timing dysfunction are also associated with high discounting rates (Allman & Meck, 2012; Berlin, Rolls, & Kischka, 2004; Meck, 2005; Rubia, Halari, Christakou, & Taylor, 2009).

In addition to timing, affect can also be manipulated to reduce delay discounting across species. Human gamblers exhibited diminished discounting when large, delayed rewards were offered in the context of a positively conditioned cue (Dixon & Holton, 2009); just as rats discounting rates were lower when they were routinely trained in a task with a cue present during the delays that had acquired the positive properties of a conditioned reinforcer (Cardinal, Robbins, & Everitt, 2000; Zeeb, Floresco, & Winstanley, 2010). In both cases, the contextual cues alter delay discounting by changing the subjects’ affective responses to the delayed reward during the learning/updating subprocess. The presence of the positively associated cues, strengthens reward associations and activates learned behavioral patterns, which together make the delayed option more valuable and likely to be pursued.

Lastly, researchers have also successfully linked working memory to intertemporal choice in humans and rodents. Not only has working memory capacity been shown to negatively correlate with delay discounting in humans and rats (Renda, Stein, & Madden, 2015; Shamosh et al., 2008); but manipulating human’s working memory by taxation or training lead to increased and decreased discounting rates, respectively (Hinson, Jameson, & Whitney, 2003; Wesley & Bickel, 2014). There is currently no consensus on the mechanism by which working memory impacts delay discounting, but one possibility is that working memory is required during the learning/updating subprocess to maintain active representations of actions across delays so that action-outcome associations are formed and values are properly learned and updated.

As mentioned, human research has identified additional psychological variables that have yet to receive attention from rodent researchers. Specifically, contextual and framing manipulations have shown that attention, feature (delay, reward) salience, and recent choice history play a role in intertemporal choice; while manipulations that changed the state of the decision maker have revealed the importance of motivation, mood, and stress. Currently, there is no evidence to suggest a relative weighting, in terms of importance, for the diverse set of psychological variables that have been identified. Rather, as a group, they deepen our understanding of the web of subprocesses that comprise the intertemporal decision making process (Figure 1B) and encourage the use of psychological approaches to treat pathological intertemporal choice.

5.0 Neural basis of intertemporal choice

To fully understand intertemporal choice, a complete functional characterization of the neural circuitry that underpins the process is also essential. First, the critical nodes of the neural circuit should be identified; and second, the contribution(s) of those structures/systems-both the computations or values they signal and the behavioral subprocesses they support-should be catalogued.

The initial work to understand the neural basis of intertemporal choice used fMRI recordings to determine which structures were active when healthy, human subjects made intertemporal decisions. They found BOLD activation of a large scale neural network that included the prefrontal cortex, amygdala and ventral stratum (Kable & Glimcher, 2007; McClure, Ericson, Laibson, Loewenstein, & Cohen, 2007; McClure, Laibson, Loewenstein, & Cohen, 2004). Beyond identifying a possible circuit, the activity patterns they observed also proposed two interesting models for value encoding during intertemporal decisions. The first model, the β-δ model, suggested that the network influenced choice through the relative activation of separate, distinct substructures that either responded to immediate or delayed options, with immediate-choice promoting activity in limbic structures (β) and delayed-choice promoting activity in prefrontal structures (δ) (McClure et al., 2007, 2004). Alternatively, the second model found correlates of subjective values (single, scaled values calculated from subjects’ choices) within both limbic and prefrontal structures (Kable & Glimcher, 2007). Those two models highlight a fundamental question about how value is computed in the brain during inter-temporal choices: are dual values integrated at the network level, are subjective values represented within structures throughout the network, or is some combinatorial approach at play? (Peters & Büchel, 2011).

Human studies have not be able to fully answer that question, nor determine whether those signals are necessary for normal levels of choice because such experiments require the use of more invasive techniques. But, rodent models have begun to shed some light on answers. Targeted neural manipulations (lesions and inactivations) have been used to map the circuit (Figure 2). Additionally, single-unit recordings have been used to assess which values are computed and signaled in the brain and to characterize neural signals in terms of their likeliest behavioral/psychological contributions.

Figure 2.

A. Anatomical circuit representation showing structures and pathways involved in rodent intertemporal decisions. Structures and DA receptors are only included if silencing with lesions, inactivation, or blockade altered rodent delay discounting. Solid black lines indicate the pathways that are required for normal discounting performance; whereas solid grey lines represent pathways that are not necessary for normal performance. Pathways marked by dotted grey lines have yet to be tested during delay discounting tasks; and the dotted black line (and asterisk) shows a pathway that has been implicated in choice of a single delayed choice but has yet to be tested during a discounting task. B. The first table lists the changes to delay discounting curves that resulted from silencing structures and disconnecting a pathway. C. The second table lists the changes to delay discounting curves that resulted from pharmacologically manipulating DA signaling. (⭡) decreased discounting; (⭣) increased discounting/impulsive choice; NE- no effect; + stimulated DA signaling; -- blocked DA signaling; BLA-basolateral amygdala; HPC-hippocampus; LHb-lateral habenula (inactivation leads to indifference (Colin M Stopper & Floresco, 2013)); mPFC-medial prefrontal cortex; NAc-nucleus accumbens; OFC-orbitofrontal cortex; VTA-ventral tegmental area. D.

5.1 Prefrontal cortex monitors and adjusts intertemporal choices

5.1.1 Orbitofrontal cortex (OFC)

The OFC is the region of the prefrontal cortex (PFC) that has been most extensively studied for its role in intertemporal choice. Although humans with OFC lesions consistently exhibit impulsive choices (Bechara, Tranel, & Damasio, 2000; Berlin et al., 2004; Sellitto, Ciaramelli, & di Pellegrino, 2010), there are discrepancies between reported effects of manipulating the rodent OFC. Following lesion or inactivation of the whole OFC, rodents’ discounting increased (Mobini et al., 2002; Rudebeck, Walton, Smyth, Bannerman, & Rushworth, 2006), decreased (Kheramin et al., 2002; Mar, Walker, Theobald, Eagle, & Robbins, 2011; Winstanley, 2004), or did not change (Abela & Chudasama, 2013; Churchwell, Morris, Heurtelou, & Kesner, 2009; Jo, Kim, Lee, & Jung, 2013; Mariano et al., 2009; Moschak & Mitchell, 2014). The reasons for the inconsistencies are not fully known, but one possibility is that the OFC was differentially engaged based on the experimental design. Specifically, differences might have arisen because rats encountered a limited number of delays (Churchwell et al., 2009); mixed delay order (Jo et al., 2013); no pre-operative training (Abela & Chudasama, 2013); lengthy post-op training (Mar et al., 2011); a cue during the delay-to-reinforcement (Mariano et al., 2009; Zeeb et al., 2010); or interactions with their basal levels of impulsivity (Zeeb et al., 2010). A second possibility is that the medial and lateral portions of the OFC exert heterogeneous influence on intertemporal choice. Mar et al (2011) showed that medial OFC lesions decreased rodent discounting while lateral OFC lesions increased rodent discounting; but it took 6 days of post-lesion retraining for those effects to be observable and inactivation of medial OFC by another group was not successful at changing rodent discount rates (Stopper, Green, & Floresco, 2014).

Single unit electrophysiological recordings have furthered our understanding of why the OFC is necessary for normal intertemporal decisions by pinpointing OFC signals that likely underpin its contribution. Specifically, two types of OFC reward signals were recorded in rats during a chamber-based intertemporal choice task in which an odor cue and the spatial location of a reward indicated whether the delay preceding the reward would be short or long. While the majority of OFC neurons exhibited reward evoked firing that reflected the delay-discounted reward value, meaning they fired stronger to the more preferred, immediate rewards than to the delayed reward (immediate-preferring neurons); a second, smaller subset of neurons showed sustained increases in firing during the delay to the large reward, i.e. a larger response to delayed rewards (delay-preferring neurons) (Roesch, Taylor, & Schoenbaum, 2006). However, neither response type could be categorized as encoding a common value because they failed to also respond differentially to reward size (i.e. immediate-preferring neurons did not fire more for the large reward when offered without delay). The authors concluded that the two reward representations support two separate processes: the delay-discounted signal broadcasts the reduced value of delayed rewards and promotes choice of the immediate reward; and the second ramping signal represents an outcome expectancy signal that tracks delay, facilitates and updates associative representations and outcome predictions in other brain areas, and promotes choice of the delayed reward (Roesch, Calu, Burke, & Schoenbaum, 2007). In support of the suggestion that OFC neurons signal outcome expectancy during intertemporal decisions, Stott & Redish (2014) observed that OFC neuron ensembles signaled covert reward representations just after the choice point of a spatial adjusted delay discounting task.

Consistent with the previously mentioned human study that showed that the degree of OFC population activation correlated with choice of delayed rewards (McClure et al., 2004), there is evidence to suggest that the two reward representations in the rodent OFC also work together to value options and guide choice. When the two reward signals were identified, the overall OFC representation favored the immediate rewards (the majority of the recorded cells were immediate-preferring) and was accompanied by a behavioral preference for the immediate rewards. Indeed, if the overall OFC reward representation is responsible for guiding choice of delayed rewards, then neural manipulations that change choices during inter-temporal decisions probably do so by changing the balance of the two reward representations in the OFC (increasing or decreasing the proportion of delay-preferring reward representation). The hypothesis has already been tested once by recording OFC neurons in aged rats that are believed to have OFC dysfunction because they exhibit age-related changes in OFC-mediated behaviors. Consistent with the hypothesis, the aged rats displayed an increased preference for delayed rewards that was accompanied by an increased percentage of delayed-preferring neurons and no change to the overall percentage of delay-sensitive reward neurons (Roesch, Bryden, Cerri, Haney, & Schoenbaum, 2012).

5.1.2 Medial prefrontal cortex (mPFC)

The mPFC is a region of the PFC that has been less-studied during intertemporal choice because it is not necessary for valuation/choice during delay discounting tasks. Although several human imaging studies have observed activation of the mPFC during intertemporal choice tasks (Ballard & Knutson, 2009; Peters & Büchel, 2011), silencing the mPFC in rodents does not affect discounting during a within-session delay discounting task. When the mPFC was lesioned, animals’ discounting curves were flattened-their choice of the large reward was reduced at short delays and increased at long delays (Cardinal, Pennicott, Lakmali, Robbins, & Everitt, 2001). At first glance, that might seem attributable to a change in discounting; but when combined with the fact that the lesion also failed to impact reward discrimination and that a recent study showed temporary mPFC inactivation had no effect on discounting, it is most likely that the mPFC contributes to timing or working memory rather than valuation (Feja & Koch, 2014). Since the mPFC is known to track interval durations (Kim, Ghim, Lee, & Jung, 2013; Meck, Church, Wenk, & Olton, 1987) and support tolerance during reaction time tasks (Narayanan, Horst, & Laubach, 2006), it is probable that the mPFC monitors delay information that in turn helps animals determine when to pursue rewards. Additionally, there is a rich literature implicating the mPFC in working memory processes that support the linking actions with outcomes over delays (Yang, Shi, Wang, Peng, & Li, 2014). While animals with their mPFC temporarily inactivated during one test session can rely on well-established delay-reward associations, which remain supported by stored temporal memory information from other neural systems (Matell & Meck, 2000); animals with lesioned mPFC, whose delay-reward associations are likely inappropriately updated during retraining (probably lengthened according to results of mPFC lesions during interval timing tasks (Meck et al., 1987)), no longer base their behavior on accurate temporal information. Consistent with a timing contribution, temporary disconnection of the mPFC and BLA reduced waiting tolerance for a single 15s delayed reward on a T-maze choice task during which the small, immediate reward was constantly available (Churchwell et al., 2009). Finally, another explanation for the discrepancy between the lesion and inactivation data may be procedural. Lesioning the mPFC is known to impair reversal learning (Salazar, White, Lacroix, Feldon, & White, 2004), so it may be the case that the adjustment or updating required between days (from the longest delay block to the zero delay block) is too extreme to accomplish without an active mPFC and inadvertently drives down preference for the large reward during the first block.

5.2 Amygdala-ventral striatal circuits promote choice of large, delayed rewards

5.2.1 Basolateral amygdala (BLA)

Even though the amygdala is primarily studied for processing negative emotions and linking environmental stimuli to aversive sensory experiences, the amygdala, and the BLA in particular, is also known to play an essential role in positive emotions and reward-related behavior (Baxter & Murray, 2002). Thus, it is not surprising that the BLA is required for intertemporal choice. Specifically, its known that the BLA is responsible for biasing choice toward the large, delayed reward because BLA-lesioned rodents shifted their choice away from the large reward during a within-session delay discounting task (Winstanley, 2004). Their increased discounting was not attributable to an inability to discriminate reward sizes nor an inability to remember the stimulus (lever)-reward association for two reasons: first, BLA-lesioned rodents in the same study retained their preference for the large reward when it was immediately available; and second, previous studies have shown the BLA is not necessary for appetitive Pavlovian conditioning (Parkinson, Robbins, & Everitt, 2000).

However, since BLA recordings have yet to be performed during a rodent delay discounting task, the precise role of the BLA in intertemporal choice remains unknown; but the most compelling hypothesis is that it contributes through its role in assigning incentive or motivational significance to instrumental cues, just as it does in other tasks. BLA neurons recorded in rodents during instrumental tasks exhibited cue and delay-related activity that differentially responded based on the motivational significance of the stimuli (Schoenbaum, Chiba, & Gallagher, 1998). Additionally, lesion studies have found that the BLA is required for animals to adjust their instrumental responses to conditioned cues when faced with changes in affective information, such as conditioned reinforcers, reinforcer devaluation, or extinction learning (Hatfield, Han, Conley, Gallagher, & Holland, 1996; Hitchcott & Phillips, 1998; McLaughlin & Floresco, 2007; Schoenbaum, Roesch, & Stalnaker, 2009). Therefore, BLA-lesioned animals probably decreased their choice of the large reward because they no longer had access to incentive signals that would normally invigorate them to pursue those rewards despite their associated delays.

5.2.2 Nucleus accumbens (NAc)

The role of the NAc in intertemporal choice has received more attention than the BLA. Time and time again, irrespective of the task used to measure delay discounting, rats with NAc core lesions displayed increased discounting that was not linked to altered reward sensitivity nor satiety (Bezzina et al., 2007; Cardinal et al., 2001; da Costa Araujo et al., 2009; Valencia-Torres et al., 2012). Curiously, when the NAc core was instead temporarily inactivated during the within-session version of the delay discounting task, the effect was the opposite – rats exhibited decreased discounting (Moschak & Mitchell, 2014). While it is possible that means temporary silencing of the NAc core affects choice differently than permanent lesions, it is more likely that it is an outlier and attributable to the fact that the dose of the GABA agonist used was low enough to, in the authors’ own words, only ‘partially’ inactivate the structure. Instead, the data indicated that the NAc is required to work with the BLA and OFC to promote choice of the large reward. The evidence for the OFC-NAc projection’s involvement comes from a disconnection study that showed that serial information transfer between the OFC and the NAc core is also required to maintain normal levels of delayed reward choices and avoid excessive discounting (Bezzina et al., 2008).

Recording studies have observed several neural responses within the NAc that further detail its contribution to intertemporal choice. Population level analyses of the NAc across species have identified reward representations that likely bias choice during intertemporal choices. Human fMRI studies found subjective value correlates as well as activity that was predictive of the choice of immediate rewards (Ballard & Knutson, 2009; Kable & Glimcher, 2007; McClure et al., 2004); while more recent recordings of rodent NAc ensembles at the choice point of a spatial adjusted delay task uncovered signals that combined reward value and chosen action – reward representations that not only distinguished reward size but also preferentially responded to the chosen reward over unchosen reward (Stott & Redish, 2014). Integration of value and action information has also been seen at the single cell level in the NAc. In a second rodent study, performed during a chamber-based intertemporal choice task with odor cues associated with short or long delayed rewards, NAc cells exhibited cue-evoked firing that was both reflective of delay-discounted value and sensitive to whether the cue-associated action was performed, meaning, for example, they fired more strongly to cues associated with more preferred immediate rewards but only when the nosepoke action was performed in pursuit of the immediate reward (Roesch, Singh, Brown, Mullins, & Schoenbaum, 2009). In addition to responding preferentially to cues whose associated rewards were obtained, the cue responses were also correlated with the speed of the nosepoke action. Thus, the study directly demonstrated how an action – dependent cue signal in the NAc core can trigger downstream motor targets and encourage the animal to perform actions and pursue a given reward. Said another way, it is a great example of how the NAc functions as a ‘limbic-motor interface’ able to integrate reward, value and cue information during intertemporal choice tasks (Mogenson, Jones, & Yim, 1980).

5.3 Hippocampus provides temporal and prospective information

The final cortico-limbic structure currently implicated in intertemporal choice is the hippocampus (HPC). In three separate within-session delay discounting experiments, HPC lesions caused rats to decrease their choice of the large, delayed reward (Abela & Chudasama, 2013; Cheung & Cardinal, 2005; Mariano et al., 2009; McHugh, Campbell, Taylor, Rawlins, & Bannerman, 2008). Their increased discounting was not specific to either subregion, as lesions of both the dorsal and ventral HPC produced the same impairment (McHugh et al., 2008); nor could the effect be interpreted as stemming from deficits in magnitude or spatial discrimination or, by extension, memory (Mariano et al., 2009). A slightly different, yet complimentary effect was observed in the spatial adjusted delay task: even though HPC-lesioned animals were eventually able to titrate around a similar indifference delay as the controls; their training took much longer and their preference for the large reward was much more variable than controls (Bett, Murdoch, Wood, & Dudchenko, 2015). Finally, HPC-lesioned rodents faced with a choice between an immediate uncertain reward and a delayed certain reward were less tolerant of delayed rewards, shifting their preference from the certain reinforcement to less certain, but immediate reinforcement (Rawlins, Feldon, & Butt, 1985). Taken together, the lesion data suggests that the HPC is required for animals to tolerate delays in order to obtain larger rewards.

A precise functional characterization of the HPC neurons during a rodent delay discounting task has yet to be performed, but based on related observations, it seems likely that the HPC contributes both temporal and prospective information to intertemporal choice. Evidence for the former role comes from the fact that HPC neurons in both primates and rodents encode the duration of elapsed time (MacDonald, Lepage, Eden, & Eichenbaum, 2011; Naya & Suzuki, 2011). Given that HPC-lesions failed to affect timing behavior in some studies (Dietrich & Allen, 1998; Dietrich, Allen, & Bunnell, 1997; Port, Romano, Steinmetz, Mikhail, & Patterson, 1986; Rawlins, Winocur, & Gray, 1983), while others have reported that lesions to the HPC or fimbria/fornix have shortened duration estimates (Meck, 1988; Meck, Church, & Olton, 2013; Olton, Meck, & Church, 1987); it is not clear what the exact role of the HPC is to timing behavior and/or whether it is even required (Yin & Troger, 2011).

The role of the HPC in prospection is more straightforward, stemming from complimentary human and rodent studies. Behavioral psychologist have long considered the HPC an important component of the imagery and prospection network because it is engaged when individuals ‘self-project’ or ‘mentally time travel’ and imagine or predict future events like reinforcement (Peters & Büchel, 2011; Peters & Büchel, 2010). In fact, when such ‘future thinking’ was stimulated by a behavioral intervention that encouraged episodic prospection during an human intertemporal choice task, subjects discounted less and their fMRI BOLD signals revealed enhanced PFC-HPC coupling (Peters & Büchel, 2010). Additionally, at decision points in a maze-based decision task, rodent HPC neuron ensembles are known to show forward sweeping spatial representations that 1) are sensitive to task demands and experience and 2) likely signal the potential forward paths and associated actions and choices (Johnson & Redish, 2007; van der Meer, Johnson, Schmitzer-Torbert, & Redish, 2010).

5.4 The dopamine system transmits expected value information throughout the decision circuit

The dopamine (DA) system is best known for its ability to broadcast reward information throughout the brain, including to structures described above. DA neurons fire in response to predictive cues and encode the magnitude of future rewards (Schultz, 2006). In fact, direct stimulation of DA neurons can invigorate reward-seeking behavior (Phillips, Stuber, Heien, Wightman, & Carelli, 2003), just as inactivation of DA neurons can impair animals ability to respond to reward-predicting cues (Yun, Wakabayashi, Fields, & Nicola, 2004). As DA neurons respond to cues that predict rewards, they continue to track reward information since they fire at higher rates for greater than expected rewards and at lower rates for smaller than expected rewards, a phenomena called reward prediction error signaling (Schultz, 2006). Finally, recordings made during two rodent delay discounting tasks showed that DA neurons’ cue responses are able to integrate delay information and signal expected value information-in this case, a single value signal that encodes both reward size and delay-discounted value (Day, Jones, Wightman, & Carelli, 2010; Roesch, Calu, & Schoenbaum, 2007). Thus, in addition to contributing to outcome evaluation, the DA neurons phasic responses provide expected value information that is transmitted throughout the decision circuit where it can update outcome predictions, modulate other decision signals, and support normal levels of delay discounting.

Pharmacological manipulations confirm that DA neurotransmission influences intertemporal choice through tonic signaling as well. Blocking DA receptors with the antagonist flupenthixol caused animals to increase discounting; whereas, stimulating DA receptors with amphetamine or a DA reuptake inhibitor decreased delay discounting (Floresco, Tse, & Ghods-Sharifi, 2008; Winstanley, Dalley, Theobald, & Robbins, 2003). Since systemic injection of the D1, but not D2, antagonist also increases discounting, it is likely that DA biases choice toward the delayed reward through global D1 receptor activation (van Gaalen, van Koten, Schoffelmeer, & Vanderschuren, 2006). Additionally, through its capacity as a neuromodulator, the DA system is also capable of influencing important signals and computations locally by modifying activity within single structures. So far, it has been reported that DA input into the OFC but not the NAc is necessary for normal levels of choice, as eliminating DA input using 6-hydroxydopamine in the OFC and not NAc caused rats to decrease choice of the delayed reward (Kheramin et al., 2004; Winstanley, Theobald, Dalley, & Robbins, 2005). Finally, two experiments found that blocking both D1 and D2 receptors in the OFC and mPFC can increase discounting (Loos et al., 2010; Zeeb et al., 2010).

5.5 A cortico-limbic-striatal circuit supports normal intertemporal choice

Human research suggests that a neural network involving the prefrontal cortex, amygdala, and ventral striatum is engaged during intertemporal choices. The use of targeted neural manipulations and single-unit recordings in rodent models has confirmed that those structures, as well as the hippocampus and dopamine system, are indeed critical nodes of the decision circuit (Figure 2). Furthermore, the diversity of single-cell responses recorded thus far suggests that many values are computed at choice and favors a combinatorial valuation model rather than the dual or subjective value models championed by some human researchers. Although much more work is needed to characterize the single-cells and population activity of every node throughout the decision making process (rather than focusing on choice points), the current data set has enough information for us to begin linking brain structures to psychological variables and populating our conceptual model. Thus, our current conception of how and when different nodes of the circuit are engaged during intertemporal choices is as follows: relying on learned associations and values stored in a similar neural circuit, HPC cells are recruited to assist with decision representation-both by timing delay lengths and/or sweeping forward in time to shape accurate outcome expectations; next DA and BLA cells compute subjective values and NAc cells help transform those value signals into meaningful action values; and finally, once the outcomes are obtained, DA neurons evaluate expectation errors and update values with the help of OFC expectancy signals.

6.0 Summary and recommendations for future rodent intertemporal choice studies

Intertemporal choices are ubiquitous and consequential to human life, and this has motivated an interest in delineating the complex processes that underpin normal and pathological intertemporal decisions. While the use of human and rodent models of intertemporal choice have permitted some advancement of our understanding of those processes, the field is still in its infancy. To clarify the problem, we have advocated for a broader conception of intertemporal choice; one that acknowledges it as a multistep decision making process rather than a standalone behavior and have 1) drawn connections between neural and psychological processes and 2) speculated about when/how neural and psychological processes are engaged during the decision making process. In this section, we will briefly discuss our recommendations for future studies.

First, we recommend that researchers continue to investigate the similarities and differences between human and rodent intertemporal choice tasks used today. As mentioned above, an important question is whether or not rodent tasks are truly comparable to human tasks (and by extension real-life experiences) in which choice are typically varied and unpredictable. Since the question is premised upon the suspicion that current tasks, which intend to model the same behavior by generating measures of delay discounting, actually model distinct behavioral processes that engage different psychological and neural processes; it should be addressed by systematically changing rodent task parameters to better approximate human choice tasks (fixed, adjusted, or intermixed delay presentations) and assessing 1) whether the same neural circuit and neurotransmitters are required during the modified tasks and 2) whether the same correlations between delay discounting and psychological measures are observed.

Next, we encourage further efforts to interpret the activity and computations of the circuit at both the single-cell and population levels. As evidenced by the literature we discussed, much of the focus of neural recordings thus far has been to understand how value is computed at the time of choice and some at the time of the outcome. However, such narrow focus might limit our understanding of which structures and signals are involved in each time point of the decision making process. Once the signals are more thoroughly mapped in time, it will then be possible to apply optogenetic techniques to assess the necessity of certain signals and identify redundancies in the system.

Third, more attention should be paid to the role of neurotransmitters. In addition to DA, the neurotransmitters serotonin, norepinephrine, and acetylcholine are also positioned to anatomically and functionally influence the decision circuit. Importantly, each is implicated in the etiology and treatment of neuropsychiatric disorders characterized by impulsive choice and is known to subserve a unique array of psychological processes. Yet, few studies have explored the contribution of their global and local neurotransmission (global: (Mobini et al., 2000; Catharine A Winstanley et al., 2005) for serotonin; (van Gaalen et al., 2006) for norepinephrine; and (Mendez, Gilbert, Bizon, & Setlow, 2012; Scattoni, Adriani, Calamandrei, Laviola, & Ricceri, 2006; Tian, Qin, Sun, Li, & Wei, 2016)for acetylcholine; local: only serotonin (Darna et al., 2015)). Such experiments may inform our understanding of those neuropsychiatric disorders and/or lead to pharmacological treatments for improving intertemporal choices.

Our final recommendation is that the field focus on extending our understanding of the dynamic interplay between the structures of the circuit. Multi-site recordings and inactivations (disconnection studies via pharmacologic, chemogenetic, or optogenetic means) should be undertaken to 1) map the direction and hierarchy of signal transfer through projections between structures and 2) catalog how signals work alone and in concert to encode the computations necessary for normal levels of intertemporal choice.

Highlights.

• Conceptual and psychological insights into intertemporal choice

• Functional roles proposed for cortico-limbic-striatal circuit involved in intertemporal choice

• Justifications for rodent models of intertemporal choice