Timing in reward and decision processes

Abstract

Sensitivity to time, including the time of reward, guides the behaviour of all organisms. Recent research suggests that all major reward structures of the brain process the time of reward occurrence, including midbrain dopamine neurons, striatum, frontal cortex and amygdala. Neuronal reward responses in dopamine neurons, striatum and frontal cortex show temporal discounting of reward value. The prediction error signal of dopamine neurons includes the predicted time of rewards. Neurons in the striatum, frontal cortex and amygdala show responses to reward delivery and activities anticipating rewards that are sensitive to the predicted time of reward and the instantaneous reward probability. Together these data suggest that internal timing processes have several well characterized effects on neuronal reward processing.

1. Introduction

Many behavioural processes are affected by rewards. Unlike the senses, time intervals cannot be directly perceived via specific receptors but must be constructed by the brain. Humans and animals estimate time intervals with great accuracy [1,2], and this accuracy decays proportionally with delay [3–5]. The ability to estimate short time intervals plays an important role in everyday behaviour. Timing is essential for predicting and planning actions. A good example is the temporal accuracy required to hit a ball in a tennis game. Although we focus presently on reward, all reinforcement processes, including aversive learning and reactions, are subjected to timing processes. The term ‘timing’ refers not only to the duration of an event but also to the moment at which the event is likely to occur (temporal prediction). Both duration estimation and temporal prediction require a metrical representation of time in which the occurrence of consecutive events is measured on a continuous scale. Recent studies indicate that temporal processing may not be centralized in one single brain structure but rather occurs across different specialized areas [6–8]. Temporal prediction can be induced by stimuli or events and also by the passing of time itself [9]. This fact is characterized by the ‘hazard function’, defined as the conditional probability that an event will occur given that it has not yet occurred [10]. We know very well examples of increasing hazard functions; the longer we wait at the bus stop, the more we expect the bus to appear at the next moment.

The time of reward has several important influences on reward processing. The economic value of reward decreases with increasing delays. This temporal discounting may lead to the preference of sooner smaller rewards over larger later rewards. Temporal discounting has been demonstrated in humans and animals in many psychological and behavioural economics studies [1,3–5,11,12], although the debate is still raging between hyperbolic, exponential and combined models. Related to value discounting, behavioural conditioning requires optimal stimulus–reward intervals and becomes less efficient with shorter or longer intervals [13,14]. Furthermore, time intervals and rates of reward are of crucial importance for conditioning [5]. This review will discuss several neuronal processes involved in these functions. The processing of temporal information about rewards seems to rely most crucially on dopamine neurons, striatum, frontal cortex and amygdala. However, each region may have a functionally discrete role.

2. Dopamine neurons

Several studies suggest an involvement of the dopamine system in the timing of events, including rewards. Administration of amphetamine, an indirect dopamine agonist that enhances extracellular dopamine concentrations, results in the overestimation of time delays, whereas the dopamine receptor antagonist haloperidol leads to underestimation of delays [15,16]. Individuals with higher time sensitivity are also more sensitive to stimulant-induced euphoria, suggesting overlapping dopaminergic mechanisms for internal clock and reward functions [16–19].

Midbrain dopamine neurons code prediction errors in reward value [20], defined as the difference between current and predicted reward value. According to temporal difference (TD) reinforcement models, current reward value at each moment is defined as the present reward plus the discounted sum of future rewards [21]. The dopamine prediction error response shows three features that reflect closely the characteristics of TD reinforcement models, which makes them candidates for effective teaching signals for reward.

First, the dopamine response occurs to any event that elicits a reward-prediction error as defined above, irrespective of the event being a primary, unconditioned reward (US) or a conditioned stimulus (CS) predicting a primary reward [22]. The response transfer from the primary reward to the reward-predicting stimulus together with the coding of reward-prediction errors at both events are hallmarks of TD reinforcement models [21].

Second, the dopamine response reflects temporal discounting, both at the time of the conditioned, reward-predicting stimulus and at the time of the reward itself [23,24]. The dopamine activation to the CS predicting a reward with a longer delay is weaker than the response to another CS predicting the same physical amount of reward after a shorter delay. This characteristic is probably due to the reduction in subjective value with the longer delay shown by intertemporal behavioural choices. Thus, dopamine neurons code the decreasing subjective reward value rather than the unchanged objective, physical value in temporal discounting. More specifically, in TD language, the dopamine response to the predictive stimulus reflects the prediction error (in subjective value) that decreases as the difference between the lower current reward value (current reward plus discounted sum of future rewards) minus the constant average reward prediction before the specific stimulus becomes smaller. By contrast, the dopamine activation following the reward itself increases with increasing reward delays after the CS. This characteristic most probably reflects the increasing prediction error at the time of the reward with longer delays, as the difference between the constant received reward minus the lower, temporally discounted, predicted value before the reward increases. This interpretation attributes the effects of temporal discounting to changes in subjective value that are induced by a timing mechanism acting on the reward system.

Third, the dopamine response is sensitive to temporal aspects of reward prediction by showing reward-prediction error responses with surprising changes in reward timing. A positive response (activation) occurs when a reward occurs at a different time as predicted, and a negative response (depression) occurs when the predicted reward fails to occur at the predicted time and it has not yet occurred (figure 1; [26]). These characteristics make the dopamine response a teaching signal for time-sensitive reward predictions. These responses comply well with basic properties of TD models whose teaching signal is time sensitive. Although earlier TD implementations failed to replicate the time-sensitive dopamine responses completely [20,27], later models using appropriate prediction resets achieved good matches with the neuronal data [28]. Reward responses of dopamine neurons are also highly sensitive to the duration of the stimulus–reward interval, and this temporal precision declines as interval duration increases [23], showing time-scale invariance [1–5]. Thus, dopamine neurons code errors in the prediction of both the occurrence and the time of reward.

Figure 1.

Responses of dopamine neurons to errors in the temporal prediction of reward. The activity of a dopamine neuron is depressed when reward is delayed and increased when the reward is delivered earlier (early reward) or delayed (delayed reward) as predicted (habitual time of reward). CS, conditioned stimulus (adapted from [25]).

Taken together, dopamine neurons are highly sensitive to time intervals. One can find temporal sensitivity in the responses to prediction errors at temporally distinct events [22], in the influence of temporal delays on subjective value responses [23,24], and in the responses to variations in temporal intervals [23,26]. The temporal sensitivity of dopamine responses is likely an important component of their potential role of teaching signal for time-sensitive behavioural processes, rather than representing a ‘clock in the brain’ which may reside elsewhere in the brain. Alternatively, the temporal sensitivity of dopamine responses may derive from a mechanism within these neurons that is dedicated to their teaching function without requiring access to a central brain clock.

3. Striatum

Several studies have demonstrated the involvement of the striatum in functions extending beyond a role in movement. Many striatal neurons process reward information. The striatal responses to reward-predicting cues undergo temporal discounting in close relationship to behavioural discounting. Cue responses in striatal neurons decrease with increasing delays to reward in parallel with decreasing preferences measured in binary behavioural choices [29]. Whereas the decreasing responses in caudate neurons reflect the differences in discounted reward value between the two choice options, the decreases in ventral striatal neurons reflect the sums of discounted choice values.

In addition to temporal discounting, the activity of striatal neurons explicitly reflects the timing of rewards. Striatal neurons in behaving monkeys show sustained activations during the expectation of external signals of behavioural significance and during the delay period preceding reward delivery, both in arm movement tasks and in oculomotor tasks [30–32]. Most of these sustained prereward activations are prolonged when reward delivery is delayed and terminate only after the reward, suggesting a relationship to reward occurrence. Interestingly, in some striatal neurons the activation begins later when reward delivery is predictably delayed (figure 2), extending the relationship to the time of the reward [33]. Orbitofrontal neurons show similar sustained reward anticipatory activity whose onset is displaced by temporal shifts of reward and then ramps up to the expected time of reward [34].

Figure 2.

Influence of delayed reward delivery on the onset of expectation-related neuronal activation. Introduction of a delay of 1 s in reward delivery, extend and delay the response onset (Adapted from [33]).

Through the experience in a particular behavioural task, a stimulus gains predictive information for the occurrence of the subsequent task components. In this way, the reception of a known stimulus would evoke a representation of the predicted event. The observation that striatal neurons are activated before the occurrence of predictable environmental events may suggest that these neurons have access to central representations of these events. These representations would include both the occurrence and the time of reward. The described sustained activity may develop when an environmental stimulus evokes the representation of a specific subsequent event. The influence of predicted reward delay on the onset of reward expectation-related activity suggests that such a stimulus may provide a temporal guide for neurons engaged in the performance of the task.

4. Frontal and parietal cortex

Responses to reward-predicting stimuli decrease in a number of cortical structures with increasing delays, reflecting temporal discounting of reward value. Premotor cortical neurons in monkeys show lower activations following visual instructions for delayed behavioural responses and rewards [35]. Reversal of cue-delay associations leads to reversal of neuronal responses, suggesting a relationship to delay rather than visual stimulus properties. About one-third of task-related neurons in monkey dorsolateral prefrontal cortex show delay-related reductions of responses to chosen cue targets in choice trials [36]. Similar temporal discounting is observed in cue responses of neurons in the orbitofrontal cortex of monkeys [37]. The same neurons code also reward magnitude, suggesting that temporal discounting reflects reduced subjective reward value rather than simple decreases in motivation. By contrast, other orbitofrontal neurons show reduced movement-related activity with increasing delays [38]. As these activities do not covary with reward value, they seem to reflect the decreased motivation induced by delayed reward. Cue responses in lateral parietal cortex decrease with increasing delays to reward in choices between fixed earlier and psychometrically varied delayed rewards [39]. Taken together, reward-related neuronal responses undergo temporal discounting in a number of cortical structures.

Neurons in frontal, parietal and temporal cortex show anticipatory activity preceding predictable visual or somatosensory stimuli or planned movements. Some of these activations reflect the timing of these events, as they vary with the hazard function of their occurrence (hazard function is defined as the conditional probability of event occurrence given the event has not yet occurred). The responses may reflect the temporal evolution of attention to a forthcoming visual stimulus in extrastriate visual cortex V4 [40], the time of a forthcoming vibratory somatosensory stimulus in prefrontal cortex [41] and the planned time of an eye movement in parietal cortex [42]. Indeed, motor preparation is intrinsically linked to the time of the movement to be executed. In being sensitive to timing, these responses have access to an internal clock. However, it is unclear whether a central, singular clock exists in the brain that provides timing information to these diverse brain structures, or whether these structures constitute parts of a clock distributed across several brain structures, or whether separate clocks in these structures are specific for the local functions such as attention and somatosensory, visual and movement processing. Such distributed clocks should be synchronized with each other through interactions without requiring a central clock.

5. Amygdala

The amygdala is a major component of the brain's reward system. Lesions of this nucleus disrupt behavioural reward processes in humans and animals [43–47]. Amygdala neurons respond to reward-predicting stimuli and reward delivery [48–53] and show activations preceding behavioural responses and reward delivery [49,54].

Populations of amygdala neurons show decreased reward responses with increasing instantaneous reward probability (figure 3; [55,56]). The less likely the reward is to occur at a given moment, the higher is the neuronal response, suggesting a relationship to temporal reward ‘surprise’. Humans and animals learn to predict the outcomes of their behaviour. Learning depends on the discrepancy or ‘error’ between received and predicted outcomes. It proceeds when there is a ‘surprise’ in the outcome, that is, when outcomes that are not fully predicted occur. The discrepancy between reward occurrence and reward prediction is termed an ‘error in reward prediction’. The surprise responses observed in the amygdala may reflect the coding of positive temporal reward-prediction errors, and thus may play a role in learning.

Figure 3.

Neuronal responses in primate amygdala reflecting reward surprise. Responses decrease with increasing reward prediction (defined by increasing instantaneous reward probability). Dotted line: responses to unpredicted reward delivered without any preceding stimulus. Second line from top (red online): reward responses during a stimulus that predicts the occurrence but not the time of a pseudorandomly occurring reward. Third line from top (blue online): responses to a fully predicted, fixed reward delivered 2 s after stimulus presentation. Bottom line (black online): activity at 2 s following a non-rewarded stimulus. (a) Single neuron. (b) Averaged population responses (n = 15 neurons) (adapted from [55]). (Online version in colour.)

Opposite to the ‘surprise neurons’, another population of amygdala neurons shows increased reward responses with increasing instantaneous reward probability (figure 4; [55]). The more likely the reward is to occur at any given moment, the higher is the neuronal response. Thus, the modulations vary positively with the temporal predictability of reward. These activations might function to maintain established, temporally specific reward predictions after reinforcement learning and thus help to avoid extinction. This response is similar to the attentional modulation seen in monkey visual cortex V4, which parallels the hazard function of visual stimulus occurrence during individual trials [40]. These data demonstrate that such temporal modulations are not restricted to attentional processes but occur also with reward.

Figure 4.

Increase of reward responses of amygdala neurons with increasing reward prediction (increasing instantaneous reward probability). Same code as in figure 3. (a) Single neuron. (b) Averaged population responses (n = 25 neurons) (adapted from [55]). (Online version in colour.)

Distinct from responses to reward delivery, a population of amygdala neurons shows activations preceding the predictable delivery of reward (figure 5; [55]). Again, reward predictability defined by instantaneous reward probability has a remarkable influence on the temporal profiles of these prereward activations. Whereas prereward activations with singular reward start late and ramp up to high peaks, the activations with flat instantaneous reward rate start earlier and maintain a modest plateau until reward probability drops with stimulus offset. The absence of ramping with flat reward rate during the stimulus reflects the flat reward expectation derived from the employed renewal process with multiple rewards. It corresponds, inversely, to the ‘rate of occurrence of failure for repairable systems’ in reliability engineering [10,57] rather than the hazard rate modelling unique, non-repeating events. Thus, the temporal profiles of neuronal activity reflect the timing of predicted rewards.

Figure 5.

Increase of prereward activity of amygdala neurons with instantaneous reward probability. Top line (blue online): activity preceding reward delivered at a fixed time at the end of a specific stimulus. Middle line (red online): activity preceding reward delivered at pseudorandom times during another stimulus. Bottom line (black online): no changes in activity during presentation of a non-rewarded stimulus. (a) Single neuron. (b) Averaged population responses (n = 86 neurons) (adapted from [55]). (Online version in colour.)

Taken together, figures 3–5 show that the activities of amygdala neurons are not only modulated by the actual or anticipated occurrence of reward but also by the predicted timing and instantaneous rate of reward. These modulations affect both the reward responses and the anticipatory activities. Thus, amygdala neurons have access to an internal clock that processes the time of future reward occurrence. The temporal influence on amygdala neurons might derive from dopamine projections [58] involved in interval timing [59,60].

In being sensitive to reward timing, amygdala neurons process a fundamental characteristic of reward function, and thus may play a wider role in reward than hitherto assumed. The observed neuronal reward signals sensitive to the expected time of reward may be involved in a wide range of behavioural functions, including allocation of behavioural resources to specifically timed rewards, planning of sequential steps of goal-directed acts, choices between temporally distinct rewards and assignment of credit to specifically timed reward during novel reward learning, value updating and economic decision making, as conceptualized by animal learning and economic decision theories.

Funding statement

Our work was supported by the Wellcome Trust, the Behavioural and Clinical Neuroscience Institute Cambridge and the European Research Council.