Abstract
Several theories of self-control (including intertemporal bargaining Ainslie (1992) and self-signaling Bodner and Prelec (2001)) imply that intertemporal decisions can be more farsighted than would be predicted by the incentive associated with rewards outside a decision context. We examined this hypothesis using behavior and functional neuroimaging. First, subjects expressed preferences between amounts of money delayed by four months and smaller amounts available that day. This allowed us to establish “indifference pairs” individualized to each participant -- immediate and delayed amounts that were equally preferred. Participants subsequently performed a reaction time fMRI task (Knutson et al, 2001a) that provided them with distinct opportunities to win each of the rewards that comprised the indifference pairs. Anatomical Region of Interest analysis as well as whole-brain analysis indicated greater response recruited by the immediate rewards (relative to the preference matched delayed rewards) in regions previously implicated as sensitive to incentive value using the same task (including bilateral putamen, bilateral anterior insula and midbrain). RT to the target was also faster during the immediate relative to delayed reward trials (p < .01), and individual differences in RT between immediate versus delayed reward trials correlated with variance in MR signal in those clusters that responded preferentially to immediate rewards (r = .33, p < .05). These findings indicate a discrepancy in incentive associated with the immediate versus the preference-matched delayed rewards. This discrepancy may mark the contribution of self-control processes that are recruited during decision-making, but that are absent when rewards are individually anticipated.
Keywords: Reward, Limbic, Decision, Motivation, fMRI, delayed gratification
Introduction
Relative to other species, human exhibit extraordinary willingness to forgo smaller sooner rewards in order to obtain larger later ones. This may be related to distinct mechanisms of self-control that are engaged during decision-making. One account links self-control to a hypothesized recursive feedback loop between one’s present choices and their anticipated future behavior (Ainslie, 1992), an idea that has been formalized in economic modeling (Bénabou and Tirole, 2004; Bodner and Prelec, 2001) but that hypothesizes no neurophysiological substrate. Another account equates self-control with the intentional suppression of goal-inappropriate prepotent responses, which is hypothesized to depend on inhibitory pathways between the lateral prefrontal cortex and the basal ganglia (Jentsch and Taylor, 1999; Barkley, 1997; Goldstein and Volkow, 2002). Still another hypothesis suggests that self-control involves the alteration of value signals that results from effortful processing of long-term contingencies, and that is primarily dependent upon the dorsolateral prefrontal cortex (Hare, Camerer & Rangel, 2009).
Common to these accounts is the notion that self-control involves distinct processes engaged during decision-making that, in the case of intertemporal choice, may attenuate the tendency to discount delayed rewards. Concretely, if self-control processes specific to decision-making affect choice, then it follows that the individual might choose, for example, $50 delayed by four months over an immediate $40, despite it being the case that when the two expectancies are evaluated individually outside of a decision context, the valuation of the immediate $40 is the higher of the two alternatives. If this is in fact the case, it entails an important challenge that must be met by any neuroeconomic model of intertemporal choice.
There are at least three ways that incentive may be measured in a non-choice context. First, individuals can be asked to introspect on the motivation associated with single rewards. Second, a behavioral correlate of incentive (e.g., speed on a reaction time task) can be measured while single rewards are pursued. Third neural correlates of incentive can be measured during the pursuit of single rewards. In the present study we included each of these approaches. We were particularly interested in testing the hypothesis that when immediate and delayed rewards are encountered in isolation, behavioral and neural markers would indicate greater incentive for the immediate, even when those alternatives are matched for value as inferred from preferences. Such a demonstration would be consistent with conceptions of self-control as involving processes that occur during decision-making, and that result in reduced delay discounting. Moreover, if a discrepancy between choice and non-choice incentive value were quantifiable, it could provide a starting point for operationally defining the extent to which self-control affects choices for specific individuals and specific decisions.
Materials and Methods
Subjects
Forty-three healthy volunteers participated in the study. All subjects gave written informed consent, and the experiment was approved by the Institutional Review Board of the University of Southern California. Before enrolling, volunteers were screened for physical and neurological disorders (using self-report questionnaires) and for current Axis I psychiatric disorders, including substance abuse and dependence (as assessed by the Mini International Psychiatric Interview). Of the forty-three subjects, three were excluded from analysis due to failure to reach stability criteria in the adaptive delay discounting task (described below). In addition, one subject was excluded subsequent to scanning due to history of stroke (not reported during initial screening), one subject was excluded due to a neurological abnormality observed (severe ventricular enlargement) and another subject was excluded due to an operational error during scanning. Among the thirty-seven subjects included in analyses, 19 were female. Ages ranged from 21 to 44 (mean 32.1 ± 6.9).
Subjects were informed that they could win bonuses up to $160 during the course of their participation and that these bonuses would be awarded in the form of Visa gift cards that they would receive at the end of the session. They were further instructed that some available bonus earnings would be delayed, and if they won delayed bonus earnings, the Visa gift cards would not register that money as available until the specified date. Finally, subjects were informed that if they won delayed earnings and lost the Visa gift cards before the specified date, they would be provided with a replacement card.
All subjects first completed a computerized version of the Monetary-Choice Questionnaire developed by Kirby et al. (1999). Subjects were presented with a fixed set of twenty-seven choices between smaller immediate rewards (ranging from $11 to $80) and larger delayed rewards (ranging in amount from $20 to $85 and in delay from 7 to 186 days). These responses were used to derive an initial estimate of the subject’s level of delay discounting using the hyperbolic discount function,
Equation 1 |
in which V is value, A is amount, D is delay in days, and k is the fit parameter that quantifies level of discounting, with k = 0 indicating no delay discounting and higher values of k indicating steeper discounting (for details of the estimation procedure utilized, see Monterosso et al., 2007). This model makes the simplifying assumption that value scales linearly with amount. In the present context in which participants choose between sooner smaller and later larger rewards, unmodeled concavity in the actual association between amount and value results in inflation of best-fit values for the k parameters (see Pine et al, 2009).
The individual estimate of discounting from the Monetary-Choice questionnaire was used as the starting value in a second computer-administered delay discounting choice task, this one employing adaptive questioning in order to gain precision in determining indifference pairs of rewards. On each trial, subjects were presented with a choice between a larger later reward (LL) and a smaller sooner reward (SS). Participants were informed that one trial would be selected from this procedure and that they would receive the alternative they selected on that trial. The delay of the LL was always 120 days and the SS was always zero delay. For half the trials, the LL was $28 ± $7 (“Low”) and for half the trials the LL was $53 ± $7 (“High”). The magnitude of the SS was initially generated by computing what would be an amount of equal value to the LL based on the fit parameter for the subject (k-value) derived from responses on the Monetary-Choice Questionnaire, as modeled using Equation 1. On each trial, if the participant chose the SS alternative, then the k-parameter associated with that reward was adjusted upward a quarter step on a log10 scale and consequently, the SS on the next trial in which that LL appeared was lower. Conversely, if the participant chose the LL, then the k-parameter associated with that reward was adjusted downward (again, a quarter step on a log10 scale) resulting in a higher SS on the next trial in which that LL appeared. This adjusting procedure continued until subjects reached stability for both reward pairs, with stability operationalized as a window of eight trials in which k-values did not deviate by more than two steps. Participants who did not reach this criterion after eight minutes (n = 3) were excluded from the study. The final indifference pairs were then generated using the geometric mean of the k-values for the eight trials during which stabilization was achieved (separately for the Low and High LL amounts). In this way, two indifference pairs were established for further investigation during fMRI. For analyses that required a single discount parameter estimate for each participant we used the geometric mean of these two k-parameter estimates.
Monetary Incentive Delay task
In the next step of the study, participants performed a variant of the Monetary Incentive Delay (MID) task (Knutson, Adams, Fong and Hommer 2001a). Prior to the scan, participants were trained to associate each of the two SS and two LLs comprising the derived indifference pairs with each of four colored shapes (LLs were always $28 in four months and $53 in four months, and SSs were individualized to the participant’s performance.) The pairing of the colored shapes with rewards was counterbalanced across participants. Subjects completed a computerized memory training program in which the four pairs of colored squares and corresponding “prizes” briefly appeared on the screen, one at a time, and were instructed to memorize the pairs. Next, subjects completed a memory test, during which the colored squares flashed on the screen, and subjects reported the corresponding prizes. Subjects were asked to provide the prize in the appropriate format (e.g., amount followed by delay, or vice versa depending on counterbalanced order) and received feedback about their responses. Upon satisfactory completion of the memory test, subjects completed a practice version of the task, similar to the fMRI version.
Each trial of the MID task began with the appearance of a colored shape that indicated which of the four rewards was available on that trial. After an anticipation period of between 4 and 4.5 seconds subsequent to presentation of the available reward stimulus, either a target appeared (the character “+”) or the words “no target” appeared (50% of the time). If the target appeared, the participant was required to respond as quickly as possible with a button press in order to win on the trial. Participants were instructed that their likelihood of winning would be greater if they responded faster, although in reality, outcome was predetermined in order to optimize orthogonality between anticipation and outcome periods (with the exception of RTs > 500 ms which were always scored as too slow, to avoid suspicion). In order to optimize power, an exponential distributed inter-trial-interval with mean 2s was used. Unlike the standard MID task, the colored shape remained on the screen until the target (or “no target” message) appeared. The critical epoch for analyses was the 4 – 4.5 seconds during which the participant was cued to the possible reward and was readying to try to obtain it. Following prior work with the task (Knutson et al., 2001a), we refer to this throughout as the “anticipation period.” In each run of the task, each of the four targets was presented 16 times. Each participant completed two runs of the task. Participants were instructed that one target trial would be selected from each run, and any money that they won on these trials would be paid, again using Visa cards, with credit activated at the specified date. If their response was not sufficiently fast on the target trial selected, then they did not win a bonus for that run of the task.
fMRI acquisition
fMRI data were collected using 3T Siemens MAGNETOM Tim/Trio scanner with a standard birdcage head-coil in the Dana and David Dornsife Cognitive Neuroscience Imaging Center at University of Southern California. For each participant, sagittal images (256×256×176) with 1×1×1mm3 resolution were obtained by a T1-weighted 3D MPRAGE (magnetization prepared rapid gradient echo) sequence (TI=900 ms, TR=1950 ms, TE=2.26 ms, flip angle=90°). Functional scanning used Echo Planar Imaging (EPI) sequence (TR=2000ms, TE=30ms, flip angle=90°, FOV=192, in-plane resolution=64×64) with PACE (prospective acquisition correction) which helps reduce head motion during data acquisition. Thirty- two axial slices were used to cover the whole cerebral cortex with no gap and the slices were positioned along anterior commissure-posterior commissure plane.
fMRI analysis
fMRI data processing was conducted using FEAT (FMRI Expert Analysis Tool) Version 5.98, part of FSL (FMRIB’s Software Library, www.fmrib.ox.ac.uk/fsl). The first four volumes before the task were automatically discarded by the scanner for T1 equilibrium. For preprocessing, the head movement which was not captured by PACE was corrected in three dimension by MCFLIRT (Jenkinson, Bannister, Brady and Smith, 2002). Six motion parameters were added into the general linear model (GLM) in order to explain variance in signal related to head motion. Data were temporally filtered by a high-pass filter with 100 s cut-off and spatially smoothed by a Gaussian kernel of full-width-half-maximum (FWHM) 5mm. The preprocessed data were then submitted to a GLM which was used to analyze the contributions of experimental factors to blood oxygen level dependent (BOLD) responses. All within-subject statistical analyses were performed in native image space, and then the statistical maps were transformed into standard space before high-level (group) analysis. The transformation into standard space was performed in two steps: EPI images were first aligned to the participant’s own MPRAGE structural scan, and then the image was normalized into standard space (Montreal Neurological Institute (MNI)) using affine transformation (Jenkinson and Smith, 2001).
Our primary analyses targeted brain signal changes during the anticipation period. Since this period ended at the onset of the target stimulus or the onset of the stimulus indicating no-target, all trials could be used in analysis of the anticipation period, irrespective of the trial outcome, resulting in good statistical power. The subsequent variation relating to the target and the outcome of the trial allowed us to better isolate anticipation period effects by reducing covariation with effects related to subsequent events. We carried out two variants of this analysis. In the first variant of this analysis, there were thirteen events modeled: four events during the anticipation period including High-Immediate, Low-Immediate, High-Delay, Low-Delay, and nine events during the feedback period including High-Immediate-Win/Loss, Low-Immediate-Win/Loss, High-Delay-Win/Loss, Low-Delay-Win/Loss and No-Target. Each event was convolved with double-gamma hemodynamic response function and temporal derivatives were added as a covariate of no interest in order to improve statistical sensitivity. Null events were not modeled. In the second variant of this analysis, the anticipation epoch was modeled using three parameters: Value (High or Low), Discount Fraction (the participant-specific denominator in Eq 1) and Amount (the undiscounted monetary amount available). Both of the latter variables were orthogonalized to Value, and Amount was additionally orthogonalized to Discount Fraction. It is important to note that the Value parameter is inferred from preference data. If there is a shift in value in the non-choice context that is a function of delay (as hypothesized), regions tracking this shift would be associated with the Discount Fraction rather than Value parameter.
For both above models, a Cross-Run High-level analysis was performed using a fixed effects model by forcing the random effects variance to zero in FLAME (Beckmann, Jenkinson and Smith, 2003). Results were input to group-level analysis using FLAME stage 1 (Beckmann, et al., 2003; Woolrich, Behrens, Beckmann, Jenkinson and Smith, 2004; Woolrich, 2008).
ROI analyses
Region of interest analyses were carried out. Regions were selected based on prior findings associated with the MID task (e.g., Knutson, et al., 2001a, 2005). Four of the selected ROIs (bilateral putamen, thalamus, caudate, and nucleus accumbens) were defined based on the automated segmentation tool FIRST, which is specifically designed to classify subcortical structures (Patenaude et al., 2007). Since this tool is not applicable to the additional ROIs selected (the left and right anterior insula, the midbrain and the supplementary motor area) we adopted an alternative strategy for these regions, drawing 6 mm spheres around the peek coordinates within the region, reported in a prior study that included a contrast isolating sensitivity to reward magnitude (Knutson et al., 2005; coordinates converted into MNI space.) In order to examine whether reward magnitude and immediacy affected MR signal change in these regions during the anticipated period, extracted beta-values for each of the four rewards were subjected to repeated measure analysis with magnitude (high versus low rewards), delay (immediate versus four months delay), and brain region included as within-subject variables. In order to examine the possibility that particular regions might show more sensitivity to delay, and others might show more sensitivity to magnitude, we repeated the analysis, with Magnitude and Delay recoded as two levels of a within subject variable (“Dimension”), each of which in turn included two levels (High and Low, and Immediate and Delayed). An interaction between Dimension and Region would provide evidence of differential sensitivity to magnitude and delay.
Whole brain analyses
The primary whole-brain analysis was based on the above model in which the anticipation epoch was modeled with four separate events corresponding to the four rewards. In this, we compared preference-matched immediate and delayed rewards (Immediate – Delayed, and Delayed – Immediate), thresholded using cluster detection statistics, with a height threshold of Z > 2.3 and a cluster probability of p<.05 corrected for search space (Worsley, 2001). Contrasts were also performed isolating sensitivity to reward magnitude (High versus Low, and Low versus High) across the entire brain. Exploratory conjunction analyses were carried out examining the overlap between High – Low contrasts (this time without correction for multiple comparisons) and both Immediate – Delay and Delay – Immediate contrasts (without search space correction). Because visualization related to this analysis was not corrected for multiple comparisons, we do not use it to support hypothesis testing, but include it as the observed pattern with this less conservative thresholding is, we think, informative.
A parametric analysis was also performed, as described above, in which Value (High or Low), Discount Fraction (the participant-specific denominator in Eq 1) and Amount (the actual monetary amount available) were used to model change in signal during anticipation. In this analysis, we were particularly interested in whether the Discount Fraction predicted response during anticipation after variance related to Value was modeled.
Subjective ratings
Subsequent to the completion of the task, subjects were instructed to rank the four rewards according to the following instructions, “Rank the prizes in terms of how each made you feel at the moment you were going for them during the game. If there are any that are exactly tied, you can give them the same rank… This is a little different than asking you which one you would choose. Don’t worry about which prize you would choose if you compared them, just think about how each made you feel at the moment you were going for the prizes.”
Post-MID choice task
In order to test for the presence of systematic drift in discounting, a subset of participants (N=15) completed a choice task after the MID task. These participants were presented with 24 choice trials in which alternatives were value-matched based on the discount parameter previously derived in the adaptive choice procedure and 24 choice trials that were mismatched based on the same previously derived discount parameter. These mismatched trials were generated by creating indifference pairs based on a k fit parameter estimate (Equation 1) that was one log unit larger (50% of trials) or one log unit smaller (50% of trials) than the participant’s actual fit parameter estimate.
Results
Intertemporal choice task
Across participants, the immediate reward amounts that formed indifference pairs with $28 delayed by four months ranged from $3 to $25, with a median of $13 (corresponding to k = .010, Equation 1). Across participants, the reward amounts that formed indifference pairs with $53 delayed by four months ranged between $6 and $52, with a median of $28 (corresponding to k = .007).
Behavioral results for the Monetary Incentive Delay task
Median reaction time data for each subject on the MID task were modeled using repeated-measures analysis of variance with magnitude and delay included as within-subject independent variables. Magnitude was coded as High for both $53 delayed by four months and for the immediate amount that was equally preferred to it (individualized to the participant). Magnitude was coded as Low for the $28 delayed by four months and for the immediate amount that was equally preferred to it (also individualized to the participant). RT was faster for the High magnitude trials than the Low magnitude (F(1,36) = 4.38, p < .05) and faster for the Immediate reward trials than for the (preference-matched) Delayed reward trials (F(1,36) = 9.1, p < .01; see Figure 1).
Figure 1.
Mean and standard error of individual median reaction time by condition on target trials (computed as distance from the overall median for each participant.) Based on repeated measures ANOVA, RT’s were significantly faster for both the High versus Low pair, and for the Immediate versus Delayed rewards.
Neuroimaging ROI analyses
Beta-values were extracted during anticipation for twelve anatomically defined ROIs for each of the four rewards (High Immediate, High Delayed, Low Immediate, Low Delayed). These values were subjected to a repeated measure analysis, with Magnitude, Delay, and Region included as within-subject variables. Significant main effects were observed for amount (F(1, 36) = 6.46, p < .05), delay (F(1,36) = 4.32, p < .05), and for region (F (11, 26) = 14.7, p < .001). When Magnitude and Delay were recoded as two levels of the variable Dimension (as described above) no interactions were observed between Dimension and Region. Difference scores highlighting the immediacy effect (beta values for the two Immediate rewards minus beta-values for the two Delayed rewards) and the magnitude effect (beta-values for the two High rewards minus that for the two Low rewards) are presented for each ROI in Table 1 (see also Supplementary Figure 1). As can be seen, all difference scores were positive (indicating higher beta-values for immediate relative to delayed rewards, and higher beta-values for high magnitude rewards relative to low magnitude.) In six of the twelve anatomical ROIs, difference scores were greater than zero at least at a trend level (2-tailed, α = .1) for both amount and delay. Two additional regions reached this threshold only for amount, and two more regions, only for delay. In general, difference scores across different regions were highly correlated for both Delay – Immediate and High – Low (Chronbach’s alpha = .95 and .92 respectively).
Table 1.
Magnitude Effect (High – Low) | Delay Effect (Immediate – Delay) | |||
---|---|---|---|---|
beta values | p -value | beta values | p -value | |
L Caudate | 4.97 ± 22.06 | .18. | 4.51 ± 21.64 | .21 |
R Caudate | 7.48 ± 20.35 | .03 | 4.71 ± 20.11 | .16 |
L Putamen | 4.43 ± 14.48 | .07 | 7.77 ± 19.92 | .02 |
R putamen | 5.19 ± 14.23 | .03 | 6.26 ± 18.4 | .04 |
L Thalamus | 6.13 ± 16.9 | .03 | 5.80 ± 19.9 | .08 |
R Thalamus | 7.74 ± 16.79 | .008 | 6.15 ± 20.52 | .08 |
L Nucleus Accumbens | 4.61 ± 25.29 | .28 | 4.27 ± 30.43 | .40 |
R Nucleus Accumbens | 7.05 ± 23.23 | .07 | 1.62 ± 27.32 | .72 |
L Insula 6 mm sphere at −30, 20, 2 | 6.53 ± 23.96 | .11 | 9.21 ± 23.83 | .02 |
R Insula 6 mm sphere at 30, 19, 10 | 4.34 ± 19.36 | .18 | 5.66 ± 16.58 | .04 |
Midbrain 6 mm sphere at 0, −18, −13 | 7.96 ± 19.03 | .02 | 8.29 ± 26.05 | .05 |
Supplementary Motor Area 6 mm sphere at 0, −24, 50 | 10.49 ± 31.00 | .047 | 14.80 ± 35.98 | .017 |
Neuroimaging whole brain analyses
We also carried out a whole-brain analysis comparing MR-signal during anticipation of immediate rewards with signal during anticipation of the preference-matched delayed rewards. We used cluster detection statistics with the height threshold of Z > 2.3, p <.05 cluster-level correction for search space. As shown in Figure 2, immediate rewards recruited greater signal change than delayed rewards in left caudate, putamen (bilateral), insula (bilateral), left pallidum, supramarginal gyrus (bilateral), anterior cingulate cortex and supplementary motor area (see Immediate > Delayed section of Table 2). Immediate rewards recruited significantly less activation in clusters within the precuneus and occipital cortex.
Figure 2.
Contrast maps for Immediate – Delayed (warm colors indicate Immediate > Delayed, cool colors Delayed > Immediate). All clusters based on whole-brain analysis with voxel threshold of Z = 2.3, and cluster level correction of p < .05. Findings for Immediate > Delayed include left caudate, putamen (bilateral), insula (bilateral), left pallidum, supramarginal gyrus (bilateral), anterior cingulate cortex and supplementary motor area. Findings for Delayed > Immediate include clusters within the precuneus and occipital cortex.
Table 2.
Whole brain | x, y, z | Max Z | |
---|---|---|---|
High-Low | R Caudate | 12,14,4 | 2.83 |
R Thalamus | 8,−4,10 | 3.15 | |
L Thalamus | −12,−22,−4 | 3.34 | |
Lateral Occipital Cortex | −18,−84,12 | 3.5 | |
Occipital Pole | −8,−98,10 | 3.37 | |
Immediate-Delayed | R Insula | 44,12,−4 | 4.17 |
R Putamen | 26,14,−2 | 3.39 | |
R Supramarginal Gyrus | 64,−34,28 | 3.37 | |
R Inferior Frontal Gyrus | 54,18,4 | 2.86 | |
L Insula | −40,16,−4 | 4.03 | |
L Putamen | −16,8,−8 | 3.54 | |
L Caudate | −10,12,2 | 2.9 | |
L Pallidum | −16,4,2 | 3.0 | |
L Supramarginal Gyrus | −60,−34,28 | 3.12 | |
Anterior Cingulate Cortex | 0,22,32 | 2.81 | |
Supplementary Motor Cortex | 0,2,56 | 4.24 | |
Delayed-Immediate | Occipital Cortex | 0,−94,10 | 4.07 |
Precuneous Cortex | 0,−56,24 | 3.26 |
We also contrasted high and low rewards in a whole brain analysis. As shown in Figure 3, cluster-level significance was reached in the right caudate, thalamus (bilateral), lateral occipital cortex, and occipital pole (see High > Low section of Table 2). Higher rewards did not recruit significantly less activity than lower rewards in any region. As an alternative approach, we also modeled the data during anticipation parametrically, as described above. The Value and Discount Fraction predictor variables used in this additional model yielded similar activation maps to those associated with Immediate – Delay, and High – Low contrasts (see Supplementary Figure 2A and 2B.)
Figure 3.
Contrast maps for High – Low rewards (red indicates High > Low; no clusters were observed for Low > High). Findings for High > Low include right caudate, thalamus (bilateral), lateral occipital cortex, and occipital pole.
For exploratory purposes, a conjunction analysis was carried out that identified all voxels that evidenced greater activity (Z > 2.3 voxel height threshold, without cluster correction) in both 1) High > Low and 2) either Immediate > Delay (Figure 4, shown in red) or Delay > Immediate (Figure 4 shown in blue.) The overlap between voxels identified in High – Low and Immediate – Delay was widespread within subcortical structures previously associated with incentive on the MID task (including brainstem, right caudate, bilateral pallidum, bilateral putamen, right thalamus, bilateral insula, and left supramarginal gyrus.) There was one large cluster in the visual cortex in which Delay > Immediate overlapped High > Low voxels. We hypothesized that this activity in visual cortex was likely related to visual attention which might be heightened for different reasons in the two contrasts. To explore this hypothesis, a functional connectivity analysis was carried out in which we used activity in the cluster that overlapped both High > Low and Delayed > Immediate contrasts as a seed to predict activity throughout the rest of the brain, contrasting connectivity with the seed in the Immediate versus Delayed trials. As shown in Supplementary Figure 3, a significant differential functional connectivity effect was observed (p < .05, cluster corrected for search space) in regions overlapping with those previously associated with incentive on the task (putamen, anterior insula, thalamus, pallidum). Specifically, there was significantly greater association with the occipital cortex seed region when rewards were immediate relative to when rewards were delayed.
Figure 4.
Conjunction whole brain maps for High – Low and Immediate – Delayed (red indicates Immediate-Delayed, blue indicates Delayed-Immediate p < .05, uncorrected for each).
Post-hoc correlational analysis indicated no correlation between individual variance in delay discounting as measured by the choice procedure and either variance in the effect of immediacy on RT (r(37) = − .19, p = .26), or of immediacy on MR signal in regions of interest (r(37) = − .10, p = .56). The immediacy effect on RT was significantly correlated with the effect of immediacy on MR signal in a priori ROIs (r(37) = .35, p < .05).
Beta values in the occipital cortex for Delayed > Immediate were not correlated with discounting (r(37) = −.14, p=.40), or with the immediacy effect on RT (r(37) = −.08, p = .62), but were inversely correlated with beta values within clusters identified in the Immediate – Delayed contrast (r(37) = −.49, p = .002).
Subjective ranking questionnaire
The mean ranking for the larger immediate reward was 1.56 ± .84, for the larger delayed reward was 1.58 ± .77, for the smaller immediate reward was 3.14 ± .90, and for the smaller delayed reward was 3.28 ± .78. When subjected to a repeated measures ANOVA, magnitude (High vs. Low) was a highly significant predictor of ranking (F(1,35) = 214.0, p <.001) and delay was not a significant predictor of ranking (F(1,35) = .16, p=.69). Although there was not an indication of a group level effect of immediacy on ranking for the preference-matched pairs, we examined the relation between individual differences in ranking based on immediacy and individual variance in the effect of immediacy on RT and on MR signal change during anticipation. To do this, we computed a difference score for the mean ranking of delayed rewards minus the ranking of immediate rewards. Although this index of the subjective effect of immediacy (given matched preference) did not predict variance in the effect of immediacy on MR signal difference (r(36) = .27, p = .11), the association with the effect of immediacy on RT was significant, and in the anticipated direction (r(36) = .36, p < .05).
Post fMRI delay discounting choice reassessment
In the post MID task reassessment of delay discounting that was administered to a subset of participants (N=15), in trials generated to be at the participant’s indifference point, the mean percentage choice of the SS was 50.3% (± 12.7%). Among trials generated to be mismatched in value, participants chose the option generated to be of higher value (based on their individual estimated discount function) on 93.5% (± 8.5%) of trials.
Discussion
We compared behavior and brain response during anticipation of long delayed (4 months) and smaller, equally preferred immediate monetary rewards (“indifference pairs”). The data unequivocally indicate observable differences between the conditions; when anticipating immediate rewards relative to delayed rewards, responses to target stimuli were faster and neural activity was greater in a network of regions previously implicated in incentive during the task. This was especially evident in the superior portion of the anterior insula (contiguous with frontal operculum) and putamen, where findings were bilateral, and evident in anatomical ROI and whole-brain analyses. In addition to its established association with feeling states (Damasio et al., 2000), the anterior insula is implicated in executive function tasks (Wager, Jonides and Reading, 2004) and may be associated with affective signals accompanying mental effort (Wager and Barrett, 2004). A similar locus of activation was reported using the MID task in contrasts identifying sensitivity to the presence (Knutson, Fong, Adams, Varner and Hommer, 2001b) and magnitude (Knutson, Taylor, Kaufman, Peterson and Glover, 2005) of reward. The putamen is implicated in reinforcement learning (Packard and Knowlton, 2002) and especially in preparation of motor responses (Alexander and Crutcher, 1990). Using the same task, activity in the putamen has also been repeatedly reported during anticipation of rewards (e.g., Knutson, et al., 2001a) and appears to be preferentially recruited during positive incentive (Knutson, et al., 2005). In addition, evidence of differential activity during immediate relative to delayed reward trials was observed in most regions previously implicated as sensitive to reward during the MID task, including the brainstem, pallidum, caudate, supplementary motor area, supermarginal gyrus, and anterior cingulate cortex. The fact that participants who completed a choice task subsequent to the MID task preferred the SS on 50.3% of trials designed to be at the participant’s indifference point indicates findings were not the product of drift towards greater discounting during the experiment.
While these data demonstrate that there was something different during anticipation of preference-matched immediate versus delayed rewards, several reasonable interpretations warrant consideration. One possible basis is that subjects, on average, valued the immediate rewards more than the delayed rewards, despite their having been matched based on revealed preference. This is consistent with conceptions of self-control that posit distinct mechanisms engaged during decision-making that generally shift preference towards greater future orientation. Accordingly, the more immediate reward within a decision-based “indifference pair” would be, on average, of higher value than the more delayed reward if each was encountered in a context where self-control was not operative, as arguably is the case in the MID task. Before returning to this interpretation, we consider two alternative accounts.
Alternatively, it could be hypothesized that the differences in MR signal result from differences in the representation of subcomponents of expected value; regions more active in the immediate reward condition might be specifically sensitive to the dimension of immediacy rather than a difference in overall incentive value. To maintain this as an explanation of observed findings, differential activation should be absent in regions sensitive to overall value. The lack of differences observed in the nucleus accumbens and ventromedial prefrontal cortex could be viewed as supportive. However, in our repeated measures analysis of signal in ROIs implicated in incentive in prior work with the MID task, we found no evidence of specificity in sensitivity to reward magnitude versus reward immediacy. That is, we observed no statistical evidence of divergence within the examined network in sensitivity to the two dimensions. Furthermore, differential activation was observed in regions that are important to the execution of the experimental task, but which are not plausibly substrates representing sub-components of expected value (e.g., the supplementary motor area.) If some of the observed findings based on the Immediate – Delay imaging contrast are related to subcomponents that contribute to value, but that imply no overall divergence in value across the conditions, then an additional explanation would have to be provided to explain activation differences in regions that are unlikely related to subcomponents of expected value, as well as to the observed difference in reaction times to the targets. The correlation between the immediacy effect across brain regions (chronbach’s α = .95) and between fMRI data and reaction time (r=.35, p < .05) make this prospect less convincing. It may be important to note that the MID task utilizes a small set (here four) of highly familiar rewards; it is possible that neural recruitment associated with value differs here from situations in which novel rewards are encountered.
A second alternative account is that observed findings reflect a conditioned response, whereby stimuli signaling immediacy potentiate a motor response (rather than discrepant valuation). Perhaps because of semantic overlap or learning history, cues of immediacy could prime action, and cues of delay prime inaction. The widespread overlap between voxels sensitive to amount and delay (Figure 4) and the absence of any statistical evidence of an interaction involving these predictor variables, fails to lend support. And while visual inspection of the thresholded images invites conjecture that the imbalance in incentive observed between immediate and delayed rewards has some particular association with the potentiation of action, signal change data across ROIs (Table 1, Supplementary Figure 1) does not suggest regional specificity in sensitivity to the two orthogonal independent variables of Magnitude and Immediacy. It is also worth noting that subjects that reported more favorable subjective rankings for immediate rewards tended to demonstrate greater reaction time superiority during immediate reward trials (r = .36, p < .05). This association would not be predicted based on the conditioned response interpretation of our findings.
Finally, it could reasonably be suggested that the primary findings might not be related to choice per se, but rather to the mere juxtaposition of multiple alternatives. Perhaps having another option for consideration, irrespective of whether it has to be chosen against, increases focus on amount rather than on delay. With respect to this possibility, we note that the four rewards we used in the MID task were presented continually in close temporal proximity and so did naturally form a frame for comparison. Also, we know of no a priori basis for supposing juxtaposition would differentially shift attention to the amount dimension over the immediacy dimension. Nevertheless, the possibility that concurrent juxtaposition shifts valuation towards amount and away from immediacy cannot be ruled out (for evidence that choice itself is a context which can change the value of options, in their case, enduringly, see Sharot et al., 2009).
On balance, we believe the data are best explained as the result of on-average higher incentive value for the immediate rewards relative to the equally preferred delayed rewards. However, one aspect of the data appears inconsistent at first blush. Differential activity was observed in a cluster in the visual cortex for High relative to Low, and also, Delayed relative to Immediate reward trials. The results of our functional connectivity provide a clue. Functional connectivity to this occipital cortex cluster (plausibly related to heightened visual attention) was significantly diminished when rewards were delayed in a network of regions implicated in incentive during the task (putamen, anterior insula, thalamus, pallidum). While High relative to Low rewards might differentially recruit visual attention because of differential incentive value (e.g., David, M., Munafò, H. et al., 2005), the reduced connectivity to basal ganglia activity during delayed rewards suggests heightened visual attention during the delay condition related to something other than value. So while the basis for greater activation in this region for delayed relative to immediate rewards remains unclear, connectivity results suggest it is not based on heightened incentive value. It should be noted that unlike the standard MID task, the incentive stimulus was visually displayed in the present study throughout the anticipation period.
Interpretation of findings
Our findings are consistent with theories that posit that self-control includes engagement of processes during decision-making (as opposed to during valuation more generally.) On this interpretation, the heightened incentive for the immediate rewards relative to the preference-matched delayed rewards reflects removal of the typically moderating influence that self-control processes had on the tendency to devalue delayed rewards (or the tendency to exhibit concavity in the association between amount and value, which in this context would similarly result in greater preference for sooner smaller over later larger). This suggests that two factors contribute to response to delay during intertemporal choices. The first factor is the direct effect delay has on incentive value, which we expect is itself complexly determined, and subject to framing effects (e.g., whether the delay is expressed as a waiting period, or by specifying the day the reward will be received, as in Read et al., 2005). The second factor consists of self-control processes engaged during decision making, which tend to push preference towards later, larger rewards. Bringing this dissociation under quantitative analysis may be illuminating. For example, to the extent that farsighted choices are based on self-control processes engaged during explicit decision-making (second of the above factors), the individual may be more vulnerable to short-sightedness in the presence of anything that selectively undermines higher cognitive functions, such as, fatigue or distraction. Alternatively, some framing manipulations could selectively influence the direct incentive value of rewards (first of the above factors) but have influence over high level decision-making, if the decision-making process explicitly ignores the frame.
Although the present work does not investigate the mechanisms underlying self-control, by examining valuation outside of a decision-context, these data provide support for the conceptions of self-control as entailing one or more processes that are engaged during decision-making, and that generally result in less shortsightedness in behavior than would be predicted by the individual incentive value of immediate and delayed rewards.
Supplementary Material
Supplementary Figure 1. 95% CI of the effect sizes for the magnitude effect (High – Low, shown in full lines) and immediacy effect (Immediate – Delayed, shown in dashed lines) of all ROI’s (bilateral ROI’s collapsed).
Supplementary Figure 2. Parametric analysis with anticipation modeled with three parameters: Value (High or Low), Discount Fraction (the participant-specific denominator in Eq 1) and Amount (the actual monetary amount available). Both of the latter variables were orthogonalized to Value, and Amount was additionally orthogonalized to Discount Fraction. Areas associated with Value (panel A) include right caudate, and thalamus (bilateral). Areas associated with Discount Fraction (panel B) in the direction of greater activity in association with the greater immediacy include, right putamen, insula (bilateral), supramarginal gyrus (bilateral), and right inferior frontal gyrus. Conversely, greater delay was associated with greater activity in clusters within the precuneus and occipital cortex. Finally, higher amounts were associated with higher activity in clusters within the occipital cortex (bilateral).
Supplementary Figure 3. Connectivity analysis using the cluster common to High > Low and Delayed > Immediate (which was within occipital cortex) as the seed region. Green areas were significantly more associated with seed region during Immediate reward trials than during Delayed reward trials. Clusters in putamen, anterior insula, thalamus, pallidum were all significant, p < .05, controlling for whole-brain search space.
Acknowledgments
The authors would like to thank Antonio Damasio, Gui Xue and two anonymous reviewers for helpful comments on an earlier version of this manuscript. The authors also thank Xochitl Cordova and Jodi Stone who carried out data collection utilized in this report. This work was supported by the National Institute of Health R01DA023176 (JM).
References
- Ainslie G. Picoeconomics. New York: Cambridge University Press; 1992. [Google Scholar]
- Ainslie G. Precis of breakdown of will. Behav Brain Sci. 2005;28:635–650. doi: 10.1017/S0140525X05000117. [DOI] [PubMed] [Google Scholar]
- Ainslie G, Monterosso J. Hyperbolic discounting as a factor in addiction: A critical analysis. In: Vuchinich R, Heather N, editors. Choice, Behavioral Economics, and Addiction. Elsevier; 2003. pp. 35–69. [Google Scholar]
- Alexander GE, Crutcher MD. Preparation for movement: neural representations of intended direction in three motor areas of the monkey. J Neurophysiol. 1990;64(1):133–150. doi: 10.1152/jn.1990.64.1.133. [DOI] [PubMed] [Google Scholar]
- Barkley RA. ADHD and the Nature of Self-Control. London: Guilford Press; 1997. [Google Scholar]
- Bem DJ. Self-Perception Theory. In: Berkowitz, editor. Advances in Experimental Social Psychology. Vol. 6. NY: Academic Press; 1972. pp. 1–62. [Google Scholar]
- Bénabou R, Tirole J. Willpower and Personal Rules. Journal of Political Economy. 2004;112(4):848–887. [Google Scholar]
- Bodner R, Prelec D. Self Signaling and Diagnostic Utility in Everyday Decision Making. In: Brocas I, Carrillo JD, editors. The Psychology of Economic Decisions. Oxford: Oxford Univ. Press; 2001. pp. 105–123. [Google Scholar]
- Broadbendt DF. Stmulus set and response set: Two kinds of selective attention. In: Mostofsky DI, editor. Attention: Contemporary Theory and Analysis. NY: Appleton; 1970. pp. 51–60. [Google Scholar]
- Beckmann C, Jenkinson M, Smith SM. General multi-level linear modelling for group analysis in FMRI. NeuroImage. 2003;20:1052–1063. doi: 10.1016/S1053-8119(03)00435-X. [DOI] [PubMed] [Google Scholar]
- Damasio AR, Grabowski TJ, Bechara A, Damasio H, Ponto LL, Parvizi J, et al. Subcortical and cortical brain activity during the feeling of self-generated emotions. Nat Neurosci. 2000;3(10):1049–1056. doi: 10.1038/79871. [DOI] [PubMed] [Google Scholar]
- David S, Munafò M, Johansen-Berg H, Smith S, Rogers R, Matthews P, Walton R. Ventral Striatum/Nucleus Accumbens Activation to Smoking-Related Pictorial Cues in Smokers and Nonsmokers: A Functional Magnetic Resonance Imaging Study. Biological Psychiatry. 2005;58(6):488–494. doi: 10.1016/j.biopsych.2005.04.028. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Goldstein R, Volkow N. Drug addiction and its underlying neurobiological basis: Neuroimaging evidence for the involvement of the frontal cortex. Am J Psychiatry. 2002;159:1642–1652. doi: 10.1176/appi.ajp.159.10.1642. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hare T, Camerer C, Rangel A. Self-contrl in decision-making invovles modulation of the vmPFC valuation system. Science. 2009 doi: 10.1126/science.1168450. [DOI] [PubMed] [Google Scholar]
- Jentsch JD, Taylor JR. Impulsivity resulting from frontostriatal dysfunction in drug abuse:implications for the control of behavior by reward-related stimuli. Psychopharmacology (Berl) 1999;146(4):373–390. doi: 10.1007/pl00005483. [DOI] [PubMed] [Google Scholar]
- Kable J, Glimcher P. The neural correlates of subjective value during intertemporal choice. nature neuroscience. 2007;10(12):1625–1633. doi: 10.1038/nn2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kirby KN, Petry NM, Bickel WK. Heroin addicts have higher discount rates for delayed rewards than non-drug-using controls. Journal of Experimental Psychology: General. 1999;128(1):78–87. doi: 10.1037//0096-3445.128.1.78. [DOI] [PubMed] [Google Scholar]
- Knutson B, Adams C, Fong G, Hommer D. Anticipation of Increasing Monetary Reward Selectively Recruits Nucleus Accumbens. The journal of Neuroscience. 2001a;21(RC159):1–5. doi: 10.1523/JNEUROSCI.21-16-j0002.2001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Knutson B, Fong GW, Adams CM, Varner JL, Hommer D. Dissociation of reward anticipation and outcome with event-related fMRI. Neuroreport. 2001b;12(17):3683–3687. doi: 10.1097/00001756-200112040-00016. [DOI] [PubMed] [Google Scholar]
- Knutson B, Taylor J, Kaufman M, Peterson R, Glover G. Distributed Neural Representation of Expected Value. The journal of Neuroscience. 2005;25(19):4806–4812. doi: 10.1523/JNEUROSCI.0642-05.2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Logue AW, Forzano LB, Tobin H. Independence of reinforcer amount and delay: the generalized matching law and self-control in humans. Learning & Motivation. 1993;23:326–342. [Google Scholar]
- Logue AW, Pena-Correal TE, Rodriguez ML, Kabela E. Self-control in adult humans: variation in positive reinforcer amount and delay. J Exp Anal Behav. 1986;46(2):159–173. doi: 10.1901/jeab.1986.46-159. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Jenkinson M, Bannister P, Brady M, Smith S. Improved optimisation for the robust and accurate linear registration and motion correction of brain images. NeuroImage. 2002;17(2):825–841. doi: 10.1016/s1053-8119(02)91132-8. [DOI] [PubMed] [Google Scholar]
- Jenkinson M, Smith SM. A Global Optimisation Method for Robust Affine Registration of Brain Images. Medical Image Analysis. 2001;5(2):143–156. doi: 10.1016/s1361-8415(01)00036-6. [DOI] [PubMed] [Google Scholar]
- McClure S, Ericson K, Laibson D, Loewenstein G, Cohen J. Time Discounting for Primary Rewards [Behavioral/Systems/Cognitive] Journal of Neuroscience. 2007;27(21):5796–5804. doi: 10.1523/JNEUROSCI.4246-06.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- McClure S, Laibson D, Loewenstein G, Cohen J. Separate Neural Systems Value Immediate and Delayed Monetary Rewards. Science. 2004;306:503–507. doi: 10.1126/science.1100907. [DOI] [PubMed] [Google Scholar]
- Mesulam MM, Mufson EJ. Insula of the old world monkey. I. Architectonics in the insulo-orbito-temporal component of the paralimbic brain. J Comp Neurol. 1982;212(1):1–22. doi: 10.1002/cne.902120102. [DOI] [PubMed] [Google Scholar]
- Miller E, Cohen J. An integrative theory of prefrontal cortex function. Annual Review of Neuroscience. 2001;24:167–202. doi: 10.1146/annurev.neuro.24.1.167. [DOI] [PubMed] [Google Scholar]
- Monterosso J, Ainslie G. The behavioral economics of will in recovery from addiction. Drug Alcohol Depend. 2007;90(Suppl 1):S100–111. doi: 10.1016/j.drugalcdep.2006.09.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Monterosso J, Ainslie G. The picoeconomic approach to addictions: Analyzing the conflict of successive motivational states. Addiction Research & Theory. 2009;17(2):115–134. [Google Scholar]
- Monterosso J, Ainslie G, Xu J, Cordova X, Domier C, London E. Frontoparietal Cortical Activity of Methamphetamine-Dependent and Comparison Subjects Performing a Delay Discounting Task. Human Brain Mapping. 2007;28:383–393. doi: 10.1002/hbm.20281. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Monterosso J, Luo S. Are there two valuation systems? The case for and against System 1 Vs. 2 competition as the basis of dynamic inconsistency. Journal of Neuroscience, Psychology and Economics, Submitted Manuscript 2009 [Google Scholar]
- Packard MG, Knowlton BJ. Learning and memory functions of the Basal Ganglia. Annu Rev Neurosci. 2002;25:563–593. doi: 10.1146/annurev.neuro.25.112701.142937. [DOI] [PubMed] [Google Scholar]
- Patenaude B, Smith S, Kennedy D, Jenkinson M. FIRST - FMRIB’s integrated registration and segmentation tool. Human Brain Mapping Conference.2007. [Google Scholar]
- Pine A, Seymour B, Roiser JP, Bossaerts P, Friston K, Curran HV, Dolan RJ. Encoding of marginal utility across time in the human brain. J Neurosci. 2009;29(30):9575–9581. doi: 10.1523/JNEUROSCI.1126-09.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Read D, Frederick S, Orsel B, Rahman J. Four Score and Seven Years from Now: The Date/Delay Effect in Temporal Discounting. Management Science. 2005;51(9):1326–1335. [Google Scholar]
- Rushworth MF, Nixon PD, Renowden S, Wade DT, Passingham RE. The left parietal cortex and motor attention. Neuropsychologia. 1997;35(9):1261–1273. doi: 10.1016/s0028-3932(97)00050-x. [DOI] [PubMed] [Google Scholar]
- Sharot T, De Martino B, Dolan RJ. How choice reveals and shapes expected hedonic outcome. J Neurosci. 2009;29:3760–3765. doi: 10.1523/JNEUROSCI.4972-08.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wager TD, Barrett LF. From affect to control: functional specialization of the insula in motivation and regulation. PsycExtra (online) 2004 [Google Scholar]
- Wager TD, Jonides J, Reading S. Neuroimaging studies of shifting attention: a meta-analysis. Neuroimge. 2004;22(4):1679–1693. doi: 10.1016/j.neuroimage.2004.03.052. [DOI] [PubMed] [Google Scholar]
- Woolrich MW. Robust Group Analysis Using Outlier Inference. NeuroImage. 2008;41(2):286–301. doi: 10.1016/j.neuroimage.2008.02.042. [DOI] [PubMed] [Google Scholar]
- Woolrich MW, Behrens TEJ, Beckmann CF, Jenkinson M, Smith SM. Multi-level linear modelling for FMRI group analysis using Bayesian inference. NeuroImage. 2004;21(4):1732–1747. doi: 10.1016/j.neuroimage.2003.12.023. [DOI] [PubMed] [Google Scholar]
- Worsley KJ. Statistical analysis of activation images. In: Jezzard P, Matthews PM, Smith SM, editors. Functional MRI: An Introduction to Methods. 2001. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplementary Figure 1. 95% CI of the effect sizes for the magnitude effect (High – Low, shown in full lines) and immediacy effect (Immediate – Delayed, shown in dashed lines) of all ROI’s (bilateral ROI’s collapsed).
Supplementary Figure 2. Parametric analysis with anticipation modeled with three parameters: Value (High or Low), Discount Fraction (the participant-specific denominator in Eq 1) and Amount (the actual monetary amount available). Both of the latter variables were orthogonalized to Value, and Amount was additionally orthogonalized to Discount Fraction. Areas associated with Value (panel A) include right caudate, and thalamus (bilateral). Areas associated with Discount Fraction (panel B) in the direction of greater activity in association with the greater immediacy include, right putamen, insula (bilateral), supramarginal gyrus (bilateral), and right inferior frontal gyrus. Conversely, greater delay was associated with greater activity in clusters within the precuneus and occipital cortex. Finally, higher amounts were associated with higher activity in clusters within the occipital cortex (bilateral).
Supplementary Figure 3. Connectivity analysis using the cluster common to High > Low and Delayed > Immediate (which was within occipital cortex) as the seed region. Green areas were significantly more associated with seed region during Immediate reward trials than during Delayed reward trials. Clusters in putamen, anterior insula, thalamus, pallidum were all significant, p < .05, controlling for whole-brain search space.