Preference for Immediate over Delayed Rewards Is Associated with Magnitude of Ventral Striatal Activity

Abstract

Discounting future outcomes as a function of their deferred availability underlies much of human decision making. Discounting, or preference for immediate over delayed rewards of larger value, is often associated with impulsivity and is a risk factor for addictive disorders such as pathological gambling, cigarette smoking, and drug and alcohol abuse. The ventral striatum (VS) is involved in mediating behavioral responses and physiological states associated with reward, and dysregulation of the VS contributes to addiction, perhaps by affecting impulsive decision-making. Behavioral tests of delay discounting (DD), which index preference for smaller immediate over larger delayed rewards, covary with impulsive tendencies in humans. In the current study, we examined the relationship between individual differences in DD, measured in a behavioral assessment, and VS activity measured with blood oxygenation level-dependent functional magnetic resonance imaging, in 45 adult volunteers. VS activity was determined using a task involving positive and negative feedback with monetary reward. Analyses revealed that individual differences in DD correlate positively with magnitude of VS activation in response to both positive and negative feedback, compared with a no-feedback control condition. Variability in DD was also associated with differential VS activation in response to positive, compared with negative, feedback. Collectively, our results suggest that increased preference for smaller immediate over larger delayed rewards reflects both a relatively indiscriminate and hyper-reactive VS circuitry. They also highlight a specific neurocognitive mechanism that may contribute to increased risk for addiction.

Introduction

People naturally favor larger over smaller rewards and rewards received sooner rather than later, but their preferences differ appreciably when given a choice between smaller immediate and larger delayed rewards. Discounting future outcomes underlies much of human decision making and figures prominently in several overlapping psychological constructs, such as self-regulation, impulse-control, delay of gratification, and intertemporal choice (Manuck et al., 2003). Moreover, individuals who strongly prefer immediate over deferred rewards of larger value are often generally impulsive or lacking in self-control and are more likely than others to engage in addictive behaviors such as pathological gambling, cigarette smoking, and drug and alcohol abuse (Madden et al., 1997; Bickel et al., 1999; Kirby et al., 1999; Alessi and Petry, 2003).

Through reciprocal cortical and subcortical connections, the nucleus accumbens and, more broadly, the ventral striatum (VS), contribute to the motivational salience of stimuli and abet appetitive or reward-dependent behaviors (Berridge and Robinson, 2003). Activity of the VS increases in response to both the anticipation and receipt of rewarding stimuli, including primary (e.g., food) and secondary (e.g., money) reinforcers (O'Doherty, 2004). Moreover, in addiction, craving and compulsive drug seeking, as well as sensitivity to drug cues, are associated with exaggerated VS activity (Kalivas and Volkow, 2005). Because the response of the VS involves an immediate response to rewards, the magnitude of VS activity may contribute to individual differences in relative preference for immediate, compared with delayed, rewards.

In experimental research on intertemporal choice, discounting of future rewards, or delay discounting (DD), is a well characterized behavioral measure of preference for immediate over delayed rewards and provides an index of impulsive tendencies in humans (Green and Myerson, 2004). Behavioral tests used to derive estimates of DD commonly ask participants to choose between multiple immediate rewards that vary in value and a constant, larger reward available after varying intervals of delay. In such tasks, rates of discounting often differ appreciably and consistently among individuals (Simpson and Vuchinich, 2000). Thus, DD represents a potentially important psychometric index of individual differences in present- versus future-oriented tendencies that may be related to the magnitude of VS activity associated with reward.

Recently, McClure et al. reported that VS activity was significantly higher when subjects chose a smaller, immediate monetary reward compared with a larger, delayed reward (McClure et al., 2004). Activity of interconnected regions of the prefrontal cortex (PFC), namely the medial PFC and orbitofrontal cortex (OFC), were also positively correlated with selection of smaller, immediate rewards. In contrast, selection of larger, delayed rewards was associated with activity in the lateral OFC and dorsolateral PFC. Although their study provided evidence that VS activity is tightly coupled with immediate rather than delayed reward, it did not examine the relationship between the magnitude of VS activity and individual differences in reward preference. In the current study, we explored the relationship between reward preference, as indexed by performance on a computerized DD task, and the magnitude of blood oxygenation level-dependent (BOLD) functional magnetic resonance imaging (fMRI) determined VS activity during a separate task involving positive and negative feedback with monetary reward. Notably, the DD measure was obtained on a separate experimental session, consistent with the idea that this measure reflects a stable individual trait (Simpson and Vuchinich, 2000). Based on previous research, we hypothesized that the magnitude of VS activity associated with positive feedback (i.e., rewarding stimuli) during our fMRI challenge would be positively correlated with preference for immediate over delayed rewards as indexed by the DD function. We also explored the relationship between interindividual variability in DD and the activity of prefrontal regions that McClure et al. (2004) identified during the selection of smaller, immediate (i.e., medial PFC and OFC) in contrast to larger, delayed (i.e., lateral OFC and dorsolateral PFC) rewards.

Materials and Methods

Subjects.

Delay discounting and fMRI data were available on 45 subjects (28 women; mean age, 43.5 years ± 6.5 SD), all of whom were participants in a parent study, the Adult Health and Behavior (AHAB) project. The AHAB project involves assessment of a wide range of behavioral and biological traits among nonpatient, middle-aged, community volunteers. All subjects provided informed consent in accord with guidelines of the University of Pittsburgh Institutional Review Board. Thirty-seven subjects were European American, 5 African American, 1 Asian, and 2 of bi-racial parentage. All subjects were in good general health and free of the following study exclusions: (1) medical diagnoses of cancer, stroke, diabetes requiring insulin treatment, chronic kidney or liver disease, or lifetime history of psychotic symptoms; (2) use of psychotropic, glucocorticoid, or hypolipidemic medication; (3) conditions affecting cerebral blood flow and metabolism (e.g., hypertension); and (4) diagnosis of any current Diagnostic and Statistical Manual of Mental Disorders IV Axis I disorder.

Assessment of delay discounting.

As a component of AHAB participation, subjects completed a computerized DD task that required choices between immediate and postponed rewards (see De Wit et al., 2007). Subjects chose between hypothetical amounts of money available that day ($0.10 to $105) and $100 available after a delay of 0, 7, 30, 90, 180, 365, or 1825 d (5 years). All combinations of immediate reward and delay interval were presented on a computer screen in randomized order. Following procedures described by Mitchell (1999), a “switch point” was defined for each delay interval (0, 7, 30… 1825 d) as the midpoint value (in dollars) between the lowest immediate reward ($0.10 to $105) selected by the subject and the next lowest immediate reward in sequence (i.e., the value of immediate reward at which the subject began consistently to select the standard $100 delayed reward). As an example, at an anticipated delay of 90 d, a given participant might choose an immediate reward of $60 or higher, but whenever the immediately available reward was $55 or less, the participant would select the alternative $100 after 90 d. The switch point value (in this example, $57.50) is taken as the amount of immediate reward at which the subject is indifferent with respect to selection of immediate or delayed outcomes.

The primary analysis of the seven indifference points (switch points) was based on their fit to the hyperbolic function: V = A/(1 + kD), where V is the value of the delayed outcome (i.e., the indifference value), A is the fixed $100 delayed reward, D is the length of the delay, and k expresses the steepness of the discount function (Mitchell, 1999; Richards et al., 1999; De Wit et al., 2007). The equation was fit to the 7 delay indifference points using a nonlinear curve-fitting program (Origin 6.1, 1991-2000; OriginLab, Northampton, MA). This curve-fitting program determined the best-fitting values for k and the coefficient of determination for delay discounting (Mitchell, 1999). Larger k values indicate that the delayed rewards are being discounted more steeply, and consequently, that the subject is more impulsive. An r² value was calculated for each subject to indicate how well the data points fit the hyperbolic function; the median r² among our 45 study participants was 0.90 (range, 0.51-.99) and nearly identical to that of the larger AHAB sample from which these subjects were derived (n = 606; median, r² = 0.92) (De Wit et al., 2007). Finally, although the rewards in this study were hypothetical, several studies have shown that hypothetical and actual rewards produce similar patterns of discounting (Johnson and Bickel, 2002; Madden et al., 2003).

Striatal reactivity paradigm.

On average, subjects participated in the fMRI study 29.30 ± 13.55 weeks (minimum, 10.43; maximum, 63.29) after they had completed the DD procedure during a separate study component. A modified fMRI paradigm based on the work of Delgado et al. (2000) was used to probe VS activity associated with positive and negative feedback in the broader context of monetary reward. Our blocked design consisted of a pseudorandom presentation of trials wherein participants played a card-guessing game resulting in positive or negative feedback for each trial. Subjects were told that their performance on the card game would determine a monetary reward to be received at the end of the game. During each trial, subjects had 3 s to guess, via button press, whether the value of an upcoming visually presented card would be > or <5 (index and middle finger, respectively). After a choice was made, the numerical value of the card was presented (higher or lower) for 500 ms and followed by appropriate feedback (green “up” arrow for positive feedback on a correct trial; red “down” arrow for negative feedback on an incorrect trial) for an additional 500 ms. A crosshair focus point was then presented for 3 s for a total trial length of 7 s. Each task block was comprised of five trials, with three blocks each of predominantly positive feedback (75% correct) and three of predominantly negative feedback (25% correct). We included an incongruent trial type within each task block (e.g., one of four trials during positive feedback blocks was incorrect, resulting in negative feedback) to prevent subjects from anticipating the feedback for each trial and maintain subject's engagement and motivation to perform well. The six task blocks were interleaved with three control blocks. During control blocks, subjects were instructed to simply make alternating button presses during the presentation of an “x” (3 s), which was followed by an asterisk (500 ms) and a yellow circle (500 ms). Each block was preceded by a 2 s instruction of “Guess Number” (for task) or “Press button” (for control), resulting in a total block length of 38 s and a total task length of 342 s. Subjects were unaware of the fixed outcome probabilities associated with each block and were led to believe that their performance would determine their net monetary gain, although all subjects received $10 upon completion of the task.

BOLD fMRI acquisition parameters.

Each subject was scanned using a Siemens (Munich, Germany) 3T Allegra scanner. BOLD functional images were acquired with a gradient-echo echoplanar imaging (EPI) sequence and covered 34 axial slices (3 mm thick) beginning at the cerebral vertex and encompassing the entire cerebrum and the majority of the cerebellum (repetition time/echo time, 2000/25 ms; field of view, 20 cm; matrix, 64 × 64). All scanning parameters were selected to optimize the quality of the BOLD signal while maintaining a sufficient number of slices to acquire whole-brain data. Before the collection of fMRI data for each subject, we acquired a reference EPI scan that we visually inspected for artifacts (e.g., ghosting) as well as good signal across the entire volume of acquisition, including the VS and ventromedial PFC. The fMRI data from all 45 subjects included in this study were cleared of such problems.

Image processing and analysis.

Whole-brain image analysis was completed using SPM2 (http://www.fil.ion.ucl.ac.uk/spm). For each scan, images for each subject were realigned to the first volume in the time series to correct for head motion. Data sets were then selected for their high quality (scan stability) as demonstrated by small (<2 mm) motion correction. Based on this criterion, data from all subjects were included in subsequent analyses. Realigned images were spatially normalized into a standard stereotactic space (Montreal Neurological Institute template) using a 12 parameter affine model. These normalized images were then smoothed to minimize noise and residual difference in gyral anatomy with a Gaussian filter, set at 6 mm full-width at half-maximum. Voxelwise signal intensities were ratio normalized to the whole-brain global mean.

These preprocessed data sets were analyzed using second-level random effects models that account for both scan-to-scan and subject-to-subject variability to determine task-specific regional responses. For each subject and scan, predetermined condition effects at each voxel were calculated using a t-statistic, producing a statistical image for each contrast: (1) positive feedback > control and (2) negative feedback > control. These contrast images were in turn used to generate a statistical image for the main effects of each task (positive feedback > control and negative feedback > control) using one-sample t tests as well as the differential effects of feedback [(positive feedback > control) > (negative feedback > control)] using ANOVA. These contrasts were made with the voxel level threshold set at p < 0.05, corrected for multiple comparisons across the volume of the bilateral VS, and an extent of at least 10 contiguous voxels. Our VS region of interest was constructed using the WFU PickAtlas Tool (version 1.04; Wake Forest University School of Medicine, Wake Forest, NC) and defined as a sphere of 20 mm in radius, centered on the Talairach coordinates of x = 0, y = 10, z = −10, encompassing the entire VS as well as adjacent regions of the caudate nucleus (i.e., head of caudate) in both right and left hemispheres.

Statistical analysis.

Before analysis, the discounting parameter, k, was normalized by logarithmic transformation. Associations between delay discounting (log[k]) and mean BOLD signal values for VS clusters identified in the main effects of task, as well as the differential effect of feedback contrasts, were evaluated by Pearson's r and multiple linear regression. Supplementary analyses using indifference values for successive delay intervals of the discounting task were conducted by nonparametric statistics (Spearman's r, Mann–Whitney U). All tests of significance were two-tailed.

Results

Consistent with previous studies employing similar paradigms (Delgado et al., 2000), BOLD fMRI showed strong bilateral striatal activity to be associated with both positive and negative feedback blocks, relative to control blocks (Fig. 1a,b). Striatal activity associated with general feedback extended from the VS to the caudate head. VS activity varied significantly by type of feedback, with bilateral activation greater in response to predominantly positive, compared with negative, feedback blocks (Fig. 1c). Overall, then, we found both (1) a generalized striatal activity associated with all trial outcomes, regardless of the valence of feedback; and (2) preferentially greater VS activation in the presence of positive feedback.

Figure 1.

Average striatal activity associated with general feedback as well as the differential effect of reward (p < 0.05, false discovery rate corrected for all contrasts; all coordinates reported in Talairach space). a, Striatal activation associated with the contrast of positive feedback > control (right cluster: 14, 12, −1; t = 7.75; left cluster: −22, 6, 0; t = 9.31). b, Striatal activation associated with the contrast of negative feedback > control (right cluster: 6, 8, 7; t = 6.91; left cluster: −6, 10, 7; t = 7.71). c, Ventral striatal activation associated with differential effect of reward from the contrast of positive > negative feedback (right cluster: 14, 13, −7; t = 2.81; left cluster: −16, 5, −9; t = 3.33). Images represent mean statistical parametric maps for each contrast overlaid onto a canonical single-subject coronal T1-weighted image.

With respect to delay discounting, individual differences in the discounting function (log[k]) covaried significantly with the magnitude of VS activity on both positive (right hemisphere cluster) and negative (left hemisphere cluster) feedback trials [r values = 0.35 (p < 0.02) and 0.30 (p < 0.05), respectively]. However, positive feedback engaged a greater extent of log[k]-correlated VS activity than did negative feedback (54 and 10 contiguous suprathreshold voxels, respectively). Moreover, the differential VS activation associated with positive, relative to negative, feedback (right hemisphere cluster) likewise correlated significantly with interindividual variability in log[k] values (r = 0.38, p < 0.011) (Fig. 2). This association was unaltered by adjustment in linear regression for demographic covariates [age, race (white/nonwhite), sex] (partial r = 0.37; p < 0.015) and remained significant when also adjusted for correlated variation in VS activity associated with positive or negative feedback [partial r values = 0.31 (p < 0.05) and 0.36 (p < 0.025), respectively]. Exploratory analyses of the relationship between interindividual variability in DD and differential prefrontal responses (positive > negative feedback) revealed positive correlations between log[k] values and activity in the medial PFC (x, y, z: 0, 49, 7; r = 0.365; p < 0.05) and negative correlations with activity in the lateral OFC (32, 13, −4; r = −0.378; p < 0.05) and dorsolateral PFC (40, 36, 28; r = −0.390; p < 0.05).

Figure 2.

Magnitude of DD is positively correlated with VS activity associated with reward sensitivity (r = 0.376; p < 0.011). The scatter plot represents the relationship between individual DD log[k] values, which index preference for smaller immediate over larger delayed rewards, and differential VS activity (right hemisphere cluster, maximal voxel coordinates: x = 8, y = 4, z = 0) from the contrast of positive > negative feedback (Fig. 1c).

Higher VS activity associated with the differential response to feedback (positive > negative) covaried with heightened preference for immediate rewards (i.e., steeper discounting) across all intervals of the delay discounting task, as indicated by the correlation of VS activity with indifference values at each time point [Spearman's r values = −0.43 (p < 0.004) at 7 d; −0.41 (p < 0.006) at 30 d; −0.42 (p < 0.004) at 90 d; −0.33 (p < 0.03) at 180 d; −0.31 (p < 0.04) at 365 d; and −0.29 (p < 0.06) at 1825 d]. This relationship is further illustrated in Figure 3, which depicts median indifference values over the seven delay intervals among subjects comprising groups of “high” (above median) and “low” (less than or equal to median) VS activity, as defined again by the differential response to positive feedback. Compared with subjects with nominally low VS activation, participants who showed high VS activity also exhibited a greater preference for immediate, relative to delayed, rewards in the delay discounting task. Consistent with the correlations reported above, group differences in the median indifference value were significant at all delay intervals depicted in Figure 3 (p values < 0.04 to < 0.003 by Mann–Whitney U test).

Figure 3.

Higher VS activity is associated with steeper discount functions regardless of delay interval. Plots represent the median indifference points at the delay intervals for subjects with either high (i.e., above median) or low (i.e., at or below median) VS activation. By Mann–Whitney U test for median differences, the indifference points of high and low VS activation groups differed significantly at all delay intervals (all p values < 0.05). Lines represent the hyperbolic functions fitted to these data points for each group. Open circles, High VS; solid circles, low VS.

Discussion

The present findings demonstrate that variability of VS activation in response to reward-related stimuli covaries significantly with individual differences in independently assessed temporal discounting. Individuals exhibiting the strongest preference for immediate over delayed rewards (i.e., who discounted future outcomes most steeply) showed the largest magnitude of VS activation associated with positive and negative feedback for monetary reward. Moreover, differential VS activation for positive, compared with negative, feedback also covaried with interindividual variability in delay discounting, and did so both: (1) independently of potential demographic confounders (age, sex, race); and (2) over and above the correlation between discounting (log[k] values) and VS activity associated with feedback regardless of valence (positive or negative).

Consistent with the study of McClure et al. (2004), activity of distinct regions of the prefrontal cortex also covaried significantly with interindividual variability in preference for immediate over delayed rewards. Specifically, log[k] was positively correlated with activity in the medial PFC and negatively correlated with activity of the lateral OFC and dorsolateral PFC. These divergent patterns echo the roles of limbic and medial PFC regions in generating stimulus-driven behavioral responses serving immediate goals and those of lateral and dorsal regions of the PFC in regulating behavior in the context of stimulus-independent long-term goals. An effective functional balance between these reciprocally connected neural regions is likely critical in mediating appropriate and advantageous behavioral responses to varying reward contingencies.

It may be noted that reward stimuli in both the delay discounting and fMRI protocols were monetary, which may limit the generalizability of our findings to other contexts of reward. At least with respect to delay discounting, however, there is evidence that variation in the discounting of monetary rewards correlates with discounting in other reward currencies (e.g., cigarettes among smokers, heroin among addicted users) (Madden et al., 1997; Bickel et al., 1999; Green and Myerson, 2004). Likewise, persons with problems of addiction (substance abuse, gambling, nicotine dependence) discount hypothetical or actual monetary rewards more steeply than healthy controls (Madden et al., 1997; Vuchinich and Simpson, 1998; Bickel et al., 1999; Kirby et al., 1999; Alessi and Petry, 2003).

Because the behavioral assessment of temporal discounting was conducted as a component of a larger data collection (AHAB project) that preceded our measurement of VS reactivity with BOLD fMRI, these two evaluations were performed at different times, and indeed were separated by an average interval of 29 weeks. The fact that we observed a correlation between temporal discounting and VS activity despite the separation in time suggests that both indicators reflect stable response propensities, possibly indicative of trait-related variation. If so, it is conceivable that this brain-behavior association may be influenced by genetic variation affecting functioning of the VS and its interconnected circuitry (e.g., dopamine-regulating polymorphisms). Prominent alterations of the dopaminergic signaling cascade in the VS have been identified in addiction (Kalivas and Volkow, 2005), and as noted above, steeper delay discounting is associated with addictive behaviors as well. In turn, this link might reflect a neurocognitive mechanism contributing to genetically biased behavioral tendencies (e.g., novelty seeking, impulsivity) that are associated with increased risk of addiction (Kreek et al., 2005).

The pattern of VS activity seen in the present study is generally consistent with that reported using the original version of this task, in which each trial was associated with monetary feedback (Delgado et al., 2000). However, the positive VS activation during negative feedback blocks compared with no-feedback control differs from the pattern of decreased activity below baseline associated with negative feedback trials in the original event-related version of our current paradigm. This difference may reflect the presence of unexpected rewards (25% of trials during negative feedback blocks) in our blocked design, which could serve as positive prediction errors that elicit striatal activation. In addition, the VS activation may more generally reflect feedback learning associated with the probabilistic nature of our blocked design (Haruno et al., 2004; Delgado et al., 2005). The extension of VS activation to the caudate head during both positive and negative feedback blocks compared with the no-feedback control is consistent with this pattern (Delgado et al., 2000, 2005). However, this potential limitation is largely avoided in our focused analyses of the relationship between DD and differential VS activation associated specifically with positive versus negative feedback, in which the probabilistic nature of each block is identical. Also, this differential VS activation did not extend to the caudate head, further suggesting that it may more closely reflect reward and not general feedback learning.

The VS activations we observed are also consistent with previous studies using various other reward incentive paradigms and stimuli (cf. Breiter and Rosen, 1999; Knutson and Cooper, 2005). However, unlike many previous studies, each trial was not associated with a monetary outcome (i.e., gaining or losing money per trial) within our modified blocked design. Rather, each trial occasioned a more general positive or negative feedback reflecting the “correctness” of the subject's guess. This aspect of our protocol limits our ability to identify VS activity associated specifically with trial-by-trial monetary outcomes (Delgado et al., 2000; Knutson et al., 2001; Galvan et al., 2005), which might be related more closely to differences in temporal discounting. Our blocked design also precludes the analysis of VS activity uniquely associated with anticipation and outcome components of each trial. Employing event-related fMRI paradigms that better allow for the dissection of anticipation and outcome, or that instantiate delay discounting specifically (McClure et al., 2004), may help further parse the relationship between reward preference and VS function.

Finally, it is possible that the relative preference for smaller, immediate rewards (steeper discounting) that we found correlated with VS activity reflects a nonspecific association between discounting and anticipation or sensitivity to any reward-related cues. While delay discounting did predict differential VS activation in response to positive feedback, for instance, it was also associated with both positive and negative feedback compared with a no-feedback control condition. This observation is consistent with other studies reporting VS activation during the anticipation of either positive (Knutson et al., 2001) or negative (Jensen et al., 2003) outcomes. Hence, an increased preference for immediate over delayed reward and the impulsivity associated with this preference (de Wit et al., 2007) may reflect variation in a VS circuitry that is relatively indiscriminate and hyperreactive to salient stimuli, contexts, and cues. In turn, such a bias may contribute to at least one important neurocognitive pathway to addiction.