Anticipating instrumentally obtained and passively-received rewards: A factorial fMRI investigation

Using dopaminergic signals in the ventromesial striatum (VS), organisms learn to approach stimuli that signal a high probability of reward [1]. During instrumental learning, single-unit activity in the striatum shifts forward from reward delivery to the presentation of reward-predictive stimuli [2]. Accordingly, human functional magnetic resonance imaging (fMRI) experiments show that regions of the striatum are activated by: 1) learned [3] cues that signal eligibility to respond for uncertain reward in monetary incentive delay (MID) tasks [4–6], 2) the salience of a reward cue [7], 3) anticipation of maximally uncertain (50%) reward relative to more certain outcomes [8], and 4) reward delivery [9, 10] that is contingent on behavior [11].

Of interest here is to use rapid, event-related fMRI to further characterize the extent to which human ventromesial striatum (VS) activation by reward anticipation is dependent on the interaction between reward delivery probabilities and instrumental response requirements. The distinction between instrumental and classical conditioning is muddled in many behavioral tasks because stimuli signaling availability to respond for reward also inherently convey information about impending reward as would a classically-conditioned (Pavlovian) cue. VS activation directly correlated with positive affective responses to reward availability [4, 5] suggesting the possibility of a Pavlovian component to this activation. Moreover, populations of VS neurons fire in response to reward-predictive cues under Pavlovian conditions [12] (reward not contingent on an instrumental response). If anticipatory VS activation by reward in instrumental tasks is primarily elicited by positive affect itself, this activation should also be elicited by cues associated with reward delivered in a Pavlovian manner.

We designed a Factorial Reward Anticipation (FRA) task (a variant of the MID task) to examine activation by linear contrasts between anticipation of potential reward versus non-reward reported previously [4–6], but here under each of Pavlovian and instrumental conditions. As a secondary objective, we wished to obtain evidence that reward-anticipatory activation previously elicited by MID tasks [4, 5] was also partially dependent on how reward for effort was not certain. Notably, Fiorillo et al demonstrated that anticipatory single-unit activity in mesolimbic dopaminergic neurons is maximal when the probability of payoff for the instrumental response is maximally uncertain (50%) [13]. The FRA included trials with an explicitly-briefed 50% probability of payoff for a successful response, where outcomes were not primarily a function of the subject’s successful behavior (as with previous MID tasks). We hypothesized that 1) anticipatory VS activation would be dependent on the requirement to respond [14], 2) VS activation would be more robust with uncertainty of reinforcement [8], and 3) VS activation would be most evident under the dual conditions of an instrumental requirement and uncertainty of reward delivery.

Ten men (mean age 34.2 ± 6.4) and 10 women (33.9 ± 6.4) free of any physical or mental illness gave written informed consent to participate in this experiment. All procedures were reviewed and approved by the NIAAA Institutional Review Board. The FRA task stimuli were white on a black background, and were projected on a screen and viewed with a head coil mirror. The six trial types (n = 18 each) were pseudo-randomly presented, and were separated by a jittered inter-trial interval (2–8 s) with fixation crosshair. Each trial lasted 6 s, and featured an instruction cue (500 ms), a target (500 ms), and feedback (2000 ms; see Figure 1). Response trials (square cue series) required the subject to respond on a button box while the subsequent target (white square) was presented. A square enclosing a “$,” a “?,” and a “0” indicated 1.0, 0.5, and 0 probabilities, respectively, of winning $1 for hitting the target. Non-response trials (circle cue series) instructed the subject to withhold response while the subsequent target was presented. A circle enclosing a “$,” a “?” and a “0” indicated 1.0, 0.5, and 0 probabilities, respectively, of passive receipt of $1 after the following target was presented. Reward presentation was not contingent on withholding a response to the target. Responding during a 500 ms target presentation was intended to promote attention to the task, to reduce variance in the timing of responses, and was twice the mean reaction time (RT) of the slowest subject in a previous study (235 ms; Bjork et al, 2004), promoting ~ 1.0 probability of hits. During feedback, both trial and cumulative winnings were presented. Each subject was trained about the reward contingencies of the six instruction cues, and performed a 4-minute practice session of the task. Subjects were shown the cash they could win.

Figure 1.

The Factorial Reward Anticipation (FRA) task. Trials lasted 6 s, and were separated by jittered intertrial intervals of 2–8 s. Subjects first saw one of six cues that signaled whether the subject needed to respond (squares) or to not respond (circles) to the target that followed, and what the consequences of the trial would be. In response trials, subjects were required to respond to the target during the 500 ms period when the target was displayed. The cue signaled a certain gain of $1 (square-$), a 50% chance of a $1 gain (square-?), or no gain (square-0) for hitting the target. In non-response trials, subjects did not respond to the targets, but viewed the identical range of signaled gain probabilities. Each trial concluded with 1.5 s of feedback of gain or non-gain..

We used a 3 T scanner (General Electric, Milwaukee, WI) and a quadrature head coil. We collected 24 3.8-mm-thick axial slices with a 1 mm interslice gap. In-plane resolution was 3.75 × 3.75 mm. Functional scans were acquired using a T2*-sensitive echoplanar sequence with a repetition time (TR) = 2000 msec, echo time (TE) =40 msec, flip = 90°. Structural scans were acquired using a T1-weighted MP-RAGE sequence (TR, 100 msec; TE, 7 msec; flip, 90°) for co-registration of functional data. Each subject’s head was immobilized by a deflatable head restraint cushion.

We analyzed blood oxygen level-dependent (BOLD) signal time-locked to instruction cue presentation. Preprocessing and statistical analyses were conducted using Analysis of Functional Neural Images (AFNI) software [15] as follows: (1) volumes were concatenated across the three task runs; (2) voxel time series were interpolated to correct for non-simultaneous slice acquisition within each volume, (3) volumes were corrected for motion. Motion-correction estimates indicated that no participant’s head moved more than 1.5 mm between volumes. We then applied a 10 mm smoothing kernel, a de-spiking algorithm, and bandpass filtering which smoothed cyclical fluctuations in signal that were greater than 0.011/sec or less than 0.15/sec.

The regression model consisted of orthogonal regressors corresponding to presentation of the six instruction cues (anticipation), outcome notifications, residual motion following volume correction, and baseline and linear trends for each run. Regressors of interest were convolved with a gamma variate function that modeled a prototypical hemodynamic response function. Statistical maps were generated by the following linear contrasts (LC: area-under-curve activation), which were planned a priori to replicate previously reported contrasts and to extend them to Pavlovian conditions: 1) anticipation of responding for certain reward (p = 1.0) versus nonreward (p = 0), 2) anticipation of responding for uncertain reward (p = 0.5) versus nonreward, 3) anticipation of passive receipt of certain reward versus certain nonreward, and 4) anticipation of passive receipt of uncertain reward versus nonreward.

We used higher-order LC to isolate activation by the combination of an incentive together with the requirement to make an instrumental response [14]. For each of the p = 0.5 and p = 1.0 trial types, this was accomplished with a higher-order LC that could be conceptualized in two ways, either: 1) activation during anticipation of responding for reward versus anticipation of passively-obtained reward, while masking out activation by the contrast: (responding for non-reward versus passive receipt of non-reward), or 2) activation during anticipation of responding for reward versus anticipation of responding for no reward, while masking out the contrast: (passive receipt of reward versus passive nonreward). Thus this comparison allows us to identify those voxels that are most activated by the combination of anticipated reward and the need to generate a motor action to obtain the reward.

Individual subject maps of linear contrast t-statistics were transformed into Z scores and warped to common Talairach space and combined into a group map using a meta-analytic formula (average Z * square root (n))[4, 5]). For each contrast, activations were objectively detected using AFNI programs AlphaSim, 3dmerge, and 3dExtrema, where: 1) voxels each exceeded a statistical significance threshold of p < .0001, and 2) activated voxels were part of a contiguous cluster of sufficient size to obtain a family-wise corrected type I error rate ≤ 0.05 using Monte Carlo simulation.

Subjects hit a large majority of targets, such that intended reinforcement probabilities in response trials were not appreciably degraded by failures to hit the targets. Omission error rates (in response trials) were 3.3% in p = 1.0 trials, 3.3 % in p = 0.5 trials, and 5.8% in p = 0 trials. Compound repeated-measures ANOVA revealed a significant increase in omission errors in the final run of the task (main effect of time (Runs 1–3) F(2,38) = 5.216, P < .01), and a trend toward greater omission errors in p = 0 trials (main effect of probability F(2,38) = 2.940, P = .065). When subjects responded to a target, hit rates were 97.2%, 98.1%, and 96.4% in p = 1.0, p = 0.5 and p = 0 trials, respectively. There was a main effect of incentive probability on reaction time (RT) (F(2,38) = 6.325, p < .001). Simple effect paired t-tests indicated that responses to both p = 1.0 targets (mean 267.2 ± 51.3 ms) and p = 0.5 targets (mean 267.8 ± 40.4 ms) were significantly faster than responses to p = 0 (mean 294.2 ± 56.3 ms) targets (p < .01). There were no main or interactive effects of time on RT (runs 1–3). The incidence of commission errors (in non-response trials) was 1.4%.

Neither the LC between anticipation of passive receipt of uncertain (p = 0.5) reward versus nonreward, nor passive receipt of certain (p = 1.0) reward versus nonreward, activated any cortical or subcortical voxels. Anticipation of responding for uncertain (p = 0.5) reward (versus nonreward) activated different regions of motor cortex, and activated the putamen bilaterally, with activated voxels extending ventro-mesially into the NAcc (See Table 1 and Figure 2A). The LC between anticipation of responding for certain (p = 1.0) reward versus responding for nonreward activated left dorsal thalamus, left parietal cortex, left putamen, cerebellar vermis, as well as insula and superior post-central gyri bilaterally (See Table 1 and Figure 2B). The VS activation was centered in the Putamen, with activated voxels extending ventro-mesially into nucleus accumbens (NAcc). A post hoc LC between certain (p = 1.0) versus uncertain (p = 0.5) reward activated mesial frontal cortex under each of response and non-response conditions (Table 1).

Table 1.

Anticipatory activations detected by linear contrasts between cues signaling differing reward probabilities

Linear contrast:	Region	Talairach Coordinates:			t-value:	uncorrected p*
Nonresponse conditions
P = 1.0 vs P = 0	No activation
P =1.0 vs P = 0.5	Mesial frontal lobe	−3	46	−7	4.017	<.0001
P = 0.5 vs P = 0	No activation
Response conditions
P = 1.0 vs P = 0	L Precentral gyrus	−26	−11	58	6.320	<.000001
	L Postcentral gyrus	−30	−30	48	6.135	<.000001
	L Cingulate motor area	−8	−11	48	5.216	<.000001
	R Postcentral gyrus	30	−34	48	5.053	<.000001
		45	−19	39	4.541	<.00001
	R Precentral gyrus	23	−15	63	4.346	<.0001
	L Superior parietal lobe	−26	−41	63	4.876	<.00001
	L Inferior parietal lobe	−53	−34	24	4.130	<.0001
	L Middle frontal gyrus	−30	34	29	4.384	<.0001
	L Insula	−34	−4	5	4.545	<.00001
	R Insula	49	4	5	4.041	<.0001
	L Nucleus accumbens	−11	8	−4	5.096	<.000001
	R Putamen (rostral)	30	4	0	4.658	<.00001
	(caudal)	26	−19	0	4.267	<.0001
	L dorsal thalamus	−8	−19	5	4.019	<.0001
	Cerebellar vermis	0	−60	−9	4.420	<.00001
P =1.0 vs P = 0.5	Mesial frontal lobe	−2	50	5	4.619	<.00001
	R superior temporal lobe	48	−19	9	5.040	<.000001
P = 0.5 vs P = 0	L Precentral gyrus	−34	−23	48	5.035	<.000001
	L Cingulate motor area	−11	8	29	4.683	<.00001
	R Superior parietal lobe	26	−49	39	4.273	<.0001
	L Inferior parietal lobe	−45	−45	39	4.882	<.00001
	R Middle frontal gyrus	15	4	48	4.312	<.0001
	L Insula	−30	8	10	4.158	<.0001
	R Putamen (rostral)	15	11	−1	5.244	<.000001
	L Putamen (rostral)	−15	8	−1	6.660	<.000001
	L red nucleus	−4	−23	−4	4.130	<.0001
	Cerebellar vermis	4	−56	−14	4.495	<.00001

^*Voxel coordinates listed in tables were the peak of a contiguous cluster sufficient to obtain a family-wise corrected type I error rate of p < 0.05 using Monte Carlo simulation.

Figure 2.

Activation by anticipation of instrumental responding for uncertain reward. (A) linear contrast between responding for uncertain (p = 0.5) reward linearly contrasted with anticipation of responding for non-reward (p = 0) (B) linear contrast between responding for certain (p = 1.0) reward contrasted with anticipation of responding for non-reward (p = 0). All statistical maps are overlaid on a sample subject’s T1-weighted image, where Taliarach coordinate of each coronal view is shown.

For each of the p = 0.5 and p = 1.0 trial types, higher-order LC revealed regions selectively activated by the interaction of the presence of potential gain (versus nongain) with the requirement to emit an instrumental response. This higher-order LC with p = 0.5 trials activated left precentral gyrus and putamen and lentiform nuclei bilaterally. (Table 2, Figure 3A). The activation in the putamen extended into the VS. The higher-order LC with p = 1.0 trials activated bilateral insula, cingulate motor area, left pre- and post-central gyrus, posterior putamen, and right cerebellum, but with no recruitment of rostral Putamen or VS voxels (Table 2, Figure 3B).

Table 2.

Activation by potential reward- specific to requirement for an instrumental response

Linear contrast:	Region	Talairach Coordinates:			t-value:	uncorrected p
Higher-order (P = 1.0 vs P = 0 Response trials) vs(P = 1.0 vs P = 0 Nonresponse trials)	L Precentral gyrus	−30	−30	48	5.503	<.000001
	L Postcentral gyrus	−41	−19	29	4.397	<.0001
		−56	−19	19	4.091	<.0001
	R Cingulate motor area	11	−19	48	4.544	<.00001
	L Cingulate motor area	−4	−8	39	4.235	<.0001
	R Precentral gyrus	19	−26	67	4.555	<.00001
	R Postcentral gyrus	26	−30	53	4.013	<.0001
	L Superior parietal lobe	−22	−45	58	5.241	<.000001
	R Superior parietal lobe	19	−45	58	4.116	<.0001
	L Inferior parietal lobe	−56	−34	24	4.083	<.0001
	L Superior temporal lobe	−53	−4	5	4.243	<.0001
	R Superior temporal lobe	56	−15	10	4.258	<.0001
	R Lingual gyrus	11	−68	−4	4.310	<.0001
	L Superior frontal gyrus	−19	11	63	4.219	<.0001
	L Insula	−34	0	10	4.824	<.00001
	L Putamen (caudal)	−26	−19	0	4.147	<.0001
	R Putamen (caudal)	26	−19	5	5.000	<.000001
	L dorsal cerebellum	11	−60	−9	4.492	<.00001
Higher-order(P = 0.5 vs P = 0 Response trials) vs(P = 0.5 vs P = 0 Nonresponse trials)	L Precentral gyrus	−34	−26	44	5.536	<.000001
	L Putamen (rostral)	−15	−8	−1	4.990	<.000001
	R Putamen (rostral)	19	−11	−1	4.780	<.00001
	R Putamen (caudal)	23	4	5	4.340	<.0001
	L Middle occipital gyrus	−38	−90	15	4.358	<.0001
	L Insula	−30	4	10	4.317	<.0001
	L Superior frontal gyrus	−15	−11	68	4.244	<.0001
	R Middle frontal gyrus	30	26	29	4.075	<.0001
	L Caudate (tail)	−26	−38	5	4.046	<.0001

Figure 3.

Activation by a higher-order combination of reward with an instrumental response requirement. A) Instrumental-specific activation during anticipation of responding for uncertain (p = 0.5) reward versus no reward (p = 0), and (B) Instrumental-specific activation during anticipation of responding for certain (p = 1.0) reward versus no reward (p = 0).

We also examined activation time-locked to trial outcome notification. Across p = 0.5 trials (response and nonresponse conditions collapsed for statistical power), the LC between BOLD signal change time-locked to notifications of wins “+$1.00” versus nonwins “+$0.00” activated anterior and posterior cingulate cortices (Table 3). Certain gains contrasted with certain non-gains activated VS and mesofrontal cortex in non-response conditions, and activated several points of motor circuitry in response conditions.

Table 3.

Activation time-locked to reward notification

Linear contrast:	Region*	Talairach Coordinates:			t-value:	uncorrected p
Nonresponse conditions
p = 1.0 wins vs	Mesial frontal lobe	5	50	5	5.291	<.000001
p = 0 nonwins	L Nucleus accumbens	−12	13	−3	5.861	<.0000001
	R Nucleus accumbens	10	11	−1	5.050	<.000001
	L Middle temporal gyrus	−59	−36	−1	4.541	<.00001
	R Middle temporal gyrus	61	−29	−2	4.677	<.00001
	L Inferior parietal lobule	− −41	−55	47	4.933	<.000001
	R Inferior parietal lobule	43	−55	44	4.423	<.00001
Response conditions
p = 1.0 wins vs	L Midbrain	5	−22	−85	4.712	<.00001
p = 0 nonwins	R Cingulate gyrus	1	−8	35	5.507	<.000001
	L Precentral gyrus	−28	−24	65	5.435	<.000001
	R Precentral gyrus	28	−25	65	4.537	<.00001
	R Cerebellum	11	−47	−18	4.961	<.000001
	Anterior cingulate	2	28	18	4.192	<.0001
(Win versus non-win notifications in all p = 0.5 trials)	L Posterior cingulate	−4	−38	34	5.270	<.000001
	R Inferior parietal lobule	41	−64	39	4.674	<.00001
	Anterior cingulate	4	27	15	4.132	<.0001

These respective patterns of activation demonstrated general support for our hypotheses that: 1) anticipatory striatal activation in a MID task is dependent on the requirement to respond and not just on imminent, potential reward delivery itself, 2) VS activation in a MID task is enhanced by the uncertainty of reinforcement, and 3) VS activation by prospective reward is sensitive to an instrumental contingency together with uncertainty of reinforcement for successful response.

The absence of significant reward anticipation activation in non-response trials is in apparent conflict with single-unit studies (e.g. [12]) that demonstrate that subpopulations of VS neurons fire in response to reward-predictive cues under Pavlovian conditions. However, we note that even under conditions where reward delivery is not contingent on an instrumental response, in [12] and other reports, a forthcoming motor response is nevertheless signaled by the reward predictive cue in that the organism must prepared to lick and swallow a liquid reward. In contrast, non-response rewards in the FRA engendered no motor response. Alternatively, the lack of Pavlovian VS activation here may have resulted from the use of an abstract monetary reward, or because BOLD signal results predominantly from local field potentials (e.g. maintenance of gradients), not action potentials [16].

Interestingly, instrumental incentive-elicited activation in other nodes of the basal ganglia-thalamocortical circuit was more extensive in p = 1.0 trials compared to p = 0.5 trials, indicating instead a direct relationship between activation and the expected value (EV; the product of reward magnitude * probability) of the instrumental response in those motor effector regions. In addition, the p = 1 vs p = 0.5 LC also activated mesofrontal cortex under both Pavlovian and instrumental conditions, in accord with mesofrontal activation by increasing payoff probability as reported in a recent fMRI study of EV [6]. Finally, this event-related experiment also replicated a previous finding of a block-design experiment [14]- that activation of putamen and other striatal regions by environmental cues for potential rewards is critically dependent on the requirement for an instrumental response.

Activation of multiple points in the motor circuit by learned incentives has been demonstrated in other reports. For example, Haruno and Kawato [17] used a choice learning task to elicit incentive-dependent activation in bilateral superior parietal, dorsolateral prefrontal, dorsal premotor and occipital cortices, thalamus, supplementary motor area, and right superior temporal sulcus. These activations specifically correlated with the degree to which a stimulus-action-reward association was learned. Similarly, Lau et al. [18], reported that presentation of a visual cue to respond on one of two buttons to win an unspecified amount (versus nongain) activated the putamen and the CMA, with additional activation in left precentral sulcus and left postcentral gyrus when the proper choice of button on which to respond was signaled to the subject in advance. Finally, the posterior mesial frontal cortex activated by instrumental reward aniticaption in the FRA task included regions shown to activate when subjects prepare an intended motor response [19].

Dreher at al recently used a slot-machine task to assess activation by reward anticipation and feedback, and reported bilateral putamen activation during outcome anticipation when maximally uncertain reward outcomes (50%) were contrasted with more certain reward outcomes (25%). Our findings of more extensive VS voxels activated by the prospect of responding for an uncertain reward versus nonreward (compared to certain reward versus nonreward, which only activated posterior regions of putamen) shares this directionality, and also suggest that uncertainty-specific activation need not require a learning context in that subjects in both the present study and in the Dreher et al study had been explicitly briefed on the probability of rewarding outcomes.

Activation time-locked to notification of monetary outcomes activated several mesial cortical regions in uncertain outcome (p = 0.5) trials, and also activated mesofrontal cortex in the p = 1 vs p = 0 contrast in non-response conditions. We caution, however, that because outcomes always followed the anticipatory cue by 4 s, activations ostensibly attributed to reward notifications in p = 1 trials may have resulted instead from a protracted hemodynamic response to the reward-anticipatory cue. Temporal jittering between events within a trial would better distinguish between anticipatory and feedback activations.

In conclusion, these findings retrospectively assist the interpretation of previous activations by MID tasks, and indicate that anticipatory VS activation by reward-predictive cues in these and similar incentive tasks is at least partially dependent on uncertainty of reward delivery, the requirement to mobilize an instrumental response, and on the interaction of these two factors.