Electrophysiological and Hemodynamic Responses to Reward Prediction Violation

Abstract

Anterior cingulate cortex has been functionally linked to the detection of outcomes that are worse than expected using both scalp electrophysiological (ERP) and hemodynamic (fMRI) responses. The current study used a reward prediction violation design acquired both ERP and fMRI data from the same participants in different sessions. Both the medial frontal negativity (MFN) ERP response and anterior cingulate cortex hemodynamic activity differentiated between reward delivery and expectation with the largest MFN and anterior cingulate cortex response when predicted rewards were not delivered. Inverse modeling placed the MFN source near the anterior cingulate cortex hemodynamic activation. The fMRI study also showed increased striatal response to rewards regardless of prediction indicating dissociation of neural processing of reward and reward expectation.

Introduction

Adaptive behavior requires individuals to evaluate the motivational properties of situations they encounter to optimize potential rewards. The dopaminergic reward system, with its source in the ventral tegmental area and widespread projections to the limbic system, striatum, anterior cingulate cortex, and prefrontal cortex, responds to violations of reward expectation to adjust behavior to optimize outcomes [1]. Schultz et al.’s animal studies [2] showed increases in the firing of ventral tegmental area dopamine neurons to unpredicted reward as well as the presentation of a cue that predicted reward delivery. However, if a predicted reward was not delivered, ventral tegmental area activity was suppressed at the time of the expected, but not delivered, reward. This reward prediction violation signal is transmitted widely to anterior telencephalic targets. Medial frontal cortex, including the anterior cingulate cortex, is one major target, and is thought to compare outcomes with expectation for the modification of ongoing behavior [3].

The medial frontal negativity (MFN) is an event-related potential (ERP) component thought to reflect this anterior cingulate cortex evaluation of how “good” or “bad” an outcome is [4]. Inverse modeling has estimated the neural source of the MFN to the ventral anterior cingulate cortex [5,6] consistent with fMRI studies that have shown anterior cingulate cortex activation to worse-than-expected outcomes [7]. However, there is little direct evidence linking the MFN to anterior cingulate cortex hemodynamic activity.

The current study collected both ERP and fMRI data from the same subjects (in separate sessions) in a reward prediction violation design. We used a passive reward prediction design modeled after studies in non-human primate work to elicit ventral tegmental area response, a design shown to elicit an MFN [8,9]. We predicted that the proposed MFN and the anterior cingulate cortex hemodynamic activity would respond like ventral tegmental area neurons, differentiating between unexpected “good” and “bad” outcomes, and that inverse modeling would place the MFN in or near the hemodynamic activation.

Methods

Participants

Twenty participants, age 18–22 (M=19.6, SD=1.35), were enrolled in both the ERP and fMRI sessions. The current ERP analyses include previously published data from the 18 subjects [9], as well as two additional subjects. Rice University’s Institutional Review Board (IRB) approved both ERP and fMRI protocols and the Baylor College of Medicine’s (BCM) IRB approved the fMRI protocol (fMRI data were collected at BCM). All participants provided informed consent and were paid $20/session.

Experimental Design and Analysis

Each trial began with a reward predicting stimulus (S1), either a lemon predicting no reward or a gold bar predicting a $1 reward, followed by a fixation and then a reward delivering stimulus (S2), again either a lemon (no reward) or a gold bar (reward delivered). S1 and S2 were the same on 80% of the trials, i.e. lemon-lemon (predicted absence of reward) or bar-bar (predicted reward). The remaining 20% of the trials delivered unexpected outcomes: lemon-bar (unpredicted reward) or bar-lemon (predicted reward not delivered). An equal number of rewarding and non-rewarding trials were presented. Feedback was given at the end of each trial informing participants of the amount won on the current trial as well as the total earnings for the current block of trials. The same design was used for both ERP and fMRI, however the fMRI study included fixation periods between trials that were 0, 2.5, or 5 seconds to allow for event-related analysis, with each fixation duration occurring on one third of the trials. Participants began with $5 and each trial cost them $0.25, like putting a quarter in a slot machine. At the beginning of each ERP block or fMRI run, participant’s totals were reset to $5. At the end of the experiment participants were paid the total from one of eight blocks or runs based on a random draw.

EEG Data Acquisition and Analysis

EEG were collected continuously, referenced to the vertex, with .1 – 100 Hz analog filtering, and digitized at 250 Hz with a 128 channel Electrical Geodesics system (EGI., Eugene, OR). The EEG was digitally filtered at 20 Hz lowpass, segmented into epochs spanning 200 ms before to 800 ms after S2 onset, screened for non-cephalic artifact, sorted by condition, and averaged to create the ERPs, which were then referenced to an average reference and baseline corrected over the 200 ms pre-stimulus period. The MFN was extracted as the mean microvolt value between 200–300 ms post S2 over a medial frontal ROI consisting of 15 EGI system channels near Fz and FCz in the 10/20 system (4, 5, 6, 7, 10, 11, 12, 13, 15, 16, 18, 19, 20, 107, 113) and subjected to a repeated-measures ANOVA with Prediction (predict reward, predict no reward) and Outcome (reward delivered, no reward delivered) as factors (ERP results from 18 of the 20 participants have been previously published [9])

FMRI Data Acquisition and Analysis

MRI scans were conducted in one of two identical Siemens 3T Allegra head-only scanners. Each head coil was equipped with a non-ferrous mirror allowing participants to view stimuli projected onto a screen located at the back of the scanner, 81 cm from the mirror. The fMRI images were acquired using an echo-planar imaging sequence with a 220 mm field-of-view and a 64 x 64 acquisition matrix, resulting in an in-plane spatial resolution of 3.4 mm. Twenty-six 4 mm-thick contiguous axial slices were acquired. The repetition time was 2.5 seconds. The task consisted of 8 runs and each run consisted of 120 volumes for a run time of 5.3 minutes. Echo time was 40 ms and flip angle was 90°

Pre-processing and statistical analyses were performed on each participant’s data using Analysis of Functional Neural Images (AFNI). Preprocessing steps included slice time correction, motion correction, and 4 mm FWHM Gaussian smoothing. Deconvolution was performed to estimate the impulse response function (IRF) for each condition over a 15 second window following stimulus onset. The response function for each condition was estimated relative to baseline (intertrial fixation). The mean of the three IRF timepoints ranging from 2.5–7.5 seconds was used as the dependent variable in the group analyses conducted at each voxel, following spatial normalization. Activation maps reflecting the locations of statistically significant differences among conditions for the group analyses were overlaid onto a T1-weighted template in Talairach space [10]. Regions of interest (ROIs) were identified functionally with threshold adjustments based on Monte Carlo simulations implemented by AFNI’s AlphaSim to determine the minimum cluster size necessary for a cluster-wise probability p<.05. With a voxel-wise p-value of .003 and a radius connectivity of 4mm, it was determined that clusters must contain at least 196 voxels. The following statistical contrasts were examined 1) Unpredicted vs. Predicted, 2) Reward vs. No Reward, 3) Predicted Reward vs. Predicted No Reward, 4) Unpredicted Reward vs. Unpredicted No Reward.

Combined ERP/fMRI Analyses

Combined analyses of the ERP and fMRI data were performed by examining group average regional source localization using the BESA 5 package (MEGIS, Germany). A single source was allowed to freely fit to the MFN from the unpredicted reward minus unpredicted no reward difference wave (subtraction to eliminate perceptual responses in common to the conditions), and a source was inserted at the center points of the fMRI anterior cingulate cortex activation to examine the amount of variance accounted for in the ERP signals of the peak MFN amplitude (260 ms).

Results

ERP Study

The MFN was largest when no reward was delivered (Outcome: F(1, 19) = 16.76, p < .001), modified by an Outcome x Prediction interaction showing that the MFN was largest when a predicted reward was not delivered and smallest when an unpredicted reward was delivered, F(1, 19) = 8.49, p < .01.

fMRI

All regions that reached significance are listed in Table 1. Overall, parietal and occipital regions showed a main effect of prediction and responded more when unpredicted outcomes were delivered compared to when predicted outcomes were delivered. The anterior cingulate cortex and caudate showed a main effect of outcome with greater responses on trials delivering rewards compared to trials delivering no reward. Furthermore on trials in which unpredicted outcomes were delivered, the anterior cingulate cortex responded more when unpredicted rewards were delivered compared to when predicted rewards were not delivered. No regions showed greater activation on trials when predicted rewards were delivered compared to when no reward was predicted or delivered.

Table 1.

Regions reaching significance with a voxelwise p<.003 and 196 contiguous voxels for within subject contrasts (clusterwise p<.05).

Contrast and Region Area	Brodmann’s	Coordinates			# Voxels
		x	y	z
Prediction Main Effect: Predicted > Unpredicted
Inferior Frontal Gyrus	47	46	17	−2	199
Precuneus	7	−1	−76	41	1120
	31	16	−67	25	432
Angular Gyrus	7	32	−65	50	1428
Occipital Gyrus	19	37	−68	−4	661
Outcome Main Effect: Reward > No Reward
ACC	32	0	53	9	1649
Caudate	−	−7	1	3	206
Unpredicted: Reward > No Reward
ACC	32	0	36	6	231
Predicted: Reward > No Reward
No Regions Found

Combined ERP/fMRI Analysis

A single dipole allowed to freely fit located to the right caudate (x, y, z = 7, 14, 5) and accounted for 91% of the variance in the unpredicted reward minus unpredicted no reward MFN difference waveform. Inserting a single regional source at the peak voxel of the fMRI anterior cingulate cortex activation (3, 40, 9) accounted for 86% of the MFN variance.

Discussion

ERP effects were the same as previously reported in the subset of 18 of the 20 participants in the current report [9] showing an interaction between reward prediction and delivery with the largest MFN when expected rewards were not delivered and the smallest MFN when unexpected rewards were delivered. The anterior cingulate cortex showed the same pattern in its hemodynamic activation pattern, with larger response when an unpredicted reward was delivered compared to when a predicted reward was not delivered. While the MFN did not freely localize to anterior cingulate cortex, the anterior cingulate was the nearest cortical area to the free dipole location, and since the ERP signal can only come from laminar (i.e. cortical) tissue, the MFN likely originated in the anterior cingulate cortex, consistent with previous studies [5,6], with ERP localization error due to the low spatial resolving power of the method. A dipole placed at the center of the anterior cingulate cortex hemodynamic activation resulted in a good fit (accounted for 86% of the variance) for the MFN, just not quite as good as the free-fit, which accounted for the slightly better 91%.

These results are functionally consistent with previous studies of monetary punishment [11], with the anterior cingulate cortex indexing unexpected negative outcomes. Previous studies have established a role of the anterior cingulate cortex in monitoring negative events, like behavioral errors [12–17] or monetary loss [18,19]. In contrast, striatal activation, with a local maximum in the caudate, showed activation to rewarding outcomes. Several studies have dissociated the neural activation associated with the anticipation of reward from the activation associated with actually receiving a reward and found that striatal activation is associated with reward anticipation, rather than the evaluation of trial outcome [20–23].

The rapid presentation of the predictor, delivery, and feedback stimuli in the current design did not allow dissociation of hemodynamic anticipatory responses to S1 and delivery and outcome responses to S2 (and possibly the feedback, although S2 completely predicted the feedback). Prediction violations could only be signaled by S2. Thus the anticipatory response could not be measured in response to S1 but had to be a reward anticipation response due to the S1–S2 pairing. Furthermore, the S1–S2 pairing is not a primary reward (i.e. lemon-bar appearing on the screen has no intrinsic rewarding properties), but rather is symbolic of a reward to be delivered later.

Conclusion

The similar anterior cingulate cortex hemodynamic and medial frontal electrophysiological response patterns (and the absence of other cortical areas with the same response pattern) and the proximity of the MFN source to the anterior cingulate cortex activation region (within the spatial resolving power of ERP) indicate that anterior cingulate cortex is the source of the MFN, that the anterior cingulate cortex responds consistently with a target of the ventral tegmental area reward prediction system, and that this system functions to evaluate outcomes in the context of expectation.

Figure 1.

a) fMRI results showing the ACC region of interest for the Unpredicted: Reward > No Reward contrast. Crosshairs represent the location of the freely fit dipole to the caudate. b) ACC hemodynamic response showing the largest activation when unpredicted rewards are delivered (UR) and largest deactivation when predicted rewards are not delivered (UNR). c) ERP MFN response in the 20 participants who participated in the fMRI session showing the largest MFN response when expected rewards were not delivered (UNR).