Perceptual Salience and Reward Both Influence Feedback-Related Neural Activity Arising from Choice

Abstract

For day-to-day decisions, multiple factors influence our choice between alternatives. Two dimensions of decision making that substantially affect choice are the objective perceptual properties of the stimulus (e.g., salience) and its subjective value. Here we measure EEGs in human subjects to relate their feedback-evoked EEG responses to estimates of prediction error given a neurally derived expected value for each trial. Unlike in traditional reinforcement learning paradigms, in our experiment the reward itself is not probabilistic; rather, it is a fixed value, which, when combined with the variable stimulus salience, yields uncertainty in the choice. We find that feedback-evoked event-related potentials (ERPs), specifically those classically termed feedback-related negativity, are modulated by both the reward level and stimulus salience. Using single-trial analysis of the EEG, we show stimulus-locked EEG components reflecting perceived stimulus salience can be combined with the level of reward to create an estimate of expected reward. This expected reward is used to form a prediction error that correlates with the trial-by-trial variability of the feedback ERPs for negative, but not positive, feedback. This suggests that the valence of prediction error is more important than the valence of the actual feedback, since only positive rewards were delivered in the experiment (no penalty or loss). Finally, we show that these subjectively defined prediction errors are informative of the riskiness of the subject's choice on the subsequent trial. In summary, our work shows that neural correlates of stimulus salience interact with value information to yield neural representations of subjective expected reward.

SIGNIFICANCE STATEMENT How we make perceptual decisions depends on sensory evidence and the value of our options. These two factors often interact to yield subjective decisions; i.e., individuals integrate sensory evidence and value to form their own estimates of expected reward. Here, we use electroencephelography to identify trial-by-trial neural activity of perceived stimulus salience, showing that this activity can be combined with the value of choice options to form a representation of expected reward. Our results provide insight into the neural processing governing the interaction between salience and value and the formation of subjective expected reward and prediction error. This work is potentially important for identifying neural markers of abnormal sensory/value processing, as is seen in some cases of psychiatric illnesses.

Introduction

The effects of salience and subjective value on choice are usually examined in isolation (Smith and Ratcliff, 2004; Milstein and Dorris, 2007; Milosavljevic et al., 2010), though some have posited that both types of decisions involve common neural computations, such as evidence accumulation (Smith and Ratcliff, 2004; Gold and Shadlen, 2007; Summerfield and Tsetsos, 2012; Philiastides and Ratcliff, 2013; Polanía et al., 2014). Several models have attempted to account for behavioral data given interacting salience/value dimensions (Gao et al., 2011; Schütz et al., 2012; Chen et al., 2013; Towal et al., 2013). In one case, investigators (Navalpakkam et al., 2010) demonstrated that decisions made by human subjects were influenced by both dimensions in a manner consistent with a Bayesian ideal observer model maximizing expected reward. An open question, however, is how neural activity relates to these models, particularly activity evoked by feedback that reflects evaluation and updating processes critical to behavior.

Event-related potentials (ERPs) have been used to investigate feedback evaluation in the human brain (Miltner et al., 1997; Holroyd and Coles, 2002; Nieuwenhuis et al., 2004a). A negative deflection of the ERP after the presentation of negative or positive feedback is referred to as feedback-related negativity (FRN), whereas the component elicited when subjects make an erroneous response is known as error-related negativity (ERN). There are alternative theories on the mechanisms of ERN/FRN generation (Gehring et al., 1993; Gehring and Willoughby, 2004; Yeung et al., 2004), though their similar source localization suggests a common neurocognitive error-detection process (Miltner et al., 1997; Nieuwenhuis et al., 2004a).

Reinforcement learning theory has been used to provide a unifying account of ERN/FRN (Holroyd and Coles, 2002; Nieuwenhuis et al., 2004a). Key is a signal that is identified as the prediction error, usually defined as the difference between expectations and rewards. It has been suggested that FRN is associated with a negative prediction error—the outcome is worse than expected. Several properties of FRN with respect to prediction error in reinforcement learning theory have been studied. First, the amplitude of the FRN is larger after negative feedback (Miltner et al., 1997). This was originally shown as the effect of valence (i.e., losses vs wins), though subsequent studies revealed that FRN could be elicited by all negative outcomes in a monetary gambling task, not just losses, when the task emphasized whether a choice was the better of two (Nieuwenhuis et al., 2004b; Yu and Zhou, 2006). Second, the magnitude of the reward showed significant effect on the feedback ERPs, although some inconsistent findings were reported (Yeung and Sanfey, 2004; Goyer et al., 2008). Third, studies have shown that the FRN amplitude is also modulated by the probability of receiving a reward (Yasuda et al., 2004; Hajcak et al., 2005; Holroyd and Krigolson, 2007; Bunzeck et al., 2011). Specifically, Cohen et al. (2007) found that feedback ERPs following wins, but not losses, were modulated by the reward probability. In these studies, the uncertainty of the feedback was manipulated by predefined probabilities associated with different stimuli. However, in natural environments, the uncertainty can also be elicited by other factors such as the perceptual salience of the stimuli.

In the present study, we designed an experiment that systematically changed perceptual salience and reward value, while simultaneously measuring EEGs from subjects performing the task and receiving feedback based on their choices. Our hypothesis is that prediction error resulting from stimulus uncertainty (e.g., perceptual salience) will also elicit similar feedback-evoked activity. Different from reinforcement learning models, the reward of each target is fixed and known by the subject in our experiment, so the uncertainty of the reward prediction is determined by the stimulus salience and the risk taken by the subject. We propose a model that can estimate subjective expected reward and prediction error on a single-trial basis by integrating value information with perceptual decision evidence, given single-trial decoding of EEGs. Based on this model, our analysis tested (1) whether trial-by-trial variability of feedback-related EEG activity correlated with prediction error magnitude and (2) how the correlation was influenced by prediction error valence.

Materials and Methods

Subjects.

Sixteen subjects (nine women and seven men, ages 18–31 years) participated in the experiment. All had normal or corrected-to-normal vision and reported no history of neurological problems. Subjects were informed that they would be paid based on their performance in the experiment (see below, Experimental design). The maximum amount of money they could possibly earn was US$39. Informed consent was obtained from all participants in accordance with the guidelines and approval of the Columbia University Institutional Review Board.

Experimental design.

We used a set of 20 face (from the Max Planck Institute Face Database), 20 car, and 20 house grayscale images (image size, 512 × 512 pixels, 8 bits/pixel). All images were equated for spatial frequency, luminance, and contrast. All images had identical magnitude spectra, and their corresponding phase spectra were manipulated using the weighted mean phase technique (Dakin et al., 2002) to generate a set of images characterized by their percentage of phase coherence. Stimuli were presented to subjects on an Apple Cinema 30 inch display, controlled by E-Prime 2.0 software installed on a Dell Precision T7500 workstation with an nVidia Quadro FX5800 graphics card.

The task for the subjects was to earn as much reward (money) as possible by rapidly selecting images associated with different amounts of money in a very brief presentation of the stimulus (Fig. 1a). Each subject performed 10 blocks in the experiment, with short breaks in between. Each block started with a self-paced presentation of sample images of all categories, i.e., face, car, and house, and the corresponding reward of each category. The reward assigned to each category was fixed during the entire block. Each trial started with an 800 ms presentation of a central fixation cross (∼0.5°) followed by a stimulus display of four different images composed of one face, one house, and two cars. Face and house images were associated with positive rewards and considered as targets. Car images were associated with zero reward and therefore considered as distractors. The locations of the targets and the distracters were randomized across four fixed locations (x eccentricity, ∼3.1°; y eccentricity, ∼3.1°; image width, ∼5.5°). The stimulus images were presented for 300 ms, followed by mask images that lasted for 1500 ms. To earn the positive reward associated with a target, subjects needed to select its location by pressing a corresponding button before the mask images disappeared. Immediately after subjects responded (or after 1500 ms of mask if no response), a feedback screen showed the reward earned on the current trial [or no reward for car (distractor) choices] and the accumulated reward summed over all trials completed in the block. The interstimulus interval was randomized in the range of 3200–3500 ms.

Figure 1.

Experimental design of reward-based perceptual decision making. a, Reward values for face, house, and car were shown to the subject at the beginning of each block. The total reward earned in the block was also presented at the end of each block. b, Twenty-five conditions were used in the experiment. Reward conditions were manipulated by changing the reward ratio between faces and houses, whereas salience conditions were manipulated by changing image phase coherence levels. We report results in term of face salience levels (1 to 5, level 5 being highest face salience, C_f = 44%, C_h = 37%), without reference to the absolute phase coherence values. c, An example face image at five different salience levels (phase coherence, 32 to 44%).

We systematically adjusted the reward and visual salience of stimuli to study how the brain integrates information across both of these dimensions during decision making. We defined five reward conditions by varying the reward ratio between faces and houses (targets) from ¼ to $\raisebox{1ex}{$4$}\!\left/ \!\raisebox{-1ex}{$1$}\right.$ in five steps (i.e., R_f/R_h ∈ {¼, ½, 1, 2, 4}). The reward ratio was fixed within one block and randomly changed across blocks. Within each block (fixed reward condition), the phase coherence of car images was fixed at a low level (35%), whereas the phase coherence of face and house images was randomly changed trial-by-trial across five levels (C_f ∈ {32%, 35%, 38%, 41%, 44%}, C_h ∈ {49%, 46%, 43%, 40%, 37%}). The face and house coherence levels were constrained to be negatively correlated in all trials (C_f + C_h = 81%). This meant the trial with highest face salience had the lowest house salience and vice versa. These phase coherence levels were determined from pilot experiments where we titrated salience levels to reduce choice bias and qualitatively reproduce the choice curves seen in the study by Navalpakkam et al. (2010).

Figure 1c displays a sample face image at five different salience levels. One may note that the overall phase coherences used for house images were slightly higher than those for faces. This was done based on pilot experiments that showed subjects' selections were slight biased toward faces when both target types were at the same coherence level. There were two repetitions of the block for each reward condition (10 blocks in total) in the experiment, with 75 trials in each block uniformly distributed across the five salience levels. Therefore, the data for each subject consisted of 30 trials in each of 25 conditions (five reward levels by five salience levels; Fig. 1b), a total of 750 trials.

Data acquisition.

Participants were seated in an electrostatically shielded room and positioned at a 1.3 m distance from the screen. EEGs were recorded using a Sensorium DBPA-1 Amplifier from 80 Ag/AgCl scalp electrodes. All channels were referenced to the left mastoid with a forehead ground. All input impedances were <25 kΩ. Data were sampled at 1000 Hz with an analog pass band of 0.01–500 Hz. Stimulus events and motor responses were recorded on separate channels.

All data were preprocessed using a 0.5 Hz fourth-order Butterworth high-pass filter to remove DC drifts and 60 and 120 Hz notch filters to minimize line noise artifacts. Trials with excessive eye blink and motion artifacts were rejected (<1% of total trials) by visual inspection. We also used independent component analysis (ICA) to identify the component that showed the strongest blinks and eye movement activity and excluded it from further analysis. All ICA processing was performed using the infomax algorithm implemented in the EEGLab toolbox (Delorme and Makeig, 2004).

Event-related potentials.

Traditional ERP analysis was performed by aligning the data to the onset of feedback, where the feedback immediately followed (was triggered by) the behavioral response of the subject. Each epoch was extracted for a period ranging from 200 ms before feedback onset until 600 ms after feedback onset. For nonequal reward conditions (R_f ≠ R_h), trials were first grouped by their feedback types: “high reward” when the subject selected targets with higher reward and “low reward” when the subject selected targets with lower reward or no reward (i.e., when the subject did not respond in time or selected cars). For example, in the R_f/R_h = ½ condition, selecting “house” resulted in higher reward feedback, whereas selecting “face” resulted in lower reward feedback. Here, trials with lower reward and zero reward feedback were grouped together and called the “low-reward” condition to obtain enough trials for this condition. Trials of all salience conditions within the same reward condition were averaged to investigate the effect of reward on the feedback activity. The reward conditions with reward ratios of ¼ and 4 were also grouped together since feedback ERPs in these two conditions were similar regardless of stimulus category labels. Here, we refer to conditions with reward ratios of ¼ and 4 as “large reward difference” conditions. Conditions with reward ratios of ½ and 2 were also grouped and referred to as “small reward difference” conditions. Statistical analyses were performed by two-way repeated-measures ANOVA on averaged ERP values of two time windows with factors of feedback type (two levels) and reward difference (two levels). These windows were selected at the time of maximal ERP difference on channel Fz. We also investigated the effect of salience by comparing high-reward-feedback ERPs for five salience levels. Two-way repeated-measures ANOVA with factors salience (five levels) and stimulus category (two levels, face and house) was performed on time windows that showed difference in their grand average ERP.

Single-trial discrimination.

Single-trial analysis using logistic regression was performed to classify EEG activity corresponding to the different feedback types (i.e., high vs low reward). We used a sliding window method (Parra et al., 2002) to obtain the variation of discrimination performance across the postfeedback time period. A training window of width 50 ms was used, and the window onset varied across the epoch in 50 ms increments from the feedback onset to 550 ms after feedback. The classifier was subsequently retrained by shifting the training window in finer steps of 10 ms around the time ranges found to be most discriminating (300–500 ms after feedback). This method enabled observation of the temporal progression of feedback-relevant components and localization of the window with maximal discrimination between high- and low-reward feedback. For a time window starting at postfeedback time τ, we used logistic regression to estimate a spatial weighting vector w_τ, which defines the direction (in EEG sensor space) that maximizes discrimination. The estimated regression function can be expressed as follows:

where X is an N × T matrix of EEG data (N channels and T trials), and the estimated mean vector y_τ is a discriminating component at time offset τ across all trials. The discriminator performance was quantified by the area under the receiver operating characteristic curve, referred to as the A_z value, using leave-one-out (LOO) cross-validation. To validate the significance of each discriminating component, we used a label permutation method to compute A_z values for the null distribution (i.e., no discriminatory information), leading to a corresponding A_z threshold of p = 0.01 significance level [false discovery rate (FDR) corrected for multiple comparison]. We also constructed forward models of our discriminating components:

where a_τ is the electrical coupling of the discriminating component y_τ identified for time window τ that explains most of the measured scalp EEG at time τ (Parra et al., 2002). This vector, a_τ, provided a spatial distribution of EEG discriminability between conditions.

We also performed a single-trial analysis of the stimulus-locked data to identify a component most discriminable of face and house images. Unlike in our previous work using a three alternative force choice decision-making experiment (Lou et al., 2011), all three kinds of images were presented at the same time, so we cannot estimate discriminating components for each category. Instead, we grouped the trials with the highest face salience level and face reward no less than house reward (R_f/R_h ≥ 1) as representative of face stimuli, since these were the trials dominated by subjects selecting “face” (Fig. 2), and grouped trials with the lowest face salience level (corresponds to the highest house salience level) and house reward no less than face reward (R_f/R_h ≤ 1) as representative of house. A sliding window method and LOO cross-validation were used to identify the optimal poststimulus timing τ of discrimination.

Figure 2.

Behavioral results. a, Each panel shows the average percentage of target choices (face or house) and distractor choices (car) in each reward condition as a function of stimulus salience levels. Each point refers to the percentage of selections of one image category under the corresponding reward and salience conditions. Error bars indicate the SE across subjects. b, Reaction times in the highest face reward (R_f/R_h = 4) and highest house reward (R_f/R_h = ¼) conditions.

Once the spatial weight vector w_τ was estimated using data derived from the optimal training window, we applied w_τ to the data across all trials at the same time. The output y_i represented the confidence of the classifier for trial i in its prediction of face versus house discrimination based on the training data (Walz et al., 2013; Lou et al., 2014). We can derive probabilities of the two targets for each trial based on the logit functions:

Computational model of prediction error.

Our hypothesis is that single-trial variability (STV) of feedback-related activity is associated with the prediction error, which is defined as the difference between expected reward and actual reward, reported by feedback. The expected reward of trial i is given by the following:

where probabilities of face and house can be obtained from single-trial analysis of stimulus-locked data (as mentioned above). We used the library lme4 from the R-statistical analysis package (Pinheiro and Bates, 2010) to perform a linear mixed effects analysis of the relationship between feedback STV and prediction error of reward. As fixed effects, we entered the prediction error into the model. As random effects, we had intercepts for subjects as well as by-subject random slopes for the effect of prediction error. Visual inspection of residual plots did not reveal any obvious deviations from homoscedasticity or normality. We obtained p values using likelihood ratio tests of the full model with the effect in question against the model without the effect in question. Blocks of large reward difference (i.e., reward ratio of ¼ or 4) and small reward difference (i.e., reward ratios of ½ or 2) were analyzed separately. The regression analysis was also performed separately on trials with negative prediction error (actual reward less than expected reward) and positive prediction error (actual reward greater than expected reward).

Results

Behavioral data analysis

We first analyzed the behavioral data to verify that the manipulation of reward and visual salience systematically affected choice. Subjects responded in >99% of the trials. Figure 2a displays the average performance of all subjects for different reward conditions. Within each reward level, the percentage of face choices (or, conversely, house choices) increased with the stimulus salience level, regardless of whether or not its relative reward was higher than that of the other target. Since house salience levels were negatively correlated with face salience levels, only face salience was indicated in the figure. We also found that the percentage of face choices (or, conversely, house choices) increased with the stimulus relative reward when fixing the salience level and comparing across reward levels. In the next step, we investigated the interaction effect on behavioral performance between reward and visual salience. We found a relatively high percentage of car choices in the conditions where the reward and salience of targets conflicted, e.g., very high face reward but very low face salience, especially in the large reward difference conditions. The likely explanation is that subjects were taking risks to guess the location of the higher-reward target, but mistakenly chosing the distractor since the salience of higher-reward target was low. In the case of small reward difference, selecting the low-reward but high-salience target was a better way of maximizing reward, and thus in this case we observed more low-reward target selections in the behavioral data. This result is consistent with a previous study of saccadic decisions among multiple objects of simple shape stimuli that suggested an ideal Bayesian observer model for maximizing expected reward in complex environments (Navalpakkam et al., 2010). This result is also consistent with a computational behavioral model that accounts for the interaction between salience and reward factors when they are incongruent (Chen et al., 2013). Figure 2b shows the reaction time for the highest face reward (R_f/R_h = 4) and the highest house reward (R_f/R_h = ¼) conditions. Two-way repeated-measure ANOVA showed the reaction time was significantly different across salience levels (p = 0.013, F_(4,64) = 3.434), as well as a significant interaction between reward and salience (p < 0.001, F_(4,64) = 99.67). We observe that the reaction time decreased with the salience level of the target that corresponded to higher reward.

Feedback ERP analysis

Next we analyzed feedback-related ERPs to compare to previous findings that have shown these to be modulated by reward level. We observed feedback-related potentials for all reward level conditions. Figure 3a shows the grand-averaged ERP following high-reward and low-reward feedback for the small reward difference condition (R_f/R_h = ½ and 2). These ERPs are shown for three representative midline electrodes, Fz, Cz, and Pz. The low-reward-feedback ERPs were more negative than high-reward ERPs in an earlier time window and became more positive in a later time window. The maximum differences between ERP curves in channel Fz were found at 321 and 437 ms after feedback. The spatial distribution of ERP difference across all electrodes at these time points is displayed in Figure 3b. We then selected two time windows around these time points (early window, 280–360 ms; late window, 360–500 ms) and tested the average amplitudes between the two feedback conditions. The differences of feedback-evoked EEG responses were statistically significant in both time windows (channel Fz, early, p = 0.025, F_(1,15) = 6.18; late, p < 0.001, F_(1,15) = 74.83). We also found a significant effect of reward difference on the late component at frontocentral electrodes (e.g., at Fz, p = 0.026, F_(1,15) = 6.07). No significant reward by feedback interactions were found in either time window.

Figure 3.

a, Grand-average feedback-locked ERPs for midline electrodes Fz, Cz, and Pz. The ERPs following low-reward feedback (red line) are more negative at an early time window (light gray) and more positive in at a late time window (dark gray) relative to what is seen for high-reward feedback (blue line). b, Topologies of ERP differences between low- and high-reward-feedback trials centered around the early (321 ms) and late (437 ms) time windows.

Source localization of feedback ERPs

We next investigated the sources that potentially generated the feedback-related activity for both the early and late time windows we identified in the ERP analysis. Specifically, we used the source localization algorithm sLORETA (Pascual-Marqui, 2002) to estimate the source distributions in the cortex. We grouped trials that had either high-reward feedback or low-reward feedback. For each subject, we found two time points (early and late time) where the ERP difference was maximum and selected two 50 ms time windows centered at these time points. We then calculated averaged ERP amplitudes for each time window. For each individual subject, current density distributions of both groups were estimated based on the average ERP, resulting in sLORETA values that were analyzed for significance via paired t tests to identify differences between groups. Randomized statistical nonparametric mapping with 5000 permutations was performed to obtain the corrected critical threshold and p values (Nichols and Holmes, 2002). The cortical areas that show significant differences at the p < 0.05 level are shown in blue and cyan in Figure 4. The difference between current densities of feedback conditions was found to be significant in both time windows, with activation following low-reward feedback being stronger. For the early component, the difference was significant (minimum log of the ratio of averages, −1.52; p < 0.01) in the supplementary motor area (SMA; BA 6), inferior parietal lobule (BA 40), and anterior cingulate cortex (ACC; BA 24). For the late component, the difference was significant (minimum log of the ratio of averages, −1.72; p < 0.01) in the anterior cingulate cortex (BA 32), supplementary motor area (BA 6), medial frontal gyrus (BA 8), and cuneus/precuneus (BA 7).

Figure 4.

a, b, The sLORETA images showing statistical differences (log of ratio of averages) between groups with high and low magnitude of optimal early discriminating components. Significant differences are observed at the SMA, inferior parietal lobule, and ACC for the early window (a), and at the ACC, SMA, medial frontal gyrus, and precuneus for the late window (b).

Impact of stimulus salience on feedback activity: ERPs and single-trial analysis

To examine the impact of salience on the feedback-related activity, we compared ERPs following high-reward feedback of five salience levels. The conditions of higher face reward were analyzed separately from conditions of higher house reward because the salience of faces and houses are negatively correlated. In the high face reward condition, we found a strong positive component at ∼400 ms after feedback when face salience was low (Fig. 5a). The amplitude of this component decreased with increasing face salience. Similar results were observed in the high house reward condition: the amplitude increased with face salience, which was equivalent to a decrease in house salience. Statistical tests were performed on averaged ERPs of the two time windows. At the 350–450 ms time window, there were significant differences found between salience levels (p < 0.001, F_(4,60) = 19.57), but not between categories (p = 0.054, F_(1,15) = 4.35). Similar effects were found for the 100–200 ms window (significant salience effect, p < 0.001, F_(4,60) = 7.19; nonsignificant category effect, p = 0.14, F_(1,15) = 2.42). No interaction effect was found between salience and category factors for either time window. Figure 5b shows scatter plots of average ERP amplitudes at 350–450 ms versus percentage of choices of the higher-reward target for all subjects in different salience conditions. Analysis revealed a significant negative correlation between ERP amplitude and behavioral performance (faces, r = 0.499, p < 0.001, F_(1,78) = 25.79; houses, r = 0.514, p < 0.001, F_(1,78) = 27.99).

Figure 5.

a, Grand-average ERP responses to high-reward feedback at five different salience levels. Conditions of higher face reward (R_f/R_h > 1) and higher house reward (R_f/R_h < 1) were analyzed separately. The amplitude in the shaded area (350–450 ms) shows significant correlation with salience levels. b, Plots of the average ERP amplitude versus the choice rate of the higher-reward target. Each dot refers to the data at one salience level of one subject.

The previous set of analyses focused on feedback-locked ERPs. In the next set of analyses, we investigate the feedback-locked EEG on a single-trial basis. For both high reward difference and low reward difference conditions and for all subjects, we were able to discriminate high-reward-feedback versus low-reward-feedback EEG trials with highly significant accuracy for multiple EEG windows. Figure 6a shows the average A_z values in the high reward difference condition across all subjects at each time point from feedback to 800 ms after feedback, with the A_z value corresponding to a significance level of p = 0.01 (FDR corrected for multiple comparison) determined by our permutation method (see Materials and Methods, Single-trial discrimination). The feedback discrimination curve displayed two peaks with local maxima at 320 and 410 ms, which matched the two components we found in the ERP analysis (Fig. 3). The scalp maps of the average forward models at these peak times are also shown in the figure. We also performed a single-trial analysis of the stimulus-locked data to find a component most discriminable of face and house images. The average discrimination curve showed only a single peak, with a maximum at 350 ms (p = 0.011), with the largest magnitude of spatial weights over central electrode sites (Fig. 6b). For each subject, the weighting vector estimated from the optimal training window was then applied to the same time window of all trials to compute the discriminating component. The outputs of this discriminating component y represented the confidence of the classifier in its prediction of face versus house EEG discrimination (Eqs. 3, 4) and were further used to calculate the expected reward (Eq. 5) and prediction error of each trial.

Figure 6.

Single-trial EEG discrimination results. a, b, Group mean discrimination performance for the high- versus low-reward feedback (a) and face versus house stimulus (b). SEs across subjects are indicated with shading. The significance level is achieved by a label permutation method (30000 times) and FDR corrected for multiple comparisons. Topographies represent the group average forward models of the discriminating components at the time of peak discrimination (referred to as optimal discriminating components). In a, time 0 ms is the time of the feedback; in b, it is the time of the stimulus.

To establish a relationship between model-based prediction error and single-trial feedback EEG-derived components, we performed a mixed effects regression analysis. Large reward difference and small reward difference conditions were analyzed separately. In this model, we focused only on face versus house discrimination and ignored car images to simplify the model. This simplification is reasonable because (1) car images were set at a very low salience level so it was very rare that subjects had strong responses to cars, and (2) there was almost no car selection when face and house rewards were equal (Fig. 2a, third panel). This means subjects could respond to either target that had a higher salience level and successfully ignore cars. Therefore, the minimum expected reward was equal to the smaller reward (2¢), and the maximum positive errors were 2¢ in the small reward difference condition and 6¢ in the large reward difference condition. We also segregated trials based on the valence of prediction error (positive error means the actual reward is greater than the expected reward). Results of the mixed effects analysis on all subjects are listed in Table 1. Both early and late feedback STV had positive correlation with negative prediction errors, but no significant correlation was found in trials with positive prediction error. Figure 7 displays outputs of the early feedback STV versus prediction error for one sample subject.

Table 1.

Summary of mixed effects analysis of the relationship between feedback STV and prediction error

Reward condition	Component	Positive error		Negative error
		p	χ(1)2	p	χ(1)2
Small	Early	0.61	0.26	0.002*	9.18*
	Late	0.67	0.19	0.001*	10.82*
Large	Early	0.09	2.93	0.011*	6.41*
	Late	0.40	0.71	0.030*	4.73*

*Significant at p < 0.05.

Figure 7.

Scatter plots of early feedback STV against prediction error from one representative subject. Trials with positive errors and negative errors are analyzed separately. Each dot represents a single trial, and each red line demonstrates the linear fits for each group of data.

EEG-informed prediction error and subsequent choice

We also investigated the influence of EEG-informed reward prediction error on the subject's choice in the following trial. Large reward difference and small reward difference conditions were analyzed separately. Specifically, this analysis was focused on “incongruent trials,” in which the high-value target was of low salience and the low-value target was of high salience. We performed analysis only on incongruent trials since incongruent trials demonstrated enough variation in the subjects' choices. For “congruent trials,” in which the high-value target was of high salience and low-value target was of low salience, subjects could easily select the high-value target and thus had very small variation in their choices. The prediction error of each trial was normalized by subtracting the average prediction error of the corresponding condition, large reward difference or small reward difference, for each individual subject. Within each condition, all incongruent trials were first grouped based on the choices: high-value target, low-value target, or zero-value distractor. Trials of choosing high-value targets and distractors were further grouped together and referred to as high risk, with the assumption that the subjects took risks to guess the location of the high-value target for both cases. Trials of choosing low-value targets were referred to as low risk, since the low-value target was of high visual salience. As shown in Figure 8, the mean prediction error was more negative for trials followed by low-risk choices. Two-tailed paired t tests showed that the difference was significant for both reward conditions (small reward difference condition, p = 0.018, t₍₁₅₎ = 2.67; large reward difference condition, p = 0.043, t₍₁₄₎ = 2.22). In other words, subjects were more likely to make a low-risk choice after a negative reward prediction error. Based on the behavioral results, we can assume that the original strategy of the subject was to try to choose high-value targets for all trials. For a certain trial, when a low-value target was chosen, it was likely that the subject changed their decision strategy for this trial. Therefore, this result further supports the notion that large negative prediction errors were more likely to make subjects adjust their decision behavior on the subsequent trial (Cohen and Ranganath, 2007).

Figure 8.

Influence of prediction errors on subsequent trials. Small reward difference (left) and large reward difference (right) conditions were analyzed separately. The normalized prediction error (PE) before low-risk choices was more negative compared to the one before high-risk decisions. Error bars indicate SE across subjects.

EEG-informed prediction error and theta power

Theta oscillations have been shown to reflect the degree of prediction error (Cohen et al., 2007; Cavanagh et al., 2010) in reinforcement learning models. We therefore tested whether a similar association exists in our paradigm for the EEG-informed prediction error. To estimate the power of theta oscillations on a single-trial basis, EEG epochs were extracted from −500 to +800 ms relative to the feedback onset for each trial and filtered at the theta frequency band (4–8 Hz). We subsequently constructed the envelope of the filtered signal via the Hilbert transform (computing the amplitude of the analytic signal), which ignored phase information and revealed oscillatory power fluctuations over time (Lou et al., 2014). To allow a direct comparison across subjects, the power estimates of individual subjects were normalized by their baseline (defined by the averaged theta power of a 30 s resting state EEG).

Similar to the effect of reward on FRN, we observed that theta power after feedback was also modulated by the reward in frontal and central areas. Figure 9a shows the averaged time series of theta power at channel Fz for high- and low-reward trials in the small reward difference condition. The scalp map shows the spatial distribution of the theta power difference averaged over the time window 300–500 ms. The averaged theta power of all reward conditions in channel Fz over this time window is shown in Figure 9b. To further demonstrate the effect of feedback on theta power, we ran a two-way repeated-measures ANOVA with factors reward difference (two levels, small and large) and feedback reward type (two levels, high and low reward). A significant effect was found for reward type (F_(1,15) = 16.89, p < 0.001), but not reward difference (F_(1,15) = 2.93, p = 0.11) or their interaction (F_(1,15) = 2.13, p = 0.17).

Figure 9.

a, Time series of theta band power (4–8 Hz) from −500 to 800 ms relative to feedback onset in channel Fz. Theta power of trials with low-reward feedback showed stronger theta power enhancement at ∼300–500 ms. The topography shows the distribution of power difference over this time window. b, Averaged theta power over 300–500 ms after feedback onset in channel Fz. Error bars indicate the SE across subjects.

We also performed a mixed effects regression analysis between EEG-informed reward prediction error and feedback theta power on a single-trial basis. For each trial, we computed the average theta power over the time window from 300 to 500 ms after feedback onset. Similar to the analysis of FRN, we segregated trials based on the valence of prediction error (positive and negative). For trials with negative prediction errors, we found significant negative correlation between prediction error and theta power (channel Fz, small reward difference, χ₍₁₎² = 13.78, p < 0.001; large reward difference, χ₍₁₎² = 9.35, p = 0.002). No significant correlation was found in trials with positive prediction error (channel Fz, for small reward difference, χ₍₁₎² = 3.6, p = 0.06; large reward difference,χ₍₁₎² = 0.29, p = 0.59).

Discussion

To explore the neural correlates of reward processing when expected reward is a function of stimulus salience and value, we conducted a study that systematically manipulated these two dimensions while collecting EEGs from subjects instructed to harvest as much reward as possible. We found that for trials following low reward, feedback-locked ERP amplitudes were more negative in an early time window and more positive in a late window (Fig. 3). We interpret this ERP modulation as a manifestation of FRN, with the result similar to those of other feedback valence studies (Yasuda et al., 2004; Hajcak et al., 2005; Cohen et al., 2007), though the typical FRN is reported in terms of a difference in reward gains versus losses. In our experimental design, we ruled out the impact of penalty/loss and focused only on reward delivery and omission. This observation is also consistent with previous studies on the modulation of reward magnitude on feedback ERP components, which show more negative early ERP when reward magnitude is small and a more positive late component amplitude if the reward magnitude is larger (Yeung and Sanfey, 2004; Wu and Zhou, 2009).

To further investigate the relationship between our results and the FRN, we performed source localization of the early and late feedback components (Fig. 4). Both components showed clusters localized in the ACC and SMA, which have been reported previously as brain regions for dipole fits of the FRN (Miltner et al., 1997; Gehring and Willoughby, 2002; Ruchsow et al., 2002; Nieuwenhuis et al., 2005; Doñamayor et al., 2011). These localizations are also consistent with fMRI analysis of reward processing and reward prediction errors (Knutson et al., 2005; Behrens et al., 2007; Dayan and Niv, 2008; Niv and Schoenbaum, 2008). Our results also showed a slight difference between the early and late components. The early component had a cluster in the inferior parietal lobule, which might suggest the early processing of visual feedback and interpretation of sensory information. In previous fMRI studies, Vickery and Jiang (2009) found that the inferior parietal lobule was involved in decision making under uncertainty, showing higher activity when humans subjects were given trial-by-trial feedback on choice outcomes compared to when feedback was not provided. The late component also showed activation in the medial frontal gyrus that has been implicated in the processing of reward prediction error (Matsumoto et al., 2007).

We also found that feedback-locked ERPs were modulated by stimulus salience. The ERP amplitude at 350–450 ms after feedback was negatively correlated with the salience level as well as the percentage of selection of high-reward targets (Fig. 5). Given that we could manipulate the difficulty of perceptual decisions and behavioral accuracy by altering the phase coherences of target images (Philiastides and Sajda, 2006; Philiastides et al., 2006), our hypothesis is that the probability of earning a high reward can also be modulated by the stimulus salience, even though the actual reward delivery rate of the target was 100%. Unlike varying reward probability in a predefined way, as seen through typical reinforcement learning paradigms, the reward probability in this task varied according to the perceptual decisions made before feedback, and thus was positively correlated with the salience level and selection percentage of the higher-reward target. This hypothesis is consistent with the findings from a previous study using a reinforcement learning model that revealed the negative correlation between reward probability and ERP amplitude following “win” feedback (Cohen et al., 2007). Our result is also consistent with another previous study of feedback processing in decisions under risk that showed larger positive potentials for positive feedback in high-risk choices (Schuermann et al., 2012), which corresponds to low-salience targets with high reward in our experiment. A similar idea was also used in a previous study to examine the shared neural basis of error detection and decision confidence (Boldt and Yeung, 2015).

To characterize trial-by-trial variability of the feedback-related evoked activity, we used a single-trial analysis method to estimate temporally localized EEG components that maximally discriminated between the high- and low-reward-feedback conditions. In a previous study of reinforcement learning models, Philiastides et al. (2010) found EEG activity correlated with prediction error valence, magnitude, and future choice. Another study using single-trial analysis found that reward motivation significantly influenced neural activity in response to target images (Hughes et al., 2013). In our study, we found two discriminating components that corresponded to feedback-related ERPs and their spatial distributions consistent with our sLORETA results. The early component showed a parietal-temporal distribution, whereas the late component showed large weights in frontal-central areas that matched the location of the ACC and SMA. Many studies on reward-based decision making have investigated the relationship between feedback-related activity and reward prediction errors, but to our knowledge, these studies all used reinforcement learning models to estimate prediction errors (Cohen and Ranganath, 2007; Cavanagh et al., 2010; Philiastides et al., 2010). Unique to this work, we estimated prediction errors on a single-trial basis by integrating value information with perceptual decision evidence, given single-trial decoding of EEGs (Eqs. 3–5). The variation of expected reward in this experiment is not due to the changes of the actual reward probability of each option. Instead, we fixed the rewards of targets and manipulated their salience levels, so the expected reward of each trial changed according to the subjective perception of targets. Specifically, we used the STV from a component that was stimulus locked, ∼350 ms after the stimulus and best discriminated face versus house trials, to estimate the expected reward. Unlike traditional FRN studies, there was no time interval between response and feedback in our design. This is because we wanted to directly link stimulus-related activity (prefeedback), representing the confidence of perceptual decision (i.e., level of stimulus evidence), with feedback-related activity, representing the evaluation of the previous decision, and avoid any confounding factors in between. Interestingly, our results showed an asymmetry with respect to the valence of prediction error in EEG correlates of feedback processing. We observed that the STV of both types of feedback activity correlated with negative prediction error, but no significant correlation was found for positive prediction error trials. Similar phenomena were found in an FRN study that reported no significant difference in FRN amplitude for positive prediction errors (Cohen and Ranganath, 2007). One explanation of this from reinforcement learning model studies is that this feedback-related activity (or FRN) was related to how subjects adjusted their decision behavior on the subsequent trial. Large negative prediction errors were more likely to make subjects change their decision behavior and thus elicited more negative FRNs. From a reward expectancy perspective, reinforcement learning theory predicts that larger FRNs should be elicited by unexpected and unfavorable outcomes (Holroyd et al., 2003), which in this paradigm are indicated by a more negative prediction error. An alternative interpretation is based on one previous study that proposed that a reinforcement learning model cannot fully interpret how FRN encodes prediction error and suggested prediction errors were associated with motivational salience rather than motivational value (Talmi et al., 2013). Without using a reinforcement learning model, our results support their idea that feedback ERP activities are stronger for negative outcomes because negative feedback is more motivationally salient (Oliveira et al., 2007; Talmi et al., 2013).

We further examined the relationship between current prediction error and subsequent behavior. Our results showed that the reward prediction error, estimated based on stimulus-locked EEG measurements, was more negative before the trial with “low-risk” choice. This observation is consistent with previous reinforcement learning studies that showed subjects were less likely to make high-risk choices after negative outcomes in a gambling task (Cohen and Ranganath, 2005). Although it was assumed that all trials were independent and the final choices were determined by the previous perceptual decisions, we still observed the influence of prediction error on the subsequent trial. However, the influence of the prediction error in this experiment was relatively small compared to typical reinforcement learning tasks, since subjects could not “learn” much from their previous decisions.

We also investigated the relationship between theta power and EEG-informed prediction error. As expected, low-reward-feedback trials had significantly higher theta power in the frontocentral cortex. This is consistent with previous findings of enhanced theta power and phase locking following losses compared to wins (Cohen et al., 2007; Trujillo and Allen, 2007; Cavanagh et al., 2009; Lega et al., 2011). More directly relevant to our findings, Cavanagh et al. (2010) indicated that the medial prefrontal cortex theta power following reinforcement cues scaled with negative prediction error, but not positive prediction error. They also found that theta oscillation reflected the degree of future behavioral adaptation through examining the relationship with immediate reaction time changes, but one should note that the behavioral adaptation was more difficult to observe in this paradigm because the reaction time was influenced more by the target salience than the feedback of previous trials. Although reinforcement learning was not integral to our paradigm, this additional analysis on theta power, together with the source analysis on the feedback ERP, supports the theory that theta oscillations over the medial prefrontal cortex, which presumably underlie the FRN component, reflect a common computation used for realizing the need for cognitive control (Cavanagh and Frank, 2014).

In summary, the present study examined feedback-related activity in an optimal reward-harvesting experiment. Our findings indicate that both rewards and perceptual salience have an impact on the magnitude of feedback-locked ERPs. Unlike traditional reinforcement learning paradigms, we fix the reward of each target in a block and manipulate only visual salience to change the uncertainty of receiving rewards. Therefore, we estimate the reward prediction error based on EEG measurements instead of parameters estimated from a reinforcement learning model. Our results suggest that the valence of prediction error is more important than the valence of the actual feedback, since only positive rewards are delivered in the experiment (no penalty or loss). Specifically, our single-trial analysis shows an asymmetric effect on the prediction error valence: STV of feedback components are correlated only with negative prediction errors, suggesting that larger negative prediction errors are more likely to be important for changing subjects' decision behavior.