Signals in human striatum are appropriate for policy update rather than value prediction

Abstract

Influential reinforcement learning (RL) theories propose that “prediction error” signals in the brain’s nigrostriatal system guide learning for trial-and-error decision-making. However, since different decision variables can be learned from quantitatively similar error signals, a critical question is what is the content of decision representations trained by the error signals. We used functional magnetic resonance imaging (fMRI) to monitor neural activity in a two-armed-bandit counterfactual decision task that provided human subjects with information about foregone as well as obtained monetary outcomes so as to dissociate teaching signals that update expected values for each action, vs. signals that train relative preferences between actions (a “policy”). The reward probabilities of both choices varied independently from each other. This specific design allowed us to test whether subjects’ choice behavior was guided by policy-based methods, which directly map states to advantageous actions, or value-based methods such as Q-learning, where choice policies are instead generated by learning an intermediate representation (reward expectancy). Behaviorally, we found human participants’ choices were significantly influenced by obtained as well as foregone rewards from the previous trial. We also found subjects’ blood-oxygen-level-dependent (BOLD) responses in striatum were modulated in opposite directions by the experienced and foregone rewards but not by reward expectancy. This neural pattern, as well as subjects’ choice behavior, is consistent with a teaching signal for developing “habits” or relative action preferences, rather than prediction errors for updating separate action values.

INTRODUCTION

According to influential theories, the dopamine system broadcasts a “prediction error” signal for reinforcement learning (RL) (Barto, 1995; Schultz et al., 1997; Dayan and Abbott, 2001; Rangel et al., 2008). However, relatively little is known about the precise action of this signal in guiding subsequent decisions, and indeed, error-driven learning can support qualitatively different decision-making strategies (Sutton and Barto, 1998; Dayan and Abbott, 2001). Two approaches differ in the content of the information learned. Policy-based (“direct actor”) methods such as the actor/critic learn a “policy” or direct mapping from situations to advantageous actions, adjusting this in light of received rewards (Barto, 1995; Sutton and Barto, 1998; Dayan and Abbott, 2001). In contrast, value-based (“indirect actor”) methods such as Q-learning produce choice policies indirectly by learning an intermediate representation: the expected reward (“value”) for each candidate action (Watkins and Dayan, 1992). These intermediate representations can then be compared to derive a policy.

These algorithms formalize important neuropsychological concepts. Policy learning parallels the notion that reinforcement “stamps in” stimulus-response habits, which is central to contemporary accounts of drug abuse (Thorndike, 1898; Dickinson and Balleine, 2002; Everitt and Robbins, 2005). However, either value or policy learning can be accomplished using teaching signals that, in typical tasks, appear nearly identical. In a typical task, where subjects repeatedly select from different options for rewards, action values can be learned from a prediction error (PE) measuring the difference between the received reward and the option’s previously predicted value. A policy can also be updated by comparing the received reward to an expected (e.g., reference or state) value prediction (Dayan and Abbott, 2001). Indeed, the actor/critic algorithm learns both values and policies using the same error signal (Barto, 1995). Attempts using such tasks to distinguish versions of these signals have produced inconsistent results (Morris et al., 2006; Roesch et al., 2007).

We studied human choices and fMRI signals in an RL task modified to distinguish signals appropriate for updating policies versus value predictions (Fig. 1A). Subjects repeatedly chose between two slot machines, associated with independent probabilities of delivering monetary reward. For each choice, the screen displayed the amount of reward subjects won, but also what they would have won, had they chosen the other option. This information should affect teaching signals for values or policies differently, allowing us to distinguish these computational strategies. If values for the two options are learned separately, then two prediction errors are needed: comparing each value prediction to its associated outcome. In contrast, the policy requires only a single update, depending only on the difference between the obtained and foregone outcomes (such that rewards need not be compared to predictions).

Figure 1.

Experimental Design. A, Timeline of a single trial. B, Example from one subject of changing reward probabilities for both slots.

Since both rewards were conditionally independent, it was unambiguous to determine their separate effects on neural activity. We focused on activity in the ventral striatum, which has been shown to correlate with PE signals similar to those seen in dopaminergic neuron recordings (Schultz et al., 1997; Berns et al., 2001; O'Doherty et al., 2003; Lohrenz et al., 2007).

Materials and Methods

Subjects

20 subjects participated in the experiment (13 female, mean age = 21.2 ± 5.0 years, age range = 18 – 39). The study was approved by the university committee on activities involving human subjects (UCAIHS), and all subjects provided informed consent before the experiment.

Counterfactual Learning Task

The task consisted of four sessions of 60 trials each, separated by short breaks. At the start of each trial, subjects were presented with pictures of two differently colored (green and purple) slot machines. Subjects could select their choice of the slot machines using two button boxes (Fig. 1A) one for each hand and the left-right ordering of the slot machines was counterbalanced across subjects but kept constant across the task for each individual. Subjects had a maximum of 1.5 seconds to enter a choice; if no choice was entered during the 1.5 sec window, a warning message “Please respond faster!” was displayed for 1 second, which was then followed by a 1–5 seconds delay before another trial started. In general, subjects responded well before the timeout, with a mean reaction time of 523 ± 20 ms (mean ± s.e.m). On valid trials, the chosen slot machine was highlighted and, three seconds later, the outcomes for both chosen and unchosen slot machines were displayed, by overlaying a picture of a $1 coin or a circle marked $0 on each machine (Fig. 1A). The outcomes stayed on the screen for 2 seconds, after which the screen was cleared. The trial sequence ended 6.5 seconds after trial onset, and a randomly jittered inter-trial interval (ITI) with a mean of 2 seconds was introduced before the beginning of the next trial.

The payoff for each slot machine (i={L, R}) on each trial (t) was either $1 or $0 (Fig. 1A), with each payoff drawn independently from a binomial distribution according to a machine-specific probability p_i,t that gradually changed over trials (Fig. 1B). At the beginning of the task (t=1), both probabilities were independently drawn from a uniform distribution with boundaries of [0.25,0.75]. Following each trial, the probabilities were each diffused either up or down by adding or subtracting 0.05 (equiprobably and independently). The updated probabilities p_i,t+1 were then reflected off the boundaries [0.25,0.75] to maintain them within that range.

Subjects were instructed that they would be paid only according to the accumulated outcomes of the slot machines that they actually chose but not the forgone choices. Subjects were also told that the final points they earned would be converted proportionally to dollars but not told the actual scaling factor (which was 0.25).

Behavioral Analysis

We first used a logistic regression analysis to estimate how a subject’s choice on trial t (the dependent variable, for all t ≥ 2) was influenced by the chosen and unchosen rewards (R_c and R_u) on the previous trial. We specified three independent variables based on the events on the preceding trial: reward on the chosen slot machine (coded as 0 for no reward, or +1 or −1 for reward following a left or right choice, respectively), reward on the unchosen slot machine (coded similarly), and the choice on that trial (a dummy variable coded as +1 or −1 for left or right choices, so as to capture any first-order autocorrelation in the choices (Lau and Glimcher, 2005)). We estimated regression weights for each subject individually using maximum likelihood, and report summary statistics for these quantities across subjects (Table 1).

Table 1.

Mean parameter fits ± S.E.M. from 20 subjects for three models

	Q-Learning		Policy-gradient		Logistic regression
QL,0	0.54 ± 0.09		--	Rc	3.44 ± 1.51
QR,0	0.61 ± 0.08	η	0.48 ± 0.06	Ru	2.09 ± .053
α	0.35 ± 0.07	α	62.23 ± 20.22	Repetition	1.89 ± 0.58
κ * α	0.24 ± 0.07	κ	0.48 ± 0.07	Intercept	−0.03 ± 0.06
β	9.08 ± 2.62	W0	−14.28 ± 35.87		--

We also fit the parameters of two learning models (detailed below) to each subject’s choices by maximizing the (log) likelihood of the choice sequence:

$${\displaystyle \sum _t\text{log}P(c_{s,t}|\mathrm{\Theta })}$$

separately for each subject s. Here, c_s,t denotes the choice made by subject s on trial t and Θ is the parameter set. We sought optimal parameters using a nonlinear optimization algorithm (fmincon, Matlab optimization toolbox), and 30 different starting search locations for each subject so as to avoid local maxima. We report negative log likelihoods (smaller values indicate better fit), both pure and penalized (Kass and Raftery, 1995) for model complexity using the Bayesian information criterion (BIC). We also report a pseudo-r² statistic (Camerer and Ho, 1999; Daw et al., 2006), defined as (r-l)/r, where l and r are, respectively, the negative log likelihoods of the data under either fit model and under purely random choices (P_c,t = 0.5 for all trials and subjects).

Reinforcement learning models: Q-Learning model

A Q-learning model (Dayan and Abbott, 2001; Sutton and Barto, 1998; Watkins and Dayan, 1992; Daw et al., 2006; Li et al., 2006) learns an expected value (“Q value”) for each option based on experienced outcomes, and then chooses accordingly. We adapted a standard model to allow learning from unchosen as well as chosen rewards; this simply updates each machine’s value on each trial according to its own prediction error. We allow the updating of the unchosen option to be controlled by a distinct learning rate, to capture any differences in attention to the two outcomes. Specifically, at each trial, both values were updated according to the feedback received:

$$\begin{array}{l}{Q}_{c,t+1}=Q_{c,t}+\alpha {\delta }_{Q_c,t}\hfill \\ Q_{u,t+1}=Q_{u,t}+\kappa \alpha {\delta }_{Q_u,t}\hfill \end{array}$$

where α is a free learning rate parameter, κ modulates the learning rate for the unchosen option, and

$$\begin{array}{l}{\delta }_{Q_c,t}=R_{c,t}-Q_{c,t}\hfill \\ {\delta }_{Q_u,t}=R_{u,t}-Q_{u,t}\hfill \end{array}$$ (1)

are prediction errors for the rewards R received on each machine.

We further assumed the probability of choosing either machine (i={L, R}) was softmax in its Q value:

$$P_{i,t}=\frac{\text{exp}(\beta Q_{i,t})}{{\displaystyle \sum _j\text{exp}(\beta Q_{j,t})}}$$

with free exploration parameter β and initial Q values (Q_L,0 and Q_R,0; see Table 1).

Policy-gradient model

A second approach to reinforcement learning maintains policy parameters specifying a preference over options, and updates this preference with feedback to achieve stochastic gradient ascent on the expected reward (Dayan and Abbott, 2001; Dayan and Daw, 2008). We again represent the selection policy as softmax, here the chance of choosing machine L is

$$P_{L,t}=\frac{1}{1+\text{exp}(-w_t)}$$

with the policy parameter w. Here, $P_{R,t}=1-P_{L,t}=\frac{1}{1+\text{exp}(w_t)}$.

If 〈r_R〉 and 〈r_L〉 are the average expected reward on either slot machine, then the expected reward given any particular policy is:

$${\langle r\rangle }_w=P_L\langle r_L\rangle +P_R\langle r_R\rangle ,$$

and its gradient with respect to the policy parameter,

$$\frac{\partial {\langle r\rangle }_w}{\partial w}=P_L{P}_R\left[\langle r_L\rangle -\langle r_R\rangle \right],$$

is proportional to the difference between the two reward rates. On each trial, then, the gradient can thus be sampled stochastically as the difference between obtained and foregone rewards; note that this does not require separately estimating the average values themselves. This algorithm instead works directly with the policy w.

To write the gradient rule in a way that relates more directly to the standard case when only the chosen reward is received, we formulate it in terms of the chosen and unchosen rewards (rather than left and right), so that

$$w_{t+1}=\{\begin{array}{l}\eta w_t+\alpha P_R{P}_L{\delta }_{w,t};c_{s,t}=L\hfill \\ \eta w_t-\alpha P_R{P}_L{\delta }_{w,t};c_{s,t}=R\hfill \end{array}$$

with error term

$${\delta }_{w,t}=R_c-\kappa R_u$$ (2)

stepsize parameter α, and decay parameter (to allow learning in the case of nonstationarity, as in the task here) η, and initial w (w₀). The final free parameter, κ, again allows for the gradient to skew (e.g. due to differential attention) toward the chosen or unchosen reward (R_c and R_u, respectively).

Imaging acquisition

Functional images (T2*-weighted echo-planar images with BOLD contrast) were collected using a 3T Siemens Allegra head-only scanner and a Nova Medical (Wakefield, MA) NM-011 head coil. To optimize functional sensitivity in the orbitofrontal cortex and temporal lobes we used a tilted acquisition oriented at 30⁰ above the AC-PC line (Deichmann et al., 2003). This yielded 33 oblique-axial slices with 3mm inter-slice thickness, 3×3 mm in-plane resolution, with coverage from the base of the orbitofrontal cortex and medial temporal lobes to the superior border of the dorsal anterior cingulate cortex. Repetition time (TR) was 2 seconds. Subjects’ heads were restrained by plastic pads to minimize head movement during the experiment. A T1-weighted structural image (MPRAGE sequence, 1×1×1 mm) was acquired after the functional run for each subject to allow localization of functional activity. High-pass filtering with a cutoff period of 128 seconds was also applied to the data.

Functional imaging analysis

Imaging data were preprocessed and analyzed using SPM5 (Wellcome Department of Imaging Neuroscience, Institute of Neurology, London, U.K.) and xjView (http://www.alivelearn.net/xjview/), except for multiple comparisons correction on final results, which was done using SPM8. Motion effects were corrected by aligning images in each run to the first volume using a 6-parameter rigid body transformation. (To account for additional residual effects of movement, the six scan-to-scan motion parameters produced during realignment were also included as nuisance regressors in the functional analysis) Mean functional images were then coregistered to the structural image, and normalized into MNI template space using a 12-parameter affine transformation (SPM5 “segment and normalize” estimated from the structural). Normalized functional images were resampled into 2×2×2 voxel resolution. A Gaussian kernel with a full width at half maximum (FWHM) of 6mm was applied for spatial smoothing.

For statistical analysis, we constructed three impulse events for each trial, at the times of slot machine presentation, choice entry, and outcome presentation. The first two events were included to control the overall variance; we focused on the outcome event here due to the fact that the key prediction error signal is associated with the outcome. In three separate GLMs, we modulated the outcome events with different parametric regressors. First, in an initial attempt to seek activity correlated with teaching signals for either policy gradient or prediction errors associated with chosen and unchosen choices, we constructed a policy gradient regressor as R_c-κR_u, the difference between the obtained and foregone reward, and also prediction error regressors for both chosen and unchosen options, R_c-Q_c and R_u-Q_u. These were entered in two different GLMs (one for the policy signal, one for both reward prediction errors) for whole-brain regression analyses (Fig. 2). Next, to more carefully differentiate effects related to either type of signal, we noted that equation (1) and equation (2) both consist of different linear combinations of the four variables R_c, R_u, Q_c, and Q_u; we thus constructed a third GLM containing each of these as parametric regressors modulating the outcome impulse event (Fig. 3).

Figure 2.

A, Neural correlates of R_c-κR_u. Overlapping view of the neural correlates of B, prediction errors of chosen choices (δ_chosen and −δ_chosen) and C, foregone choices (δ_unchosen and −δ_unchosen) (p < 0.05).

Figure 3.

A, Effect of decision variables in striatum. BOLD activity positively correlates with R_c and −R_u (p < 0.05) but not with –Q_c or Q_u, even at a loose threshold (p = 0.01, uncorrected). The dotted boxes surround the pairs of effects expected to be significant for value-based learning (R_c-Q_c and –(R_u-Q_u)) and the solid box surrounds those for policy learning (R_c- κR_u). B, Similar activities (bilateral insula, anterior cingulate cortex (ACC) and DLPFC) were positively correlated with –R_c and R_u (p < 0.05).

For all these analyses, the chosen and unchosen rewards were taken directly from the outcomes experienced by the subject, and the Q values were those implied by the Q learning model on each trial, using the outcomes experienced by the subject and the free parameters for the model estimated to best fit the subject’s choice data (Fig. 3). We then convolved these regressors with SPM5’s canonical hemodynamic response function, computed parameter estimates for each subject, and took these estimates to the group random effects level for statistical testing (Friston et al., 1995).

We report significance of activations correcting for whole-brain multiple comparisons using cluster-level false discovery rate (“topoFDR”) algorithm implemented in SPM8, on maps generated at an underlying uncorrected threshold of p<.001. (Note that we did not employ voxel-level FDR, which has recently been argued to be invalid (Chumbley and Friston, 2009)). Accordingly, except where noted, activations are rendered for display using the p<.001 uncorrected threshold, but retaining only those clusters that pass the p<.05 cluster size correction.

Finally, for the ROI regression analysis (Fig. 4), we first identified two ROIs in striatum using the conjunction (Nichols et al., 2005) of R_c and -R_u (thresholded at p < 0.001, uncorrected; Fig. 4A). We used the average activity from each of these regions for the subsequent regression between the neural effect of R_u (the per-subject regression weight for that variable) and the behavioral effect (the per-subject estimate of κ from the best fitting policy gradient model). Note that this approach largely skirts the problem of multiple comparisons in inter-subject regression analyses, since the initial analysis to identify the ROI (the existence of a conjunction effect, in the mean across subjects) does not bias the subsequent test for the between-subject pattern of variation. Significance levels need thus be corrected only for two comparisons (two ROIs, left and right; Fig. 4A) rather than for the whole-brain multiple comparisons used to select the regions.

Figure 4.

A, Error signaling ROI in left ventral striatum, identified from the conjunction of R_c and –R_u across subjects (p < 0.001, uncorrected). B, In left striatum (circled in Figure 4A), the neural effect size for –R_u was positively correlated, across subjects, with the weight for the unchosen reward, κ, estimated from choice behavior (p = 0.018, Bonferroni corrected; r = 0.57).

All results reported herein were qualitatively the same when Q values were computed using a common set of parameters across subjects, taken as the average over all subjects of those from the individual fits. Results were also invariant to changes in the ordering of the entry of regressors in the design matrix (which, due to serial orthogonalization of parametric regressors in SPM, might hypothetically have impacted their relative significance).

RESULTS

Foregone Reward and Action Selection

First, we assessed subjects’ behavioral sensitivity to the experienced and foregone rewards by using logistic regression to predict each subject’s choices as a function of the feedback she received on the previous trial (Table 1). Across subjects, both experienced (t₁₉=2.27, p <.02) and foregone (t₁₉=3.94, p < .001) rewards significantly predicted the subject’s next choice, with no significant difference in the strength between the two influences (paired samples, t₁₉=.84, p > 0.4). This result is consistent with the previous literature on counterfactual effects, indicating that humans’ choices are affected not only by “what was” but also by “what might have been” (Camille et al., 2004; Coricelli et al., 2005; Lohrenz et al., 2007).

We used two RL models to examine more detailed hypotheses about how experience drove choices via learning (Sutton and Barto, 1998; Dayan and Abbott, 2001; Bhatnagar et al., 2008). A standard value-based model, Q-learning (Watkins and Dayan, 1992), separately tracks the expected value for each option and compares these predicted values at choice time to derive a policy for action selection. However, a policy-based model, the direct actor, learns a policy representing the probability of choosing either action, with an update that is determined by stochastic gradient ascent on the overall expected reward (Dayan and Abbott, 2001; Bhatnagar et al., 2008). Both models were straightforwardly adapted to incorporate foregone rewards, and both included an additional free parameter, κ, to allow for a possible difference in weighting (e.g. attention) experienced rewards and foregone rewards (see Materials and Methods and Table 1). We estimated free parameters and compared the models’ fits by maximizing the likelihood of each subject’s choices (Table 1). The policy-based model generally performed better than the value-based model. Individually the policy-based model outperformed the Q-learning model for 19 of 20 subjects (Table 2) according to the Bayesian Information Criterion.

Table 2.

Qualities of behavioral fits of both models

	Direct Actor	Q-Learning
−LL	107.6	114.3
Pseudo-R 2	0.3534	0.3131
# parameters	4	5
BIC	118.6	128.0

Neural Correlates of Q-Learning and Policy Gradient

Choice behavior was thus most consistent with a policy-based strategy. However, because the behavioral predictions of the two learning strategies are qualitatively similar to one another, we next tested for neural signatures of teaching signals, about which the two hypotheses make quite qualitatively distinct predictions. Specifically (see Materials and Methods and Camerer and Ho, 1999), learning the value of each option separately requires two independent prediction error signals, measuring the difference between the rewards received (R_c and R_u) and expected (Q_c and Q_u) for both chosen and unchosen options (R_c-Q_c and R_u-Q_u). In contrast, due to the symmetry of the task, updating action preferences requires a unitary signal (formally, the gradient of the expected reward with respect to the policy) proportional to the difference between the obtained and foregone rewards, R_c-κR_u.

As a first step, we verified whether these three candidate error signals correlated with blood-oxygen-level-dependent (BOLD) activity in each trial at the time when outcomes were revealed and learning was expected to take place for both Q-learning and policy gradient models. These three signals were extracted from each model using the parameters that best fit the behavior. We focused on ventral striatum, where many previous studies using fMRI have shown error-related neural activity (Berns et al., 2001; O'Doherty et al., 2003; McClure et al., 2003; O'Doherty et al., 2004; Delgado et al., 2005; Daw and Doya, 2006). Accordingly, both the prediction error for the chosen action’s value in the value-based model, R_c-Q_c, and that for the policy in the policy-based model, R_c-κR_u, correlated positively with BOLD activities in the ventral striatum (Figs. 2A, B; p <.05; unless otherwise noted, all statistics concerning fMRI activations are corrected at the cluster level for false discovery rate due to whole-brain multiple comparisons); the prediction error for the unchosen option R_u-Q_u correlated negatively with activity in a similar region (p <.05; Fig. 2C).

The finding that error signals from both models were correlated with BOLD activity in ventral striatum presumably reflects the fact that the candidate error signals are themselves mutually correlated, necessitating finer investigation to separate them apart. For instance, the chosen value error (R_c-Q_c) and policy error (R_c-κR_u) signals might potentially each correlate significantly with striatal BOLD in virtue of sharing the same term R_c. More particularly, all these signals consist of different linear combinations of the same variables: the chosen and unchosen rewards, R_c and R_u, and the expected rewards Q_c and Q_u. Taking advantage of this decomposition and the additivity of the general linear model (GLM) to examine more carefully which error signal was most consistent with striatal BOLD activity, we estimated a GLM that included these four factors as separate regressors. The policy-based model predicts that R_c and R_u should together modulate any teaching-related brain activity in the same voxels but with opposite directions (see Materials and Methods): the weighted combination of these two effects corresponds to the policy error signal. In contrast, the value-based model (Q-Learning) makes no particular claim about spatial overlap or relative direction of R_c and R_u effects (which according to this hypothesis are part of separate error signals that are not necessarily spatially distinct). However, this value-based model does predict that each effect (R_c or R_u) should be accompanied by spatially overlapping activity correlated with its correspondingly associated predictions (Q_c or Q_u), with opposite signs for the R and Q components (R_c-Q_c and R_u-Q_u).

Activity in bilateral ventral striatum positively correlated with R_c (Figure 3A; p <.05), which is a component of teaching signals in both hypotheses. However, as predicted by the policy model, R_u was negatively correlated with BOLD activity in substantially the same region of striatum (Fig. 3A; p <.05). A conjunction analysis confirmed that these two effects occurred in overlapping voxels (shown bilaterally at p <.001 uncorrected in Fig. 4A; the cluster on the right survived whole-brain correction at p <.05). Furthermore, we tested for differences in the spatial expression of these effects by using the contrasts R_c> -R_u and R_c< -R_u to identify voxels where the positive effect of R_c and the negative effect of R_u differed in size. Consistent with the policy model, no such differences were found in striatum (p <.05). Finally, we tested whether value prediction signals Q_c and Q_u were correlated with striatal BOLD activity in the directions opposite to their associated rewards (Fig. 3A), as predicted by the value-based model. However, no such correlations were found for either signal, even at a much weaker threshold (shown in Fig. 3A at p < 0.01 uncorrected) and in either positive or negative directions. In all, the results are consistent with net ventral striatal BOLD activity expressing a single teaching signal proportional to R_c-κR_u, as predicted by the policy gradient algorithm, but do not demonstrate any evidence for a pair of value prediction errors for each of the chosen and unchosen options, either overlapping or spatially separate (Lohrenz et al., 2007). (Indeed, the results also are not consistent with an alternative value-based decision variable, i.e. the relative action value [Q_c - Q_u], since the prediction error, [R_c - R_u] - [Q_c - Q_u], for this combined quantity also predicts effects of both Qs.) The lack of significant value prediction-related activity in our study seems to contrast with many other studies in which striatal BOLD activity was demonstrably modulated by value expectation, as for prediction errors (Berns et al., 2001; O'Doherty et al., 2003; Tanaka et al., 2004; Hare et al., 2008). This apparent difference may be due to the design of the current task, which differs from many others in that, because of its symmetric form and the inclusion of counterfactual feedback, the teaching signal for the policy contains no reward expectation term.

Similarly, and strikingly, throughout the rest of the brain, activity correlated with R_c and R_u was observed in almost the same set of neural pathways (Fig. 3B), but with opposite directions of effect. Evidence for spatially non-overlapping effects was found (R_c > -R_u, p <.05) only in posterior portions of the brain: occipital visual cortex and fusiform areas. Over the whole brain, no activity was found to correlate positively or negatively with Q_c or Q_u even at a lower threshold (p <0.01 uncorrected).

Correlation of Behavioral and Neural Sensitivities to Foregone Reward

Finally, we compared neural and behavioral variation across subjects to investigate whether foregone outcome signaling in ventral striatum was related to choice behavior. In particular, we tested whether, across subjects, there was covariation between estimates of the weight (e.g. attention) given to the foregone outcome as assessed from choice behavior (the parameter κ from the policy-gradient model) and from neural error-related activity in striatum (the effect size for the foregone reward). We first identified areas of error signaling in left and right ventral striatum (using the conjunction of R_c and –R_u effects; Fig. 4A), then tested within each cluster whether the contribution of R_u to this signal covaried across subjects with the weight to foregone rewards estimated from choice behavior. The predicted correlation between the behavioral and neural effects of the foregone reward was significant in left striatum (Fig. 4B, circled in red in Fig. 4A; p = 0.018; Bonferroni corrected for two comparisons, left and right striatum), and trended in the same direction, though not significantly so, in right striatum (p=0.45 corrected; data not shown), suggesting that both neural and behavioral analyses consistently tapped a common learning process.

DISCUSSION

A longstanding question in psychology – dating back to early debates surrounding behaviorism (Thorndike, 1898; Tolman, 1949; Dickinson and Balleine, 2002)– is the representational question: what exactly is learned from reinforcement? Error-driven RL theories are surprisingly ambivalent on this issue. Value-based (Q-Learning) models propose that the difference between obtained and predicted rewards is used to update expected action values, and choice policies are subsequently derived by comparing these intermediate quantities (Barraclough et al., 2004; Daw et al., 2005; O'Reilly and Frank, 2006; Hare et al., 2008; Boorman et al., 2009). Policy-based approaches instead update a choice policy directly, though often alongside a value representation. What makes these two approaches difficult to differentiate is that in most circumstances the policy update signal, derived from the gradient of the expected rewards with respect to the policy, also takes the form of a difference between obtained and expected rewards. Indeed, the signals are so similar that the prominent actor/critic algorithm actually uses the same error signal to update both state values and policies.

Here, we studied a task in which this is not the case, allowing us to distinguish a teaching signal for direct policy preferences from signals for learning value predictions. In particular, foregone reward takes the place of the expected reward in a policy teaching signal, but not in an action value teaching signal. We found evidence that net outcome-related BOLD activity in the striatum is appropriate to learning policies, but no similar evidence for signals appropriate for updating separate action values. That said, the latter (negative) conclusion relies on a stronger test for PE signaling than is often used in the literature. In many studies, and ours as well (Fig. 2), striatal BOLD activity correlated with a PE signal for Q values. We decomposed the activity to show that in our task this correlation was likely due only to outcome- and not prediction-related activity, supporting the policy model. Although most previous authors did not report this particular analysis, many showed other evidence that the striatal BOLD response was modulated by predictions as well as outcomes (e.g. by including outcome as an effect of no interest or contrasts between expected and unexpected outcomes (Berns et al., 2001; McClure et al., 2003; Li et al., 2006; Hare et al., 2008; Li et al., 2011)). Interestingly, Behrens and his colleagues (Behrens et al., 2008; see also Behrens et al., 2007) separately tested both effects in the same manner we do, and also found no effect of predictions. This may be because in their task, like ours, foregone rewards were known, so that expectancies would also disappear from the policy update signal.

A number of recent studies suggest that error-related BOLD activity in striatum may reflect, at least in part, the dopaminergic input from midbrain (Pessiglione et al., 2006; Knutson and Gibbs, 2007; Schonberg et al., 2010). Of course, since the fMRI BOLD signal is a generic metabolic signal not specific to a single underlying neural cause, unit recordings will be required to determine whether our results generalize to the prediction error responses of midbrain dopamine neurons.

Our results do, however, provide positive evidence in the human brain for a teaching signal specifically appropriate for updating action policies. Unlike error signals previously reported in other tasks, which did not probe the distinction since value and policy errors coincided (Morris et al., 2006; Daw et al., 2006; Hampton et al., 2006; Schonberg et al., 2007; Schonberg et al., 2010), this signal cannot alternatively be interpreted as a prediction error for values. Perhaps the best previous evidence for a policy-specific update signal was an influential report of spatially distinct correlates in striatum of a prediction error during a free choice compared to an instructed choice condition (O'Doherty et al., 2004, also see Tricomi et al., 2004). Although in that task also, modeled value and policy teaching signals were substantially the same, activity in the dorsal striatum was specific for the free-choice condition, and interpreted as a policy teaching signal on that basis. Interestingly, standard actor-critic models do not obviously predict that policy teaching signals will be specific to free-choice conditions; instead, they assert that both actor (policy learning) and critic (value learning) modules just use a common error signal (Barto, 1995). Here, we skirt this interpretational difficulty by investigating whether the information carried by the signal itself is appropriate for training policies or values.

The comparison with the O'Doherty et al. study also points to another unresolved puzzle in both datasets (and in studies of striatal error signaling in humans more generally), which is that lesion work in rodents identifies stimulus-response policy learning specifically within the dorsolateral striatum (Yin et al., 2004), which corresponds to the dorsal putamen in humans. Most PE-related activity in humans, including, surprisingly, the policy errors in the present study, is instead focused more ventrally in ventral putamen and caudate (though see Schonberg et al., 2010). Meanwhile, in the O'Doherty et al. (2004) study, the putative policy error in the free-choice condition was localized in the dorsal caudate (dorsomedial striatum), which is also not consistent with stimulus-response learning in rats (Yin et al., 2005). Thus, further work remains to detect counterparts in human striatal error signaling of the striatal suborganization suggested by rodent work (Tricomi et al., 2009; Schonberg et al., 2010).

Alongside our positive evidence for a policy update signal, our accompanying failure to detect evidence for value update signals in the present task should not be interpreted as contradicting the overwhelming evidence that neural signals, including those in the nigrostriatal system, reflect action value expectations in many other circumstances (Barraclough et al., 2004; Daw et al., 2005; Samejima et al., 2005; O'Reilly and Frank, 2006; Rangel et al., 2008; Hare et al., 2008; Seo and Lee, 2008; Boorman et al., 2009). First, we cannot rule out that our results are idiosyncratic to the present task. For instance, subjects’ brains might have adopted the direct actor strategy here because it’s particularly straightforward and efficient due to the symmetric information structure of our task design. Anyway, a true neural value representation would be required even under the wildest possible extrapolation from the current results – that all previously reported error signals, too, can be understood as policy rather than value update signals – since in those tasks even the error signal for policies itself contains value predictions.

The necessity of value predictions for producing an error signal (for policies) does not imply that those value predictions are themselves also learned from that error signal, although the latter is also assumed in standard models like the actor/critic. For instance, even if value predictions are not trained by a nigrostriatal error signal, they may instead arise from prediction processes involving other brain systems. A longstanding idea in psychology and cognitive neuroscience is that the brain relies on multiple learning and memory systems, which can each learn different representations (Knowlton et al., 1996; Poldrack et al., 2001; Dickinson and Balleine, 2002; Daw et al., 2005). In addition to values directly learned from prediction errors, value predictions are also believed to be constructed from still more fundamental information – e.g., from “action-outcome” associations, cognitive maps, or “model-based” RL, often associated with prefrontal or declarative memory systems (Poldrack et al., 2001; Dickinson and Balleine, 2002; Daw et al., 2005). These predictions may, in turn, inform nigrostriatal signals for policy update (Doll et al., 2009; Simon and Daw, 2011; Daw et al., 2011).

Such “model-based” evaluation processes likely played a more prominent role in a pair of recent studies of counterfactual effects on choice (Camille et al., 2004; Coricelli et al., 2005), because in those studies subjects had to evaluate new options on each trial based on a pictorial description, rather than learn a choice policy by trial-and-error as in the present task. The requirement for such explicit evaluation may help to explain why, in those studies, the level of “regret” related to foregone options was specifically associated with distinct neural correlates in orbital prefrontal cortex, whereas both actual and foregone rewards engaged largely identical brain networks in the present study.

In another related study, Lohrenz et al. reported correlations in striatal BOLD with “fictive error signals” in a decision task in which subjects learned how much to bet on a simulated trading market (Lohrenz et al., 2007). This “fictive error” signal (the difference between the obtained reward and the largest possible reward) can be viewed as another instance of the policy gradient signal reported here, and indeed the authors noted this potential interpretation. However, because that task involved only a single dimension of action (how much to bet) with a single reward (the movement of the market), rather than two actions with two independent rewards, it would have been difficult to tease apart the constituents of the signal as we have done in our task, so as for instance to distinguish a counterfactual policy update signal (how much to move the bet policy toward the all-in bet) from other correlated quantities such as a counterfactual value update signal (e.g., updating a stored value prediction for the all-in bet).

Our findings thus provide specific new evidence about the computational form and function of error-driven learning in the human striatum, adding to an accumulating body of evidence about these processes. The suggestion that these signals directly reinforce choice policies, rather than training value predictions that serve as intermediate quantities in evaluating candidate choices, may have particular relevance for dysfunctions involving compulsive action, such as drug abuse, gambling, and impulse control disorders (Dickinson and Balleine, 2002; Everitt and Robbins, 2005).