What is dopamine doing in model-based reinforcement learning?

Abstract

Experiments have implicated dopamine in model-based reinforcement learning (RL). These findings are unexpected as dopamine is thought to encode a reward prediction error (RPE), which is the key teaching signal in model-free RL. Here we examine two possible accounts for dopamine’s involvement in model-based RL: the first that dopamine neurons carry a prediction error used to update a type of predictive state representation called a successor representation, the second that two well established aspects of dopaminergic activity, RPEs and surprise signals, can together explain dopamine’s involvement in model-based RL.

Introduction

The reward prediction error theory of dopamine proposes that dopamine neurons signal the difference between how good the future was expected to be, and how good it turned out to be, taking into account both immediate and anticipated long-run rewards^1,2. As a normative theory that explains a diverse range of experimental data, the influence of RPE theory is hard to overstate. However, an increasing body of findings suggest that RPE is only be a partial account of what dopamine is doing. Signals carried by dopamine neurons appear more heterogeneous than predicted by the theory ^3–6, including responses shaped by movement^7–11, and to aversive or threatening stimuli^12,3,13. A second set of unexpected findings concerns dopamine’s involvement in model-based reinforcement learning^14–23 (RL).

Model-based RL uses an internal model of the world that predicts future states given chosen actions, and evaluates the long run value of different options by simulating their likely consequences². This contrasts with model-free reinforcement learning, which stores estimates of the long run value of states and actions, and updates these directly from experience using RPEs. These distinct algorithms have different strengths. Model-based RL uses information efficiently, but is slow and computationally expensive, as it must simulate many possible futures. Model-free RL is quick and computationally cheap, at the cost of less flexible decision making when the environment changes. Brains are thought to use both methods in parallel to exploit the strengths of each^24–26.

As RPE is the key teaching signal in model-free reinforcement learning, RPE dopamine theory was initially conceived with respect to model-free RL. However, several lines of evidence implicate dopamine in model-based RL. First, dopaminergic RPEs do not simply reflect model-free cached values of directly observable events, but rather are informed by inferences based on rich models of the environment^{27,14,15,20,22,28}. This is still compatible with an RPE account of dopamine function, but expands the sources of value information that inform RPEs. More troubling for RPE theory is that dopamine neurons respond to events that are surprising but not directly rewarding or aversive, i.e. sensory or state prediction errors (SPEs)^{29–32,23,33}. This is fundamentally at odds with RPE theory, because RPEs only occur when outcomes are better or worse than expected. Arguably most strikingly, both causal manipulations^16,18,21,34 and natural variation^17,19 in dopamine function affect model-based learning and decision making. For a detailed review of these findings see Langdon et al ³⁵.

Thus, while there is compelling evidence implicating dopamine in model-based RL, its precise role remains unclear. Here we examine an intriguing recent proposal³⁶ that dopamine encodes a type of SPE used to learn a predictive state representation called the successor representation (SR). We suggest that some aspects of dopamine anatomy and physiology are hard to reconcile with this account, and propose instead that dopamine’s involvement in model-based RL can be explained through two well established aspects of dopamine activity; RPEs and surprise signals.

Dopamine as successor representation prediction error

The basic idea of the successor representation is to learn the long-range predictive relationships between states – i.e. which states generally follow after which other states^37–39. Specifically, the SR for state X is a vector whose elements indicate the expected discounted future occupancy for each state after starting in state X. This separates the problem of predicting long run value into two components: the expected future states given current state – summarized by the SR – and the immediate rewards available in each state. Long run values can be computed by multiplying the SR with the immediate reward available in each state. This allows value estimates to be updated quickly and simply if the immediate reward available in each state changes (for example due to a change in motivational state or current goal), subject to some important limitations due in part to the dependence of the SR on the specific policy followed while it was being learnt³⁸. The SR has attracted substantial recent interest in neuroscience, both as a mechanism for goal-directed behaviour^38,40, and to account for response properties of neurons⁴¹.

As the number of possible discrete states of the world is infinite, practical application of the SR requires working with state features^36,42. These could be low-level sensory features like colours or high-level abstract features like ‘at work’. A feature-based SR is a matrix, mapping the set of features describing the current state of the world onto those features that are likely to be observed in the future. This can be learned using temporal difference methods, based on updating estimates when observations differ from expectations, very similar to those used in model-free RL to learn values. The key difference is that rather than updating a scalar value estimate using a scaler RPE, the SR for a given state feature is a vector indicating how strongly it predicts the future occurrence of each feature (Figure 1A,B). As the prediction is a vector rather than a scalar, the prediction error is also a vector.

Figure 1. Neuron numbers, signal dimension, and algorithm.

A) Number of cortico-striatal, striatal and midbrain dopamine neurons in the rat brain, estimates from ^43,44,48. The disparity in neuron numbers strongly suggests that the signal carried by dopamine is much lower dimensional than the cortical input to striatum. B) Diagram illustrating temporal difference (TD) value learning using linear function approximation. A vector s_t of state features active at time t, is multiplied by a weight vector θ_t to give a vector v_t whose elements are the contributions made by each state feature to the scalar value v_t of the state. The scalar reward prediction error δ_t, used to update the weights, is computed as δ_t = r_t + γv_t − v_t−1, where r_t is the immediate reward at time t, γ is a discount rate and v_t−1 the value at the previous time step. Irrespective of the dimension of the state representation, the prediction error is a scalar, hence if s_t is represented by cortical neurons, θ_t by their synapses in striatum, and δ_t by dopamine, the algorithm is consistent with much smaller signal dimension for dopamine relative to cortical input. C) Temporal difference learning of a feature based successor representation (SR). The state vector s_t is multiplied by a weight matrix Θ_t to give a matrix M_t whose elements $m_t^{i,j}$ are the contributions made by feature i of the current state to predictions about the future occurrence of feature j. Summing the contribution of all features of the current state gives the vector m_t, the SR for the current state. As the SR is a prediction of future state features, rather than rewards as in TD value learning, the feature vector s_t takes the role played by reward r_t in value learning. As the prediction is a vector of dimension given by the number of state features, so is the prediction error δ_t. This algorithm does not appear consistent with the massive difference in signal dimension between cortical and dopaminergic input to striatum. D) The one dimensional prediction error signal in standard TD value learning is inconsistent with the observed heterogeneity of dopaminergic responses. One possible explanation is that parallel cortico-basal-ganglia loops (labelled 1 & 2) independently learn value estimates, each using only a subset of state features. For clarity we have shown the extreme case of no crosstalk between loops. E) Recent data suggest that rather than predicting scalar reward, the basal ganglia predict multiple axes of reinforcement (reward and threat) in loops involving different striatal regions (nucleus accumbens and tail of striatum). Here we have shown two loops which use partially overlapping sets of state features to predict different components of a multi-dimensional reinforcement vector r_t.

The SR account of dopamine activity proposes that dopamine encodes the vector valued SPE used to update a feature-based SR³⁶. This accounts for several observations not straightforwardly predicted by RPE theory. Firstly, it predicts heterogeneity of dopamine neuron responses, as different dopamine neurons encode prediction errors for different state features. Secondly, it accounts for dopamine neuron responses to surprising events that are neither appetitive or aversive, as these can still cause SPEs. Thirdly, it predicts that dopamine activity is necessary for learning stimulus-stimulus relationships. Finally, it is argued to explain the ability of dopamine neuron stimulation to drive learning about stimulus-stimulus relationships in situations where it would not normally occur^21,36. We will return to this last claim later.

Despite these strengths, there are some complications with the SR account of dopamine. The first is dimensionality. The prediction error used to update the SR has the same the dimensionality as the SR itself, i.e. the number of state features. Computational accounts of basal ganglia typically assume that the cortical input represents state, while prediction errors are carried by dopamine neurons. The rat brain contains approximately 17 million cortico-striatal projection neurons⁴³, but only around 70 thousand midbrain dopamine neurons⁴⁴. Given this vast disparity in neuron numbers, it seems inconceivable that the dimensionality of the dopamine signal is the same as that of the cortical state representation. Furthermore, each dopamine neuron densely innervates a substantial volume of striatum⁴⁵, and dopamine neurons communicate predominantly via volume transmission^46,47, further reducing the dimensionality of the dopamine signal received by post synaptic neurons. This massive dimensionality mismatch is not a problem for RPE theory, as the RPE is scalar. It is for an SR account. Could the dimensionality mismatch be avoided if only a subset of the cortico-striatal input participates in the SR computation? This does not appear to align with the anatomy, as both the cortical and dopaminergic innervation are distributed across the whole striatum. Nor are dimensionality considerations invalidated by emerging evidence for a degree of heterogeneity across dopamine neurons^3–6, or that they carry some information about surprising reward flavours³³ (albeit with weak selectivity) – as the argument is not that the dopamine signal is one dimensional, only that it is much lower dimensional than the cortical input representing state. Accurately estimating the dimensionality of activity is challenging, both because it requires large recorded populations, and because the dimensionality of task related activity is constrained by that of the behavioural task used^49,50. However, while we lack precise quantitative estimates of the dimensionality of cortical and dopaminergic input to striatum, the neuron number and other considerations outlined above suggest the disparity is large. This constrains the algorithm that the circuit may be implementing.

A second issue is that an SR account does not straightforwardly predict the diverse array of experimental data consistent with RPE theory, including experiments where a large fraction of dopamine neurons qualitatively and quantitatively behaved like an RPE^1,51–54 or that dopamine neuron stimulation can be strongly reinforcing^55–59. Gardner et al. suggest a resolution to this is to treat reward as just another salient stimulus dimension, such that a subset of dopamine neurons encode a reward prediction error as part of a broader SPE. But the point of the SR is to separate the problem of long run value prediction into one component that learns long run predictions about future states (the SR) and a distinct component that learns about the immediate reward available in those states. This necessitates that reward enters into SR based algorithms in a fundamentally different way from state features. Therefore, while RPE-consistent data can be shoehorned into an SR account this way, RPE signals are not predicted by SR theory on normative grounds.

A final issue concerns the proposal that the SR account explains elegant data demonstrating that optogenetic stimulation of dopamine neurons can unblock learning of stimulus-stimulus relationships²¹. In these experiments, rats received presentations of an audio-visual compound stimulus ‘AC’ that predicted the subsequent presentation of a different auditory stimulus ‘X’ (denoted as AC→X). If the rats had previously learnt that visual stimulus A alone predicted X (A→X), this blocked learning of the C→X association during subsequent presentation of the compound AC→X. However, if during presentations of AC→X, dopamine neurons were optogenetically activated at the onset of X, learning about C→X occurred. Under an SR account, this could be caused by dopamine stimulation acting as an additional SPE driving learning³⁶. The problem with this is the specificity of learning. In order for dopamine stimulation at the time of X presentation to selectively promote learning C→X, the dopamine neurons recruited must be specifically those that represent errors in the prediction of X. One might argue that these neurons had sub-threshold excitation from the presentation of X itself, and stimulation turned this into super-threshold spiking. However, this seems unlikely as the robust stimulation parameters employed in the experiment would be expected to recruit a large population of dopamine neurons irrespective of any other excitatory input.

Therefore, despite the elegance of the SR account, we think these issues reduce its plausibility, leading us to consider other potential ways that dopamine may interact with model-based RL.

Well informed RPE + surprise = model-based dopamine?

In addition to reward prediction errors, it is well established that dopamine neurons often respond to surprising, novel or salient sensory stimuli^{29,31,30,32,23,13,60,61}. We suggest that many observations linking dopamine to model-based RL can be accounted for by the combination of these two aspects of dopamine activity. Specifically, we suggest that dopamine responses to new information comprise an RPE, which takes into account the best rapidly available value estimates (which may extend beyond model-free values, as we describe below), and a scalar-valued surprise signal. The former can be used to update values stored at cortico-striatal synapses, while the latter is permissive for learning state-state predictions but is not in itself a vector-valued SPE indicating how state predictions should be updated. We will examine each of these ideas in turn.

Dopaminergic RPEs occur at short latency when new information becomes available, so the value information used to compute them must be rapidly available. There are several mechanisms through which internal models could inform value information within the available timeframe: state inference, offline planning, predictive representations, and minimal rollouts (Figure 2).

Figure 2. How do internal predictive models contribute to RPE signals?

Diagrams showing different ways in which predictive internal models could contribute value information to dopaminergic RPEs. A) Model-based planning uses roll-outs along different possible future trajectories (black) from the current state (red) to calculate the long run value associated with different options. The short latency of dopamine responses likely precludes repeated roll-outs using new information from occurring in time to inform the RPE, though offline planning during rest or sleep may update cached values that inform future RPEs. B) The successor representation caches a diffuse prediction of likely future states given the current state, which averages over previously experienced behavioural trajectories. This allows for rapid computation of new long run values when the immediate rewards associated with states change, but see ³⁸ for some limitations. C) A minimal rollout of only the most probably future trajectory could potentially provide useful value information rapidly in near deterministic settings. D) In addition to predicting the future, internal models may help to disambiguate between possible current states that can only be disambiguated by considering the extended history. If the different states have different cached values associated with them, such state inference will affect RPEs.

As current sensory input only partially constrains the current state of the world, brains must infer the world’s state by combing recent sensory data with internal models learned over a lifetime. Elegant studies have shown that dopaminergic RPEs reflect inferences about the current state of the world^14,22,28,62, for example inferring reward probability for one stimulus based on sampling another whose reward probability is anticorrelated¹⁴. Inferences about the world’s current state are ‘model-based’ in the colloquial sense that they depend on internal models. However they are conceptually distinct from ‘model-based RL’ as defined by Sutton and Barto², which refers more narrowly to algorithms which use predictions about future states to compute values⁶³. As predictions of future states do appear to shape dopamine responses^15,20, we must consider how model-based RL could inform values within the available timeframe.

One possibility is that model-based value computations which contribute to dopaminergic RPEs, do not occur at the time of the RPE, but rather during prior ‘offline’ planning. This uses simulated experience from a model to refine cached value estimates², to improve both future decisions and RPEs that use the cached values. Hippocampal sharp wave ripples, which occur during inactivity and sleep, are thought to implement offline planning^64,65. During ripples, cell assemblies in hippocampus are activated in sequences representing possible behavioural trajectories^64,66–68. VTA activity is modulated around ripples, and reward responsive VTA neurons are preferentially recruited when the hippocampal sequence represents reward locations⁶⁹. This ripple associated dopamine activity may update values stored at striatal synapses. A theoretical model which proposes that offline planning is focused on those states that will most improve future choices, explains diverse observations about the content of hippocampal ripples⁶⁵. Behavioural and brain imaging data also suggest that offline planning affects future choices in humans^70,71. Closed loop electrical stimulation of medial-forebrain bundle (which recruits dopamine neurons) locked to the spiking of a particular place cell during sleep, causes rats to visit the location represented by the place cell on awakening⁷², demonstrating artificially induced offline value updating.

The other possibility is that online prediction of the states that will follow an event, allows value associated with them to shape the dopamine response to it. While there is behavioural and neural data consistent with online planning during decision making^73,74, it is unclear whether online planning could inform the dopaminergic RPE to new information. The challenge is that accurate computation of model-based values generally requires repeated roll-outs (i.e. simulated trajectories) along multiple branches of the decision tree⁷⁵, which does not appear plausible in time to shape dopamine responses. The successor representation is one possible solution to this problem. It avoids the need for repeated roll-outs by caching a single diffuse prediction of the likely future which averages over trajectories experienced in the past. This allows for rapid value computation by multiplying the SR with the expected immediate rewards in each state. Values estimates derived from an SR could therefore plausibly contribute to dopaminergic RPEs in the available timeframe.

More speculatively, it is worth considering the possibility that a minimal roll-out of the most likely future trajectory could contribute to the dopamine response. Though there is little evidence to suggest sensory events trigger sharp wave ripples at short latency, the hippocampal theta oscillation can be reset by stimuli, particularly when task relevant^76–78. During each theta cycle, hippocampal neurons activate sequentially, mapping out a trajectory from the current position to that predicted a short time in the future. At decision points, theta sequences explore different possible future paths, suggesting a role in online planning^73,79. Each theta cycle lasts 120-250ms, while the latency from onset to peak RPE encoding by dopamine is around 150-400ms⁸⁰, so theta phase resetting by stimuli could in principle allow the hippocampus to roll-out at least one possible future trajectory informed by the new information. Values derived from a single sample trajectory would be inaccurate when the future was stochastic, but useful in more deterministic settings, particularly if they incorporated an accuracy estimate derived from the predicted trajectories’ probability.

Surprise, model-learning, and cortical hierarchy

While there are plausible mechanisms through which model-based value can inform dopaminergic RPEs, this does not explain the observation of dopamine responses to surprising but neutral events^{29,31,32,23,61}, the necessity of dopamine activity for learning stimulus-stimulus predictions, or that stimulating dopamine neurons can unblock such learning²¹. Consistent with previous accounts^81,82,80, we suggest that dopamine neurons carry a surprise or novelty signal in addition to the RPE, which indicates that something unexpected has happened, independent of valence. We propose that this surprise signal upregulates new learning about predictive relationships, and is responsible for dopamine’s causal role in stimulus-stimulus learning.

The anatomy and physiology of the dopamine system suggest a particular arrangement of this surprise signal with respect to cortical hierarchy. Unlike other neuromodulatory systems, dopamine innervation of cortex is most dense to medial frontal and medial temporal lobe regions positioned high in the hierarchy, and comparatively sparse in sensory regions. However, dopamine responses to sensory surprise or novelty can occur at very short latencies (40 – 100ms), comparable to those in V1 and earlier than information about stimulus identity is available in higher cortical regions⁸². This suggests that dopamine neurons carry a surprise signal evaluated low in the sensory hierarchy, close to ‘ground truth’ sensory data, but transmit it to regions high in the hierarchy working with highly abstracted representations.

Surprise could promote model updating by increasing learning rates and down weighting prior knowledge relative to new information. Consistent with this, dopamine responses to novel conditioned stimuli (CS) promote learning in Pavlovian conditioning, while stimulating dopamine projections to prefrontal cortex during presentation of a familiar CS accelerates learning^61,83. Additionally, dopamine projections to amygdala mediate surprise induced attention to preceding cues, which promotes subsequent learning about them^84,85. Upregulation of model learning by dopamine can potentially explain the effect of dopamine manipulations on stimulus-stimulus learning^21,34. If model updating in frontal regions is upregulated when a prediction failure is signalled by dopamine, inhibiting dopamine neurons will impair stimulus-stimulus learning. Conversely, where prior learning of A→X blocks subsequent learning of C→X during presentation of the compound AC→X, artificially creating a surprise signal at the time of X presentation may down-weight previously learnt predictions, unblocking new learning of C→X. Unlike the SR account of these data, this does not require stimulation to recruit dopamine neurons specific for predictions about X; rather, the surprise signal simply indicates that something unexpected has happened, promoting new learning.

We note that there is substantial overlap between our proposals concerning dopaminergic surprise signals upregulating learning, and previous accounts acetylcholine and noradrenaline function⁸⁶. Exploration of differences between these neuromodulators is beyond the scope of this review. Our aim is rather to point out that surprise signalling in dopamine neurons is well established, that upregulating learning is a normative response to surprise, and hence may account for recent data implicating dopamine in stimulus-stimulus learning.

Mixed signals and heterogeneity

Several questions are raised by our account. Firstly, why mix surprise and RPE signals? One possibility is that though these signals are conceptually very different, the normative responses to each have substantial overlap. Functionally coupling surprise and RPE may drive exploration of undiscovered aspects of the environment, exposing new opportunities^87,88. Empirically, surprise or novelty can reinforce behaviour, as when rodents press levers to turn on lights⁸⁹, a behaviour that is dopamine dependent⁹⁰. Conversely, while any prediction failure suggests that the model should be updated, failure to predict reward has particular behavioural relevance. Coupling RPE and surprise signals may focus the model’s representational capacity on those bits of the state-space that are important for behaviour. A second and not mutually exclusive possibility is that there is sufficient separation of these signals in time and/or space to allow downstream regions to demultiplex the two components and differentially respond to them⁹¹.

A second challenge for our account is the increasing evidence for heterogeneity of the dopamine signal, as both RPE and surprise signals are scalar. Computing RPE using a vector valued state representation requires summing the contribution to value estimates from each state feature (Figure 1B). Standard RPE theory assumes that this summation happens prior to the dopamine neurons, so they all carry the same signal. However, given the strong topographical organisation of projections in cortico-basal ganglia loops^92–94, summation prior to dopamine is likely only partial, such that each dopamine neuron combines value information from a subset of state features, generating heterogeneous responses⁹⁵ (Figure 1D). As individual dopamine neurons innervate large regions of striatum, a degree of summation will occur downstream of release via volume transmission, though bulk dopamine measurements across different striatal regions indicate that substantial heterogeneity remains at this level^4,13,96.

Though some heterogeneity in a conceptually scalar RPE signal may be accounted for by partial summation of value over state features, it is hard to reconcile the radically different response profile of dopamine neurons projecting to the tail of the striatum, which respond to high intensity sensory stimuli but not unexpected reward, and are aversive when stimulated¹³. Menegas et al. propose that these neurons represent a threat prediction error, forming a separate axis of reinforcement from the value prediction errors carried by dopamine projections to ventral striatum (Figure 1E). Segregating learning about rewards and threats makes sense due to the very different behavioural responses they require. It remains an open question whether other aspects of dopamine heterogeneity, such as movement responses, can be accounted for by partial summation of value information, or indicate the presence of additional axes of reinforcement.

Conclusions

We have proposed that dopamine interacts with model-based RL through three mechanisms. Firstly, dopamine neuron activity during offline planning refines cached values stored at striatal synapses, affecting both future behaviour and RPEs. Secondly, dopaminergic RPEs during behaviour incorporate model-based value information where this is rapidly available; from earlier offline planning, predictive representations such as the SR, and possibly from minimal rollout following stimulus onset. Thirdly, surprise signals carried by dopamine neurons upregulate new learning in predictive models instantiated in cortex and hippocampus. We therefore suggest that, while data implicating dopamine in model-based RL expands the scope of its computational role, and points to a tight integration of model-based and model-free RL mechanisms, it remains compatible with RPE theory and does not require complete re-evaluation of dopamine’s role in learning and action.