The ubiquity of model-based reinforcement learning

Abstract

The reward prediction error theory of dopamine function has enjoyed great success in the neuroscience of learning and decision-making. This theory is derived from model-free reinforcement learning in which choices are made simply on the basis of previously realized rewards. Recently, attention has turned to correlates of more flexible, albeit computationally complex, model-based methods in the brain. These methods are distinguished from model-free learning by their evaluation of candidate actions using expected future outcomes according to a world model. Puzzlingly, signatures from these computations seem to be pervasive in the very same regions previously thought to support model-free learning. Here, we review recent behavioral and neural evidence about these two systems, in attempt to reconcile their enigmatic cohabitation in the brain.

Introduction

The reward prediction error (RPE) theory of dopamine (DA) [1, 2] has been a remarkably influential account of neural mechanisms for learning from reward and punishment. This marriage of computer science and neuroscience posits that dopaminergic bursts and dips transmit plasticity-modulating teaching signals to cortico-striatal circuits, training the selection of rewarded actions and avoidance of punished ones.

The computational end of this theory is known as ‘model-free’ reinforcement learning (RL), and derives its name from the fact that the learner need not attempt to understand the sequential transition structure of the task (a ‘world model’) to maximize reward. Instead of utilizing this structure to assess future outcomes, these models learn which actions are beneficial through direct experience with their rewarding consequences. This learning scheme has clear benefits in terms of computational efficiency, but there are costs as well. The capacity to learn about the world’s structure, lacking in these approaches, is most beneficial (and empirically most apparent) when flexible changes in behavior are required, e.g., when goals or aspects of the environment change, necessitating rapid reevaluation.

Psychologically, model-free algorithms reinvent Thorndike’s [3] early law of effect, the reinforcement principle according to which rewarded actions tend to be repeated. In many ways, the refutation of this demonstrably incomplete behaviorist principle sowed the seeds of the cognitive revolution [4]. Even rats can do more than repeat successful actions: they can, for instance, learn the layout of a maze (a prototypical example of a world model) and plan novel routes in it.

However, it is easier to demonstrate that the brain is smart than it is to understand how it manages this sophistication. Fortunately, just as model-free RL provides a theory specific enough to be refuted, the engineering literature offers a second class of algorithms that have recently been identified as promising candidates for formalizing a more flexible alternative [5, 6, 7]. Such ‘model-based’ RL approaches learn the sequential contingencies of events and actions in a task (which outcomes follow which actions, e.g., where different paths in a maze lead), which can be used adaptively and dynamically to compute ideal actions by simulating their consequences. It is this sequentially structured world model, and its use in forward-looking computations, that distinguish model-based from model-free RL.

Here, we review recent efforts that leverage model-based methods to uncover the neurobiological underpinnings of more flexible decision making, and to dissociate them from their putative model-free counterparts.

From goals and habits...

The model-based vs. model-free dichotomy was proposed to capture a long-standing distinction in psychology between two classes of instrumental behavior known as goal-directed and habitual [5, 8]. This distinction is operationalized with tasks that use revaluation probes, such as training an animal to lever-press for food when hungry, then testing performance when full. Revaluation interrogates whether choice of an action (lever-pressing) is affected by consideration of its outcome (the food, now worthless) or merely determined by previous reinforcement (which occurred when hungry). If behavior instantly adjusts to reflect the new value of the outcome that the action would obtain, this demonstrates that the choice is ‘goal-directed’, i.e., derived from a representation of the action’s specific consequences, or in RL terms, model-based. Insensitivity to revaluation instead indicates ‘habits’, choices made without regard for any representation of outcome identity but instead selected based on previously realized value, as with model-free RL.

In experiments, each account prevails under different circumstances: animals can either respect or ignore the new outcome value. Together with lesions dissociating these functions (below) and the strong, pre-existing theory associating DA and model-free RL, these revaluation results suggested that the brain contained separate, competing systems for model-based and model-free RL [5]. This view also offers a theoretical explanation for when each approach should dominate. Given estimates from both systems, one favorable strategy is to select the least uncertain among them [5]. A recent theory framed arbitration more explicitly in terms of the costs (time) and benefits (better reward harvesting) of performing model-based evaluation [9]. The newer formulation has stronger decision-theoretic foundations and better process-level plausibility, but the accounts are similar in effect because the reward increment expected for model-based evaluation depends on uncertainty. Thus both frameworks correctly predict many circumstances in which putatively model-free behavior dominates, including after overtraining [10] and in conditions when action-reward contiguity is low [11] or reward variability is high [12], as well as one direct test of uncertainty’s role in arbitration [13]. The cost/benefit analysis (specifically, the time cost of model-based evaluation) may also explain why chronic stress in rats [14] and acute stress in humans [15, 16] shift both species toward habitual behavior.

The revaluation assay can dissociate not only behavior but also its neural underpinnings in rats [8]. Recently, this approach has been extended to humans, suggesting conservation of brain and behavior across species [17]. Lesions in the rodent indicate that integrity of the dorsolateral striatum is key for the formation of habitual behavior [18]. The role of (at least this portion of) striatum resonates with the model-free RPE theory of its phasic DA input, which knockout and optogenetic studies demonstrate is necessary for some measures of behavioral conditioning [19, 20].

The more puzzling counterpart to these results, however, is that goal-directed (putatively model-based) behavior requires adjacent dorsomedial striatum [21]. It is difficult to reconcile the structural similarity of these parts of striatum with the substantial differences in their theorized computations. Thus, these results paint a confusing picture, whereby a dopamine-rich cortico-striatal loop, analogous to that commonly thought to implement model-free RL, appears to be involved additionally in model-based RL. More particularly, standard model-based RL has no use for an RPE of the sort usually associated with striatum’s dopamine input, and instead uses quite different teaching signals [22]. One route to a solution might be potential differences in the properties of the areas’ dopamine inputs. Indeed, different striatal regions are irrigated by dissociable groups of midbrain DA neurons [23]. However, relatively few electrophysiological recordings have suggested systematic variation in signaling properties across them [24, 25], and the interpretation of such variation can be further complicated by challenges in identifying which cells are dopaminergic [26].

A similarly mixed picture arises from studies investigating dopamine’s causal effects in revaluation tasks. Indeed, habits can be induced using DA-agonizing drugs of abuse in place of natural rewards like food [27]; while for natural rewards, deafferention of DA cells targeting dorsolateral striatum [28], and deletion of NMDA receptors in DA neurons in knockout mice [29] prevent transition from goal-directed to habitual behavior. Conversely, pharmacological blockade of DA has no effect on measures of goal-directed instrumental learning [30] at doses that affect (presumably model-free) Pavlovian conditioning. On the other hand, recent results using a human task inspired by traditional associative learning models of the revaluation paradigm seem to suggest a counterintuitive role for DA in goal-directed rather than habitual behavior [31, 32]. One possibility is that these results reflect dopamine’s involvement in (and goal-directed behavior’s reliance on) prefrontal cognitive functions besides reinforcement, such as working memory. However, unlike the standard devaluation paradigm, the central manipulation of this task — involving different combinations of the same stimuli as both cues and outcomes for responses — is not clearly interpretable in terms of model-based and model-free computations, so these results may not easily generalize.

...to model-based and model-free RL

Indeed, model-based vs. model-free, as the key dimension distinguishing learning strategies, extends beyond its proposed operationalization in goal-directed and habitual behaviors. The computational distinction applies also, for instance, in Pavlovian conditioning, spatial navigation, and other cognitive problems, especially those from the engineering literature where it arose. Accordingly, numerous recent studies, mostly using fMRI, have aimed explicitly to dissociate model-based from model-free reinforcement learning using learning tasks inspired by the computational RL literature. These can roughly be grouped into two classes. One is sequential decision tasks — mazes or more abstract multistep sequences — in which model-based methods can learn the sequential transition structure and leverage it to evaluate actions (Figure 1)[22, 33, 34, 35, 36, 12, 37, 38, 39].The second involve explicit or implicit counterfactual structure, where information about rewards not actually received can be inferred or observed [40, 41, 42, 13, 43, 44]. A typical example is a serial reversal contingency, where a drop in the value of one option implies an increase in the other’s value. Purely reinforcement-based model-free RL would be blind to such structure. Note, however, that while such tasks go beyond model-free RL, they don’t as directly exercise the key affirmative features of model-based RL as we have defined it, i.e. the computation of values using a sequential transition model of an action’s consequences.

Figure 1.

Sequential task dissociating model-based from model-free learning. (a) A two-step decision making task [33], in which each of two two options (A1, A2) at a start state leads preferentially to one of two subsequent states (A1 to B, A2 to C), where choices (B1 vs. B2 or C1 vs C2) are rewarded stochastically with money. (b, c) Model-free and model-based RL can be distinguished by the pattern of staying vs switching of a top level choice following bottom level winnings. A model-free learner like TD(1) (b), tends to repeat a rewarded action without regard to whether the reward occurred after a common transition (blue, like A1 to B) or a rare one (red). A model-based learner (c) evaluates top-level actions using a model of their likely consequences, so that reward following a rare transition (e.g., A1 to C) actually increases the value of the unchosen option (A2) and thus predicts switching. Human subjects in [33] exhibited a mixture of both effects.

From both sorts of studies, the overall sense is that model-based influences appear ubiquitous more or less wherever the brain processes reward information. The most expected of these influences are widespread reports about model-based value signals in ventromedial prefrontal cortex (vmPFC) and adjacent orbitofrontal cortex (OFC), which have previously been identified with goal-directed behavior using devaluation tasks [45, 46]. vmPFC has been proposed to be the human homologue of rat prelimbic cortex, which is required for goal-directed behavior [8]. OFC is also implicated in model-based Pavlovian valuation in rats and goal values in monkeys [47, 48], though understanding this area across species and methods is plagued by multiple factors [49]. More unexpectedly, several reports now indicate that RPE correlates in the ventral striatum — long thought to be a human counterpart to the DA response and thus a core component of the putative model-free system — also show model-based influences [33, 34, 44]. Even DA neurons, the same cells that launched the model-free theories due to their RPE properties [1, 2], communicate information not available to a standard model-free learner [41].

The harder part of this hunt, then, seems to be for neural correlates of exclusively model-free signals, which are surprisingly sparse given the prominence of the model-free DA accounts. The most promising candidate may be a region of posterior putamen that has been implicated in extensively trained behavior in a habit study [17] and a sequential decision task [37], and may correspond to the dorsolateral striatal area associated with habits in rodents [18]. The foundation of both fMRI results, however, was overtraining (a classic promoter of habits), rather than whether these areas reflect values learned or updated by model-free methods. Indeed, value correlates in a nearby region of putamen have been reported to follow model-based rather than model-free updating using the computational definition [34].

A different, promising path for isolating model-based RL is neural correlates related to the model itself. Representations of anticipated future states or out-comes — rather than just their consequences for reward — are what defines model-based RL. Hippocampal recordings in the rat have shown evidence of forward model ‘lookahead sweeps’ to candidate future locations at maze choice points [35]. These data fit well with the spatial map-encoding properties of hippocampus [50], and may permit striatum to signal value for simulated rather than actually experienced outcomes [36]. Hippocampus is similarly implicated in a study that examines learning predictive world models outside the context of a decision task [51]. A number of cortical areas have also been observed with correlates related to model learning [22] and evaluation [34]. Work in learning tasks with task-relevant hidden structure may also speak to the construction of world models [52]. These studies have implicated the lateral PFC, an area often associated with working memory function, in discovery of this structure.

The way forward

Why the ubiquity of evidence for model-based RL? Several factors probably contribute, all of which point to important opportunities for progress.

First, the resolution and other limitations of the BOLD signal may conceal distinctions that would be visible using more invasive techniques. The explosion of human studies explicitly examining model-based RL is recent enough that analogous animal electrophysiological studies are largely not yet available. A strength of fMRI is to develop tasks and analyses, and to locate areas for further study; the time is now ripe to test similar tasks in animals. For instance, further dissociations might be found in the signaling properties of different DA cell groups and projections [23], potentially elucidating the potential co-expression of model-based and model-free signals by DA cells.

Second, the brain’s RPE systems may be smarter than they have been made out to be, yet still essentially model-free. Of the different characteristics that have been taken as hallmarks of model-based RL, some are easier than others to accommodate in a lightly modified model-free system. In particular, a model-free learner can generalize learning from one state to another, without additional experience, if its inputs corresponding to those states overlap (Figure 2). This is a particularly plausible explanation for seemingly model-based inference in serial reversal and similar tasks [40, 41, 44], as indeed the authors of some of these studies have pointed out. If counterfactual updating in these tasks occurs implicitly, due to generalization, then it wouldn’t involve forward modeling of future states. In this respect, tasks involving sequential contingencies are stronger and more canonical tests of model-based RL, but variants of the same representational trick can in principle apply even there [53]. For instance, if actions are represented in terms of their associated outcomes – for instance if the representation for a lever that produces food overlaps with that for the food itself – and if these inputs (themselves now, in effect, a sort of world model) are mapped to values using even model-free RL, then the learned value will be substantially shared between the lever and the food. In this case, if the food is devalued, the leverpress value will also decline immediately, and the resulting behavior will appear goal directed. This approach might help to explain the involvement of similar striatal circuits in both goal-directed and habitual behavior. More elaborate versions of this scheme can apply to arbitrary sequential tasks, but such a strategy is easier to spot, and potentially to rule out, in tasks with deeper sequential structure and changing transition contingencies [34].

Figure 2.

Learning through value generalization (left) and model-based forward planning (right). In a reversal learning task (left), the rat has just taken an action (lever-press) and received no reward, and so updates its internal choice value representation to decrement the chosen value’s option. Because the unchosen value’s option is represented on the same scale, inverted, it is implicitly incremented as well. Implemented this way, learning relies on model-free updating over a modified input, and does not involve explicitly constructing or evaluating a forward model of action consequences. In a model-based RL approach to a maze task (right), the rat has an internal representation of the sequential structure of the maze, and uses it to evaluate a candidate route to the reward.

Third, there may be hitherto unanticipated crosstalk or integration between model-based and model-free systems. It is reasonable to imagine that model-based capacities evolved on top of an earlier model-free system, rather than separately and in parallel. One related proposal that has received some recent support [54, 55] is that explicit representations (putatively PFC and hippocampus dependent) can directly bias an underlying model-free learner, permitting higher order contingency information to be learned by the model-free architecture [56]. A model-based system in the brain might similarly leverage a model-free learner, as with some model-based algorithms that incorporate model-free quantities in order to reduce computational overhead [57, 58, 59]. Different modes of behavior may simply reflect different aspects of a more complex, integrated learning system. For example, there is evidence for one type of model-based learning that embeds a Pavlovian system, giving rise to some aspects of both sorts of behavior simultaneously [60].

The explosion of model-free RL approaches in psychology and neuroscience has led to tremendous progress over the past 15 years. Model-based approaches hold a similar promise, but with their complexity comes the need for close attention to the specific computations and predictions these models make, and a re-evaluation of their relationship with established model-free approaches.

Highlights.

• *Model-free RL is a successful theory of corticostriatal DA function.
• *Flexible model-based RL methods offer to enrich understanding of brain and behavior.
• *Data suggest extensive overlap between putative neural correlates of these RL systems.