A mosaic of cost-benefit control over cortico-striatal circuitry

Abstract

Dopamine contributes to cognitive control through well-established effects in both the striatum and cortex. Although earlier work suggests that dopamine impacts cognitive control capacity, more recent work suggests that striatal dopamine may also impact cognitive motivation. We consider the emerging perspective that striatal dopamine boosts control by making people more sensitive to the benefits versus the costs of cognitive effort and discuss how this sensitivity might impact competition between controlled and prepotent actions. We propose that dopamine signaling in distinct cortico-striatal subregions mediates different kinds of cost-benefit tradeoffs and also discuss mechanisms for the local control of dopamine release, enabling selectivity among cortico-striatal circuits. In so doing, we show how this cost-benefit mosaic can reconcile seemingly conflicting findings about the impact of dopamine signaling on cognitive control.

Cognitive control, motivation, and dopamine

Dopamine has long been implicated in cognitive control and working memory, with clear relevance for Parkinson’s disease, ADHD, schizophrenia, and depression [1–3]. Dopaminergic drugs can remediate control deficits, but their mechanisms of action remain unclear. Their cognitive enhancing effects are commonly ascribed to prefrontal cortex modulation, where the slower time constants of dopamine clearance and degradation are well-suited to support temporally-protracted working memory maintenance [4–7]. Yet, dopamine also plays a complementary role in the striatum where fast dynamics shape working memory gating and task shifting [7–13].

While prior work has implicated cortical and striatal dopamine in cognitive control capacity, another, previously under-appreciated possibility, is that striatal dopamine regulates the willingness to exert control [3,14–18]. This perspective fits with a large body of work demonstrating striatal dopamine’s motivational effects on learning and performance of motor tasks [19–23] and growing evidence that, like physical effort, cognitive control is costly and requires motivation [16–18,24–33].

Here, we consider the hypothesis that striatal dopamine boosts willingness to exert cognitive control by increasing sensitivity to the benefits versus the costs of effort across a range of domains from motor to cognitive actions. We start with a brief synopsis of the historical perspective implicating prefrontal and striatal dopamine in cognitive control. Next, we highlight studies demonstrating the impact of dopaminergic drugs on cost-benefit decision-making about control. In the final section, we consider the hypothesis that dopamine signaling is not homogenous across the striatum but is instead characterized by rich spatiotemporal dynamics impacting different kinds of decision variables across striatal sub-regions. This more heterogeneous account has the potential to explain a host of findings not accommodated by monolithic theories of striatal dopamine signaling, while still retaining a core cost-benefit interpretation in any given sub-region.

A brief history of dopamine and cognitive control

Cognitive control, generally, refers to the ability to override prepotent responses with flexible, rule-guided behavior. It requires working memory to update and maintain rules and information about context [34]. Adaptive behavior arises from hierarchical mechanisms by which higher-level task rules maintained in rostral prefrontal cortex bias lower-level, task-relevant processing pathways in more caudal sensorimotor regions. Experimental data [4,6], and computational models [5] implicating prefrontal dopamine in the stability of working memory representations of rules also, thereby, implicate cortical dopamine in cognitive control of responses. For example, early connectionist models centered on aberrant cortical dopamine function to explain dysfunctional rule representations in schizophrenia, e.g. [35].

Concurrently, striatal dopamine was implicated by the recognition that fronto-striatal interactions are well suited for governing what should be gated into (and out of) working memory, and in which contexts, to maximize rewards and minimize punishments [8,36]. To drive adaptive behavior, the prefrontal cortex maintains task-relevant information to bias sensorimotor processing and acts in concert with the striatum to gate working memory contents based on expected reward [7,12,36–38]. This account of striatal dopamine in working memory gating builds on its analogous role in adaptive motor behavior and helps explain a variety of findings linking striatal dopamine to working memory operations and cognitive flexibility [8–10,13,39–43].

Cognitive control is motivated

According to early models, cognitive control is recruited automatically in response to conflict or errors [44,45]. These models predict that control is engaged in proportion to task demands, to maximize expected reward. When response conflict is perceived, for example, a control rule will be gated and that rule will exert top-down influence over lower-level pathways to process task relevant over task-irrelevant information. In contrast, an emerging perspective posits that cognitive control is inherently costly, and must be motivated [16–18,24–33,46]. Thus, rather than being reflexive, control results from a kind of cost-benefit decision-making, balancing factors like control efficacy and reward magnitude against cognitive effort costs [28,47].

Why cognitive control is costly remains unresolved. Unlike physical effort, control does not incur clear metabolic costs [48,49] - though, perhaps, the brain is sensitive to the accumulation of metabolic by-products of control processes [50]. An alternative class of explanations argues that control is normatively costly because it biases responses which are unpracticed and uncommon relative to well-trained habits with more predictable returns; in a sense, control is costly because it is more risky [51]. Yet even when we know we can do tasks properly, using control for one set of tasks would prevent us from doing other tasks. In other words, control incurs opportunity costs [49]. Accordingly, control should be curtailed when opportunity costs rise [52,53]. Consistent with opportunity cost models, control is diminished when average reward rates are high (opportunity costs increase with the payoff per unit action), or when cognitive capacity is low (each unit of allocation accounts for a larger fraction of available resources) [54,55]. Conversely, withholding resources is beneficial because it affords greater flexibility for task-switching as alternative opportunities arise [53]. This tradeoff could explain why people avoid task engagement more in environments prioritizing flexibility over stability (in a recent preprint: [56]).

Although the basis of cognitive effort costs remains unresolved, there is nevertheless consensus in the existence of a cost function governing control allocation. Indeed, people largely choose lower over higher task demands, all else being equal [25,57,58] (though individual differences matter [32]), and discount goals as a function of rising demands [26,27,30,31,33,59]. Rewards are discounted, for example, when received in the context of higher demands for working memory [60], task switching [61], or response conflict [62,63].

A cost function implies that incentives could promote control by offsetting costs. There is ample evidence that incentives enhance various forms of control [16,28] from proactive response preparation [64,65], to task switching [66], to distractor resistance [19,42]. Notably, incentive effects vary as a function of genes and drugs that affect striatal dopamine transmission [19,37,43,63,67,68].

Dopamine impacts cost-benefit decisions about cognitive effort

The performance-enhancing effects of incentives and dopamine drugs in cognitive control tasks have traditionally been understood to reflect heightened capacity. For example, more striatal dopamine facilitates flexible gating [43,66]. An alternative explanation is that incentives and concomitant dopamine release increase motivation to expend cognitive effort [17,18]. To test whether motivation is altered requires asking whether dopamine impacts explicit decisions about cognitive effort. Explicit economic choices provide direct evidence about motivation, while avoiding the pitfalls inherent in inferring cognitive effort from task performance – which is jointly determined by capacity, motivation, and controllability [69].

To test the hypothesis that striatal dopamine promotes motivation, we combined dopamine synthesis capacity imaging (18[F]fluoro-dopa PET) with methylphenidate, in a study of explicit economic choices [70]. First, we asked participants to choose between high-load working memory tasks for more money, or low-load tasks for less money. We found that people with lower dopamine synthesis capacity in the caudate nucleus were less willing to accept high-cost, high-benefit offers (Fig. 1b). We also found that methylphenidate increased willingness to expend effort, but more so for those with low dopamine synthesis capacity – implicating striatal dopamine (in the caudate nucleus, Fig. 1a,b) in cognitive motivation.

Figure 1. Striatal dopamine and cognitive effort selection in two human pharmaco-imaging studies.

a) Two effort-based decision-making tasks ([70]: red frame, [96]: blue frame) and a mask image of striatal sub-regions including the caudate nucleus (red), the putamen (green) and the ventral striatum (blue). On placebo, higher dopamine synthesis capacity, b) in the caudate nucleus, predicts greater selection of high-effort, high-benefit versus low-effort low-benefit offers to perform working memory tasks for money and, c) in the ventral striatum, predicts greater selection of free time for less money versus performance of a visual working memory task for more money. Methylphenidate increases selection of cognitive effort, selectively among b) those with low dopamine synthesis capacity in the caudate nucleus and c) those with high dopamine synthesis capacity in the ventral striatum implying that dopamine signaling in different striatal sub-regions will have distinct effects on different types of decisions about cognitive effort. d) Dopamine synthesis capacity predicts higher effort selection by increasing sensitivity to the relative benefits, and decreasing sensitivity to the relative costs of a hard task for more money versus an easy task for less money, as qualitatively predicted by a computational model of striatal dopamine [71].

Next, we tested the more specific hypothesis that dopamine alters cost-benefit tradeoffs [71]. To test this hypothesis, we examined eye gaze patterns (to track attention to cost or benefit information), and choice behavior, as participants decided between offers tailored to their individual reward-effort tradeoffs. Consistent with our hypothesis, we found that dopamine (correlationally via PET and causally via pharmacology) both increased sensitivity to benefits and decreased sensitivity to the costs of cognitive effort. Furthermore, we found that empirical choice patterns could be predicted from simulations of a computational model [71] of striatal dopamine’s effects in cost-benefit action selection (Fig. 1d). Note that our inference about dopamine promoting motivation is based on the simplistic assumption that higher dopamine synthesis capacity and dopamine transporter blockade amplify post-synaptic dopamine signaling. Future studies with more temporally-resolved methods (e.g. [72]) could elucidate the precise dopamine dynamics regulating cognitive motivation.

Our results help explain the findings from other recent human pharmacology studies examining explicit choices. In one study, methylphenidate increased selection of high versus low task-switching demands, among some participants [58]. Critically, the drug effect depended on trait impulsivity, elsewhere linked with striatal dopamine function. In another study, Parkinson’s disease patients were more likely than controls to choose lower attentional demands for lower rewards over higher demands for higher rewards [73]. Yet, patients who took their Parkinson’s medications matched controls’ preferences for high-cost, high-benefit options. Our results suggest that such effects may reflect increased striatal dopamine signaling which made patients weight benefits (e.g. of bonus points) more strongly relative to costs (e.g. of control demands).

Our study indicates that dopamine is sufficient to mediate cognitive motivation, which was hitherto unclear given the possibility that other neurotransmitters might do so. Noradrenaline, for example, has been implicated in multiple aspects of physical effort including effort selection and force production [74,75] and provides an alternative explanation of medication effects in the previous Parkinson’s disease work. Acetylcholine has also been proposed to explain why psychostimulants alter cognitive effort choice in rats [76], yet neither noradrenaline nor dopamine-selective agents impact choice in the same task [77]. On the other hand, it is possible that dopamine drugs did not alter rodents’ preferences because offers were not sufficiently close to indifference to detect a systematic effect across animals. In our study, dopamine’s impact on sensitivity to costs and benefits was most apparent on trials when offers were closest to participants’ individual indifference points.

Our finding implicating striatal dopamine does not rule out an effect of cortical dopamine on increasing sensitivity to benefits versus costs – though it is less clear how cortical dopamine could mediate such an effect. By contrast, our results accord with an extensive literature implicating striatal dopamine in promoting effort [15,20,23,78,79], and the weighting of benefits versus costs [71,80,81]. Namely, our results are consistent with a canonical model whereby instrumental action learning and selection are mediated by two opposing sets of neurons that coordinate striatal output, alternately expressing dopamine receptor subtypes that make the cells more (D1) or less (D2) sensitive to cortical inputs [8,82,83]. Given that unexpected rewards (positive prediction errors) are signaled by phasic increases, and unexpected punishments (negative prediction errors) by phasic decreases in dopamine release [84], the cortico-striatal synapses of D1 and D2 neurons will, in the course of learning, come to reflect reward and punishment statistics, respectively. Thus, dopamine signals train cortico-striatal synapses onto D1 and D2 neurons to reflect the expected benefits and costs of actions, respectively [71,81,85–87].

There is some question over whether dopamine cell firing and dopamine release encode effort costs – strictly defined in terms of metabolic demands of physical exertion [88], see also [89]. Nevertheless, many factors associated with cognitive and physical effort demands (risk, delays, etc.) dampen phasic responses to reward cues [21,81,90]. Phasic dopamine release in the rat ventral striatum, for example, encodes the net subjective value of reward magnitude, discounted by either delay to reward, or the number of lever presses required, as revealed by fast-scan cyclic voltammetry [81]. Critically, optogenetic stimulation of dopamine neurons can influence the degree to which animals later choose to work for reward, consistent with a causal role for dopamine in training cortico-striatal synapses to reflect physical effort costs and benefits [81].

There is also indirect evidence that dopamine mediates learning about cognitive effort costs. For example, human fMRI studies reveal that while learning about cognitive task demands, brains compute the kinds of reward prediction errors conveyed by phasic dopamine signals in other domains [32,91]. Also, people discount rewards received in the context of cognitive demands [61,63], and this effect is modulated by both the D2 receptor agonist cabergoline, and individual differences in a gene (DARPP-32) linked with the striatal D1 versus D2 pathway balance [63]. These data are consistent with our hypothesis that striatal dopamine signaling shapes cortico-striatal synapses to reflect the relative benefits and costs of cognitive effort.

Effort costs, opportunity costs, and dopamine

Our decision-making study described above indicates that striatal dopamine can reduce sensitivity to cognitive effort costs. Yet, another hypothesis proposes that striatal dopamine encodes an average rate of rewards that signals an opportunity cost of time [92–94]. Critically, because control is slow relative to fast and efficient habits, greater opportunity costs (corresponding with greater environmental richness) may decrease, rather than increase preference for control [52]. Indeed, reward rate manipulations increasing opportunity costs can make people faster and less accurate in cognitive tasks, consistent with a down-regulation of cognitive control [55].

Could dopamine signaling mediate an opportunity cost effect, biasing fast prepotent action over slow control? Rather than biasing selection among competing actions, dopamine signaling opportunity costs was originally proposed to increase behavioral vigor [92]. This focus on behavioral vigor is emphasized in recent accounts proposing that striatal dopamine release primarily determines action latency (e.g. [95]). Nevertheless, as noted above, there is considerable evidence that striatal dopamine can also determine which action to choose (i.e., reward-maximizing or punishment-minimizing) and not just how fast. Thus, dopamine may plausibly bias cost-benefit selection among competing opportunities.

To test the hypothesis that dopamine release can cause people to avoid (rather than engage) demanding tasks in response to high opportunity costs, we conducted a study asking participants to choose greater reward for more time on task or less reward for unconstrained free time, and again used methylphenidate to manipulate dopamine and PET to measure dopamine synthesis capacity [96]. Mirroring our first study, we found that methylphenidate increased high-effort selection and that drug status interacted with individual differences in dopamine synthesis capacity. This pattern supports the general conclusion that striatal dopamine can increase willingness to work for reward, even with respect to unconstrained opportunities.

There were, however, key differences. First, in contrast to our first study, high rather than low dopamine synthesis capacity individuals avoided effort the most, and were most sensitive to the effects of methylphenidate on choice (Fig. 1c). Second, these individual difference effects depended primarily on ventral rather than dorsal striatal dopamine. It is possible that these differences reflect distinct computations mediated by dopamine signaling in distinct striatal subregions [97–101]. Namely, we speculate that in our second study, individuals were differentiated chiefly by ventral striatal dopamine reflecting opportunity costs, because they were tasked with deciding between work and unconstrained opportunities comprising the expected value of the state [71,102] at large. Conversely, in the first study, individuals were differentiated chiefly by sensitivity to effort cost-benefit tradeoffs, and dorsal striatal dopamine because they were tasked with weighing fully explicit costs and benefits of opponent actions (cf. [98]).

Our inference that the functional impact of dopamine signaling depends on striatal subregion accords generally with conclusions from recent rodent imaging work showing that striatal subregions exhibit spatiotemporal variations in dopamine release as a function of task demands [99]. In particular, dopamine release is preferentially directed toward dorsomedial regions during instrumental but not Pavlovian tasks, allowing the animal to adapt behavior to changing tasks demands [99]. Similarly, in human imaging studies, striatal reward prediction errors are amplified in those subregions most related to a given task structure [103]. Collectively, such results motivate a mosaic model, which we will turn to in the next section, by which dopamine signaling impacts the weighting of benefits versus costs of different dimensions in distinct striatal subregions.

Our results converge with foraging studies [93,94] towards the inference that (ventral) striatal dopamine can shift the balance in favor of disengagement in the context of high opportunity costs (when the average, expected benefits of alternative opportunities is higher). Yet, this inference is tempered by the results of another human pharmacology study implying limits on the kinds of tradeoffs which dopamine can alter, with respect to the opportunity cost of time [23]. Namely, participants do not squeeze a handgrip harder to save opportunity costs (time-on-task) on levodopa versus placebo, even though levodopa causes them to squeeze harder for larger rewards. These results are consistent with the broader hypothesis that dopamine can sensitize people to benefits versus costs of actions, even if they do not necessarily influence the weighting of opportunity costs [23]. It is possible that a functional segregation across striatal subregions, which we turn to next, also implies limits on the degree to which dopamine can mediate the tradeoff between different commodities in distinct subregions (see Box 1 for a discussion).

Box 1. Dopamine’s Effects on Choice versus Performance.

Dopamine appears to sensitize people to opportunity costs in our task [96] and in foraging studies [93,94], but not impact the tradeoff between handgrip force and opportunity costs in another pharmacology study [23]. Could differing results reflect a distinction between explicit choices (e.g. between high cost/benefit and low cost/benefit options) and performance (e.g. force used to squeeze a handgrip) such that the effect of dopaminergic medication on one does not predict an effect on the other? Evidence against this explanation comes from a study in which patients with Parkinson’s disease, off medication, both chose high effort options less often and also exerted less force while squeezing [79]. Critically, computational modeling revealed that both effects of dopamine depletion on choices and performance could be explained by a common cause – namely, a reduction in sensitivity to reward benefits [79]. Similarly, in our own study [70], dopamine drugs tended to increase saccade velocities more for those participants who also showed a greater increase in high-effort choice. Thus, explicit choices and performance appear to provide overlapping information about dopamine-dependent motivation.

Differences in the nature of the costs being weighed, such as time versus effort costs, could be another reason why levodopa did not cause people to squeeze harder to save opportunity costs. Our overarching hypothesis is that dopamine biases different kinds of tradeoffs in distinct striatal subregions. Spatial heterogeneity provides specificity, but may also impose limits on the kinds of tradeoffs that dopamine can mediate across different subregions. Perhaps, for example, the opportunity costs of time are mediated by ventral striatal dopamine [21,96], while choices between greater or lesser handgrip force are mediated by dorsal striatal regions, which play a key role in mapping effort costs to motor kinematics [120]. If the two commodities are represented in distinct circuits, a global pharmacological boost may both make people more sensitive to the benefits of alternative opportunities one region, and less sensitive to effort costs in another, without impacting how the two trade off. Accordingly, dopamine might not cause people to squeeze harder to save time, but it could cause them to squeeze faster.

A mosaic of striatal dopamine cost/benefit calculations across subregions

Collectively, effort choice studies imply that striatal dopamine can promote cognitive effort by making people more sensitive to effort benefits and less sensitive to effort costs (Fig. 2a). On the other hand, they also suggest that dopamine may signal opportunity costs and thus shift the balance in the opposite direction, away from effortful cognitive control when the average value of alternative opportunities is higher.

Figure 2. Dopamine shapes both the learning and performance of action policies in cortico-striatal-thalamic circuits.

a) Specifically, dopamine binding at D1 versus D2 receptors respectively increases and decreases striatal neuron excitability to cortical drive. Phasic reward prediction error signaling thus shapes synapses at D1- versus D2-receptor expressing neurons to reflect the benefits versus the costs of actions, respectively. In turn, high dopamine release can also instantaneously amplify benefit versus cost information encoded in cortico-striatal synaptic weights at the time of choice. Adapted from [119]. b) Midbrain dopamine projections shape the learning and performance of cognitive and motor actions in a conserved, hierarchically-structured, cortico-striatal architecture for action selection [14,37,38]. Dopamine projections to the prefrontal cortex modulate the stability of recurrent working memory representations. Abstract, higher-level goal representations in more rostral cortex bias increasingly concrete, lower-level representations, including specific cognitive and motor actions, in more caudal cortex. At the highest level of abstraction, this could include selection of effortful cognitive tasks themselves. Working memory contents are also determined by gating policies in the striatum, where dopamine both trains policies and alters their expression, by modulating sensitivity to afferent cortical projections. A reciprocal, cortico-striatal circuit is replicated across the dorsal frontal cortex and dorsal striatum, thus constituting a spiraling, dopamine-modulated pathway for hierarchical action selection. Abbreviations: DS – dorsal striatum, VS – ventral striatum, VTA – ventral tegmental area, SNc – substantia nigra pars compacta, 1PFC – lateral prefrontal cortex, ACC – anterior cingulate cortex, DA – dopamine.

Striatal dopamine has been shown to undermine control in a number of other contexts as well. For example, it can promote working memory flexibility which is detrimental when tasks demand stability to resist distractors [104]. Robust dopamine-mediated incentive signaling can also undermine performance among people with higher baseline dopamine function in control-demanding tasks, even when the same incentives are beneficial for healthy adults with lower baseline dopamine function [105] or for Parkinson’s patients [106]. Large, dopamine-mediated incentive signals may also amplify response vigor, undermining fine motor control [107].

A mechanistic account of such effects is that striatal dopamine sensitizes D1- versus D2- receptor expressing neurons diffusely, lowering the threshold for gating multiple candidate actions, thereby potentiating both costly control, but also the cheap habits that control was intended to override [71,108]. Moreover, the degree to which dopamine potentiates habits versus control may itself depend on the strength and availability of the habit being proposed relative to control rules retrieved from long term memory [108]. This prediction stems from a core aspect of competition between control which is slow relative to fast, available habits.

How, then, could dopamine selectively promote control signals over habits, or prioritize alternative opportunities in rich environments? A complementary proposal, which we consider here, is that the consequences of dopamine signaling will depend on where and when dopamine is released in the striatum. An architecture of spiraling cortico-striatal loops implies heterogeneity in information processed across different subregions [14,37,38,109] (Fig. 2b). Thus, dopamine signaling in distinct subregions is proposed to mediate different kinds of cost-benefit tradeoffs [109]: e.g. impacting sensitivity to opportunity time costs in the ventral striatum, or willingness to expend cognitive effort in the dorsomedial striatum [101]. Also, circuits are hierarchically nested in the sense that task-sets gated in more medial and anterior prefrontal cortex can contextualize lower-level action selection in lateral and posterior cortex [36,38]. For example, ventral striatum-anterior cingulate cortex circuits may select relevant cost-benefit variables which can then constrain the selection of task representations in dorsal striatum-lateral prefrontal cortex circuits [110]. Accordingly, cortico-striatal interactions could affect cost-benefit selection, based on whether it is worth it, to 1) engage in a Stroop task relative to other potential opportunities, and then 2) gate rules which direct attention to the Stroop color dimension over the text dimension – rules which are more beneficial once the Stroop task is engaged.

Crucially, there are mechanisms for local control of dopamine release, enabling selectivity among cortico-striatal circuits [22,83,99–101]. A key mechanism involves transient suppression of tonically active cholinergic interneurons, distributed throughout the striatum, thereby gating release from local dopamine terminals [83,100,101], and both the learning and performance of specific, motivated behaviors [111,112]. These cholinergic interneurons are, in turn, partly driven by projections from prefrontal cortex, suggesting access to rich, high-level information to shape their activity selectively with respect to the animal’s belief about their state [113].

The functional consequences of local control were demonstrated in a recent study in which axonal activity and dopamine release were imaged across the rat dorsal striatum [99]. Specifically, imaging revealed “waves” of release which, by virtue of their spatiotemporal mapping, selectively promote different behavioral controllers (e.g. Pavlovian versus instrumental control) in different striatal subregions [99]. Data were consistent with a model whereby regions in which dopamine was signaled sooner after reward cues became more strongly associated with reward, and the control signals they processed were more likely to govern subsequent behavior. Their results also had direct relevance for how animals learn to exert physical effort: waves originating in dorsomedial striatum predicted greater subsequent effort (when effort is instrumental toward achieving rewards), while waves originating in dorsolateral striatum predicted less (when effort is not related to rewards, in a Pavlovian context).

Extending this notion, we propose that spatiotemporally precise dopamine signaling to distinct striatal regions supports the learning and performance of complex control-demanding tasks. For example, dopamine release may preferentially target dorsal striatal regions interconnected with anterior prefrontal cortex when gating decisions are made at the level of a task [114] that needs to be sustained for the duration of its execution [38]. Conversely, dopamine release may be targeted to regions interconnected with lateral and posterior prefrontal cortex when effort cost / benefit-based gating needs to conducted to support local task operations [38], such as working memory updating and manipulation, when flexibility is required [43,104]. Concurrently, dopamine release in the ventral striatum tracking the value of the current state [21,96,109] could promote decisions about whether and how vigorously to persist with effort in the current task [20,115] or to switch to alternative actions when their average value is high. Future work is needed to elucidate the extent to which the spatiotemporal dynamics of dopamine release have the resolution to support complex cognitive control demands (e.g., model-based goal pursuit; see Box 2).

Box 2. Striatal Dopamine Signaling and Model-Based versus Model-Free Deliberation.

Model-based control relies heavily on both flexible and stable working memory updating and maintenance for protracted intervals, and typically compete with cheap and fast model-free associations. Yet, there are a number of reasons to think that dopamine dynamics have the requisite complexity and precision to support model-based deliberation. First, phasic dopamine signals appear to train model-based associations and could thus shape internal models of the environment and state transitions [121]. Second, because model-based deliberation is effort-costly [122], dopamine release may offset costs, thereby promoting model-based versus model-free choice. Consistent with this hypothesis, both higher striatal dopamine synthesis capacity [123] and administration of the dopamine precursor l-dopa [124] shift the balance to model-based over model-free choice – though this may reflect a reduced weighting on model-free action values [124]. Third, sustained dopamine ramping, e.g. [21,22,99] operates at timescales that could promote protracted deliberation and could, in principle, be driven by hierarchically inferred progress [125] in model-based deliberation. Critically, dopamine release reflects an animal’s beliefs about their present state [126], and dopamine ramps likewise depend on an animal’s beliefs about progress inferred from an internal model of their position with respect to goal states [127]. Ramps may reflect an accumulation [128] of dopamine-mediated “pseudoreward” prediction errors [129] which could, in principle, sustain protracted deliberation. Fourth, although very high dopamine release (e.g. at the pinnacle of a ramp) could plausibly produce detrimental flexibility, goal progress itself increases the likelihood of on- versus off-task gating, given that goal state associations will be increasingly salient as goals near [18]. Moreover, striatal dopamine can facilitate re-engagement with sequences of instrumental motor behavior [130], in a way that suggests that dopamine would also promote re-engagement with sequences of deliberative cognition following distraction. Future work should test these predictions about whether and how dopamine could promote and sustain protracted, model-based deliberation.

Notably, this cost-benefit mosaic account can reconcile seemingly conflicting findings regarding dopamine’s effects in motor vigor. While higher dopamine release has traditionally been associated with greater vigor [92], recent work suggests that dopamine dynamics afford considerable selectivity for precise and bi-directional control. During learning, dopamine signals can train either slower or faster responding, whichever better predicts reward [116]. Moreover, studies manipulating reward contingencies suggest that dopamine can up-regulate vigor when reward depends on it, and suppress vigor when it does not [117,118]. In a saccade task, for example, Parkinson’s patients off dopamine medications showed faster saccades for larger incentives, whether incentives were performance-contingent or delivered randomly [118]. Yet on dopamine medications, patients, like healthy controls, only increased speed for rewards contingent on fast saccades, and were slower to respond when rewards were not contingent on speed. Thus, dopamine can both promote vigor that is instrumental, and suppress vigor that is not goal-directed. These action-selective effects are generally consistent with the model framework we advance here, whereby dopamine modulates sensitivity to benefits over costs of competing actions, respecting goal hierarchies, rather than producing indiscriminate vigor. Such selectivity likely reflects rich spatiotemporal dynamics of dopamine release in striatal sub-regions governing varying levels of behavioral control.

Concluding remarks

The work reviewed here supports that cognitive control is costly and is regulated by cost-benefit decision making. It implicates dopamine in promoting control by increasing sensitivity to benefits versus costs of cognitive actions, consistent with well-established effects on physical effort, and a conserved cortico-striatal architecture governing both cognitive and motor action selection. It also suggests that dopamine signaling could have different consequences in different striatal sub-regions, impacting the weighting of costs and benefits across a hierarchy of behavioral control. Future work should investigate the spatiotemporal dynamics of dopamine release and their functional implications for alternately enhancing control, or the actions that control might otherwise override (see Outstanding Questions).

Outstanding Questions Box.

1. Why is cognitive control effort-costly?

2. How and what does the brain monitor to compute opportunity costs? Are opportunity costs signaled by striatal dopamine release?

3. What are the functional implications of dopamine release for cognitive control in specific subregions including the ventral, dorsomedial, and dorsolateral striatum?

4. Does dopamine signaling mediate action policy learning for cognitive control? For example, do phasic dopamine prediction error signals encode cognitive effort costs, and what variables are tracked?

5. How do prefrontal-basal ganglia circuits orchestrate midbrain dopamine dynamics to sustain sequences of operations in protracted working memory tasks? Might dopamine dynamics in particular circuits be prioritized as a function of key meta-parameters computed in cortex, such as environmental controllability and state value?

6. Under what conditions do dopamine drugs promote control versus impulsivity and distractibility?

Highlights.

1. Striatal dopamine can promote cognitive control by increasing sensitivity to the benefits and decreasing sensitivity to the costs of cognitive effort.

2. The opportunity costs of time may also be signaled by striatal dopamine, biasing disengagement from control-demanding tasks in rich environments.

3. A hierarchical, cortico-striatal architecture for action selection implies spatial heterogeneity in the kinds of cost-benefit tradeoffs mediated by dopamine signaling in distinct striatal sub-regions.