Dopamine and Proximity in Motivation and Cognitive Control

Abstract

Cognitive control – the ability to override a salient or prepotent action to execute a more deliberate one – is required for flexible, goal-directed behavior, and yet it is subjectively costly: decision-makers avoid allocating control resources, even when doing so affords more valuable outcomes. Dopamine likely offsets effort costs just as it does for physical effort. And yet, dopamine can also promote impulsive action, undermining control. We propose a novel hypothesis that reconciles opposing effects of dopamine on cognitive control: during action selection, striatal dopamine biases benefits relative to costs, but does so preferentially for “proximal” motor and cognitive actions. Considering the nature of instrumental affordances and their dynamics during action selection facilitates a parsimonious interpretation and conserved corticostriatal mechanisms across physical and cognitive domains.

Introduction

Cognitive control is essential for flexible, context-sensitive planning and decision-making. Recent studies have shown that striatal dopamine (DA) signaling can alternatively promote cognitive control, boosting accuracy and speeding reaction times, and undermine it, yielding impulsivity. Here, we review evidence for these opposing effects and propose a novel hypothesis to explain why DA sometimes promotes and sometimes undermines cognitive control, in terms of cortico-striatal action selection mechanisms and biases.

DA offsets effort costs, promoting control

Although cognitive control is necessary for flexible, adaptive functioning, it is also subjectively costly [1–3], causing demand avoidance [4] and reward discounting [5,6]. Control is thought to be recruited in proportion to potential benefits less effort costs [7]. The nature of subjective effort costs is yet unresolved: they may reflect mechanisms to reduce cross-talk interference among multiplexed control signals, or opportunity costs incurred by resource allocation [2]. Nevertheless, the consequences are real: higher subjective costs erode control under fatigue [8], and in advanced cognitive aging [5]. Deficient motivation may also partly account for cognitive deficits in schizophrenia [9–12] and disorders including depression and ADHD [1].

Incentives, conversely, promote cognitive control [3], and these effects are likely mediated, in part, by dopamine (DA) signaling in the striatum [13]. Phasic DA signals train cortico-striatal synapses to gate cognitive actions, such as working memory updating and task-set selection, according to their relative reward and punishment histories, by modulating synaptic plasticity in direct and indirect pathways, respectively [14,15]. Extracellular DA can also convey momentary motivation, biasing high-benefit, high-cost actions over low-benefit, low-cost actions during action selection [15–21]. Momentary, DA-mediated motivational signaling explains both why incentives boost apparent control for speed, accuracy, and distractor resistance in a saccade task, and also why these incentive effects are attenuated in Parkinson’s disease [22].

Importantly, striatal DA dynamics during goal-directed behavior suggest that they are well-suited to convey an evolving willingness to work over extended intervals that cognitive control requires [21,23]. Key features include protracted ramps during goal approach which adapt to unanticipated state transitions, encode temporally-discounted rewards, and predict action likelihood [21,24]. Computational theory has highlighted the influence of costs in arbitrating between cheap and efficient “model-free” (MF) action selection, and precise but costly “model-based” (MB) planning over complex state-action-outcome transitions [25–27]. Evolving striatal DA dynamics may thus be important for conveying the expected values of costly MB processes. Indeed, decision-makers rely more on MB over MF control when the stakes are higher [28] and with increased striatal DA signaling [29–31].

DA also undermines control

If DA promotes control by conveying incentives that offset effort costs, there is also evidence that it undermines control. Notably, the DA precursor levodopa yields impulse control disorders in 17% of Parkinson’s disease patients [32] and may also drive impulsive responding to irrelevant stimuli as a function of patients’ trait impulsivity [33]. Trait impulsivity itself has been linked with higher adolescent DA function [34], higher striatal D2 receptor density in healthy adults [35], and D2 autoreceptor density and amphetamine-induced DA release [36]. Experimentally, DA can both promote and undermine control within a single task: during a Stroop task, trial-wise incentives enhance performance (reduce conflict costs) for those with low striatal DA synthesis capacity, while incentives undermine performance (increase conflict costs) for those with high synthesis capacity [37]. Beyond altering control performance, DA can also increase the degree to which individuals explicitly choose to avoid high versus low control-demanding tasks. Specifically, the DA transporter blocker methylphenidate caused high trait-impulsive participants to avoid control demands more [38], suggesting that DA may undermine control by altering when individuals choose to exert it.

DA may also undermine control, in part, due to DA’s well-established effects on behavioral vigor [39–43]. In short, higher extracellular DA tone in the striatum increases the likelihood, and reduces the latency of action commission [21,24,40,41]. Thus, prepotent actions that control is intended to override (e.g., reading a Stroop word) are also potentiated by higher DA, just like controlled actions. That is, DA can potentiate actions that require incentive motivation for overcoming effort costs, but also actions which do not require motivation. Indeed, DA-mediated incentives can simultaneously potentiate both performance-contingent and non-contingent behaviors like speeding saccades both when rewards depend on reaction times and when they do not [44].

DA interacts with proximity to modulate control

What determines when DA will promote control and when it will undermine it? One suggestion comes from an elegant series of studies which implicate both DA and spatial proximity in conditioned approach to instrumental apparatus [45]. Subpopulations of striatal neurons respond to discriminative stimuli and their activity determines whether rats approach and engage instrumental apparatus. Critically, this activity is DA-dependent [40,46] and is modulated by spatial proximity: more proximal apparatus evoke more firing, greater likelihood of approach, and shorter latency reaction times [47]. As a consequence, rats are biased towards closer low-cost, low-reward levers, even if they otherwise prefer a high-cost, high-reward lever [47].

Striatal proximity effects themselves reflect early cortical dynamics of competing action proposals (e.g. in pre-motor cortex) that are evoked as instrumental affordances and filtered by mutual inhibition, biased by multiple factors predicting action probability [48–50]. Filtered actions are then “proposed” to the striatum where they are gated, via thalamic disinhibition, according to the relative activity in the direct and indirect pathways [14,15]. Thus, actions which are proposed more rapidly and robustly, will be earlier and stronger candidates for action gating. Generalizing to any factor which causes cortical action representations to be evoked rapidly and robustly, we can see that spatial proximity as well as attention, salience, prepotency, familiarity, concreteness, etc. will all have similar effects, thus proximity is hereafter used to refer to psychological rather than strictly spatial proximity.

Striatal DA tone will also interact during action selection by increasing direct versus indirect pathway excitability [15,21,24,40], functionally equivalent to more benefit and less cost evidence across all candidate actions [14,19]. If an instrumental apparatus is more proximal, then, it will be an earlier candidate for potentiation by striatal DA tone. We therefore propose the following hypothesis concerning the interaction of DA and proximity: DA will potentiate action commission, by up-weighting benefit over cost evidence, preferentially for proximal actions (Fig. 1). Thus, when DA tone is high, proximity will strongly determine output. If no actions are uniquely proximal, high-benefit, high-cost alternatives (e.g. controlled over automatic responses) will win out. However, as one action becomes relatively more proximal, it is more likely to be selected. Conversely, when DA tone is low, preferences will shift towards low-benefit, low-cost alternatives, but, since the general likelihood of action commission is reduced, proximity effects will also be attenuated. Moreover, even under high DA tone, proximity biases can be overcome by raising the gating threshold when detecting the need for cognitive control, via recruitment of prefrontal-subthalamic nucleus circuits [51,52], as less proximal actions will have more time to compete.

Figure 1.

Proximity and DA shape competition between low-value, proximal, and high-value, distal actions. Higher DA amplifies value differences, favoring high-value actions more when none are uniquely proximal, but will selectively potentiate low-value actions when early cortical dynamics confer a strong proximity advantage. Here, using DDM-style simulations, we simulate the effect of higher DA as an increasing drift rate towards the high-value action when the proximity advantage is small, while a large proximity advantage is simulated as a large starting point bias in the direction of the proximal action, which is itself amplified by DA. Relative distribution densities reflect probability of proximal and non-proximal action at a given time-point.

We can formalize key features of the proposed DA-proximity interaction in a choice between a low-cost, low-benefit action b, that has a proximity advantage ΔP over a high-cost, high-benefit action a. Specifically, we can write the net action value of a as the linear combination of activity evoked in the direct (D_a) and indirect pathway (I_a; following [15]):

$${\mathit{Act}}_a={\beta }_D{D}_a-{\beta }_I{I}_a$$ (1)

where β_D and β_I weights reflect D1 and D2 receptor effects and increase, and decrease with striatal DA levels, respectively. Accordingly, we can express the probability of executing action a (vs. b) with a softmax function, modified such that the net cost-benefit information for action a is attenuated by its proximity disadvantage (ΔP) to capture the proposal that DA potentiates action values preferentially for proximal actions:

$$p(a)=\frac{\rho e^{{\mathit{Act}}_a}}{\rho e^{{\mathit{Act}}_a}+(1-\rho )e^{{\mathit{Act}}_b}}$$ (2)

where ρ is:

$$\rho =\frac{1}{2}{e}^{-\mathrm{\Delta }P(DA)}$$ (3)

Note that in Eqn. 3, the proximity advantage itself can be amplified by striatal DA, capturing the prediction that very high DA levels can amplify a proximity bias in action selection¹.

Importantly, since cognitive actions are selected by similar cortico-striatal loops as those involved in physical action selection [14], a proximity bias may shape cognitive action selection just as it does for physical actions. This is particularly relevant considering that while physical apparatus are often proximal (levers at hand, stairs underfoot, etc.), controlled cognitive actions are, by definition, psychologically distant. Controlled actions are not automatically evoked by the environment. Instead, they must be constructed slowly by combining current percepts with rules retrieved from long-term memory, and maintained over protracted intervals, withstanding interference from the environment [53,54]. Conversely, action proposals which are automatically evoked by the environment, or prepotent actions, enjoy a selection bias which controlled actions must overcome.

The neural mechanisms of inhibitory control provide a useful blueprint [52]: prepotent actions are rapidly evoked in sensorimotor cortex (e.g. frontal eye fields, FEF), while controlled actions arise more slowly by the conjunction of afferent input from the environment and internal (e.g. lateral frontal) rule representations. Critically, errors are predicted by earlier and more robust FEF unit activity corresponding to the prepotent action, and slower and attenuated activity in controlled action units. By contrast, correct responses are predicted by faster and more robust activity in controlled response units. Since proximity sets the speed and intensity with which action proposals are evoked, proximity shapes the competition between controlled and prepotent responses. Thus while high DA favors high-benefit, high-cost controlled actions, it can also potentiate action prepotency, undermining control.

A normative account of DA-proximity interactions

Though we motivated a DA-proximity interaction with choice behavior and neurophysiological data, there are functional motivations as well. Multiple theories posit a role for striatal DA in optimizing foraging as a function of environmental richness, uncertainty, and effort costs [55–57]. An older, influential account [55] suggests that tonic DA mediates the tradeoff between the opportunity cost of failing to act quickly in reward rich environments, and the higher effort costs of behavioral vigor. By this account, higher tonic DA signals increasing average reward, a proxy for environmental richness, and potentiates vigorous action. Recent studies support that higher DA tone promotes vigor [20,21,24,39–42,58], reward rates predict vigor [39,42,43], and DA mediates the reward rate-vigor relationship [39,42]. Another account posits that DA arbitrates between exploration and exploitation where, according to Marginal Value Theory, when local reward rates falls below the long-run average, it is better to explore than exploit [57,59]. By this account, tonic DA is again proposed to signal the long-run average and thus bias exploration when higher and exploitation when lower. Indeed, higher DA availability predicts exploration over exploitation in foraging tasks [59,60].

There is a tension between the proposal that higher DA promotes vigorous exploitation of an instrumental apparatus on one hand, and disengagement and exploration on the other. Interactions with proximity help resolve this tension. If a valuable reward is near (and both DA and proximity are thus high), devoting resources to exploration will detract from reward pursuit. Consider, e.g., that choking is more likely when there is very high incentive motivation, distraction from relevant task features and diminished cortico-striatal coordination for task performance [61]. Moreover, distracted planning increases the risk that someone else will grab a proximal reward before you can. Thus, a proximity bias is desirable to promote focus on the immediate task. If, however, there are valuable rewards in the environment, but none enjoys a uniquely strong proximity advantage, then DA can promote costly but valuable exploration. This maps to the observation that while rats are biased to approach proximal levers, they appear to engage in more “cognitive” forms of cost-benefit comparisons when far from all levers [45], perhaps indicating prospective, cognitive exploration across future outcomes [62]. In sum, a DA-proximity interaction is normative in that it promotes quick, focused action when rewards are close at hand and opportunity costs are high, while it promotes wider exploration when rewards are equidistant and it is more important to pick the best opportunity before committing.

Among the many functions performed by cognitive control, MB planning and control provide a particularly illustrative case. In comparison with MF habits, MB planning is slow and computationally expensive (i.e., MF actions have a built-in proximity advantage over MB actions), but offers the capacity for prospective, cognitive exploration to maximize reward over future states, actions, and transitions [62]. Hence, an optimal forager should be biased towards MB exploration when the environment is rich, but not when there is a uniquely proximal reward at hand. The proposed DA-proximity interaction predicts this pattern exactly (Fig. 2). First, higher striatal DA signaling high average reward promotes disengagement for exploration [59]. While this disengagement might itself be a MF habit, our hypothesis predicts that higher DA signaling high average reward should also promote either costly prospection or costly physical travel required for exploration. Second, the prediction that higher striatal DA offsets effort costs is supported by data showing more costly exploration and MB versus MF control when stakes are higher [25,28], or with greater striatal DA [30,31,59,63]. More MB versus MF behavior may obtain due to more MB control [30,31], or reduced coupling between MF action values and choice, i.e., lower softmax inverse temperatures [63]. Though not systematically explored, reduced coupling between MF action values and choice may itself be due to proximity effects (e.g. momentary attention drives choice rather than reward/punishment history). Third, critically, proximity should arbitrate between MF and MB control of behavior when DA is high. While MF action values are pre-computed and thus have a built-in proximity advantage, factors which influence the size of that advantage should arbitrate between MF and MB control. Support for this prediction includes a computational model of arbitration in which choice behavior is more MF to the extent that the MF system initially converges on a dominant action [64]. Conversely, the proximity advantage of the MF system will shrink if MB planning mechanisms are more robust. This prediction is consistent with evidence for more MB control with increasing working memory capacity for MB planning [63,65,66].

Figure 2.

MF actions will dominate unless both striatal DA tone is high, and the relative proximity advantage of MF actions is small, supporting more MB action. The surface was plotted according to Eqns. 1—3 assuming action weights that vary symmetrically as a function of striatal DA: β_G = 1 − ε ^−DA and β_N = ε ^−DA.

Conclusions

We think it is possible to resolve paradoxical effects of striatal DA signaling in both promoting and undermining control, in part, by accounting for psychological proximity and precise temporal dynamics in cortico-striatal-thalamic action selection. Importantly, a DA-proximity interaction is not only consistent with extant data, but offers an adaptive mechanism for mediating between fast, habitual action, and slow planning and control over complex action sequences. While future work is needed to directly test key predictions, it will also be important to consider the implications of a DA-proximity interactions for multiple domains including not only MF and MB control, but also incentive effects on choking [61], saliency effects in intertemporal choice [67], and Pavlovian biases [68] and familiarity biases [69] in information search. Finally, articulating the parameters of a DA-proximity interaction may prove crucial for optimizing drug therapies that promote desirable cognitive control without also promoting impulsivity.

Highlights.

• Striatal dopamine can both promote cognitive control and undermine it.

• Opposing effects may be resolved considering striatal action selection biases.

• Dopamine may undermine control by amplifying a bias for proximal actions.

• Dopamine-proximity biases may be adaptive in foraging contexts.