Computing motivation: Incentive salience boosts of drug or appetite states

Abstract

Current computational models predict reward based solely on learning. Real motivation involves that but also more. Brain reward systems can dynamically generate incentive salience, by integrating prior learned values with even novel physiological states (e.g., natural appetites; drug-induced mesolimbic sensitization) to cause intense desires that were themselves never learned. We hope future computational models may capture this too.

Redish et al. provide a valuable and comprehensive analysis of addiction models. They deserve gratitude from the field. Their sophisticated assessment of alternative models and explanatory mechanisms is admirably wide-ranging and thoughtful. In their fine scholarly effort, we would like to highlight what they note remains a significant gap unfilled by any available computational model. As the authors put it, “[C]omputational models of the planning system are insufficiently detailed to lead to specific predictions or explanations of the mechanisms by which outcomes are overvalued” (sect. 3.2.1, para. 4). We think this touches a central problem of dynamic motivation: the generation of dynamic incentive salience motivation from static learned values.

Current computational models that Redish et al. elegantly describe are powerful, but they are based purely on associations and memories. They act solely on what they know. In most reinforcement-based models, motivation is encoded as a part of an environmental state associated with the learned value of rewards, based on previous experiences. Motivational states may serve as occasion-setting contexts to modulate the value of rewards that have been previously experienced, and they may also modulate the unconditioned impact of a reward via alliesthesia. But the learned incentive value of the reward is never directly and dynamically modulated by the motivational state of the animal, without an additional learning process to intermediate.

However, evidence indicates the brain does something more when controlling desire. It dynamically generates motivation too, sometimes in surprising ways, by integrating static learned values with changing neurobiological states, some of which may never have before been experienced. Modulation of incentive value by new physiological/pharmacological states can be very potent – even the first time a relevant state occurs (Fudim 1978; Tindell et al. 2005a; 2005b; Zhang et al. 2005).

From the computational point of view, novel integrations of learned cues with new physiological states requires a kind of coupling that has not yet been satisfactorily modeled. Such coupling must connect associative values that have been previously learned and stabilized with current physiological/pharmacological states that can change from moment to moment. This falls into their Vulnerability 4, the one that we feel is particularly relevant to drug addiction. In particular, we concur with their assertion that new models are needed to address this point.

Addiction is recognized to usurp natural reward mechanisms, and even natural appetites offer dramatic demonstrations of dynamic generation of motivation (Berridge 2004; Toates 1986). Salt appetite is especially exemplary, because it can be produced as a novel state, as most humans today and most laboratory animals have never experienced a sodium deficiency. Salt appetite transforms the value of an intensely salty taste that normally tastes nasty (such as a NaCl solution that is triple the concentration of seawater). The intense saltiness becomes nice in a sodium-depleted body state, and the same triple-seawater becomes as hedonically positive as a sucrose solution (Tindell et al. 2006).

But what if a rat were not given the newly liked salt taste on its first day in a salt appetite state? What if it were instead given only a Pavlovian conditioned stimulus (CS) that had previously been paired with a salt unconditioned stimulus (UCS) when it was nasty? Cues for the previously nasty salt should have no incentive value according to most models. All the learning models described by Redish et al. make the same prediction here: the CS should elicit only negative reactions on the first trials of sodium deficiency. Any cached value of the stimulus-response (S-R) habit system obtained via a temporal difference mechanism must remain strongly negative, established by the previous pairings with punishing saltiness. Contextual knowledge does not yet exist about the potential goodness of salt in a sodium-deficient state. Even a cognitive-tree search mechanism has no way to infer the new value: its cognitive tree contains only memories of unpleasant saltiness. It lacks a branch for “liked” saltiness, at least until the rat is allowed to taste NaCl in its new physiological state.

Yet data from our lab and others show clearly that the incentive value of relevant cues can be modified on-the-fly based on homeostatic state. Indeed, we find the motivational value of the CS for salt is transformed to positive on the first day in the new state, even before saltiness is experienced: the cue becomes avidly approached and consumed, and it becomes able to fire limbic neurons like a cue for sweetness (Berridge & Schulkin 1989; Fudim 1978; Tindell et al. 2005b; 2006).

Bizarre as this reversal of cue valuation by a natural appetite may seem, nearly the same mechanism is exploited by drugs of abuse to cause addiction (Robinson & Berridge 1993; 2003). For example, other data from our laboratory show that drug-induced sensitization of mesolimbic systems, or acute amphetamine elevation of dopamine levels, causes certain relevant reward cues to dramatically become more “wanted,” eliciting more incentive salience (Tindell et al. 2005b; Wyvell & Berridge 2001). The elevation in CS incentive value occurs before the UCS reward has ever been experienced while amphetamine was in the brain, or while the brain was in a sensitized state. In addicts, such sensitized “wanting” is posited to cause intense cue-triggered motivation for drugs that far outstrips their previously learned values (Robinson & Berridge 2003).

The implication of these examples is that desire is not reducible to memory alone. Brain mesocorticolimbic systems are designed to dynamically modulate previously learned incentive values, and they do not necessarily require new learning to do so. As Redish et al. have so admirably shown, current models give a fine account of how previously learned values of reward are recalled and coordinated to predict rewards based solely on previous experiences. We hope future models may also generate new motivation values of the sort we have described in order to more fully capture addiction.