From prediction error to incentive salience: mesolimbic computation of reward motivation

Abstract

Reward contains separable psychological components of learning, incentive motivation and pleasure. Most computational models have focused only on the learning component of reward, but the motivational component is equally important in reward circuitry, and even more directly controls behavior. Modeling the motivational component requires recognition of additional control factors besides learning.

Here I will discuss how mesocorticolimbic mechanisms generate the motivation component of incentive salience. Incentive salience takes Pavlovian learning and memory as one input and as an equally important input takes neurobiological state factors (e.g., drug states, appetite states, satiety states) that can vary independently of learning. Neurobiological state changes can produce unlearned fluctuations or even reversals in the ability of a previously-learned reward cue to trigger motivation. Such fluctuations in cue-triggered motivation can dramatically depart from all previously learned values about the associated reward outcome.

Thus a consequence of the difference between incentive salience and learning can be to decouple cue-triggered motivation of the moment from previously learned values of how good the associated reward has been in the past. Another consequence can be to produce irrationally strong motivation urges that are not justified by any memories of previous reward values (and without distorting associative predictions of future reward value). Such irrationally strong motivation may be especially problematic in addiction. To comprehend these phenomena, future models of mesocorticolimbic reward function should address the neurobiological state factors that participate to control generation of incentive salience.

Introduction

Associative learning and prediction are important contributors to motivation for rewards. Learning gives incentive value to arbitrary cues such as a Pavlovian conditioned stimulus (CS) that is associated with a reward (unconditioned stimulus or UCS). Learned cues for reward are often potent triggers of desires. For example, learned cues can trigger normal appetites in everyone, and can sometimes trigger compulsive urges and relapse in addicts.

But learned associations contain only information; that is, mere knowledge about reward. The knowledge may be relatively procedural in the form of cached Pavlovian prediction errors, or explicit in the form of declarative representations that model the world. But knowledge by itself, no matter what kind, is never motivation. Something else is required to translate remembered knowledge into motivation that can actually generate and control behavior. That something else is the topic of this paper.

One reflection of the nonequivalence between knowledge and motivation is the observation that learned cues are inconstant in their motivating power. The same drug cue that potently triggers addictive relapse on a dismal occasion, spiraling an addict out of recovery, may have been successfully resisted on many previous encounters. And for everyone, reward cues vary across hours and days in their ability to evoke desire. Food cues are potent when you are hungry, but not so potent when you have recently eaten. Relevant states of physiological appetite, states of stress, or – for compulsive consumers -- trying to take ‘just one’ hit or just one taste of a palatable treat, can all enhance the temptation power of reward cues. Motivation fluctuation is nearly ubiquitous in daily life, both for normal reward operation and in pathologically extreme addictions. Fluctuation in the temptation power of learned cues needs to be addressed in computational models.

How can such fluctuations in temptation power be generated by the brain or be computationally modeled? Fluctuations in motivation intensity triggered by a reward cue are generated in large part by neurobiological state fluctuations acting within mesocorticolimbic reward circuits that react to the cue. These circuits include mesolimbic dopamine projections to the nucleus accumbens that have been the focus of computational models of reward learning, as well as larger mesocorticolimbic loops that use cortical glutamate and other neurochemical signals. Indeed, dopamine level fluctuations, both tonic and phasic, are among the most potent modulators of cue-triggered temptation, as will be discussed below.

Fluctuations in incentive salience evoked by a constant cue happen because the motivation is generated anew in each moment of encounter with a previously learned cue for reward. The level of motivation is not simply a passive function of learned associations carried over from stored memory caches of previous outcome values. Changes in neurobiological states dramatically shift relevant motivation intensities, and even new neurobiological states can shift cue-triggered motivation without need of re-training and before any further learning occurs about the reward outcome.

Incentive salience (‘wanting’) as a distinct psychological process

What is incentive salience? Incentive salience or ‘wanting’ is a specific form of Pavlovian-related motivation for rewards mediated by mesocorticolimbic brain systems (Figure 1) (Robinson & Berridge, 1993; Berridge, 2007). ‘Wanting’ typically gives a felt ‘oomph’ to declarative desires, but can also occur unfelt as a relatively unconscious process (Berridge & Winkielman, 2003; Winkielman et al., 2005). ‘Wanting’ typically coheres with ‘liking’ (hedonic impact) for the same reward, but ‘wanting’ and ‘liking’ can be dissociated by some manipulations, especially those that involve dopamine (Berridge & Robinson, 1998; Berridge, 2007; Smith et al., 2011). And ‘wanting’ can also be distinguished from learning about the same reward, especially as ‘wanting’ is much more likely to dynamically fluctuate whereas memories obtained via learning remain relatively constant (at least until new learning occurs) (Zhang et al., 2009; Smith et al., 2011).

Figure 1. Incentive salience distinguishes ‘wanting’, ‘liking’ and learning about the same reward.

A cue’s learned associations (CS) or a UCS reward are each an input to potentially trigger ‘wanting’ (top) and ‘liking’ (bottom). Natural appetite or satiety states act as kappa factor in Zhang equation to modulate both ‘wanting’ and ‘liking’ for relevant reward UCS and CS. Dopamine drug and mesolimbic sensitization act more selectively to modulate only incentive salience because of the special dopamine relation to ‘wanting’ mechanisms. Re-drawn from Robinson & Berridge (1993), based on concepts from Toates (1986) and Bindra (1981).

Incentive salience integrates two separate input factors: (1) current physiological neurobiological state; (2) previously learned associations about the reward cue, or Pavlovian CS+ (Toates, 1986; Robinson & Berridge, 1993; Berridge, 2004)(Figure 1). Integrating current physiological state with learned cues allows behavior to be guided dynamically by appetite-appropriate stimuli even without need of further learning (e.g. Pavlovian cues associated with food are immediately more attractive to a hungry animal). Incentive salience can readily be triggered by encountering Pavlovian reward CSs or by vivid imagery of reward. ‘Wanting’ can also be triggered by encounters with a reward UCS itself that activate mesolimbic systems (a reason why it’s hard to consume just one small treat). Whether triggered by CS or UCS, incentive salience typically occurs as temporary peaks of ‘wanting’, relatively transient and lasting only seconds or minutes, and tied to encounters with the physical reward stimuli.

Incentive salience as Pavlovian motivation or ‘wanting’ has several neural and psychological features that distinguish it from more cognitive forms of desire (wanting in the ordinary sense of the word). Ordinary cognitive wanting neurally depends more heavily on cortically-weighted brain circuits, computationally conforms better to model-based systems, and psychologically is more tightly linked to explicit predictions of future value based on declarative remembered previous values in episodic memory (e.g., as conscious episodic memories) (Berridge, 2001; Dickinson & Balleine, 2002; Dickinson & Balleine, 2010; Kringelbach, 2010; Liljeholm et al., 2011). Such cognitive desires are based more firmly on explicit representations of the predicted goodness of future outcome, predictions which in turn are often based on declarative memories of previous pleasure of that outcome (Dickinson & Balleine, 2010).

One way of describing the difference between cognitive desire and incentive salience ‘wanting’ is in terms of predicted utility and decision utility (Kahneman et al., 1997; Berridge & Aldridge, 2008). Predicted utility is the expected value of future reward, a prediction of outcome. Decision utility is the motivating value of that outcome, as revealed in actual behavior such as choice, pursuit or consumption. In those terms, for cognitive desires, decision utility = predicted utility, and predicted utility = remembered utility (Berridge & Aldridge, 2008). Cognitive wanting is relatively stable and not prone to fluctuation in the absence of new experiential learning about the outcome (retasting), and not as directly modulated by mesolimbic dopamine fluctuations or related circuitry manipulations (Dickinson et al., 2000; Wassum et al., 2011).

By contrast, incentive salience carries several recognizable features. For incentive salience, under conditions of dopamine-related stimulation, situations can exist where cue-triggered decision utility > remembered utility from the past, and similarly decision utility > predicted utility for future reward value (Berridge & Aldridge, 2008). In other words, it is possible to ‘want’ what is not expected to be liked, nor remembered to be liked, as well as what is not actually liked when obtained. The mesolimbic mechanism makes such irrational ‘wanting’ possible (Robinson & Berridge, 1993). Two recognizable features of incentive salience are often visible that can be used in neuroscience experiments. These features are 1) UCS-directed ‘wanting’: CS-triggered pulses of intensified ‘wanting’ for the UCS reward, and 2) CS-directed ‘wanting’: motivated attraction to the Pavlovian cue, which makes the arbitrary CS stimulus into a motivational magnet.

Cue-triggered ‘wanting’ for the UCS

A brief CS encounter (or brief UCS encounter) often primes a pulse of elevated motivation to obtain and consume more reward UCS. This is a signature feature of incentive salience. In daily life, the smell of food may make you suddenly feel hungry, when you hadn’t felt that way a minute before. In animal neuroscience experiments, a CS for reward may trigger a more frenzied pulse of increased instrumental efforts to obtain that associated UCS reward in situations that purify the measurement of incentive salience, such as in Pavlovian-Instrumental Transfer (PIT) experiments (Wyvell & Berridge, 2000; Wyvell & Berridge, 2001; Holland, 2004; Pecina et al., 2006; Talmi et al., 2008; Corbit & Balleine, 2011; Ostlund & Maidment, 2011). Similarly, including a CS can often spur increased consumption of a reward UCS by rats or people, compared to consumption of the same UCS when CSs are absent (Weingarten, 1983; Cornell et al., 1989; Everitt et al., 2001; Caggiula et al., 2002; Holland & Petrovich, 2005; Petrovich, 2011). Thus Pavlovian cues can elicit pulses of increased motivation to consume their UCS reward, whetting and intensifying the appetite. However, the motivation power is never simply in the cues themselves or their associations, since cue-triggered motivation can be easily modulated and reversed by drugs, hungers, satieties, etc., as discussed below.

Cue as attractive motivational magnets

When a Pavlovian CS+ is attributed with incentive salience it not only triggers ‘wanting’ for its UCS, but often the cue itself becomes highly attractive – even to an irrational degree. This cue attraction is another signature feature of incentive salience. The CS becomes hard not to look at (Wiers & Stacy, 2006; Hickey et al., 2010a; Piech et al., 2010; Anderson et al., 2011). The CS even takes on some incentive properties similar to its UCS. An attractive CS often elicits behavioral motivated approach, and sometimes an individual may even attempt to ‘consume’ the CS somewhat as its UCS (e.g., eat, drink, smoke, have sex with, take as drug). ‘Wanting’ of a CS can turn also turn the formerly neutral stimulus into an instrumental conditioned reinforcer, so that an individual will work to obtain the cue (however, there exist alternative psychological mechanisms for conditioned reinforcement too). Many studies measure the attractive ‘motivational magnet’ properties of a ‘wanted’ CS as sign-tracking or autoshaping. In animal sign-tracking experiments, for example, motivational magnet properties are seen when rats seek, approach and intensely sniff, nibble and bite a Pavlovian CS that predicts sucrose UCS, even if that nibbled cue is only an inedible piece of metal which has previously poked into the chamber through the wall to predict delivery of each sucrose pellet (Uslaner et al., 2006; Saunders & Robinson, 2010; 2011). Likewise pigeons may attempt to eat-peck a CS that is an illuminated bit of plastic previously associated with grain UCS, or drink-peck another CS bit of plastic associated with water UCS (Jenkins & Moore, 1973; Allan & Zeigler, 1994). Similarly, for a CS that predicts a sexual female partner UCS, a male quail may attempt to copulate with a piece of stuffed terrycloth (Domjan, 2005; Cetinkaya & Domjan, 2006). In people, for a drug UCS, human crack cocaine addicts have been known to ‘chase ghosts’, meaning to scrabble on the ground on hands and knees after white specks that are only sugar grains or stone pebbles (Rosse et al., 1993; Rosse et al., 1994), or even to put white pebbles from the ground into their crack pipes and try to light them (S.V. Mahler, personal communication 2008). And some human cigarette smokers may actually prefer to puff on a nicotine-free cigarette (i.e., consume CS alone, without UCS) rather than to receive an injected bolus of nicotine directly into their veins (receive drug UCS alone, without CS) if those are the choices offered (Rose et al., 2010). Again, however, the attraction to CS is not purely in the learned association, but instead can fluctuate with neurobiological factors (Mahler & Berridge, 2009; DiFeliceantonio & Berridge, 2010; Piech et al., 2010; Robinson & Berridge, 2010).

In such Pavlovian cases where cues elicit desire, the incentive motivation seems at first glance almost entirely learned and carried by the CS as a simple property of its associative history. That purely-learned appearance is an illusion. Because incentive salience is generated afresh by mesocorticolimbic circuits each time the stimulus is re-encountered, room arises in the brain computation of CS-triggered ‘wanting’ to incorporate a second source of input besides previously-learned cached values of the reward outcome in the past. That second source of motivation is the neurobiological state of mesolimbic circuits at the moment the cue is re-encountered, and it is always ready to go to work whenever opportunity arises. For example, as a general rule, cue-triggered ‘wanting’ for UCS elicited by a CS can be dramatically intensified in reencounter if dopamine levels in nucleus accumbens are elevated above normal at that moment by a drug such as amphetamine or by neural sensitization induced by previous exposures to amphetamine or related drugs (Wyvell & Berridge, 2000; Wyvell & Berridge, 2001; Tindell et al., 2005; Smith et al., 2011). Neurobiological state also includes physiological factors stemming from natural appetites of hunger or thirst or salt appetite, even if new and never experienced before (induced by orexin, leptin, angiotensin II, aldosterone and related hormones), and these natural appetites are of course the evolutionary reason why a second source of motivation input exists (Fudim, 1978; Berridge & Schulkin, 1989; Tindell et al., 2009). The neurobiological state factor also includes drugs of abuse (e.g., intoxication priming by a drug on board), as well as longer term consequences of withdrawal especially involving permanent sensitization states (Wyvell & Berridge, 2000; Wyvell & Berridge, 2001; Tindell et al., 2005; Mahler & Berridge, 2009; Smith et al., 2011). Finally, the neurobiological state factor also includes stress states involving elevated CRF signals in mesocorticolimbic circuits that can recruit dopamine participation (Pecina et al., 2006; Berridge et al., 2010; Dallman, 2010). All of these states can amplify the dynamic translation of a relevant Pavlovian CS for reward into motivation at the moment the cue is re-encountered, intensifying the level of ‘wanting’. Conversely, cue-triggered ‘wanting’ can be suppressed by drugs that block dopamine receptors (Dickinson et al., 2000; Wassum et al., 2011), and ‘wanting’ for CS or UCS can be virtually eliminated if dopamine is removed by 6-OHDA lesions or prevented by genetic mutation that induce severe Parkinsonism (Berridge et al., 1989; Berridge & Robinson, 1998; Robinson et al., 2005). An implication of dopamine or related modulations of ‘wanting’ means that at the moment of CS encounter, decision utility can exceed (when dopamine is elevated) or dive below (when dopamine is suppressed) the learning-based predicted utility of a future outcome generated from cached memories of previous values of the reward UCS (Kahneman et al., 1997; Berridge & Aldridge, 2008).

Some experimental demonstrations of neurobiological amplification will be described later. For now, the important point is that such neurobiological states all share in common the ability to amplify ‘wanting’ triggered by the next relevant CS encountered in the state, suddenly elevating incentive motivation to a higher level. The amplification of CS-triggered ‘wanting’ can reach levels higher than ever before associated with the UCS. Such fluctuations in motivation need to be modeled in order to accurately capture the psychological function of mesocorticolimbic circuits.

Incentive Salience Model of Zhang et al. (2010)

An initial attempt to model such fluctuations in temptation power of a previously-learned reward CS was recently made by Jun Zhang and his colleagues in a dynamic model of incentive salience (Zhang et al., 2009). I will refer to this as the Zhang equation or Zhang model of incentive salience. The Zhang model of incentive salience is unique in that it incorporates a dynamic physiological factor κ (kappa), which can change as rapidly as appetite, satiety or drug-state changes, and which modulates motivation generated from the learned value of a relevant CS for reward (r_t) without requiring any new learning about its UCS value in the new physiological state (Figure 2).

Figure 2. Simulations of upshifts in CS temptation power.

Zhang equations simulate actual enhancements in CS attractiveness induced by increases in kappa factor (Zhang et al., 2009). Multiplicative amplification of level of ‘wanting’ elicited by reward CS shown on left, induced by a new intoxication state (e.g., amphetamine) or by a mesolimbic sensitization state existing at the moment of cue re-encounter. Valence reversal from negatively aversive to positively ‘wanted’ shown at right, induced by sodium appetite, modulates incentive salience of CS previously associated with triple-seawater concentrated salty taste UCS. Simulated data based on (Wyvell & Berridge, 2001; Tindell et al., 2005; Smith et al., 2011; (Krieckhaus & Wolf, 1968; Fudim, 1978; Berridge & Schulkin, 1989; Stouffer & White, 2005; Tindell et al., 2009; Robinson & Berridge, 2010).

In the Zhang model, the cue-triggered incentive salience or motivational value is defined as Ṽ(s_t). Computationally,

$$\stackrel{\sim }{V}(s_t)=\stackrel{\sim }{r}(r_t,\kappa )+\gamma V(s_{t+1})$$

The level of Ṽ(s_t) is triggered by encounter with the CS that carries a previously learned association (r_t) with reward UCS. The κ factor incorporated into the Zhang model reflects current neurobiological state relevant to the UCS that has been associated with the CS (hungers, satieties, drug states, etc.) at the moment of CS re-encounter. This κ factor acts to transform the learned memory or cached value of r into a particular level of incentive salience. For convenience, the κ state that held during previous learning trials (i.e., during CS-UCS training) is assumed to be κ=1. As long as nothing changes, state can remain 1 and Ṽ(s_t) = (r_t).

What is important is the κ state at the subsequent moment of CS re-encounter. Only if κ =1 continues to be true at re-encounter, and physiological state remains essentially unchanged, will ‘wanting’ triggered by the CS match the previously learned value. Any departures of κ from previous value of 1, (i.e., any changes in relevant neurobiological state), will let the level of ‘wanting’ at the moment of CS re-encounter be dynamically modulated. If state declines (e.g., natural satiation state or pathological loss of dopamine), so that κ<1, the shift produces a decrement in incentive motivation below the previously learned level. Conversely, if relevant state rises (e.g., an increase in hunger or taking a priming dose of addictive drug), so that κ>1, the shift enhances CS-triggered levels of motivation above the previously trained amount (Figure 2). As an aside for readers wondering about an unconditioned reward UCS, although the model does not explicitly address UCS reward ability to trigger ‘wanting’ (e.g., priming by actual food or actual drug), it is possible to imagine inserting a UCS value such as, say, U_t value (U_t for unconditioned UCS attractive features) as level of innate attractiveness in place of the learned cache r_t value. That would let the model apply to encounters with UCS as well as to encounters with CS. The value of U_t would be relatively innate rather than learned, but still modulated by κ. The U_t value presumably would equal or exceed in magnitude the maximum possible r_t value, and the U_t would be modulated by kappa in the equation in exactly the same way as r_t for the CS (e.g., for UCS, Ṽ(s_t) = [U_t, κ]).

Let’s return to a learned CS for reward. The interaction of κ with r_t modulates the motivation elicited by the CS, potentially producing a new value that differs from all previously learned values. But where does the learned value come from? To generate the cached learned value, Zhang et al. adopted the temporal difference model of reinforcement learning (Sutton & Barto, 1981; Schultz & Dickinson, 2000; Dayan & Balleine, 2002; McClure et al., 2003b; O’Doherty et al., 2003; Schultz, 2006; Niv & Schoenbaum, 2008; Redish et al., 2008b; Glimcher, 2011). The temporal difference model is also known as the prediction error model, and sometimes called a ‘model-free’ algorithm because it contains only a cached value as memory associated with CS, with no explicit representation of the UCS or its place in the world (though noncomputational readers may find confusing the notion of an apparently self-contradictory ‘model-free model’ of reward learning). Zhang et al.’s adoption of the temporal difference value cache associated with CS for the learned input (r_t) was mainly out of deference to popularity of prediction error concepts in computational modeling during the past decade and use in describing dopamine neuronal firing.

However, in my view, nearly any other model of Pavlovian reward learning could be equally well used in place of the temporal difference model to generate the learned (r_t) value associated with CS and be plugged into the Zhang model to interact with κ. As new Pavlovian algorithms are suggested to replace the temporal difference model, they could quite replace the associative inputs in the Zhang equation. What is important for motivation during the moment of cue re-encounter is the interaction of κ with r_t, and not so much the particular Pavlovian learning algorithm that generates the previously-learned value of r_t.

Whatever learning model is used as input in generating incentive salience, one general rule arises from the Zhang interaction. Whenever the same physiological state is used for CS testing as for Pavlovian training of CS-UCS association, K= 1, and the level of incentive salience triggered by the cue corresponds to the previously learned value of reward on prior UCS experiences. In other words, learning and ‘wanting’ levels can appear identical as long as neurobiological state is kept constant. But actually, learning and motivation are confounded whenever κ=1state is kept constant across learning and cue test, at least from the point of knowing which controls neural activation or behavior. That is, when the two CS values (learned cache and incentive salience) do not diverge, scientists or modelers cannot tell whether purely learned value or transformed motivation value is reflected in a mesocorticolimbic neural activation triggered by the cue or in a behavioral response to the cue, and the debate will be endless between learning advocates and motivation advocates. It is only when relevant physiological state is changed between training and CS test do possibilities arise for experimental dissociations between the prediction error cache (r_t) and incentive salience Ṽ(s_t). Only when state is changed can an experimenter tell whether a neural activation encodes stable associative memory (learning) or transformed incentive salience (‘wanting’).

Distinguishing univalent changes in ‘wanting’ intensity from bivalent reversals between ‘wanted’ and ‘unwanted’

The Zhang equation above simply gives a generic model for modulating incentive salience at the moment of stimulus reencounter. However, describing actual encounters with reward stimuli require more specific interactions. The Zhang model elaborated the specific form of interaction between CS r_t and κ state into two further forms, each applying to its own situations. The first form, which is usually applicable, is multiplicative (A; equation 3a in Zhang et al):

$$\text{A}:\stackrel{\sim }{r}(r_t,\kappa )=\kappa \ast r_t$$

This form applies to the amplification of ‘wanting’ for a reward by addictive drug administration or by permanent drug-induced neural sensitization. It also applies to most amplifications of ‘wanting’ for food by a hunger state. It applies to any positive reward for which the level of motivation varies only quantitatively between zero and high (i.e., never goes negative to become aversive). The multiplicative form makes a CS for a pleasant UCS reward more or less attractive than was previously learned (but always remaining positive in valence if it was learned as positive). This multiplicative form of the Zhang equation generates incentive salience as:

$$\stackrel{\sim }{V}(s_t)=\stackrel{\sim }{r}(r_t\ast \kappa )+\gamma V(s_{t+1})$$

In this multiplicative form, a positive incentive cache for the cue (r_t) could be raised into higher or lower incentive salience than was learned by corresponding increments and decrements in the κ factor. The change in Pavlovian motivation would apply instantly to the next encounter of the CS even if the UCS had never been experienced in the new physiological state. That is, the motivation response to the CS would no longer match its previously learned level. Examples will be given below.

However, a second specific form of the Zhang equation was designed for a few special situations that actually reverse valence from positive to negative, or from negative to positive (Figure 2).. Certain types of state do not merely shift the positive quantity of a relevant reward to more or less positive, but actually reverse CS/UCS valence between nice and nasty. For example, an upshift from negative to positive can be produced by certain specific appetites such as salt appetite (Tindell et al., 2009; Robinson & Berridge, 2010). This upshift will be described in detail later on.

Modulations of ‘wanting’ that reverse valence between negative and positive, such as salt appetite or learned taste aversions, cannot be dealt with by a purely multiplicative interaction, except by positing negative multiplication values (e.g. r_t * (−1)) to reverse valence. Such a negative multiplication would introduce complications of its own that would distort real-world reversals. Changing the +/− sign of a multiplicative transform would invert the rank order among CSs whenever several different reward stimuli were in the same family and all reversed together across zero (e.g., 3 CSs for 3 different concentrations of UCS reward). For example, reversing valence from positive to negative in a multiplicative model would cause the reward that was originally most highly liked and ‘wanted’ of all out of a group of several related rewards, to become the most highly disliked or repulsive after devaluation of the group’s category. Conversely, a most aversive cue among several for different concentrations of hypertonic saltiness might become the most ‘wanted’ when valence was reversed from negative to positive. Such re-ordering is unrealistic. When valence actually flips in life, an originally most liked reward usually still remains the best of a bad lot. Likewise, the least aversive stimulus of a group of negative stimuli may after reversal into positive valence become the most ‘wanted’.

Zhang and colleagues tried to offer a better computational expression of such valence reversals in the form of a log-based transformation in their equation 3b:

$$\text{B}:\stackrel{\sim }{r}(r_t,\kappa )=r_t+log\kappa$$

So in this form of the Zhang equation, incentive salience is generated as:

$$\stackrel{\sim }{V}(s_t)=\stackrel{\sim }{r}(r_t+log\kappa )+\gamma V(s_{t+1})$$

Changes in the logκ term away from 1 can move the incentive salience value across the zero boundary during the CS encounter, shifting the value either up (i.e., increases ‘wanting’, κ>1) or down (i.e., decrease ‘wanting’, κ<1). This allows polarity reversal from a negative value to a positive value (with κ much larger than 1), or vice versa (with κ closer to 0), without requiring negative multiplication.

The logarithmic or additive version allows several reward stimuli belonging to the same reward family (e.g., different concentrations of saltiness) to all reverse valence together yet maintain their relative rankings. Thus the least disliked of several intensely salty tastes becomes the most liked during salt appetite, and so on.

Valence reversal would similarly encompass cases in which dopamine-related generation of motivation switches between desire and dread (Faure et al., 2008; Reynolds & Berridge, 2008; Richard & Berridge, 2011). For example, the same glutamate signal disruptions in locations within medial shell of nucleus accumbens in rats can generate either intense appetitive motivation to eat or intense actively fearful motivation, depending on whether the current environment is comfortable and familiar or over-stimulating and aversive. Both the appetitive incentive salience and the fearful salience involve an interaction between endogenous mesolimbic dopamine and the localized glutamate disruption, and the valence reversal shifts the relative balance between D1 and D2 types of dopamine receptors at the site that mediate the interaction (Richard & Berridge, 2011). The additive Zhang model can capture such dopamine-related shifts that reverse valence between incentive salience attributed to the perception of food, which makes the food attractive and appetizing, and fearful salience attributed to people and objects in the room, which makes them threatening so as to elicit active anti-predator behaviors or distress calls and escape attempts. Related reversals of valence may be clinically involved in schizophrenia or drug psychosis, where enhancements of fearful motivation toward some stimuli may transition to enhanced appetitive motivation toward other stimuli, or from appetive to aversive (Kapur, 2003; van Os & Kapur, 2009; Buckholtz et al., 2010; Dowd & Barch, 2010; Treadway & Zald, 2011).

Limitations of the Zhang model of incentive salience

These features above let the Zhang model capture dynamic shifts in incentive salience level or motivation valence, even shifts that depart from all previously learned values of a CS, without requiring any new re-learning about the UCS. However, the model still leaves out several important features of incentive salience. It also leaves out all learning and motivation processes beyond incentive salience. These omissions are discussed next.

1) Directedness of ‘wanting’ towards specific reward stimuli?

Incentive salience is not merely a nonspecific activation of arousal, vigor, or response energy without direction. While dopamine may contribute to general effort/vigor processes (Niv et al., 2007; Salamone et al., 2007), cue-triggered ‘wanting’ is much more directionally focused onto particular CS and UCS stimuli. The CS smell of food makes you want to eat, not to want something else. A cigarette lit by someone across the room makes a smoker specifically want to smoke. And when cues become motivational magnets, they pull behavior directionally toward themselves. In humans, a visual CS for reward elicits visual attention and eye fixation. It is hard not to look. The eye fixation is automatic and involuntarily, and can either help or hinder an intentional goal depending on whether the person is searching for reward cues or for a different stimulus (Hickey et al., 2010b; a; Tapper et al., 2010; Anderson et al., 2011). Likewise, physiological sodium appetite makes a rat trace a beeline to a physical CS for saltiness as a motivational magnet (even if the CS has always been avoided in normal state, and its UCS always ‘disliked’), but does not change the attractiveness of CSs for other rewards, such as sugar, nor make attractive a CS- that signals nothing (Robinson & Berridge, 2010). And a CS motivational magnet becomes specifically ‘wanted’ in a manner tailored to its particular UCS: sniffed, nibbled and bitten if the CS has food as its UCS, but behaved towards quite differently if the UCS were water, sex or drug.

The Zhang model does not explicitly try to capture the directionality of ‘wanting’ towards a particular reward’s CS in a landscape that contains additional CSs for other UCSs. The model never directly contrasts ‘wanting’ for one UCS versus another UCS. Rather it considers just one CS-UCS association at a time, and deals only with the intensity of ‘wanting’ triggered by that CS in the current state relevant to its UCS.

To approach directional specificity of appetites, Zhang et al. do at least posit that each neurobiological appetite state (e.g., caloric hunger vs salt appetite) has its own qualitatively unique K distinctive to its UCS, which specifically modulates only its own CSs, and need not modulate CSs for other rewards. This helps recognize the specificity of CS-state-UCS interaction. An appetite-specific κ transforms only the r_t of a CS for its relevant UCS, and not necessarily the CSs for other UCSs. Thus caloric hunger can transform CSs for food, but not CSs for intense saltiness UCS or for sex or drugs, and so on.

2) Narrowness vs. breadth of directional focus

A related omission from the Zhang model is that it lacks any parameter to express the width of focus in the directed beam of incentive salience attributed to targets (narrow vs. broad). Focus width can vary between narrowly making just one single stimulus intensely ‘wanted’, while other stimuli may even become simultaneously ‘less wanted’, to broadly making several incentives or even many incentives more ‘wanted’ at the same time. For example, in human clinical situations, a drug addict may narrowly ‘want’ only drugs, or even just one particular drug (Flanagan, 2011). Likewise a cued binge eater may ‘want’ only foods or one particular food above all (Gearhardt et al., 2009; Pelchat, 2009). Similarly, in animal neuroscience studies, width of CS focus for incentive salience can be narrowed at the whim of the experimenter, via neurochemically stimulating limbic-related circuits that translate a particular remembered Pavlovian association into motivation toward a single incentive cue. For example, Stephen Mahler and Alexandra DiFeliceantonio each found in our lab that stimulating opioid circuitry within the central nucleus of the amygdala (a component of extended amygdala circuitry that translates learned associations into motivation), or within particular patch compartments of dorsal neostriatum (which have limbic functions related to reward projections from ventromedial regions of pre-frontal lobe), focused cue-triggered ‘wanting’ more narrowly in winner-take-all fashion (Mahler & Berridge, 2009; DiFeliceantonio & Berridge, 2010). Under those conditions, a single favorite reward-related CS became a super-potent ‘motivational magnet’ that pulled virtually all attraction toward itself, while another competing CS simultaneously became oppositely less wanted at the same time. This was demonstrated in a sign-tracking/goal-tracking experiment, in which two CSs (sign vs goal) compete to attract. In that CS-as-motivational-magnet paradigm, some rats approach a sign CS (a suddenly inserted lever that predicts delivery of UCS reward), whereas other rats approach a goal CS (dish where sucrose UCS appears) (Boakes, 1977; Flagel et al., 2009; Yager & Robinson, 2010; Saunders & Robinson, 2011). Most normal rats sometimes approach their alternative stimulus too. However, drug microinjections into central amygdala or neostriatum that stimulate mu-opioid neurotransmission makes all individuals narrow their focus further, doubling approaches to their preferred CS, while simultaneously making somewhat fewer to the alternative. Sign-trackers emit more intensely focused sign-tracking, while goal trackers emit more intensely focused goal-tracking (Mahler & Berridge, 2009; DiFeliceantonio & Berridge, 2010). Thus cortico-amygdala-striatal state can induce a laser-like narrowing of the beam of incentive salience focused on one target, at the expense of attraction to other targets even if related to the same reward.

Conversely, incentive salience focus can broadened in other brain states to encompass several more targets at once. This makes multiple incentives become more ‘wanted’ together. For example, hunger can potentiate incentive motivation to obtain deep brain electrical stimulation as reward, as well as to obtain food reward (Carr, 2011). Amphetamine sensitization in rats can amplify the incentive value of both food and sex rewards for some individuals (but not drugs), but enhancing incentive attraction to drug-associated stimuli for other individuals (but not food or sex) (Nocjar & Panksepp, 2002). Hunger and sexual cues may both increase ‘wanting’ for money (Briers et al., 2006; Knutson et al., 2008) Human drug addicts may ‘want’ several different drugs, and some may also be hypersexual or prone to other compulsions (Washton & Stone-Washton, 1993; Benotsch et al., 1999; Leeman & Potenza, 2011). Likewise, in compulsive dopamine dysregulation syndrome that afflicts some Parkinson’s disease patients when taking dopamine-stimulating medication, a patient who compulsively ‘wants’ to gamble may also experience excessive urges to shop or to pursue sex, or engage in a hobby, especially while taking the medication (Benotsch et al., 1999; O’Sullivan et al., 2011). Electrical brain stimulation of mesocorticolimbic circuitry may also elicit quite broad ‘wanting’ of many targets, especially initially (Valenstein, 1971; Berridge & Valenstein, 1991; Heath, 1996; Kringelbach et al., 2010). For example, deep brain electrical stimulation in the subthalamic nucleus engaged mesocorticolimbic circuits in a woman to generate intense feelings of being “in love with two neurologists, and tried to embrace and kiss people”, and then went on to engage in binges of “unrestrained buying of clothes” in which she spent so much money that her alarmed family wished to take away her credit card (p. 1383)(Herzog et al., 2003). That is broad attribution of incentive salience indeed. But often enhancements are much more focused onto a particular target. For now, the Zhang model neglects width of focus and deals explicitly with only the intensity level of incentive salience evoked by a particular CS. Width of focus on target and directedness towards particular CS/UCS targets deserves further investigation and computational modeling in future.

Reboosting and irrational preservation of CS ‘wanting’ after UCS revaluation

Sometimes CS incentive salience does not change so suddenly as the Zhang model suggests when UCS undergoes revaluation (especially devaluation), but rather the CS value lags behind. ‘Wanting’ under some conditions is thus more persistent, resisting instant modulation by neurobiological state changes or goal revaluation.

The first reported lag was ‘extinction mimicry’ after neuroleptic administration, originally advanced by Wise and colleagues in 1970s–80s as evidence for the anhedonia hypothesis: a gradual falling off in pursuit or consumption of a well-established reward under the influence of a low to moderate dose of a dopamine antagonist drug, such as pimozide (Wise et al., 1978; Wise, 1982; 1985; Ettenberg, 1989). The signature feature of extinction mimicry was that systemic administration of a neuroleptic dopamine antagonist at a dose too low to produce either movement impairment or instant motivation impairment would typically fail at first to suppress previously learned goal-directed behavior to obtain or consume the reward, but gradually performance would decline as trials proceeded under the drug (sometimes revealed by a delayed impairment in a drug-free test the next day) (Wise, 1985). That is the pattern that would be produced by real extinction or omission of UCS reward: at first the trained animal would still work, but gradually fall off as its instrumental efforts went unrewarded (though it was pointed out that extinction mimicry is often not a perfect mimic (Salamone et al., 1997)).

The slow or delayed decrements of extinction mimicry were re-interpreted as ‘deboosting’ of incentive salience (neuroleptic prevention of normal reboosting) by my colleagues and I as an alternative explanation to anhedonia 20 years ago (Berridge & Valenstein, 1991; Robinson & Berridge, 1993; Berridge, 1996),. To explain slow suppression of motivated behavior, my colleagues and I posited that reboosting ‘of ‘wanting’ normally occurs on CS-UCS rewarded trials whenever UCS is received, as an incremental mechanism of incentive salience maintenance that operates trial-by-trial. More formally, a learning-based computational model for reboosting of incentive salience was later suggested by McClure and colleagues in the form of a pure temporal difference model (McClure et al., 2003b). Their model proposed that the CS incentive salience value corresponded exactly to the cached value of learned reward acquired over previous trials, and could be maintained or decremented by neuroleptic or by real extinction (omission of UCS) on a trial-by-trial basis. Normal reboosting has several ‘wanting’ effects, and impairment of reboosting can detach CS ‘wanting’ value from UCS ‘liking’ value in a gradual fashion.

An immediate effect of UCS reboosting of incentive salience is priming of increased motivation to consume more of that UCS. Taking a single hit of cocaine can make person report higher craving to take more cocaine a few minutes later (Jaffe et al., 1989), and taking a single alcoholic drink can make a person more likely to take another drink soon after(de Wit & Chutuape, 1993). The increase is often directed specifically towards the same reward target. People who have eaten a full sandwich lunch and afterwards given a taste of pizza are then likely to choose and eat considerably more pizza if allowed, whereas people given a taste of ice cream after their lunch choose and eat more ice cream (Cornell et al., 1989).

A more sustained effect of reboosting is to increment the incentive salience assigned associatively to the CS, which becomes evident on future trials. Reboosting is positively incremental (to maintain constant an already learned level of incentive salience) whenever CS is reinforced by UCS reward. The process turns into negative deboosting to decrement incentive salience when CS is extinguished alone or when mild pharmacological disruption of mesolimbic function is introduced, such as by low-dose dopamine blockade (too low to additionally produce an instant kappa decrement in the Zhang model). That selective disruption of reboosting prevents the UCS from reboosting the CS value. However, higher doses of antagonist or a complete 6-OHDA loss dopamine would both impair reboosting and produce an instant downshift in CS incentive salience in accordance with Zhang’s kappa downshift (Dickinson et al., 2000; Ostlund & Maidment, 2011; Wassum et al., 2011). The upshot of all this is that sometimes CS value can detach from UCS value, and gradually trend downwards by itself if reboosting of incentive salience is selectively impaired.

Irrationally high and persistent ‘miswanting’

In an opposite direction, persistent excessive ‘wanting’ for a previously learned CS, when in the interim UCS has been devalued, is another failure to dynamically shift kappa state that departs from the Zhang model (such as after some cases of taste aversion learning or after bad drug experiences for an addict) (Wilson et al., 1981; Holland, 2004). In such cases, the individual still pursues the CS, or encounters with CS still spur pursuit of the reward UCS (at least until the unwanted UCS is actually obtained) even though the UCS would not be consumed if it were present. In the case of overly persistent ‘miswanting’ after UCS devaluation, the cue-triggered ‘want’ seems irrational, because the UCS may no longer be ‘liked’. Speculatively, irrational persistence of CS ‘wanting’ may occur especially when the UCS devaluation, conducted after the CS-UCS reward relation is originally learned, is induced by a second stage of associative learning mechanisms (e.g. by associative pairing of food with nausea to induce a conditioned taste aversion), rather than more directly by physiological states of satiety or drug manipulations. Adding layer upon layer of associations to the UCS may let CS value once strongly acquired decouple from subsequent shifts in UCS value. Persistence has been reported most often after UCS devaluation by associative taste-aversion learning, in which the original UCS food reward is paired with nausea illness in the absence of the auditory or visual CS that previously signaled the food (Wilson et al., 1981; Holland, 2004). In such cases, the food becomes ‘disliked’ yet the rat may continue to run in the maze to the food’s location. Irrational persistence is also facilitated by over-training of the original CS-UCS association when the UCS is still rewarding (Holland, 2004). Such persistence is often called habitual, but sometimes the persistence cannot be explained by S-R habits. For example, the CS may trigger high pulses of ‘wanting’ expressed as increases in instrumental effort for a ‘disliked’ UCS in an extinction PIT test, where a habit between the CS and the action can be excluded because the CS motivates an action that is new in its presence, having never before been paired as response with CS (Holland, 2004). Rather than S-R habit, I interpret that persistence at least sometimes as excessive enduring cue-triggered ‘wanting’, which has detached from UCS value. Such cases might require actual encounter with aversive UCS to deboost the incentive salience attributed to the CS (Dickinson & Balleine, 2010). In clinical applications, perseveration of cue-triggered ‘wanting’ might also be important in addiction and related compulsive pursuits of incentives, where intense cue-triggered ‘wanting’ may persist even in cases where the available drug or incentive is known by the addict to not be particularly pleasant (especially if driven by sensitization of incentive salience). Of course we need a better understanding of how detachments of CS motivation from UCS value can occur in order to incorporate persistence more fully into future computational models of incentive salience.

Limitations beyond incentive salience: Learned cognitive expectancies and rigid habits

A final noteworthy omission is that the Zhang model also leaves out all other (nonPavlovian) forms of incentive motivation and other types of reward learning, such as cognitive model-based predictions of future reward value and of act-outcome understanding of how to obtain the reward (Dayan & Balleine, 2002; Dickinson & Balleine, 2002; Dickinson & Balleine, 2010). It also leaves out simpler S-R habits. Those are important phenomena too, but no model can address everything at once.

Habits and cognitive learning about rewards are quite different from the Pavlovian learning that powers incentive salience. Perhaps the distinction between Pavlovian learning in incentive salience and cognitive learning in goal-directed desires may seem subtle, but they are quite distinct psychologically and computationally, and even have separable brain substrates (Dayan & Balleine, 2002; Dickinson & Balleine, 2002; Dickinson & Balleine, 2010; Boureau & Dayan, 2011). In particular, cognitive forms of reward learning and desire are more related to what are called model-based computational systems, such as tree-search models (called model-based because they contain models or maps of real-world relationships among stimuli, actions, and goals) (Dayan & Balleine, 2002; Daw et al., 2005; Niv et al., 2006; Dayan & Niv, 2008; Redish et al., 2008a; Balsam & Gallistel, 2009; Dickinson & Balleine, 2010). However, cognitive values and expectations of future outcomes may make relatively weak contributions to the control of incentive salience (Berridge, 2001; Dayan & Balleine, 2002; Balleine & O’Doherty, 2010; Dickinson & Balleine, 2010).

The difference between cognitive desire and Pavlovian-triggered incentive salience can produce cases of irrational ‘wanting’ in which an individual ‘wants’ what they cognitively expect not to like as well as actually do not like when they get it (or conversely, fail to ‘want’ what they expect to like and actually do ‘like’ when they get it) (Robinson & Berridge, 1993; Berridge & Aldridge, 2008). For example, a sensitized addict in recovery (no longer suffering from withdrawal) may sincerely and accurately believe the drugs currently available will not be very hedonically pleasant, and not nearly worth the painful consequences that taking drugs will be certain to bring. Yet that not-quite-ex-addict may still be susceptible to relapse via a pulse of overpowering incentive salience if encountering a drug cue when in a temporary state of kappa ≫ 1. Such a person could be said to have an irrationally intense ‘wanting’ to take drugs that they do not cognitively want and do not expect to like, and do not actually like very much when eventually consumed. Incentive salience ‘wanting’ can thus diverge at times from rational cognitive desire.

Choosing between models for dopamine’s role in reward: Some relevant empirical evidence

Perhaps the biggest objection to the incentive salience model of mesolimbic function suggested here comes from modelers who believe that dopamine instead mediates some other component of reward. During the 1980s–1990s, many investigators believed that dopamine caused the hedonic impact of rewards: pleasure ‘liking’ (Wise, 1985; Koob & Le Moal, 1997). Although a relative few scientists may still believe that dopamine causes pleasure, the majority opinion in the field seems no longer to accept the dopamine = hedonia view (Wise, 2006; Barbano & Cador, 2007; Berridge, 2007; Daw, 2007; Nicola, 2007; Niv et al., 2007; Salamone et al., 2007; Koob & Volkow, 2010; Aarts et al., 2011). Much evidence has accumulated against the hypothesis that dopamine mediates pleasure. Evidence from affective neuroscience studies has shown: 1) dopamine is not needed for normal ‘liking’ reactions to sensory pleasure, so dopamine blockade or even complete destruction of mesolimbic dopamine via extensive 6-hydroxydopamine neurotoxin lesions in rats leaves ‘liking’ to sweetness pleasure unimpaired (Berridge et al., 1989; Berridge & Robinson, 1998). 2) Likewise, normal pleasure ratings are given to sweet tastes by Parkinson’s patients who have extensive dopamine depletion (Sienkiewicz-Jarosz et al., 2005), and normal pleasure ratings are given to cocaine by normal people who are in a drug-induced state of dopamine depletion or blockade (Brauer & de Wit, 1996; 1997; Leyton et al., 2005; Leyton et al., 2007; Leyton, 2010). 3) Sucrose rewards and cocaine/morphine drug rewards are still behaviorally reinforcing in the ‘stamping-in’ sense of becoming learned about and preferred after experiences with those rewards in the absence of any brain dopamine in dopamine-deficient mutant mice (Cannon & Palmiter, 2003; Hnasko et al., 2005; Robinson et al., 2005; Hnasko et al., 2007). 4) Even elevations in dopamine do not seem to be pleasure mechanisms. Amphetamine microinjections directly into the hedonic hotspot of nucleus accumbens to raise synaptic dopamine fails to enhance sucrose ‘liking’ reactions or neural signals to hedonic impact in meso-corticolimbic-pallidal circuits of rats (Smith et al., 2011). 5. Similarly, sucrose ‘liking’ reactions are not enhanced by gene-induced elevation of synaptic dopamine in nucleus accumbens or striatum [via knockdown of dopamine transporter], nor by systemic amphetamine administration, drug-induced mesolimbic sensitization, or deep brain stimulation of mesolimbic systems via an electrode in the medial forebrain bundle (even though all of these increase ‘wanting’ for the same food reward that is not more ‘liked’; as does the amphetamine microinjection of #4) (Berridge & Valenstein, 1991; Pecina et al., 2003; Tindell et al., 2005; Smith et al., 2011). 6) In humans, elevations in striatal-accumbens dopamine levels can become uncorrelated with subjective pleasure ratings of liking for cocaine/amphetamine/L-Dopa drug rewards, but remain correlated strongly and directly with pulses of intense ‘wanting’ ratings caused by the drugs (Leyton et al., 2002; Evans & Lees, 2004; Evans et al., 2006; Leyton, 2010; O’Sullivan et al., 2011). For these and other reasons, the original pleasure proponents by and large no longer advocate the hedonia-anhedonia hypothesis of dopamine (Wise, 2006). For example, even Roy Wise, the primary architect of the original anhedonia hypothesis, by the late 1990’s was quoted to say “I no longer believe that the amount of pleasure felt is proportional to the amount of dopamine floating around in the brain” (p.35) (Wickelgren, 1997).

Instead, from late 1990s to the present, anhedonia has been largely replaced in the reward neuroscience literature by the alternative idea that dopamine causes learning about rewards. The dopamine=learning hypothesis has usually taken form of teaching signals or prediction errors in the temporal difference model, or of stamping-in new associations between stimulus-stimulus or stimulus-response pairs, and of recruiting neuronal molecular cascades for memory formation, such as in long-term potentiation or long-term depression of synaptic signaling (Schultz et al., 1997; Hyman et al., 2006; Wise, 2006; Schonberg et al., 2007; Niv & Schoenbaum, 2008; Glimcher, 2011). So it seems most useful here to turn to evidence that helps answer whether dopamine actually causes reward learning or instead causes incentive salience ‘wanting’ for learned (and unlearned) rewards. In a nutshell, I conclude that although dopamine signals often seem to code learning (just as many studies once reported that dopamine seemed to code pleasure), a closer analysis indicates that dopamine does not actually cause learning about reward. Even the coding of learning may be partly illusory, because it occurs primarily under confounded conditions that conflate learning with motivation. These conclusions are unpacked below.

Prediction error in electrophysiological-neuroimaging results: Prediction confounded with motivation?

Two decades of studies have reported that phasic dopamine activations in the brain encode prediction error signals in a way that often conforms to temporal difference models (Schultz et al., 1993; Schultz et al., 1997; Schultz, 1998; Schultz & Dickinson, 2000; McClure et al., 2003a; Pessiglione et al., 2006; Niv & Schoenbaum, 2008; Glimcher, 2011). However, I believe that clamping of physiological state in such studies, typically by employing similar deprivation/satiety states in both training and testing phases, has often produced conditions ripe for confounding previously learned value with current incentive value. Observers may have mistaken some mesolimbic motivation signals for current value to be pure prediction signals. Remember that when κ=1 in the Zhang equation the incentive salience output mimics a temporal difference model (which provides half the input to incentive salience). That is because the r_t associative input is not transformed when κ=1, but rather is copied directly from the cached memory input value to become the motivational output. Whenever the physiological state in training is nearly replicated in subsequent testing, κ=1. That same-state repetition has often been precisely the case in many electrophysiological and neuroimaging studies that reported prediction error results. For example, the original and classically elegant studies by Schultz and colleagues that provided the impetus for temporal difference models of dopamine function, typically trained and tested monkeys in a relatively constant state of moderate thirst. The thirst level was induced by removing water for about 20 hours before each training and test session (Schultz et al., 1992; Schultz et al., 1993; Mirenowicz & Schultz, 1994). Always somewhat thirsty, the monkeys always found fruit juice UCS to be highly valuable, and always found CSs to be highly and equally motivating once the Pavlovian association was learned. When physiological motivation parameters are clamped into such a narrow range across training and testing phases of an experiment, then only learning is left as an input factor to control changes in the motivational impact of a CS. Under those conditions, a CS-triggered incentive salience signal will track learning inputs, and will appear to be a purely learned prediction signal even if actually a motivation output. Firing that moves from UCS to a CS looks like pure learning, but equally well could represent movement of the onset of a surge of incentive motivation. Likewise, negative decrements in firing to an expected but omitted UCS could reflect motivational disappointment as easily as a negative teaching signal. That is, dopamine firing that codes Pavlovian incentive salience might often masquerade as coding learned prediction errors – at least when physiological states are kept constant across training and test. The same principal applies to neuroimaging studies of prediction error coding in humans: while not thirsty, participants are usually in an ordinary physiological state that does not much vary across phases of learning and testing.

Real life is not so constant. Physiological states often do change between thirst, hunger and satiety, between undrugged and intoxicated, or medicated vs unmedicated, normal vs. sensitized, etc. Fortunately, some experiments have begun to explore the effect of shifting a physiological state on mesocorticolimbic output signals. Those results will be described below, but before that perhaps it is best to say a word about how dopamine manipulation experiments compare to phasic dopamine firing experiments. In experiments using drugs to manipulate kappa state below, the level of dopamine is pharmacologically stimulated to higher levels tonically over minutes to hours. By contrast, the brain is normally more frugal in dopamine use: as suggested for example by the phasic dopamine firing elevations originally described by Schultz, and confirmed in results of others, the normal brain seems to saves dopamine elevations for brief phasic moments of actual CS or UCS onset (Schultz et al., 1997; Roitman et al., 2004; Aragona et al., 2009; Flagel et al., 2011; Glimcher, 2011). That natural phasic pulse is well timed to facilitate interaction with cue-triggered glutamate signals at the same moment. But the end consequence may be the same: namely, elevating kappa in the Zhang equation, and so the level of incentive salience triggered by the learned reward cue at the moment of re-encounter.

Manipulating kappa state with natural appetites and satiety

One way to approach the question of whether neural CS signals reflect pure learning or transformed motivation is to manipulate κ by creating a new physiological appetite or satiety state during the CS test, or by giving drugs or brain manipulations to create a new stimulated dopamine-related κ state or a new suppressed dopamine state. By a prediction error or related model of pure learning, the CS value on first test should equal its previously learned value. By the incentive salience hypothesis of dopamine function, these manipulations may alter κ state in the Zhang equation to change CS value.

For proof of principle, the emphasis must be on new κ states. Old familiar states, learned about from experience, are too ambiguous for deciding the question. Advocates of dopamine=learning hypothesis might prefer to interpret a shift between two familiar states as not shifting κ state but rather as shifting from one learned context or occasion setter to another, which may conditionally gate learned CS value via contextual modulation of cached prediction errors gathered in the past (i.e., a different cache for each state). So let’s escape that argument about familiar hungers, satieties or drug states, and instead turn to stronger evidence for testing κ state shifts: new physiological states that are completely novel, never having been experienced before in an individual’s life and so that carry no learned information at all.

Proof of principle: novel salt appetite

As a novel yet natural physiological state, my colleagues and I have especially favored induction of salt appetite in experiments to manipulate kappa (Tindell et al., 2009; Robinson & Berridge, 2010). A state of salt appetite is quite novel for most individuals, because most modern humans and laboratory rats have grown up eating salted diets that contain more than enough NaCl, and so have never experienced a strong sodium depletion state in their lives. Those people and rats have absolutely no learned information about how the physiological state would change salty reward values. Yet sodium deficiency was frequently encountered by our ancestors and is today by many wild animals, and mesolimbic-related brain circuits can robustly generate salt appetite under particular hormonal conditions that mimic aldosterone stimulation to activate brain limbic circuitry (especially when facilitated by simultaneous angiotensin II stimulation) (Fluharty & Epstein, 1983; Schulkin, 1991; Lucas et al., 2003; Liedtke et al., 2011). That hormone combination can be mimicked by injections of deoxycorticosterone and furosemide, inducing robust salt appetite within 24 hours. Thus a novel salt appetite provides a strong test as proof of principle for the hypothesis that a κ state shift can directly transform the level of incentive salience elicited by a relevant CS. The same motivation principles, once clarified with the novel natural appetite, can be seen to apply also to more familiar appetites and to drugs of abuse that act on mesocorticolimbic circuitry.

Our studies have used novel salt appetite to ask whether a physiological shift in κ state, never before been experienced in the life of the individual, can appropriately transform the incentive salience attributed to a CS, which was previously associated with intense saltiness UCS, on first re-encounter with CS alone (Tindell et al., 2009; Robinson & Berridge, 2010). The taste of intense saltiness is normally unpleasant at NaCl concentrations much above isotonic bodily level (bodily concentration is just less than 1% NaCl [0.9%; 0.15 M]). A mouthful of seawater that is three times bodily levels is generally perceived as nasty (3% NaCl). A mouthful of much more concentrated 10% NaCl solution that is ten times bodily levels is nastier still. Ten percent salinity is roughly equivalent to concentration of NaCl in the Dead Sea (Dead Sea = 10% NaCl + 20% other salts), and 10% NaCl is the taste concentration we have used for a salty UCS. On training trials in a normal state, each auditory or lever CS is immediately followed by a squirt of 10%NaCl solution into a rat’s mouth. For such an intensely salty UCS squirt, rats normally gape and flail their forelimbs to the taste as aversively as if the squirt were bitter quinine (Tindell et al., 2006). Accordingly, rats quickly learn to turn away from a CS that predicts the Dead Sea taste (Robinson & Berridge, 2010). When a strong salt appetite state is induced, the UCS taste of intense saltiness suddenly becomes pleasant, no longer eliciting gapes and instead evoking positive facial expressions such as lip licking, just as does the taste of sucrose (Berridge et al., 1984; Flynn et al., 1991; Tindell et al., 2006).

But what if the rats that learned the CS-UCS association as nasty aren’t allowed to retaste the NaCl as a nice UCS in the new κ state before re-encountering its CS? What if they are simply presented again with the CS (alone, in extinction), which has always predicted unpleasantness on many occasions in the past? This question can be answered either from the brain’s point of view, by monitoring neuronal signals from mesocorticolimbic outputs, or from the psychological point of view, by monitoring motivated behavior toward the CS. Both answers give the same conclusion: CS incentive salience is directly transformed in accordance with the Zhang equation, so that the Ṽ(s_t) interaction becomes equivalent to (r_t + log κ) (Tindell et al., 2009; Robinson & Berridge, 2010).

From the brain’s point of view, mesocorticolimbic signals of CS value from the nucleus accumbens travel most heavily to the ventral pallidum (VP) (following classic projection patterns of striato-pallidal ordering) (Swanson, 2005; Heimer et al., 2008; Thompson & Swanson, 2010). Firing of neurons in a ventral pallidum hotspot for reward was recorded by Amy Tindell and Kyle Smith in the laboratory of my Michigan colleague J. Wayne Aldridge (Tindell et al., 2009). Tindell and co-workers found that when rats are in a normal physiological state similar to training, VP neurons fire vigorously to a sweet-associated CS that predicts an always-’liked’ sucrose taste, but not to a different salt-associated CS that predicts intense 10% NaCl taste, which has always been ‘disliked’. However, on the first time the CSs are re-encountered in a novel state of sodium appetite (induced by hormone injections 24 hours earlier), the neurons suddenly fire as robustly to the salt-associated CS that predicts NaCl taste UCS as to the sweet-associated CS for sucrose UCS (Tindell et al., 2009). Even though the salty UCS taste has never yet been tasted by the rat as ‘liked’ in the novel κ state, the new κ state (κ >1) transforms Zhang-style the brain’s reaction to the previously learned r_t carried by CS for salt. Suddenly on that appetite day, the auditory CS for triple-seawater saltiness elicits intense neuronal firing peaks for the very first time, and with no Ṽ(s_t) need for re-learning about the associated UCS, and the CS incentive salience becomes transformed as (r_t + log κ).

From the psychological point of view, a CS for nasty salt that has been learned about as aversive in the normal body state also becomes instantly attributed with high incentive salience in the sense of becoming behaviorally attractive and motivationally ‘wanted’ upon first re-encounter in a novel state of sodium appetite (Krieckhaus & Wolf, 1968; Fudim, 1978; Berridge & Schulkin, 1989; Stouffer & White, 2005; Tindell et al., 2009; Robinson & Berridge, 2010). For example, Dr. Michael Robinson in our laboratory showed that a lever CS for Dead Sea saltiness, which always elicited avoidance during training, becomes suddenly and intensely ‘wanted’ as rats avidly approach and attempt to ‘consume’ a metal CS that predicts a salty UCS on its very first encounter in the new κ >1 state (Robinson & Berridge, 2010). Behavioral attraction to CS was measured using an autoshaping or sign-tracking paradigm. Normally rats avoid a CS metal lever when they’ve learned its appearance predicts a UCS squirt of Dead Sea NaCl concentration into their mouth: a rat turns its face away and retreats a further distance. But on the first, presentation of the metal lever CS in the salt appetite state, rats eagerly approach and sniff and nibble the metal CS object with consummatory ingestive behaviors, even though they have never yet tasted the NaCl as nice in the new appetite state, and on all previous CS-UCS presentations in a normal physiological state the same rat had always avoided the CS object (Robinson & Berridge, 2010).

Thus a novel relevant kappa state can transform incentive salience into a highly positive value, behaviorally attractive and neurally able to activate mesolimbic circuits just like a sweet-associated CS that carries a positive r_t, from sucrose pairings, even for a salt-associated CS that has negative r_t. Though the CS was never paired before with a rewarding prediction error, but instead always with a punishing UCS, the CS instantly becomes an attractive motivational magnet when re-encountered in the new κ >1 state. Transformation of a repulsive CS into a ‘wanted’ CS provides the strongest possible proof of principle for incentive salience generation.

The general point is that Pavlovian motivation is not simply a passive readout of cached r_t values, but rather dynamically generated by integrating the CS r_t with a relevant κ state prevailing at the moment. It is impossible for a temporal difference model to have generated the positive status of a salt-associated CS on first re-encounter: the learned r_t from all previous prediction errors was negative. At that moment, a temporal difference model is weighed-down by a cache of accumulated learned aversion. To become positive, a TD model would require extensive re-training with the CS being multiply paired with the newly-’liked’ UCS in order to return from negative to neutral, plus more re-training beyond that to learn a new positive value. Even a model-based learning system would be hard pressed to generate any positive value without explicit instruction that a salty CS should become valuable in a sodium deficiency state, and no such instruction is available without experience. But incentive salience mechanisms have no problem in instantly reversing the Pavlovian CS value operating in a manner consistent with the Zhang model. Reversal is accomplished by combining a stimulus-stimulus sensory representation of salty UCS triggered by the CS, together with the incentive salience postulate that a CS takes on motivational properties of its UCS, including the property of ‘wanting’ and ‘liking’ being directly modulated by a relevant physiological state (Bindra, 1978; Toates, 1986; Berridge, 2001).

Human demonstrations of appetite modulation

In humans, a related (κ, r_t) interaction by physiological appetite/satiety state might apply to demonstrations by Farooqi and O’Rahilly and colleagues, who have studied brain activations elicited by food cues in leptin-deficient people (Farooqi et al., 2007; Farooqi & O’Rahilly, 2009). These individuals have a genetic inability to generate the satiety hormone leptin, and consequently grow up obese after a childhood spent demanding and consuming excessive quantities of food. The crucial observation is that these individuals show entirely different patterns of mesolimbic neuroimaging activations to food cues when they are given exogenous leptin medication as adults. Without leptin, they show an always-hungry type brain signature of intense mesocorticolimbic activation when viewing foods, whether they are actually hungry or have recently eaten, unlike an ordinary person who would show intense activation when hungry but reduced activation after satiety. Of course, a normal person might have learned to suppress neural activation when sated due to repeated experiences that food is less rewarding when full, so suddenly giving leptin to a leptin-deficient adult is especially instructive in nearly the same way as novel salt appetite. Leptin administration, when combined with eating a full meal, dampens the level of mesocorticolimbic activation induced by food cues (Farooqi et al., 2007; Farooqi & O’Rahilly, 2009). That is, exogenous leptin essentially opens a gate to more normal modulation by a full stomach, allowing the physiological satiety signals from recently eating to dampen mesolimbic brain response to food cues in a more normal fashion. This interaction might be understood by positing the combination of leptin and meal satiety signals to reduce κ to <1, suppressing the ability of food cues to elicit excessive incentive salience, after a lifetime spent in a state in which κ ≫1 always for food cues.

Drug modulation of CS incentive salience: animal evidence

Drugs of abuse tap into κ–state amplifications of cue-triggered motivation that evolved originally for natural appetites. Drugs typically engage the multiplicative form of the Zhang equation, so that incentive salience V becomes transformed as (r_t * κ).

Dopamine and opioid drug potentiation of incentive salience can be seen in comparable experiments to the above, where (r_t * κ) amplifies the level of incentive salience triggered by a previously-learned CS. As in salt appetite, drug (r_t * κ) interaction can elevate ‘wanting’ upon first cue re-encounter in a drugged state even if that kappa state is totally novel (Wyvell & Berridge, 2000; Tindell et al., 2005; Smith et al., 2011). Likewise, enduring neural sensitization caused by a previous series of drug binges can similarly amplify CS-triggered levels of incentive salience on the next re-encounters (Wyvell & Berridge, 2000; Wyvell & Berridge, 2001);(Wyvell & Berridge, 2000; Tindell et al., 2005; Smith et al., 2011). Drug-induced κ–state amplification of incentive salience triggered by a previously-learned CS’s r_t can be seen from the brain’s point of view, in mesolimbic neuronal output signals, as well as from the psychological point of view in motivated behavior and human subjective ratings.

For example, from a brain point of view, drug amplification of cue-triggered mesolimbic motivation signals were found in the neuronal firing patterns of limbic output electrophysiological recording studies by Amy Tindell and by Kyle Smith in the Aldridge lab at Michigan. By measuring electrophysiological firing of ventral pallidum neurons that receive nucleus accumbens projections, they showed that neuronal signals carrying incentive salience triggered by a previously-learned reward CS were increased by microinjection into the nucleus accumbens of a drug that stimulates dopamine neurotransmission (amphetamine) or drug that stimulates mu opioid neurotransmission (DAMGO) immediately before test, as well as by enduring mesolimbic sensitization induced by a series of drug experiences interposed between Pavlovian training and the test (Tindell et al., 2005; Smith et al., 2011). Tindell and Smith further showed that amplified ‘wanting’ motivation could be teased apart from an unchanged learned prediction of the upcoming reward, which was embodied in separate neuronal signal. That separation of Pavlovian ‘wanting’ from learned prediction signals was accomplished by using a serial CS Pavlovian procedure, in which a reliable sequence of CS1-CS2-UCS reward events always occurred in a fixed pattern (Figure 3). The sequence is easily learned. In a temporal difference model that learns such a fixed sequence, prediction gradually moves backwards in time to the CS1 because prediction error increments back-propagate earlier and earlier to the first predictive stimulus, stopping at CS1 because there is nothing before that to predict occurrence of reward (Schultz et al., 1993; Glimcher, 2011). Likewise, in traditional information theory (Attneave, 1959), prediction value is expressed as the reduction of uncertainty or surprisal value (h) contributed by a cue (surprisal value expressed as h = log2 1/p; where p = probability of reward). The initial CS1 contributes a large prediction increment in the form of reduction of uncertainty (H) about whether a reward will come: from roughly 1 out of 14 chance of success in predicting reward before CS1 (log₂ h=3.88) in the Smith and Tindell studies to nearly zero uncertainty immediately after CS1 (h=0; implies nearly 100% prediction or certainty of reward) (Smith et al., 2011). By contrast, the redundant CS2 adds almost no new predictive information (uncertainty about UCS remains near h=0), yet CS2 is associated with maximal motivated responses, and carries high incentive salience, for example expressed in high levels of eager approach and sniffing of the dish where sucrose UCS will arrive. Each CS elicits its own firing signals in ventral pallidum neurons, and so the order of the serial CSs thus helps tease apart reward prediction from incentive salience signals. Dopamine stimulation in nucleus accumbens (caused by microinjections of amphetamine on the day of test) amplified neuronal firing signals of incentive salience elicited by CS+2, to levels 1.5 X normal levels, whereas neuronal prediction signals to the CS+1 remained unchanged (Tindell et al., 2005; Smith et al., 2011) (Figure 3).

Figure 3. Dopamine/opioid amplification of neural signals for incentive salience.

Neural signals of nucleus accumbens outputs were recorded in ventral pallidum. Two serial CSs had been previously learned by rats to predict sucrose UCS. Amphetamine (dopamine stimulation) or DAMGO (opioid stimulation) microinjections did not enhance prediction signal of upcoming reward elicited by first CS1 (which predicted CS2 and reward UCS with 100% certainty) (Smith et al., 2011). Drug stimulation did amplify >50% neural signals for incentive salience triggered by CS2 (associated with maximal motivation, though merely redundant predictor) microinjections. In a similar experiment (right), after Pavlovian training in a normal state, mesolimbic sensitization added incrementally to the acute effects of amphetamine on board in amplifying intensity of incentive salience triggered by sucrose CS (Tindell et al., 2005). Reprinted by permission.

In a similar way to amphetamine, opioid stimulation of nucleus accumbens, via microinjection of a mu opioid agonist into a cubic-millimeter hedonic hotspot in medial shell (which elevates ‘wanting’ as well as ‘liking’), multiplied incentive salience signal but not the prediction signal: phasic neuronal firing triggered by CS2 was amplified, but not firing to CS1 (Smith et al., 2011).

Further, more permanent mesolimbic sensitization as a model of drug addiction can be compared to these temporary ‘wanting’ signal amplifications caused by drugs on board at the moment of CS re-encounter. Sensitization enhances phasic neuronal signals for incentive salience in the same pattern as (r_t * κ), but just in a more persistent way (Tindell et al., 2005). Neural sensitization was induced in rats by a series of amphetamine drug binges weeks before, interposed between Pavlovian training and the crucial CS re-encounter test. Sensitization amplified the CS2 incentive salience neural signal but not the CS1 prediction signal (even though no drugs were left in brain at the time of sensitized test) (Tindell et al., 2005). Finally, when sensitization and amphetamine on board were combined together, by giving another drug injection to sensitized rats on the day of CS re-encounter, the two manipulations added cumulatively to produce a super-amplification of the neural CS2-triggered ‘wanting’ signal greater than either single enhancement alone (Tindell et al., 2005) (Figure 3). That extra amplification may explain why an addict who tries to take a single hit of a drug so often relapses into a binge of further drug consumption: a sensitized brain that perceives reward cues while intoxicated receives a one-two punch of excessive temptation signal to continue. The combined state would make relevant reward cues even more irresistible to an addict, and so more likely to precipitate a prolonged binge of further consumption.

All these amplifications of CS2 neural signals appeared full blown on the very first presentations of Pavlovian cues in the new κ>1 state, when the CS was presented by itself in extinction (without UCS). Indeed, when the UCS sweet reward was finally delivered on later trials, neither amphetamine-elevated dopamine nor sensitization amplified UCS-triggered neural signals, suggesting that teaching signals or prediction errors were not even amplified when CS-UCS pairings was allowed (Tindell et al., 2005; Smith et al., 2011). Only opioid stimulation of the hedonic hotspot in nucleus accumbens amplified neural firing triggered by subsequent tastes of sucrose UCS (and incidentally, the failure of amphetamine microinjections in the hotspot to do so provides further evidence that high tonic dopamine does not raise hedonic ‘liking’ nor amplify a UCS-triggered prediction error signal, if either of those were subsequently relayed through ventral pallidum) (Smith et al., 2011).

Yet perhaps these neuronal signals are more difficult to interpret in psychological terms than I have implied. Psychological confirmation could be desired. Does amplification of neural signals for incentive salience actually translate into higher cue-triggered consequences for real motivated behavior? The answer is yes, as indicated by several behavioral studies of cue-triggered ‘wanting’ in our lab. For example, using Pavlovian-Instrumental transfer (PIT) procedures, Cindy Wyvell found that amphetamine/sensitization procedures dynamically amplified the intensity of cue-triggered ‘wanting’ surges, in nearly exactly the same (r_t * κ) way as the neural enhancements above (Wyvell & Berridge, 2000; Wyvell & Berridge, 2001). For example, microinjection in nucleus accumbens of amphetamine just before the moment of cue re-encounter amplified the height of peak surges of increased effort to obtain the UCS that occurred each time its CS was presented and that lasted about a minute afterwards (Figure 4). An equivalent amplification of cue-triggered ‘wanting’ surges was produced by drug-induced neural sensitization experienced weeks before (but after Pavlovian and instrumental training), even if no amphetamine at all was on board during the CS test. Similar amplifications of cue-triggered ‘wanting’ can be produced by other elevations in kappa state, such as by raising mu opioid stimulation levels in nucleus accumbens or central amygdala, or by raising stress-related CRF levels in nucleus accumbens (Pecina et al., 2006; Peciña et al., 2006; Mahler & Berridge, 2011). Conversely, κ<1 suppression of the surges of cue-triggered ‘wanting’ in PIT can be rather selectively produced by administration of dopamine antagonists just before the CS extinction tests in rats (Dickinson et al., 2000; Ostlund & Maidment, 2011; Wassum et al., 2011). All these results illustrate a converging substrate in mesocorticolimbic circuits for state-induced modulations of Pavlovian incentive salience.

Figure 4. Behavioral consequences of dopamine-amplified ‘wanting’: CS triggers higher pulses of ‘wanting’ for reward UCS.

Cue-triggered ‘wanting’ for reward was assessed using Pavlovian-Instrumental transfer. Testing was in extinction (no sucrose UCS, so responding decreases as trial proceeded). Amphetamine microinjections in nucleus accumbens selectively amplified the peak height of cue-triggered pulses of increased motivation to obtain sucrose reward. Prior sensitization similarly produced selective amplification of CS+-triggered motivation to obtain reward (red; separate rats used for sensitization; sensitized rats tested here in the absence of amphetamine-on-board). The CS had never been paired with response of lever pressing prior to test, eliminating CS-press stimulus-response habit explanations. Amphetamine failed to enhance baseline levels of pressing (i.e., increasing pressing peak height, but not raising base plateau that peaks sit on). The pattern indicates dopamine stimulation did not increase any stable prediction of reward value throughout the session, but rather selectively amplified the phasic pulse of motivation elicited by each CS that lasted about a minute. Redrawn data from (Wyvell & Berridge, 2001); redrawn from (Zhang et al., 2009) by permission.

Drug modulation of CS incentive salience: human evidence

In humans, comparable evidence that dopamine stimulation produces a kappa amplification of the incentive salience attributed to reward cues at re-encounter has recently emerged.

Some of the most compelling new evidence comes from Parkinson’s patients, some of who show addiction-like symptoms when given dopamine-stimulating drugs. These Parkinson’s cases are especially compelling because such patients escape most of the usual learning-based alternative explanations that could be offered for ordinary addicts. For example O’Sullivan and colleagues reported that administering L-DOPA medication that promotes dopamine synthesis to Parkinson’s patients who show dopamine dysregulation syndrome (a syndrome involving various impulsive-compulsive motivation symptoms under medication) produced amplification of the incentive impact of reward cues. In this study, the reward cues were photos of tasty foods, sex images, drug scenes, gambling scenes, etc., and incentive impact was measured neurally as increases in mesolimbic dopamine elicited by cues (dopamine release assessed by raclopride displacement using PET neuroimaging). The visual cues elicited larger phasic increases in dopamine release in nucleus accumbens/ventral striatum when L-DOPA was administered to these patients (O’Sullivan et al., 2011). In my view, the L-DOPA administration can be viewed as raising their kappa value in the Zhang equation to κ>1, multiplying the photo-triggered level of incentive salience – and correspondingly the brain level of dopamine surges elicited by the cues.

Importantly, many of the photos were novel to the patients, and not their own idiosyncratic compulsion, and so not the product of associative learning by repeated CS-UCS pairings in their previous experiences. Yet dopamine pulses were still elevated when triggered by the novel photos. The dopamine rise was not merely pharmacological: by itself, L-DOPA administration alone failed to elicit the high surges of endogenous dopamine release in the absence of cues, just as amphetamine failed to increase baseline levels of limbic neuronal firing in the animal studies above. That is, high kappa by itself is not incentive salience. Cues are also needed. Likewise, cues alone were not very effective in raising dopamine in the absence of any L-DOPA. Rather the cues and L-DOPA were needed simultaneously, synergistically interacting to control dopamine release in a (κ * r_t) fashion (Evans et al., 2006). In an earlier study, Evans and colleagues reported such patients who excessively over-consumed their L-Dopa medication in an addictive-like manner showed high correlations between the intensity of dopamine release induced by L-Dopa and their subjective ratings of wanting-to-take-more-drug (Evans et al., 2006). In such cases, the sights and other sensations and acts related to drug taking might provide the cue that interacts with kappa state.

Likewise, Claassen and colleagues reported that Parkinson’s patients with related impulse control disorders who compulsively pursued hobbies, shopping or sex, under the influence of their dopamine-stimulating medications (direct D2–D3 receptor agonists) made riskier decisions to win money in a gambling game than when nonmedicated (Claassen et al., 2011). Those patients gambled by squeezing a handgrip to inflate a computer image of a balloon, which paid more money the larger it got but which also ran a greater risk as it grew of popping and losing all payoff. Dopamine agonist medication made the patients take riskier gambles, inflating the balloons more to the point of popping and increasing the risk of losing. Interaction between dopamine stimulation and risk taking during the game suggests that dopamine stimulation amplified kappa to increase incentive salience of the gamble, interacting with their individual vulnerability and with the exciting perception of the growing balloon (Claassen et al., 2011).

Not just cues for drugs and gambling, but also cues for natural food rewards may interact in a similar (κ * r_t) way. Drug-on-board can amplify human dopamine signals elicited by foods in obese binge eaters and in drug addicts. For example, several studies by Volkow and colleagues report that phasic dopamine increases elicited by palatable foods are amplified by giving the mild amphetamine-related stimulant methylphenidate just before test to obese binge eaters (Volkow et al., 2002; Wang et al., 2011). That is, under methylphenidate binge eaters display greater dopamine surges elicited by the tasty foods (measured in PET by raclopride displacement), than the same individuals show without methylphenidate (drug factor), and higher also than in other individuals even under the drug (individual difference factor; discussed below). Methylphenidate is a pre-synaptic dopamine agonist that promotes release similar to amphetamine. The drug could create a κ>1 neurobiological state to amplify a reward CS input’s translation into incentive salience at that moment. The higher surges in dopamine release were seen especially in the dorsal neostriatum elicited by tastes of their favorite foods, raising interesting possibilities for incentive salience mechanisms that spread above the nucleus accumbens. The higher dopamine surges were only revealed by methylphenidate being on board when the cues were presented, and not evident in binge-eaters without the dopamine boost, illustrating again the transforming power of the kappa state manipulation.

Individual differences indicate phasic dopamine signals incentive salience (not learning per se)

Brains are not all alike when it comes to (κ * r_t) amplifications of incentive salience. Individuals differ in their attribution of incentive salience to particular cues, and in their capacity to have cue-triggered ‘wanting’ amplified by drug-on-board or physiological appetites. Some individual differences arise innately; others may be induced later in life by experience (addictive drug sensitization; traumatic stress); and some innate differences may bias the susceptibility to further modification by later experience.

Several of the examples already discussed involved important individual differences. Obese binge eaters differed from other individuals in (methylphenidate* food cue) interaction, Parkinson’s patients with dopamine dysregulation or impulsive/compulsive syndrome differed from other patients in (L-DOPA/agonist * drug/gambling cue) interaction, and sensitized rats that had been exposed to a series of amphetamine binges differed from non-sensitized rats in (spontaneous/amphetamine-at-test * sucrose cue) interaction. Likewise, the incentive-sensitization theory of drug addiction posits that human drug users whose mesolimbic brain systems become sensitized by the drugs will be at risk to show compulsive ‘wanting’ to take drugs again, even if the drugs are not particularly ‘liked’, and even if the addict is no longer in withdrawal (Robinson & Berridge, 1993).

Such patterns suggest that trait-like individual differences exist in mesolimbic systems that contribute in persisting ways to κ states. Individual difference traits may interact with temporary physiological or pharmacological factors to determine final κ states that interact with reward cues to produce incentive salience.

Similarly, individuals with high reward-motivation personality traits (“I go out of my way to get what I want”; BAS traits) may also show higher neural mesocorticolimbic activations elicited by cues for palatable foods (Beaver et al., 2006). And individuals who report stronger urges to smoke elicited by cigarette cues compared to other individuals may also be the ones to report stronger urges to eat elicited by food cues (Mahler & de Wit, 2010). All of these individual differences might be thought of involving a more responsive κ factor in the Zhang (κ * r_t) equation, heightening mesolimbic attribution of incentive salience to particular cues in those individuals compared to other individuals.

A particularly elegant study by Flagel and colleagues that exploited apparently endogenous individual differences in rats has recently helped to tease apart the role of mesolimbic dopamine in reward learning versus incentive salience (Flagel et al., 2011). Flagel and colleagues exploited an individual difference that occurs spontaneously in rats when trained in Pavlovian ‘sign-tracking’ or ‘autoshaping’ situations. Some individual rats (sign-trackers) attribute intense incentive salience to the CS cue (insertion of a metal lever into a chamber through the wall) that predicts food reward UCS. For those individuals, the CS becomes attractive, and the metal lever is approached and eagerly sniffed and nibbled whenever it appears. Other rats (goal trackers) learn the CS-UCS association equally well, but do not approach the CS; instead goal-trackers show a Pavlovian CR of going directly to the dish location where the UCS will arrive, whenever the lever appears. Sign-trackers also ‘want’ the CS in the sense of being willing to work harder to obtain it (instrumental conditioned reinforcement), whereas goal-trackers may not work for the lever cue (Flagel et al., 2011).

Most crucially, Flagel and colleagues reported that sign trackers showed large phasic increases in dopamine levels in their nucleus accumbens when the CS lever was presented (measured by in vivo electrochemistry). However, goal trackers showed much less elevation in nucleus accumbens dopamine, despite having equally well learned that the CS predicted reward (Flagel et al., 2011). Flagel and colleagues’ concluded that dopamine surges reflected incentive salience of the CS rather than its learned prediction. This ‘lever-wanting’ conclusion was bolstered by their further finding that only the sign-tracking response was impaired by systemic dopamine blockade, while the goal-tracking response remained relatively intact (Flagel et al., 2011). These findings complement an impressive set of related studies from Terry Robinson’s laboratory that have shown sign-trackers to be more inclined to attribute high levels of incentive salience to discrete CSs for food or drug rewards, showing greater cue-triggered relapse of cocaine seeking or food seeking, and higher self-administration of cocaine UCS when the drug receipt is accompanied by those CS cues (Saunders & Robinson, 2010; Yager & Robinson, 2010; Lovic et al., 2011; Meyer et al., 2011; Saunders & Robinson, 2011). In general, sign-trackers seem to possess an important higher kappa bias to attribute incentive salience to a discrete reward CS, resulting in stronger ‘wanting’ attraction to the predictive cues.

Altogether, these results indicate that enduring individual traits and short-term drug intoxication states and natural appetites key in interactive ways to modulate the temptation power of reward CSs. Although learning, memories and learning-based predictions of future outcomes need not be changed at all by these states, they often powerfully change the motivation intensity triggered by a cued reward memory. All seem to fit a Zhang-style (κ, r_t) interaction of motivation potentiation, in the sense that the traits or states elevate κ levels, without needing to raise the learned prediction r_t of the Pavlovian CS-UCS link associated to the cue. The consequence is to raise the intensity of incentive salience Ṽ(s_t) at the moment of cue re-encounter while preserving prediction accuracy. The CS thus becomes a more potent trigger of temptation, eliciting stronger ‘wanting’ for the cue and its associated UCS reward even if the learned prediction of future reward has not altered.

Dopamine is unnecessary for learning or remembering rewards only for incentive ‘wanting’

Finally, it seems important to briefly mention an opposite negative source of evidence against the dopamine=learning alternative hypothesis: namely, evidence that dopamine is not needed to learn new reward values nor to remember previously-learned reward values. This further eats away at the idea that dopamine surges somehow are teaching signals or prediction errors that are causally needed for reward learning, rather than for incentive salience that motivates reward value.

Going back more than a decade, results of animal experiments have indicated that even a dopamine-free brain can learn quite normally about rewards. For example, Terry Robinson and I found that rats still learned about rewards after losing mesolimbic dopamine, due to 6-hydroxydopamine (6-OHDA) neurotoxin lesions that selectively destroyed 98% of dopamine neurons in nucleus accumbens and neostriatum (Berridge & Robinson, 1998). The rats were still able to learn a Pavlovian devaluation of a sweet reward induced by taste aversion learning as competently as intact rats. After dopamine lesions, the rats were presented with a novel and distinctively sweet CS that they initially ‘liked’: the taste of a saccharin and polycose solution infused into their mouths. On first infusion, the sweet CS elicited normal high numbers of purely positive ‘liking’ facial expressions (e.g., licking of the lips) whether rats had no dopamine or were normal (i.e., more evidence that dopamine is not needed for normal hedonic ‘liking’). To next train a Pavlovian taste aversion, the CS sweet flavor was paired associatively several times with nausea as UCS, caused by injection of lithium chloride (LiCl). Finally, to test if the associative devaluation had been successfully learned, the CS flavor was presented by itself in a final test. All rats showed they had learned a complete devaluation of CS from nice to nasty: all the rats now emitted intense disgust reactions (e.g., gapes) in the post-training test, and their original positive ‘liking’ expressions had nearly vanished (Figure 5). The 6-OHDA-lesion rats with dopamine-free brains learned the Pavlovian ‘dislike’ for CS taste as strongly as the intact rats that had 100% normal dopamine (Berridge & Robinson, 1998). However, of course, the 6-OHDA rats lacked any incentive motivation to eat or drink, and would have starved voluntarily despite being surrounded by palatable food (and despite apparently retaining basic capacity for eating movements though having motor symptoms of Parkinson’s disease) if they had not been intra-gastrically fed. Without dopamine, the 6-OHDA rats had normal ‘liking’ and learning, but no ‘wanting’.

Figure 5. Normal learning by rats without dopamine: Pavlovian reward devaluation.

After rats lost >98% of dopamine concentrations from nucleus accumbens and neostriatum, due to 6-OHDA lesions, a Pavlovian taste-aversion learning was used to devalue a sweet CS flavor. Tastes of a distinctive and originally palatable saccharin-polycose solution were associative paired as CS with injections of LiCl to induce nausea as UCS. On the first presentation, the CS taste elicited purely positive ‘liking’ facial expressions (e.g., rhythmic lip licking) in a taste reactivity test. The affective reactions of rats with dopamine-free brains were essentially as hedonically positive as normal control rats. After 3 Pavlovian training pairings with LiCl illness, affective expressions to the sweet CS taste were reversed in valence. Negative disgust reactions (e.g. gapes) were elicited by the CS after learning, indicating a learned shift to ‘disliking’. The magnitude of the learned devaluation was equivalent in dopamine-free 6-OHDA rats and normal control rats. Thus, rats with virtually no dopamine in their brains were still capable of learning a new value for a reward. Data redrawn from (Berridge & Robinson, 1998).

Positive upshifts in reward learning also have been reported to occur in the absence of brain dopamine, most notably by Richard Palmiter and colleagues using dopamine-deficient (DD) mutant mice (Cannon & Palmiter, 2003; Hnasko et al., 2005; Robinson et al., 2005; Hnasko et al., 2007). Those DD mice congenitally lack the gene for enzymes to produce dopamine. Consequently DD mice show full blown symptoms of Parkinson’s disease soon after birth (and so require daily injections of L-DOPA medication to enable brief meals to survive). However, despite lacking dopamine, DD mice still appear able to learn some basic Pavlovian reward associations, such as between a CS drinking spout and UCS liquid sucrose reward (so as to prefer the CS spout later), or between a CS place and drug UCS reward even in absence of L-DOPA (so as to return to the place later) (Cannon & Palmiter, 2003; Hnasko et al., 2005; Robinson et al., 2005; Hnasko et al., 2007). DD mice also can learn an instrumental maze or spatial map association needed to find food reward, at least with the assist of caffeine (Robinson et al., 2005), and all presumably without any phasic dopamine teaching signals.

Of course, advocates of dopamine-learning theories may reply that only some forms of reward learning require dopamine according to their hypothesis, not all learning, and so these tests may have missed the particular forms of learning that need dopamine. That is indeed possible, but what particular forms of learning would those advocates suggest remain to need dopamine? Pavlovian reward learning was the original source of the dopamine prediction error hypothesis, and the temporal difference model is frequently presented in the context of Pavlovian predictions of reward (Schultz et al., 1997; Dayan & Balleine, 2002; O’Doherty et al., 2003; Glimcher, 2011). But Pavlovian learning of reward re-valuation seems quite possible without phasic dopamine prediction errors. If not for Pavlovian reward learning, then for what learning is dopamine needed?

Still, I must grant that dopamine-free DD mice have deficits in avoidance learning tests and in learning instrumental lever-press tasks (Fadok et al., 2009; Darvas & Palmiter, 2010; Darvas et al., 2011). Also dopamine manipulation by drugs have been reported to change learning and learned performance in people (Pessiglione et al., 2006; Moustafa et al., 2008; Samson et al., 2010). However, I suspect many of these learning modulation reports have alternative explanations in the form of motivational/attentional/effort effects, which only indirectly influence learning, and might not actually change learning mechanisms after all (Salamone et al., 2007; Aarts et al., 2011). Most human studies so far that have reported dopamine blockade to interfere with learned performance, or psychostimulants to enhance performance, never explicitly asked whether their apparent change in learning might be alternatively explained by a drug-induced change in motivation or attention. Instead they simply ask whether acquisition of the learned task was altered by the drug, and take any positive answer to be support for the prediction error model, without considering alternative explanations. That sets the evidence bar too low.

Fortunately a few human studies now are beginning to compare alternative explanations when evaluating prediction error hypotheses. As just a couple examples, Nagy and colleagues recently reported that administering direct dopamine D2/D3 agonist medications to Parkinson’s patients helped them to learn a reward prediction task (Nagy et al., 2011). However, the learning improvement appeared indirect and mediated secondarily by a more direct enhancement of incentive salience attributed to reward predictive cues. That was revealed in the finding that even some irrelevant or nonpredictive cues were attributed with aberrant incentive salience after medication, leading to illusory associations and correlated also with psychotic-like symptoms of hallucinatory experiences and perceptual aberrations. The results led Nagy and colleagues to conclude that “dopamine agonists do not selectively enhance adaptive reward learning, but increase general salience signals leading to illusory and aberrant stimulus–reward associations (p. 8) (Nagy et al., 2011).

Similarly, Sharot and colleagues gave L-Dopa to normal people before asking them to imagine a potential future vacation in Greece or Thailand, and on another day asked them to imagine the alternative vacation without drug (Sharot et al., 2009). Later, when asked to choose which vacation they would prefer, participants picked the one they had imagined while under L-Dopa, and guessed that they would be happier in the drug-associated location than in the other location. Sharot et al. noted that their results suggested that “dopamine… enhanced a prediction of pleasure associated with a future event”, perhaps because it had strengthened learning of an association on the drugged day, consistent with increased prediction error (Sharot et al., 2009). But, alternatively, they noted their results could equally have been explained by the hypothesis that “administration of L-Dopa might be construed as enhancing incentive salience attributed to imagined stimuli” (Sharot et al., 2009). Which explanation is correct? Sharot et al. concluded “The current study cannot distinguish between these two possibilities”. That is exactly right, and the authors deserve credit for acknowledging the interpretation remains an open question. After all, generations of students have taken amphetamine and related psychostimulant drugs to help them study and take tests. But most readers might think those drugs aided chiefly to keep the tired students interested, alert, attentive and quick of mind – not by mimicking phasic teaching signals or prediction errors in their mesolimbic systems. Similar recognition of ambiguities have led to recent suggestions that prediction error, learning and hedonia formulations of other pathologies such as schizophrenia or depression may be better reformulated as hypotheses based more on motivation and attention features of those conditions (Kapur, 2003; van Os & Kapur, 2009; Barch & Dowd, 2010; Buckholtz et al., 2010; Treadway & Zald, 2011).

Conclusion

Pavlovian CSs associated with reward must be dynamically translated into incentive salience each time the CS is re-encountered to motivate reward seeking and consumption. The translation of CS association into motivation is influenced by current neurobiological states of mesocorticolimbic systems, including dopamine states, in a Zhang-style (κ, r_t) interaction at the moment of cue re-encounter. Natural appetite or satiety states, stress states, pharmacological states related to addictive drugs and related medications, and individual differences, all powerfully modulate the κ factor in the interaction and so change the level of incentive salience produced by the r_t of the CS at that moment Ṽ(s_t). Such modulation helps explain fluctuations in the power of a given cue or context to trigger temptations. Of course, much more remains to be done to create models that more realistically capture the translation of learned knowledge into motivation. Exciting opportunities exist for future models to better incorporate neurobiological-state modulation of cue-triggered temptation, so as to better capture the reality of how mesocorticolimbic circuits generate motivation for rewards.