Appetitive conditioning: neural bases and implications for psychopathology

Abstract

Appetitive conditioning is the process through which new rewards are learned and acquire their motivational salience. Although it has the same evolutionary survival significance as aversive conditioning, appetitive conditioning has rarely been studied in humans. This gap may be explained by the difficulty to find in humans suitable appetitive stimuli that can elicit physiological responses similar to those elicited by aversive stimuli.

To help remedy this gap, we review the literature on conditioning, with emphasis on appetitive conditioning. This review comprises three parts. First, we examine the different forms of conditioning. Second, we review the neural basis of appetitive conditioning, particularly from a functional neuroimaging perspective. And third, we demonstrate how perturbations in processes involved in appetitive conditioning can be involved in psychopathologies and suggest neurobiological models underlying these pathologies.

The ultimate goal of this review is to stimulate new avenues of research that have direct links to molecular biology, and thus could prove to be invaluable to progress in the understanding and treatment of psychiatric disabilities.

1. Introduction

Appetitive conditioning is a form of associative learning and is the process by which new rewards are learned and are imbued of a motivational salience. In this process, neutral stimuli acquire a new motivational significance through their association with a reward. Understanding appetitive conditioning is important for elucidating mechanisms of both learning and motivational processes. Such knowledge can help foster potentially critical research on the pathophysiology of psychiatric conditions that manifest perturbed motivation.

Although the conditioning of appetitive stimuli has the same evolutionary significance as the conditioning of aversive stimuli, appetitive conditioning has rarely been studied in humans, in contrast to the large literature on aversive conditioning. This gap between aversive and appetitive conditioning can be explained by the difficulty to find suitable appetitive stimuli that can elicit a physiological activation similar to the one elicited by the painful or fear stimuli used in aversive conditioning.

The aim of this work is to provide an overview of the research on the neural bases of appetitive conditioning and to integrate the findings obtained in humans, mostly from functional neuroimaging studies, with the animal literature. Furthermore, we hope to stimulate interest in these processes as a line of research in the neurobiology of psychopathology.

This review is organized in three sections: 1) Description of the major forms of conditioning, including classical, evaluative and operant conditioning; 2) Presentation of the current state of knowledge regarding the neural bases of appetitive conditioning; 3) Implications for understanding the neurobiology of psychiatric diseases that are characterized by core symptoms of perturbed motivation.

2. Appetitive conditioning: Definitions and types of conditioning

2.1. Classical conditioning

In classical conditioning, a neutral stimulus (NS) is repeatedly paired with an unconditioned stimulus (US). Through this repeated association, the NS will trigger the same reaction as the US; in other words, the unconditioned reaction (UR) that was produced by the US is now elicited by the NS. The NS is then called conditioned stimulus (CS) and the reaction to the CS is called conditioned reaction (CR) (see figure 1A). ‘Unconditioned’ means that the stimulus and the response are naturally connected, i.e., the association is not learned.

Figure 1.

Classical conditioning

A. Basic process of classical conditioning: through the repeated association of a neutral stimulus (NS) with an unconditioned stimulus (US), the neutral stimulus will produce the same reaction as the US. The unconditioned reaction (UR) that was produced by the US is elicited by the NS after the conditioning process and is called conditioned reaction (CR).

B. Second-order conditioning: Once the learning process has taken place and the CS is able to elicit the CR, the CS can be paired with a new neutral stimulus, which will then also elicit the CR. CS1 refers to the CS, which is originally paired with the US and can elicit the CR1, which was originally elicited by the US. CR2 refers to the second CS, which is able to elicit the CR1 through its association with the CS1. However, the response evoked by the second order CS is not necessarily the same as that elicited by the first order CS, i.e., CR1. Often no overt response is evoked or the form of the CR can be very different if the CS 1 and CS 2 are of different modalities (Gewitz and Davis, 2006).

One main condition for classical conditioning is that the US elicits an innate reflex, the unconditioned reaction, and that the unconditioned reaction occurs at a physiological level (Pavlov, 1927). Another important condition is that the US and CS must be presented in close temporal proximity (Pavlov, 1927). This way, the CS will predict the occurrence of the US and the subject learns that both stimuli are linked (Hamm and Vaitl, 1996). Pavlov’s famous experiments on classical conditioning in dogs used salivation as the innate physiological response (UR) to the presentation of food (US). He paired repeatedly and in closed temporal proximity the sound of a bell or a flashing light (NS) with the delivery of food, and showed that the sound of the bell or the flashing lights alone elicited salivation (CR) (Pavlov, 1927). However, the pairing between US and CS alone is not sufficient to produce conditioning. According to contemporary theories of learning the critical factor is whether the CS signal has a predictive relationship to the US (Rescorla and Wagner, 1972). As Rescorla (Rescorla, 1966) initially showed, pairings can be offset by enough CS-alone and US-alone trials that degrade the information value of the CSs.

Once the learning process has taken place and the CS is able to elicit the CR, the CS can be paired with a new neutral stimulus, which will then also elicit the CR. This process is called second order conditioning (figure 1B) and is at the core of the acquisition of new rewards, an essential aspect of adaptive motivated behavior.

A number of conditions modulate the strength of the association between CS and US. Among them, extinction, latent inhibition, and devaluation are the most relevant to potential sources of disturbances in the emergence of psychiatric symptoms.

Extinction occurs when the CS is repeatedly presented alone (without US). This leads to a decline in the elicitation of the CR (Mackintosh, 1983). Latent inhibition refers to the delay in learning when either the CS or US has been presented prior to the CS-US conditioning trials (Lubow, 1995). The occurrence of the CS-to-be without the presence of the US progressively reduces the mobilization of attentional resources directed towards the CS-to-be, which becomes irrelevant. This disturbs the ability of individuals to notice that the CS-to-be is occurring in the presence of the US, which delays the formation of the association. Finally, devaluation occurs after conditioning, when the significance of the US has changed (Everitt et al., 2003a). This process is one of the most studied features of associative learning because it permits to identify neural changes associated with discrete and specific manipulations of the process of conditioning.

Originally, classical conditioning was conceived as the establishment of new units of behavior (Pavlov, 1927). In the last century, alternative models have been proposed. One of the current models conceptualizes classical conditioning as the acquisition of associations using representations of stimuli stored in memory (Holland, 1990). Signal learning is another model. The first component of signal learning is the acquired association between CS and US, which involves the acquisition of a propositional-declarative knowledge, i.e., a conscious knowledge that can be verbalized, about the association between US and CS. This way, the CS becomes a signal for the US and generates an active expectancy for the occurrence of the US. This mechanism allows individuals to detect reliable predictors for significant environmental events. The second component of signal learning refers to discriminative learning (Dawson and Shell, 1986, Davey, 1987), which introduces the concepts of CS+ and CS−. Discriminative learning is a learning process, in which the occurrence of a specific stimulus, the CS+, predicts the immediate occurrence of a positive or negative event, and other stimuli, the CS−, predict the non-occurrence of this event. The principles of discriminative learning are represented in figure 2.

Figure 2.

Discriminative conditioning

In discriminative learning, the occurrence of a specific stimulus, the CS+, predicts the immediate occurrence of a positive or negative event, i.e., the US, which is in turn associated with an unconditioned response (UR) and another stimulus, the CS⁻, predicts the non-occurrence of this event. After conditoning, the CS⁺ can elicit the same reaction as the US, now called the conditioned reaction (CR), while the CS⁻ does not elicit this response.

The most current view of classical conditioning postulates a multilevel process, based on the multi-dimensional characteristics of the US and UR. The CS is thought to generate a number of associations through its pairing with the US, which are possibly established through different neural mechanisms. For example, a US can trigger a set of responses, including preparatory responses (e.g., orienting reaction) and consummatory responses, (e.g., salivation to food). Furthermore, the CS holds associations with the specific and distinct properties of the US, such as its physical and affective characteristics. These characteristics are coded by different neural systems and their representations at the neural level are presumed to be stored in different brain regions. As mentioned above, the discrete manipulation of the features of the US after conditioning is a powerful tool to study the various neural systems involved in classical conditioning.

2.2. Evaluative conditioning

Evaluative conditioning (EC) is the process via which the valence of a stimulus (positive, negative, or neutral) can be transferred to other stimuli when these are repeatedly presented together (figure 3). In other words, evaluative conditioning can be seen as the learning process through which likes and dislikes are conditioned. Although first introduced by Martin & Levey (Levey and Martin, 1975, Martin and Levey, 1978), this concept has been described several decades ago (Watson and Rayner, 1920, Razran, 1954, Staats and Staats, 1957).

Figure 3.

Evaluative conditioning

The valence of an unconditioned stimulus (US), described as unconditioned valence (UV) in the figure, can be transferred to a neutral stimulus (NS), when it is repeatedly presented together with the US. The NS is then associated with the valence of the US and becomes a conditioned stimulus (CS) with a conditioned valence (CV).

Classical conditioning and evaluative conditioning differ on several points. The fundamental difference resides in the nature of the US. By definition, in classical conditioning, the US is biologically significant and elicits a physiological reflex, whereas in evaluative conditioning, the US is characterized by its valence, positive or negative, which is transferred to an initially neutral stimulus (De Houwer et al., 2001). According to De Houwer et al. (De Houwer et al., 2001), evaluative conditioning typically uses second-order CS as US or US that are not biologically relevant.

Other differences of EC compared to classical conditioning include principally 1) its resistance to extinction (Baeyens et al., 1988, Baeyens et al., 1989, Baeyens et al., 1995a, De Houwer et al., 2000), 2) the weaker influence of the number of pairings between CS and US (Bayens et al., 1993, Bayens et al., 1996, Bayens et al., 1998), and 3) the independence of contingency awareness (De Houwer et al., 1997, Field, 2000, De Houwer et al., 2001).

Based on these differences, Baeyens et al. (Baeyens et al., 1995b) suggested that distinct processes are engaged in classical and evaluative conditioning, a view not unanimously shared (Davey, 1994, Field and Davey, 1999). According to Baeyens et al. (Baeyens et al., 1995b), the CS-US association would rely on a referential system in evaluative conditioning and an expectancy system in classical conditioning. More specifically, the CS in classical conditioning would activate the US representation and generate an expectation of the occurrence of the US (expectancy system), whereas the CS in evaluative conditioning would activate the US representation without generating an expectation (referential system).

Most EC studies found a similar learning effect for pleasant and unpleasant US. However, some EC studies of taste preferences reported a learning effect only for aversive US, and not for positive US (Bayens et al., 1990, Rozin et al., 1998). This observation could be related to the difficulty in finding pleasant taste stimuli which are liked as intensely as the unpleasant stimuli are disliked.

2.3. Operant conditioning

The concept of operant or instrumental conditioning has been introduced by Skinner (Skinner, 1937), who distinguished type S and type R conditioning. The type S conditioning matched classical conditioning, which is characterized by a reinforcing stimulus (e.g., food) contingent upon a stimulus (e.g., the bell). The type R conditioning referred to operant conditioning, which is characterized by a reinforcing stimulus (e.g., the food) contingent upon a response (e.g., press lever) (see figure 4).

Figure 4.

Operant conditioning

In operant conditioning, a reinforcing stimulus, the consequence (C), e.g., food, is contingent upon a response (R), e.g., press lever, and not upon another stimulus as in classical conditoning. The consequence can be reinforcing (C^R) and can increase the probability of occurrence of a response as illustrated in A; or the consequence can be punishing (C^P) and can decreases the probability of occurrence of a response.

The central mechanism underlying operant conditioning is reinforcement. For Skinner, a stimulus is reinforcing when it has the power to enhance the probability of occurrence of a response (Skinner, 1938). Positive reinforcers, such as rewards, increase the probability of a response when they are presented in a temporal relation with the stimulus or response, whereas negative reinforcers increase the probability of a response when they are omitted. Reinforcers can consist of stimuli whose motivational value has previously been learned by means of classical conditioning. In such instances, classical conditioning interacts with operant conditioning, and can influence goal-directed behavior.

Reinforcers are distinguished from punishment. Punishment decreases the probability of occurrence of a response. It involves the presentation of a negative consequence to a behavior, or the omission of a positive reinforcer following the behavior.

In summary, the learning processes involved in classical conditioning are centered on the contingency between two stimuli, whereas those involved in operant conditioning are centered on the contingency between a stimulus and a response. Presently, we will focus on appetitive conditioning through the review of studies of classical and evaluative conditioning. We will address findings of studies using operant conditioning, when they contribute to the understanding of the mechanisms of appetitive conditioning and its role in maladaptive behavior.

3. Neural bases of appetitive conditioning

The neural bases of appetitive conditioning have been investigated extensively in animal studies, using electrophysiological (single neuron recording) and neurobehavioral (lesion) methodologies. Findings suggest that a neural network including the anterior cingulate cortex, the orbitofrontal cortex, the amygdala, the nucleus accumbens and the mesolimbic dopamine system underlies Pavlovian approach behavior (Everitt et al., 2000, Schoenbaum et al., 2003b).

Few neuroimaging studies in humans have examined the neural correlates of appetitive conditioning. Many of them involved evaluative conditioning rather than classical conditioning. These studies used conditioning to different valences (pleasant and unpleasant) with first-order US, such as odor (Gottfried et al., 2002) and taste (O’Doherty et al., 2003), or second-order US such as money (Kirsch et al., 2003, Cox et al., 2005). Other works employed classical conditioning. These studies assessed US that are associated with a reflex behavior such as pain relief (Seymour et al., 2005), or juice delivery (McClure et al., 2003, O’Doherty et al., 2004, O’Doherty et al., 2005). With the exception of the functional magnetic resonance imaging (fMRI) work by McClure et al. (McClure et al., 2003), all these studies assessed how well subjects could discriminate between two learned stimuli, or CS (discriminative conditioning paradigm, see figure 2).

Human and animal studies revealed discrepant results. Species, types of conditioning, and methodology could account for these differences. Particularly, the use of evaluative conditioning in humans and classical conditioning in animals for the study of appetitive associative learning may be a critical source for these discrepancies, suggesting that distinct mechanisms may underlie these processes.

3.1. The amygdala: Attentional processes and attribution of emotional value

The amygdala is one of the most consistently investigated regions in the processing of associative learning and seems to be involved in the assignment of emotional significance to events and in attentional processes. Animal studies show that the different nuclei of the amygdala, in particular the basolateral (BLA) and the central nuclei (CeN), could serve different subsystems and be involved in distinct aspects of the learning process. The role of these nuclei appears to differ in appetitive and aversive learning.

The BLA is involved in the conditioning of the affective value of the US in appetitive conditioning (see figure 5) and the encoding of the CS-US association in aversive conditioning (Everitt et al., 2003a, Everitt et al., 2003b, Gabriel et al., 2003). In contrast, the CeN may be involved in the encoding of pavlovian stimuli-responses associations (SR) in appetitive conditioning (see figure 5) and in the promotion of the expression of the conditioned reactions (CR) in aversive conditioning.

Figure 5.

Model of the neural bases of appetitive conditioning. This model summarizes the findings of animals and human studies about the neurocircuitry involved in appetitive conditioning. This model is far from being exhaustive. The absence of connections between two regions doesn’t mean that these regions are not connected, but simply that the role of these connections in the network supporting appetitive conditioning is either not known or still unclear. We describe here only regions and connections, whose functions have been thoroughly investigated and are well established.

ACC: anterior cingulate cortex, BLA: basolateral amygdala, CeN: central nucleus of the amygdala, Nacc: nucleus accumbens, OFC: orbitofrontal cortex, PCC: posterior cingulate cortex, VTA: ventral tegmental area.

The hypothesis of a differential role of the BLA and CeN in appetitive conditioning is supported by findings of studies in rats. Lesions of the BLA have been shown to impair processes related to the association between the CS and the value of the US, such as the acquisition of second-order CR and the effect of US devaluation on the responses to the CS (Hatfield et al., 1996, Setlow et al., 2002a). These results were confirmed by neurophysiological studies on discriminative learning, evidencing a firing of BLA neurons in response to changes in the outcomes predicted by odor cues (Schoenbaum et al., 1998, Schoenbaum et al., 1999). In contrast, lesions of the CeN impaired the acquisition of the association between the CS and the preparatory responses elicited by the US, such as first and second order orienting reactions (Hatfield et al., 1996). Furthermore, studies on autoshaping, a procedure measuring the approach behavior elicited by classical conditioning, showed that CeN lesions, but not BLA lesions impaired the acquisition of the conditioned approach reaction (Parkinson et al., 2000a), suggesting that the CeN is also involved in the association between the CS and the consummatory unconditioned response (UR) associated with the US.

Finally, lesions of the CeN, but not the BLA, impaired pavlovian-instrumental transfer, another process affecting the association between CS and UR. This association is thought to require the integrity of the core of the nucleus accumbens and its dopaminergic innervation (see below section on nucleus accumbens). Lesions of the CeN, but not the BLA, impaired the enhancing effect of intra-accumbens amphetamine infusions on responses to conditioned reinforcement (Burns et al., 1993b, Robledo et al., 1996). Furthermore, the intact functioning of the CeN is a condition for the increase of extracellular level of dopamine in the nucleus accumbens in response to food (Ahn and Philipps, 2003). Since the CeN has no direct connections to the nucleus accumbens, it is postulated that the CeN regulates the dopaminergic projection from the ventral tegmental area of the midbrain (VTA) to the nucleus accumbens through its own projections to the VTA.

In addition to its role in the acquisition of the CS-UR association, the CeN seems to modulate attentional aspects of stimuli through its projections to the reticular formation. The CeN also appears to regulate of the capacity of stimuli to be conditioned through its projections to cholinergic neurons in the nucleus basalis magnocellularis, and from there to the parietal cortex (Gallagher and Schoenbaum, 2003).

On the basis of these results, Everitt et al. (Everitt et al., 2003a) developed a functional model of the amygdala for conditioning. This model postulates that the BLA enables the CS (e.g., light) to access the affective value of its particular US (e.g., food) and that this information is used to control the CeN and its projections to produce a behavioral response. However, the CeN can directly encode stimulus-response pavlovian associations and influence the conditioned responses through its projections to the midbrain, hypothalamus and brain stem. This model, principally based on appetitive conditioning studies, differs from the conclusions led from the research work done on aversive conditioning, which indicates that the association between CS and US is coded in the lateral and basolateral nuclei of the amygdala, but not in the CeN. This information is then sent to the CeN, which promotes the execution of the CRs (Gabriel et al., 2003). The main differences between both models reside in the role of the CeN, which participates in the formation of the association between CS and UR and drives the CR in appetitive conditioning, but only participates at the execution of the CR in aversive conditioning (Everitt et al., 2003b).

Although animal research clearly established a role for the amygdala in appetitive conditioning, the results of neuroimaging studies in human are less consistent. Only two studies found activation in the amygdala in response to appetitive CS+ when compared to CS− (Gottfried et al., 2002, Seymour et al., 2005). Another study reported a sub-threshold correlation between neural activation in the amygdala and post-conditioning pleasantness ratings associated with cues (CS) that predicted the delivery of different juices (US) (O’Doherty et al., 2006). In contrast, studies using money as US did not find activation in the amygdala in the processing of appetitive conditioning (Kirsch et al., 2003, Cox et al., 2005). Therefore, the role of the amygdala in appetitive conditioning in humans remains to be clarified

3.2. The orbitofrontal cortex: Encoding outcome expectancies

Another key region in associative learning, especially in appetitive conditioning, is the orbitofrontal cortex (OFC). Anatomically, the OFC is strongly connected to the BLA, suggesting that both structures could work together for the acquisition of new associations (Carmichael and Price, 1995, McDonald, 1998, Saddoris et al., 2005). Recently, a renewed interest in the role of the OFC in appetitive conditioning has emerged in animal studies. Lesion studies suggest that the OFC could serve a dual function in associative learning (Schoenbaum and Roesch, 2005). On one hand, the OFC could encode outcome expectancies, and, on the other hand, it could facilitate associative learning in the BLA. Indeed, rats and monkeys with OFC lesions exhibit normal conditioned responses, but do not show a normal decrease of responses after food devaluation (Gallagher et al., 1999, Baxter et al., 2000). These results are similar to those reported in animals with BLA lesions. To disentangle the specific roles of both regions, Saddorris et al. (Saddoris et al., 2005) and Schoenbaum et al. (Schoenbaum et al., 2003a) examined the effects of OFC lesions on BLA function. OFC lesions were found to impair the outcome-expectant firing and associative encoding of BLA neurons. These results are consistent with the idea that the OFC encodes outcome expectancies, and that this information is then used to facilitate the associative encoding in the BLA (Schoenbaum and Roesch, 2005). In short, the BLA encodes the association between the CS and the current value of the US, which is represented in the OFC.

Most neuroimaging studies in humans find OFC activation during appetitive conditioning. A study on olfactory learning reported activation in the medial OFC in response to both aversive and appetitive CS+, when compared to CS− as well as a valence specific activation, in the anterior OFC for appetitive and in the lateral OFC for aversive learning (Gottfried et al., 2002). Studies using money, or just positive verbal feedback, as US found lateral OFC activation in response to CS+ (Kirsch et al., 2003, Cox et al., 2005), suggesting an implication of this region not only in learning, but also in the mediation of the value of the US regardless of its valence. Finally, a study showed that activation in the medial OFC followed the predictions obtained with the temporal difference model (O’Doherty et al., 2003). This model, based on neurophysiological studies (Schultz, 1998), posits that the prediction error (PE), i.e., the difference between expected and actual outcome, should occur at the US presentation before learning, but should shift to the CS during learning. These results support the role of OFC in learning processes.

Taken together, findings of human studies corroborate results of animal studies, and support the notion that the OFC is a central region for appetitive learning. At this point of our knowledge, it is too early to be able to specify subregional specializations within the OFC. More information will become available as new findings emerge from the continued research on regional specialization within the OFC (Ongur et al., 2003)

3.3. The anterior cingulate in discriminative learning

The anterior cingulate (ACC) has been repeatedly implicated in situations of conflict (Carter et al., 1998, Botvinick et al., 2004), which are manifest when one has to differentiate among alternatives, such as in discriminative processes. Accordingly, the ACC appears to be involved in discriminative learning (Parkinson et al., 2000a). Animals with ACC lesions are impaired in tasks involving the discrimination of multiple stimuli according to their association with reinforcers (Bussey et al., 1997, Cardinal et al., 2003). The loss of discrimination among CS is expressed by an increase in approach behaviors following CS− presentations. Studies of the neural bases of discriminative avoidance learning show enhanced ACC neuronal firing in response to CS during the early stage of conditioning. However, the late stage of conditioning, once discriminative responses are established, engages the posterior cingulate (Gabriel et al., 2003). These results suggest the existence of an early and a late system in the cingulate gyrus, which could reflect a primary and a secondary (post-consolidation) memory circuit (Gabriel et al., 2003). It is not clear, however, whether a similar differentiation exists for appetitive conditioning.

Interestingly, ACC neuronal firing in response to discriminative avoidance learning has been shown to be affected by amygdala lesions (Poremba and Gabriel, 1997). The CeN might be particularly involved as it influences the early acquisition of CR’s during discriminative avoidance learning, and modulates indirectly the anterior cingulate through its projections to the ventral tegmentum. On the other hand, the BLA seems to interact mostly with sensory and ACC regions to mediate the instrumental part of discriminative avoidance learning. As mentioned earlier, the BLA would help to maintain the affective value of the US representation on line (Gabriel et al., 2003).

Neuroimaging studies in humans are not as clear as the animal work on the role of the ACC. Two fMRI studies using money as US found ACC activation that was specific to the appetitive CS+ (Kirsch et al., 2003, Cox et al., 2005). The ACC was engaged during the presentation of a positive verbal feedback (US) vs. no feedback (Kirsch et al., 2003, Cox et al., 2005). Furthermore, the study using pain relief as US showed an activation of the genual ACC in response to aversive learning only (Seymour et al., 2005). The lack of ACC activation in response to appetitive conditioning in studies using non-monetary US (e.g., (Gottfried et al., 2002, Seymour et al., 2005) suggests that this region is involved in appetitive conditioning in humans only if the US represents a second-order, but not a primary reinforcer. A critical caveat with regards to a synthetic rendition of the function of the anterior cingulate function is the recognition of distinct roles of this structure as a function of topography. For the sake of space and clarity, and also because too many questions remain to be clarified, we will not detail these putative functional maps of the cingulate cortex.

Taken together, the results of the neuroimaging studies in humans seem to confirm the role of the ACC in discriminative appetitive learning, particularly in the processing of the appetitive cues. The potential role of the ACC in aversive learning processes, especially in avoidance learning as shown by (Seymour et al., 2005) is in accordance with the results of animal studies. As already mentioned, we do not specify the subregional specializations within the ACC (e.g., anterior, dorsal, subgenual ACC), which are still under investigation, and are not being addressed by all the studies reviewed herein.

3.4. The striatum: output regions and implication in second-order conditioning

The striatum, especially the nucleus accumbens (Nacc), a key region for the processing of rewarding stimuli in humans and animals (Knutson et al., 2001, Elliott et al., 2003, Schultz, 2004), is a prime candidate to be involved in appetitive conditioning.

Neurophysiological studies in rodents and primates have shown learning-related striatal neuronal firing in caudate nucleus, putamen and Nacc, consisting of increased response to CS predicting reward during the learning process. Such neuronal responses are strongly diminished after extinction (Tremblay et al., 1998, Aosaki et al., 1994b), and are modulated by dopamine (Aosaki et al., 1994a). Once learning is established, striatal neurons continue to respond to the CS that predicts reward in discriminative conditioning experiments (Kimura et al., 1984, Hollerman et al., 1998, Kawagoe et al., 1998, Shidara et al., 1998). Tonically active striatal interneurons (TANs), exhibit firing related to the coding of prediction errors, i.e., higher firing rate to unpredicted than to predicted rewards and no response when rewards are omitted (Schultz et al., 2003, Apicella et al., 1997).

The striatum, particularly the ventral striatum, may receive information from the ACC to generate appropriate behavioral output. Like the amygdala, the Nacc is composed of subcomponents, the core and the shell, with distinct functional specialization (e.g., (Di Chiara, 2000)). Lesion studies indicate that the core of the Nacc is a major site of corticostriatal projections from the anterior cingulate (Parkinson et al., 2000a). Lesions of the Nacc core, but not of the Nacc shell, impair the acquisition of pavlovian approach behavior (Parkinson et al., 2000a). Furthermore, the disconnection of the ACC from the Nacc impairs pavlovian approach behavior, suggesting that both ACC and Nacc are required for associative learning, and that the nucleus accumbens is necessary for the acquisition of a conditioned appetitive response (Parkinson et al., 2000b).

In contrast to lesions of the amygdala, lesions of the Nacc do not affect the ability of rats to react properly to changes in the contingency between CS and US, to detect changes in the value of the US, or to develop habitual behaviors by overtraining (stimulus-response associations) ((Parkinson et al., 2000a). However, the functional disconnection of the Nacc from the BLA (unidirectional BLA-Nacc projections) has been shown to impair second-order conditioned responses, a process involving the original value of the US (Setlow et al., 2002b). Taken together, these findings suggest that the Nacc has a dual role, (1) the modulation of ongoing motor performance using information of Pavlovian appetitive conditioned stimuli provided by the ACC and (2) the processing of information about learned motivational value in connection with the BLA.

In human neuroimaging studies of appetitive conditioning, the striatum has received much attention. Activation of the dorsal striatum, including putamen and caudate nucleus, and the ventral striatum (mostly Nacc) has been reported in response to primary and secondary CS+ (e.g., odors, money or positive verbal feedback). In addition, the Nacc has been found to be activated in a time-dependent fashion during appetitive learning (Gottfried et al., 2002, Kirsch et al., 2003, Cox et al., 2005). The ventral putamen also was shown to be activated during appetitive Pavlovian conditioning, in a way that was compatible with the predictions obtained with the temporal difference model (McClure et al., 2003, O’Doherty et al., 2003, O’Doherty et al., 2004). Furthermore, this region was shown to respond equally strongly to CS predicting the least preferred US and the most preferred US in contrast to other regions, such as the midbrain which showed a linear positive correlation between its activation and the valence of the US. This result could indicate that the ventral striatum encodes the relative salience of stimuli rather than their valence (O’Doherty et al., 2005).

3.5. The mesolimbic dopaminergic system: potential neurochemical bases

A large body of evidence implicates the dopaminergic system as a key mediator of appetitive conditioning. Most neurochemical animal studies use the place preference paradigm to demonstrate appetitive learning, particularly in the field of addiction. In brief, appetitive conditioning is demonstrated by the preference of an animal to remain in the place where the appetitive pairing previously occurred (e.g., addictive drug administration) relative to a place where a pairing was done with a neutral stimulus (e.g., placebo administration). Place-preference studies show that most addictive drugs, when injected in the Nacc, can induce place preference (McBride et al., 1999). Furthermore, bilateral Nacc lesions abolish a previously acquired conditioned place preference for food (Everitt et al., 1991).

In addition to its role in maintaining already acquired conditioned place preference, the Nacc, through modulation of dopamine (DA) stimulation, can amplify conditioned reinforcement. As already mentioned in the amygdala section, this amplification is impaired by CeN lesions. The administration of d-amphetamine in the Nacc potentiates the ability of a conditioned reinforcer to reinforce the acquisition of a new response (Taylor and Robbins, 1984). DA depletion or DA receptor blockade in the Nacc impairs this effect, but does not affect the acquisition of new instrumental responses with a conditioned reinforcer (Taylor and Robbins, 1986, Wolterink et al., 1993). These results suggest that DA in the Nacc can amplify the responding to conditioned reinforcement, but is not necessary to its acquisition, and that conditioned reinforcement is related to the amygdala input to the Nacc. Lesions of the subiculum, a region that forms the more interior portion of the hippocampus with input to both Nacc and amygdala (O’Mara, 2005), also impair the potentiation of conditioned reinforcement by intra-Nacc amphetamine administration (Burns et al., 1993a). This finding implies that the glutamatergic projections from the ventral subiculum to the striatum could modulate the potentiative effect of DA (Parkinson et al., 2000a). Both CeN and subiculum could transmit to the Nacc information about the motivational values of stimuli, which modulates Nacc dopaminergic transmission and affects the strength of conditioned reinforcement (Parkinson et al., 2000a). Other neurotransmitter systems are likely to contribute to appetitive conditioning, such as cholinergic function, probably through its role in memory and attention (Vazdarjanova and McGaugh, 1999, See et al., 2003, Crespo et al., 2006).

Summary

In summary, a circuitry involving amygdala, ACC, OFC, and striatum seems to be involved in appetitive conditioning. These regions are highly interconnected providing both functional redundancy and specialization for different aspects of the learning process (see figure 5). This process certainly involves more regions, such as the cerebellum, as evidenced in neuroimaging studies, but little is known about their specific role in appetitive associative learning. The results obtained in human (summarized in table 1) and in animal research are slightly different. The discrepancies apply particularly to subcortical structures such as the amygdala. There is more evidence for the key role of the amygdala in animals than in humans, which can reflect species or methodological differences.

Table 1.

Regions activated in response to appetitive conditioning in human fMRI studies. This table summarizes the results of human fMRI studies of appetitive conditioning and describes the activation coordinates and contrasts, in which the regions discussed were activated. Only brain regions pertaining to brain circuitry described in the section on the neural basis of appetitive conditioning are reviewed

Brain regions	Authors	Coordinates	Contrasts	Comments
	Gottfried, O’Doherty, & Dolan (2002)	-20 -14 -12 (MNI)	Appetitive CS+ versus CS− (olfactory learning)
		14 -12 -22 (MNI)
		16 -10 -12 (MNI)
Amygdala		-16 -12 -24 (MNI)	Time dependent decrease in contrast appetitive CS+ versus aversive CS+
	Seymour, et al. (2005)	-20 2 -26 (MNI)	Appetitive prediction error of pain relief
	O’Doherty, Buchanan, Seymour & Dolan (2006)	-18 -3 -24 (MNI)	Linear correlation with cue (CS) preference	Activation below the threshold for significance
	Gottfried, O’Doherty, & Dolan (2002)	14 46 -18 (MNI)	Valence independent olfactory learning	Only the regions showing activation at p < 0.001 uncorrected or p< 0.05 corrected for whole brain volume are presented
		-18 46 -12 (MNI)
		28 46 -6 (MNI)
		18 44 -16 (MNI)	Appetitive olfactory learning
		-32 52 -12 (MNI)	Aversive olfactory learning
		14 46 -18 (MNI)
Orbitofrontal Cortex	Kirsch, et al. (2003)	-30 20 -14 (TAL)	Appetitive learning: monetary CS+ versus CS+ associated with verbal feedback
		-24 17 -11 (TAL)
		27 14 -16 (TAL)	Appetitive learning (monetary CS+ versus CS−)
	O’Doherty, Dayan, Friston, Critchley, & Dolan (2003)	-21 45 -9 (TAL)	Fit to temporal difference model of learning and of prediction error
		-24 54 -18 (TAL)
	Cox, Andrade, & Johnsrude (2005)	0 44 -18 (TAL)	Presentation of reward versus negative feedback during conditioning.
		-16 42 -18 (TAL)
		-18 38 -20 (TAL)
Ventral striatum	Gottfried, O’Doherty, & Dolan (2002)	14 6 2 (MNI)	Appetitive olfactory learning	Putamen
		-8 4 -2 (MNI)		Nacc
		-10 16 -12(MNI)	Time dependent response to aversive learning	Nacc
	Kirsch, et al. (2003)	-11 8 -5 (TAL)	Appetitive learning: monetary CS+ versus CS−	Nacc
		13 8 -4 (TAL)		Nacc
		-21 6 -5 (TAL)		Putamen
		15 9 -3 (TAL)		Putamen
		12 8 -6 (TAL)	Appetitive learning: monetary CS+ versus CS+ associated with verbal feedback	Nacc
		-18 8 -11 (TAL)		Putamen
		16 8 -3 (TAL)		Putamen
	O’Doherty, Dayan, Friston, Critchley, & Dolan (2003)	-27 3 -9	Fit to temporal difference model of learning and of prediction error
		27 -9 -9
		-27 15 -3
	O’Doherty, Dayan, Schultz et al. (2004)	-26 8 -4	Reward prediction error responses	Putamen
		20 6 -8
	Cox, Andrade, & Johnsrude (2005)	-16 8 -12	Presentation of reward versus negative feedback during conditioning	Putamen
		28 0 -10
	O’Doherty, Buchanan,	24 9 -15	Predicitive responses to appetitive CS+ according to their preference	Putamen
		-30 9 -12
	Seymour, O’Doherty, Koltzenburg, et al.(2005)	18 8 0 (MNI)	Combined appetitive-aversive prediction error	Putamen
		-18 8 -12 (MNI)
Dorsal striatum	McClure, Berns, & Montague(2003)	-18 4 8 (MNI)	Positive prediction error
		-18 1 8 (MNI)	Positive prediction error
Anterior Cingulate	Kirsch, et al. (2003)	-9 8 44 (TAL)	Appetitive learning: monetary CS+ versus CS−
		6 16 35 (TAL)
		-9 8 44 (TAL)	Appetitive learning: monetary CS+ versus CS+ associated with verbal feedback
		6 16 35 (TAL)
	Cox, Andrade, & Johnsrude (2005)	-4 -18 40	Presentation of reward versus negative feedback during conditioning

TAL: Talairach coordinates, MNI: Montreal Neurological Institute coordinates, Nacc: Nucleus accumbens

4. Implications for psychopathology

In this section we present a brief review of the literature of empirical research studies that have investigated the role of appetitive conditioning in a variety of psychopathologies. A sampling of research focusing on the role of appetitive conditioning in substance abuse disorders, eating disorders, depression, and schizophrenia will be discussed. This section is not intended to be an exhaustive exploration of the literature of classical conditioning and psychopathology, nor is it meant to provide a comprehensive explanation of the etiology and maintenance of the psychological disorders listed below. Rather, this section seeks both to provide succinct examples of empirical support for the role of appetitive conditioning in psychopathology that has been theorized in this paper, and to encourage its consideration in both the research and treatment of psychological disorders.

4.1. Substance Abuse

Substance use disorders represent the prototype of psychiatric disorders related to deficits in associative learning. The study of the origin and maintenance of substance abuse disorders during the past half-century has centered on the role of operant conditioning (London et al., 2000, Martin-Soelch et al., 2001, Kalivas and Volkow, 2005, Hyman et al., 2006). However, relying solely on the tenets of operant conditioning misrepresents the complexity of addiction and discounts the role of classical conditioning. Within the framework of classical conditioning, the addictive drug can represent the unconditioned stimulus (US) that automatically elicits pleasurable (appetitive) bodily sensations.

Robinson and Berridge (Robinson, 1993) point out that individuals with substance abuse often use their substances of choice (US) in a wide variety of situations and environments (NS), which become associated with the rewarding consequences of the drugs (CS). These situations and environmental stimuli become cues for substance use. These originally neutral environmental cues have been thought to become imbued with an excessive motivational value through repeated dopamine release in the Nacc (Di Chiara, 1999), which is produced by most substances of abuse. Such prevalent and pervasive cueing facilitates the abuse of the substance, and triggers relapse following treatment (O’Brien et al., 1998, Weiss and Porrino, 2002).

Furthermore, Volkow, Fowler, and Wang (Volkow, 2004) propose that the exquisite saliency of the effects of an abused substance, and the cues that have been paired with it, may dull the saliency of other naturally occurring appetitive stimuli. Kelly (Kelly, 2004) suggests that receptors, when becoming responsive to addictive substances, such as alcohol or opioids, may lose their responsivity to other naturally occurring appetitive stimuli, perhaps reflecting competing influences of substance use and naturally occurring appetitive stimuli on the reward-related circuitry.

The power of environmental cues to elicit craving or relapse was demonstrated in several human and animal studies (Grant et al., 1996, Childress, 1999, Katner and Weiss, 1999, Braus et al., 2001, Ciccocioppo et al., 2001, Mann et al., 2001, Schneider et al., 2001). These studies show that stimuli previously associated with a drug can elicit craving symptoms and/or relapse in abstinent subjects, suggesting a role of appetitive conditioning in the addiction process. From a neurobiological perspective, the ability of environmental cues to induce craving (i.e., strong desire for the reinforcing property of the US) seems to be associated with a neural network that underlies appetitive conditioning (see figure 5). Neuroimaging studies in human show activation in brain regions involved in appetitive conditioning, including the amygdala and the ventral putamen, in response to drug related cues in different groups of former drug users (Grant et al., 1996, Childress, 1999, Braus et al., 2001, Schneider et al., 2001). Similarly, animal studies find that both the exposure to environmental drug-related cues and the relapse triggered by these cues implicate the ventral striatum and the mesolimbic dopaminergic system (Weiss et al., 1993, Gonzalez and Weiss, 1998).

Two recent studies by Wong et al. (Wong et al., 2006) and Volkow et al. (Volkow et al., 2006) evidenced a significant increase of dopamine release in the dorsal and not in the ventral striatum in response to cocaine-cue videos in cocaine addicted subjects. This change in the dopaminergic transmission was correlated with self-reports of craving and with addiction severity. These results are in agreement with the hypothesis of distinct roles of the dorsal and ventral striatum in response to reinforcement. The ventral striatum seems to be involved in reward and motivation (Cardinal et al., 2002) and particularly in outcome anticipation (Seymour et al., 2004, O’Doherty et al., 2004, Knutson and Cooper, 2005), while the dorsal striatum seems to be involved preferentially in motor and cognitive control, specifically in the acquisition of stimulus-response (habit learning), and stimulus-response-reward associations (Montague et al., 1996, Packard and Knowlton, 2002, O’Doherty et al., 2004, Knutson and Cooper, 2005). In drug addiction, it has been postulated that the dorsal striatum is involved in the mediation of the habit-based nature of drug seeking behavior, and the correlation observed between dopaminergic transmission in the dorsal striatum and cue-induced craving could suggest that craving may be an automatic response (habit-based) in drug addiction (Tiffany, 1990, Robbins and Everitt, 1999, Volkow et al., 2006) Wong et al., 2006).

The administration of dopaminergic receptor antagonists can reverse the ethanol-seeking behavior elicited by ethanol related cues in dependent rats, a similar effect was found after administration of opioid receptor antagonists (Katner and Weiss, 1999, Wilson et al., 2000, Ciccocioppo et al., 2002, Liu and Weiss, 2002) and the administration cannabinoid receptor antagonists can decrease ethanol intake in mice and rats (Arnone et al., 1997, Freedland et al., 2001) and inhibit the reinstatement of cocaine-seeking behavior elicited by environmental cues (De Vries et al., 2001), suggesting that several neurotransmitter systems are involved in learning-related aspects of addiction. Taken together, these findings support a central role of classical appetitive conditioning, in addition to operant conditioning, in the development and maintenance of addiction.

4.2. Eating disorders

Similarly to their role in substance abuse disorders, conditioned cues (CS) strongly influence the emergence and maintenance of some eating disorders (Wardle, 1990, Jansen, 1992, Jansen, 1998). Food intake may be viewed as an unconditioned stimulus (US) which elicits unconditioned responses (UR) in the form of metabolic changes. Food intake is also cued by sensory stimuli (e.g., aroma, sight of food). These cues, within the framework of classical conditioning, will become CS that trigger the desire for food which is experienced as craving, and lead to more food intake (Jansen, 1994).

Jansen and Van den Hout (Jansen, 1991) show that dieting women who were exposed to the aroma of food items (CS) prior to their presentation (US), were unable to maintain control over their food intake more often than dieting females who had not been exposed to this cue. This loss of control was attributed to the strength of food craving (CR).

Exposure to food related cues (CS) has been shown to enhance food intake in animals, even in a satiated state (Weingarten, 1983). Using classical conditioning, a light and buzzer tones were simultaneously associated with food presentation. After conditioning was established, i.e., the light and buzzer became the CS eliciting food intake (CR), rats were allowed unlimited access to food to create satiation. Once satiated, they continued to be exposed to free access to food in two conditions, with and without CS presentation. Rats consumed significantly more food during CS days, when compared to non-CS days, suggesting that exposure to food cues enhanced food intake even in a replete state.

These findings are directly relevant for the understanding and treatment of binge eating in bulimia nervosa. Individuals who exhibit binge eating behavior may have unusually strong craving sensations, and/or, as a result of their repeated binging, have developed many food cues (Jansen, 1991, 1998). Treatment could conceivably use the reconditioning of these associations between cues and food. Indeed, Jansen, Broekmate, and Heymans (Jansen, 1992) find that obese subjects diagnosed with bulimia nervosa show a 100% reduction in binge eating after receiving a treatment that includes non-reinforced cue exposure (i.e., presentation of a food-related cue without subsequent food consumption). In other words, this non-reinforced cue exposure treatment leads to the extinction of the learned association between the cue and food. This suggests that the reconditioning of food-related cues should be included in the treatment of eating disorders. In addition to eating disorders, similar extinction approaches to smoking cessation have been explored (Rose and Behm, 2004).

4.3. Depression

Depression, characterized by anhedonia or the loss of the ability to experience pleasure, may also manifest deficits in appetitive conditioning, resulting in the failure to form or maintain positive associations between normally appetitive US and NS. Supporting this hypothesis, cerebral blood flow differences between depressed patients and controls have been evidenced in brain regions involved in appetitive conditioning, such as the amygdala, ventral striatum, ACC, prefrontal and lateral OFC (Drevets, 2001). Preliminary results from an fMRI study of appetitive conditioning show dysfunctional learning in both appetitive and aversive learning conditions associated with a pattern of dysfunction of amygdala, lateral OFC, striatum (caudate nucleus) and ACC in patients with major depressive disorder (MDD) (Martin-Soelch et al., 2006).

In animals, using an animal model of depression, similar evidence for a dysfunctional response to appetitive stimuli and to appetitive learning is provided by DiChiara et al.’s earlier work (DiChiara, 1999). In this study, rats are subjected to chronic mild stress (CMS), a procedure inducing behavioral abnormalities consistent with those observed in human depression (Willner et al., 1992). CMS rats show a blunting of DA response to appetitive stimuli in the Nacc and prefrontal cortex (PFC). In contrast, aversive stimuli not only fail to inhibit DA response in the Nacc, as seen in healthy rats, but actually elicit a DA response in the Nacc and potentiate the DA response in the PFC. Di Chiara et al. (DiChiara, 1999) conclude that depression could be the consequence of dysfunctional appetitive and aversive learning, which could be related to changes in the responsiveness of DA transmission in the NAcc and PFC.

4.4. Schizophrenia

A core feature of schizophrenia is cognitive disturbance, which frequently manifests as attention and information processing deficits. These deficits are thought to be related to the inability to ignore or disregard irrelevant stimuli. Accordingly, individuals with schizophrenia have impaired latent inhibition (LI) (Lubow, 1995, Vaitl, 2002). Latent inhibition refers to a delay in the learning of an association between a CS or US, when either the CS or US has been presented prior to the CS-US conditioning trials (Lubow, 1995) (see section on classical conditioning). For example, Vaitl, et al.(2002) compare levels of physiological arousal (i.e., skin conductance) among healthy controls, medicated and unmedicated schizophrenics while they respond to novel (non-pre-exposed US) and familiar (pre-exposed US) stimuli. Unmedicated subjects with schizophrenia failed to show latent inhibition, normally indexed by a greater physiological arousal response to non-pre-exposed US. This inability to differentiate between relevant and irrelevant stimuli (Lubow, 1995, Vaitl, 2002) is pertinent to the role of classical conditioning in schizophrenia. Individuals with schizophrenia may continuously experience high levels of arousal, which may lead to the maintenance of associations after US are no longer reliably preceded by CS cues (i.e., resistance to extinction) An alternative explanation is that external cues become irrelevant due to the lack of a predictive relationship with chronic, constant hyperarousal.

5. Conclusions

The aim of this review is to provide an overview of appetitive conditioning including its underlying neural bases and potential contribution to pathological processes.

The neural bases of appetitive conditioning have been extensively investigated in animal studies, in contrast to the few studies conducted in humans (table 1). The results of these studies show discrepancies that need to be reconciled in future work. Here, we attempt to integrate these findings into a model of the neural circuitry of appetitive learning, which includes amygdala, Nacc, OFC, ACC and midbrain regions (figure 5).

This model is far from being exhaustive, and, in fact, shows conspicuous gaps in potential functional links among regions. Their absence speaks to our lack of knowledge rather than their non-existence, and further highlights areas in need of research. Indeed, we hope that this review will stimulate and help direct work on the neurobiology of appetitive conditioning.

From a clinical perspective, the consideration of disturbances in appetitive conditioning for the formation of psychiatric symptoms is promoted through the selected examples we provided in the last section of this review. This approach could open the way to new visions of the understanding of pathological conditions, in which motivation is perturbed, and could inform new treatment strategies.

Above all, this review underscores the contrast between the paucity of work dedicated to understanding the mechanisms underlying appetitive conditioning in humans, and the potentially critical importance of this process in the genesis of psychiatric disorders.