The Origins and Organization of Vertebrate Pavlovian Conditioning

Abstract

Pavlovian conditioning is the process by which we learn relationships between stimuli and thus constitutes a basic building block for how the brain constructs representations of the world. We first review the major concepts of Pavlovian conditioning and point out many of the pervasive misunderstandings about just what conditioning is. This brings us to a modern redefinition of conditioning as the process whereby experience with a conditional relationship between stimuli bestows these stimuli with the ability to promote adaptive behavior patterns that did not occur before the experience. Working from this framework, we provide an in-depth analysis of two examples, fear conditioning and food-based appetitive conditioning, which include a description of the only partially overlapping neural circuitry of each. We also describe how these circuits promote the basic characteristics that define Pavlovian conditioning, such as error-correction-driven regulation of learning.

Pavlovian conditioning is the process by which we learn relationships between stimuli, but it is commonly misunderstood. Two examples, fear conditioning and food-based appetitive conditioning, illustrate this.

BASIC CONCEPTS OF ASSOCIATIVE LEARNING

Well before the birth of modern psychology and neuroscience, philosophers suggested that the way the mind creates ideas is by forming associations between events. Matters experienced would be joined because of their temporal proximity, common spatial locations, or perceived similarity. More complex thoughts would, in turn, be built from these basic associations. Although less discussed, the resulting associations would have to be stored in memory to impact cognition and action. Thus, there is a long history that acquired associations are at the core of the way the mind represents the world and that such associations provide the structure of memory itself.

Environmental Relationships

Early in its history, psychology also emphasized the importance of acquired associations in shaping behavior. Associations arose from experiencing events in close temporal proximity. Experience with two types of environmental relationships fostered association formation. One relationship was when two stimuli were experienced close in time (Pavlov 1927); the other was when a behavior was followed closely by a stimulus (Thorndike 1898). Thus, we recognize two classes of associations, one caused by stimulus relationships the other caused by relationships between actions and the environment. This work focuses on the former class, stimulus-based associations. The modern neuroscientific study of associations began with the work of Ivan Pavlov, who was concerned with stimulus associations and, therefore, the conditions that fostered such associations are appropriately called “Pavlovian conditioning.” The latter class is called instrumental conditioning because, in such situations, behavioral action was instrumental in obtaining an outcome.

The Procedure—Process-Mechanism Distinction

This instrumental versus Pavlovian distinction is based on the events that are experienced, the things that happen in the environment that cause associations to form. In his laboratory, to cause association formation, Pavlov paired two stimuli together; for example, a tone might be immediately followed by food. Hence, we are defining Pavlovian conditioning by a procedure. Evidence for the formation of an association was provided by a change in behavior to the first stimulus. The tone never caused salivation until it was paired with food. Note that this procedural definition is neutral with respect to what happens inside the organism to link experience with behavior. The theoretical construct used to explain this is what has been termed the “process.” Some early proponents of associative learning suggested that a common internal process underlies both instrumental and Pavlovian associative learning (Watson 1916; Hull 1942). Others suggested that each procedure produced its behavioral effects through different psychological processes (Konorski and Miller 1937; Spence 1956). And there were those that suggested that we should focus only on procedures and not delve into processes, as only procedures and behaviors were observable (Skinner 1938). Psychological process models emphasize how certain components of the procedure are isomorphic with the mediating events. For example, Hull (1942) suggested that, regardless of whether the procedure was Pavlovian or instrumental, associations formed when a rapid temporal sequence of neutral stimulus→response→biologically significant stimulus was experienced. This experience caused a connection (association) between the mental representation of the stimulus and response. We can distinguish such process models from mechanistic models, which describe how synapses within specific neurocircuits change with experience (i.e., the “mechanism” of learning). Of course, psychological process models can provide a framework for the discovery and understanding of the brain mechanisms of learning and the observed brain changes following learning can inform process theory. We take such an approach.

The Learning—Performance Distinction

The only way to know that an association has formed is to observe a change in behavior following experience. Although behavior is a reflection of learning, it is, however, not learning itself. Learning resides in the process and/or mechanism that mediate the formation of associations between environment and behavior. But behavior will be affected by factors other than learning. For example, Pavlov's dog salivated more or less depending on its hunger status. Similarly, learning may occur but it may not alter behavior. A classic example of the learning–performance distinction is Tolman's latent learning experiment (Tolman 1951). In this experiment, a rat was allowed to explore an empty maze. The rat's behavior was aimless wandering about the maze trial after trial—there was no obvious change in behavior as a function of experience. But when food was suddenly introduced in one location, the rat immediately went to that location on the next trial. This did not happen if the rat never explored the maze without reward. Thus, the rat learned the stimulus configuration of the maze (i.e., formed a cognitive map) during its apparently aimless wandering, but never expressed this learning in performance until motivated to do so.

The Misdefinition of Pavlovian Conditioning

Most definitions of Pavlovian conditioning are similar to this one taken from the Oxford Dictionary: “A learning process that occurs when two stimuli are repeatedly paired; a response that is at first elicited by the second stimulus is eventually elicited by the first stimulus alone.” Virtually every aspect of this definition is incorrect. Here we point out the fallacies, later we will provide a more accurate, modern, definition.

1. The conditional response (CR) and the unconditional response (UR) are not the same; the learned response is often different from the one elicited by the unconditioned stimulus (US) (see section on topography).

2. Repeated: Some of the most robust forms of conditioning occur with a single trial. The two most notable examples of conditioning with a single trial are fear conditioning (see section on fear conditioning) and taste aversion. In taste aversion, a novel taste is followed by an illness-producing stimulus. Following just a single experience there is a hedonic shift such that the novel taste, even if it was initially pleasing, becomes distasteful.

3. Pairing: This goes to the heart of why Pavlov used the term conditioning. In our example with a tone and food, Pavlov called the tone a conditional stimulus (CS) to differentiate it from the US, food. The response to the food, the UR, was “inborn” because the pathway that leads from stimulus to response “is already complete at birth.” The tone, which initially does not produce the response of interest, is the CS and with experience comes to elicit a new response, which is labeled the CR. There is considerable controversy over the choice of the term conditioned versus conditional. In Anrep's translation, which we use here, the term conditioned is most frequently used. However, the literature has Pavlov saying the use of the term conditioned is “fully justified” because “these new reflexes actually depend on very many conditions.” This accords better with the term conditional. Indeed, the stimulus relationship that produces association is well described as a situation in which the occurrence of the US depends, or is conditional, on the US.

Rescorla (1968) put this question to an empirical test. He asked whether co-occurrence of the CS and US was sufficient for learning or whether it was the conditional (dependent) relationship between the CS and US. He trained rats with a tone CS and a shock US. Several groups of rats received exactly the same number of CS–US pairings (e.g., 40% of the CSs were paired with shock). What differed between groups was the likelihood of the US in the absence of the CS; some groups had additional shocks delivered during the intertrial interval ([ITI] the time between CSs). When no shocks occurred during the ITI conditioning, the CS was at the maximum detectable level for the measure used; 40% CS–US pairing was sufficient to produce strong conditioning. However, if the probability of shock during the ITI was 0.10, conditioning was reduced by about half. And when the probability of shock during the ITI was the same 0.40 as during the CS, no conditioning occurred. Thus, the level of Pavlovian conditioning was determined by the “conditional” relationship of the CS and US. Therefore, the term “conditional” not only squares better with Pavlov's intention, it is also empirically supported. Converging evidence points to the fact that pairings are not the critical variable that determines conditioning.

WHAT CAUSES CONDITIONING

Pairing versus Contingency

During its early history, it was thought that contiguity of CS and US was the necessary and sufficient condition to cause association formation. This is the idea of pairing discussed above—all that matters is whether or not a sufficient number of pairings has occurred. If so, conditioning should occur; if not, there was no conditioning. Rescorla's (1968) experiments, showing that conditioning can be degraded by additional unpaired US presentations, severely challenged this view. There were several additional findings that also suggested pairing alone was not sufficient. One originally described by Pavlov (1927) was overshadowing. In overshadowing, a CS that conditions well (i.e., comes to produce a strong CR) on its own shows less conditioning when it is accompanied by another CS. The overshadowing effect is greater the more intense the added stimulus (Mackintosh 1976). Mackintosh suggested that this occurred because of competition for attentional resources (Sutherland and Mackintosh 1971). Intense CSs grab limited attentional resources away from an overshadowed CS that would alone command those resources. This account also fit well with a finding by Wagner and colleagues showing that a CS that was paired only 50% of the time with a US conditioned well if it was the best available predictor of the US, but failed to condition well when a better predictor was available (Wagner et al. 1968). Wagner and colleagues suggested that the subject allocated attentional resources to the most valid predictor in the situation and as a result learning about the less valid predictors suffered. In both cases, pairing alone was insufficient to explain the strength of conditioning. This shifted the interpretation of conditioning away from an emphasis on the US's ability to automatically reinforce association formation (contiguity theory) toward an attentional view that emphasized CS processing.

The idea of pairing was also challenged by a seminal series of experiments by Kamin (1969). Kamin followed a compound CS of noise and a light with a US and found that conditioning to the light varied with the prior history of training with the noise. If the noise had been paired with the US before compound training, no conditioning to the light would occur, a sort of exaggerated overshadowing effect (Table 1). The finding that prior conditioning to one element of a compound “blocks” conditioning to the other element also, at first blush, fits with the limited attention view. Because the subject first learns that the noise is a good predictor of the US, it grabs the attentional resources that could have been split between the two elements of the compound. However, a second experiment by Kamin dispelled that account. If US intensity was increased in the compound phase (relative to that during pretraining), the added element conditioned well (so called “magnitude unblocking”; Table 1). If attention was directed to the pretrained CS, then conditioning to the added element should still suffer. The fact that conditioning occurred to this element indicated that it must have garnered sufficient attention to support learning. Kamin offered an interpretation that tilted theory back toward an emphasis on the US. He suggested that a US only reinforces learning to the extent it is surprising. In the blocking design, the pretrained stimulus already predicts the US so the added element is never paired with a surprising US. In the unblocking design, increasing the US intensity means that the pretrained CS does not fully predict the new US that is paired with the compound; hence, US surprise is restored and the novel element receives the reinforcement that causes learning.

Table 1.

Conditioning phenomena: Several conditioning arrangements and their effects are schematized

Phenomenon	Pretraining	Training	Test: Conditioning to light
Control	None	L+	Strong
Overshadowing	None	TL+	Weak
Blocking	T+	TL+	None
Unblocking	T+	TL+	Strong
Overexpectation	T+, L+	TL+	Weak
Supernormal	V+, VT−	TL+	Very strong
Latent inhibition	L−	L+	Weak

Conditional responses (CSs) are indicated by letters (T, tone; L, light; V, vibration). Two letters together (e.g., TL) indicate stimuli presented simultaneously. Reinforcement by pairing with an unconditioned stimulus (US) is indicated by a +; the bold + in unblocking indicates a larger US (e.g., more food or a more intense shock). Testing is always to the L stimulus and results are all relative to the control.

The Rescorla—Wagner Model

Rescorla and Wagner formalized Kamin's notion of surprise and showed that this approach could account for both Rescorla's contingency effects and the attention-like phenomena described in the previous section (Rescorla and Wagner 1972). The model dictated that a given US could only support a limited amount of associative strength and that this limit was determined by the intensity of the US. Surprise was the difference between the limit (λ) and the amount of conditioning that had already occurred to the stimuli present to support expectation of the US (V = the amount of associative strength). Earlier contiguity models made similar assumptions (Hull 1942; Bush and Mosteller 1955), but Rescorla and Wagner put a slight twist on the idea that made a world of difference. Rather than assuming the change in conditioning to a specific stimulus on a trial was the difference between λ and the associative strength of the stimulus in question, as previous contiguity models had, they postulated that the change in associative strength was the difference between λ and the sum of the associative strength of all CSs present on that trial. The resultant model was that the change in associative strength to stimulus A on a trial (ΔV_A) is a proportion (α) of the difference between the intensity of the US on that trial (λ) of associative strength already conditioned to the CSs present on the trial (V_Σ).

Thus,

$$\mathrm{\Delta }{V}_A=\alpha \ast (\lambda -V_{\mathrm{\Sigma }}).$$

Kamin's blocking effect is easily predicted by this model. Because of the initial training the associative strength of the noise CS would be at or near λ. The novel light would enter the second phase with no associative strength, but because the associative strength of the noise contributes to V_Σ, the quantity (λ − V_Σ) is near 0, so the light receives no increment in its associative strength. Unblocking occurs because the increase in US intensity causes an increase in λ. The model obtained additional power from the assumption that the learning rate parameter α was determined by CS intensity or salience. This allowed the model to explain why overshadowing increases with CS intensity. In essence, the model put CSs in competition with each other for the reinforcing value of the US.

Another important aspect of the Rescorla–Wagner model was that it focused attention on the context as a significant contributor to conditioning. In any conditioning situation in addition to the explicit CS, there are also the situational cues within which conditioning takes place (i.e., the context). These cues would also naturally compete with CSs. Thus, Rescorla's contingency effects could be explained by the models prediction that giving the subject unpaired USs would drive the associative strength of the context and make it a significant competitor with the CS.

The Rescorla–Wagner model made predictions not only about increases in associative strength, but also decreases. Whenever the value of (λ − V_Σ) is negative, a decrement in associative strength occurs. This most frequently happens when an expected US is omitted, as in extinction, because the value of λ drops to 0. This means that in some situations a stimulus can have a negative associative strength. Such CSs are Pavlovian inhibitors that have the ability to suppress a CR.

Further impact of the Rescorla–Wagner model came from its ability to predict new phenomena. Two examples are overexpectation and superconditioning (Table 1). One example of overexpectation occurs when two CSs are trained independently and then are put together as a compound that is reinforced with the same US that each previously independently predicted. Because each CS is near λ alone, (λ − V_Σ) will be negative and result in a decrement in associative strength. Empirically, the prediction that the CR to the elements of the compound decrease in a manner proportional to their salience has been confirmed (Kamin and Gaioni 1974). Additionally, as predicted, if a novel stimulus is added to the compound, that novel stimulus becomes an inhibitor despite the fact that it was consistently reinforced (Kremer 1978).

Error Correction and the Neural Instantiation of the Rescorla–Wagner Model

Kamin, Wagner, and Rescorla reframed the idea of a US and provided a modified version of contiguity theory. A US is defined as a surprising event and any stimulus contiguous with that surprise will be learned about. Indeed, in more recent versions of his theory, Wagner postulated that any surprising stimulus, even one that is hedonically neutral, will reinforce its associations with contiguous events (Wagner and Brandon 1989). This has the benefit of explaining a phenomenon such as sensory preconditioning in which two neutral events become associated. However, if a neutral stimulus is predicted it will be less able to promote association formation. For example, if a neutral stimulus is presented alone in a context, the context will come to predict the stimulus. If that previously preexposed stimulus is subsequently paired with a US, because the CS is not surprising, it will not enter into association with the US—a phenomenon called latent inhibition. Importantly, if the preexposed stimulus is rendered again surprising by presenting it in a novel context it will form associations, and latent inhibition is lost.

Bolles and Fanselow (1980) looked at surprise in a somewhat different way. They emphasized that the CS should produce accurate expectancies of the US and that inaccurate expectancies must be corrected. Their model of conditioning stated that “any error in the expectation is fed back so as to reduce future errors. If the amount of correction is directly proportional to the size of the error, then one has a learning system that will sooner or later correct its errors and generate accurate predictions.” Thus, they described Pavlovian learning as an error-correction system driven by negative feedback. In this model, a comparison is made between the CS generated expectancy of the US and the actual US received. To match this to the Rescorla–Wagner equation, V is the expectancy, which is also the strength of the CR, and this value is subtracted from the actual US received (λ). The negative feedback could be any CR that has the ability to oppose the reinforcing power of the US. Bolles and Fanselow (1980) also suggested a neural mechanism for this error correcting negative feedback. One CR to a fear CS is an analgesic response (Fanselow and Baackes 1982). Because shock conditions fear proportional to its painfulness, an analgesic CR would undermine the reinforcing effectiveness of the US. Such a finding is also consistent with the reduction in the UR that frequently accompanies conditioning (Fanselow 1984; Canli and Donegan 1995). Additionally, pharmacological antagonism of endogenous opioids prevents error correction (Fanselow 1986a). We elaborate on this specific circuit in our detailed analysis of fear conditioning. Subsequently, a negative feedback circuit for eyeblink conditioning has been identified in the form of a γ-aminobutyric acid (GABA)ergic projection from the deep nuclei of the cerebellum, in which the CS–US association is formed, to the inferior olive, which is where the ascending reinforcement signal from the US is processed (Kim et al. 1998). Although the negative feedback circuit has not been fully identified in appetitive conditioning, groups of dopaminergic cells that behave as if they carry the error-correction signal (i.e., λ − V_Σ) have been described (Waelti et al. 2001) (see section on dopamine and reward prediction error).

Beyond the Rescorla–Wagner model

As a heuristic device, the Rescorla–Wagner model captures a tremendous proportion of the variance found in conditioning phenomena. However, there are certain findings that the model, as initially proposed, has trouble with.

Time

To perform the iterative calculations, the model breaks a conditioning session into a series of CS-length chunks. This makes predictions easy, but CS length is a rather arbitrary variable so there is no representation of exact time in that model. Sutton and Barto have developed a model that incorporates time within a Rescorla–Wagner-like calculation in their temporal difference reinforcement-learning model (Sutton 1988; Sutton and Barto 1990). Absolute time, in terms of weighted stimulus degradation parameters, features prominently in a model developed by Wagner and Brandon (1989).

Attention

As mentioned above, preexposing a CS before conditioning reduces the rate of learning once that CS is reinforced (Lubow and Moore 1959; Rescorla 1971). The basic Rescorla–Wagner model does not predict this, because the absence of the US means that λ is 0, leaving associative strength (V) at the same 0 value it started with. Because associative strength is the only value carried over from preexposure to reinforcement, there is no reason for preexposure to have an effect. Rescorla (1971) recognized this early on and suggested that the learning rate parameter (α) must change because of nonreinforcement—whereas α should start at a value defined by the CSs salience; this value will decrease with nonreinforcement. The idea that this learning rate parameter changes over the course of learning is central to attentional models proposed by Mackintosh (1975) and Pearce and Hall (1980). For Pearce and Hall, associability (α) decreases for stimuli that predict no change in reinforcement and increases when there is a surprising change in reinforcement. Another approach, in Wagner's subsequent models described above, states that preexposure allows the context to predict the CS and because the CS is not surprising during the conditioning phase it receives less processing.

Extinction

Perhaps the major problem with the Rescorla–Wagner model is extinction. Although the model accurately predicts decrements in performance following extinction, it does so by causing a reduction in associative strength (unlearning). However, as first shown by Pavlov (1927) and substantially developed by Bouton (1993), associative strength is still intact after extinction. This is shown by recovery phenomena in which after extinction has weakened responding to the CS, the CR returns following a change in context (renewal), time elapsing between extinction and test (spontaneous recovery), and the administration of unpaired USs (reinstatement). Bouton (1993) suggests that rather than unlearning the original CS–US association, extinction causes the acquisition of an inhibitory CS–no US association that is context specific. This idea is formally incorporated into the Pearce–Hall model (Pearce and Hall 1980), although there is no specific account of why this CS–no US association is context specific. We will address these issues when we consider the mechanisms underlying fear conditioning.

THE CONTENT OF LEARNING

Shortly after the discovery of the Pavlovian conditioning phenomenon, a large group of scientists became intensely interested in the study of the procedures and behavioral output of conditioning (Skinner 1950). Many others, however, including Pavlov himself (Pavlov 1932), and quite prominently Jerzy Konorski (1948, 1967), argued that Pavlovian and other conditioned behaviors were a window into the brain mechanisms of behavior.

Associative Structure and Its Diagnosis

Historically, associative conditioning has been thought to involve the formation of nodes (presumably in the brain) between the conditioned components. One primary theory was that Pavlovian (and instrumental) conditioning involves the formation of a stimulus–response (S–R) bond. The CS serves as the S node and with learning becomes capable of directly activating the motor program (the R node) innately generated by the US itself (Hull 1943; Spence 1956). This was encouraged by findings that the CR is often identical to the original UR. Interestingly, however, the CR can differ from the UR and, in some cases, can be entirely opposite. For example, morphine elicits an analgesic UR, but a morphine-predictive CS can actually elicit hyperalgesia (Siegel 1975a), a finding not easily reconciled by an S–R Pavlovian association because the hyperalgesia response “node” is never present during conditioning. At further odds with S–R theory, a Pavlovian CR can still develop if access to the US is blocked, preventing UR execution (Zentall and Hogan 1975). Also unfavorable to S–R theory is sensory preconditioning. In these experiments, two neutral stimuli are paired together (S1–S2) in the absence of any US. Later, S2 is paired with a US, and then in a third phase presentation of S1 alone elicits a CR (Brogden 1939; Rizley and Rescorla 1972; Rescorla 1980). S–R theory cannot explain this result because there was never an opportunity for S1 to become linked to the CR.

To reconcile these findings, Bolles (1972) argued for a more cognitive conditioned association (he was not the first to suggest this, c.f., Koffka 1935; Lewin 1936; Kohler 1940; Tolman 1949). Bolles suggested that, rather than an S–R relationship, during Pavlovian conditioning subjects form a stimulus–outcome (S–O) association (also termed stimulus–stimulus; S–S*) in which a link between the mental nodes representing the CS (S) and the specific US (i.e., outcome [O]) with which it is paired (Bolles 1972). This account was supported by many other learning theorists of the time (e.g., Rescorla 1973a) and suggested that Pavlovian CRs are elicited by a cognitive expectation of the predicted US. As a result, CRs can be more flexible. Indeed, this account allows the Pavlovian CR to take a form different than that directly elicited by the US itself, because it does not rely on a conditioned association to this original response. Sensory preconditioning is also well explained by this account by presuming that the neutral stimulus (S1) elicits a representation of the stimulus with which it was initially paired (S2), which in turn generates the response via a mental connection between the S2 and the US.

Inflation and Devaluation

The essential difference between the theorized S–R and S–O associative structures is that details of the US's identity are encoded in the latter, but not the former. Therefore, the critical test between these theories is to evaluate CRs after making a specific change to the US, for example, altering its value (Rozeboom 1958; Pickens and Holland 2004). Postconditioning devaluation of the US will modify CRs if such responding is guided by an S–O association, but not if it is guided by an S–R association (Pickens and Holland 2004). Rescorla put this to the test (Rescorla 1973b, 1974). In one experiment, a light CS was paired with a loud noise US. The US was then repeatedly presented alone to lower its value by habituation. Although the CS was never paired with the devalued US, its ability to generate a fear CR was reduced. Such behavior could only be generated if the CS aroused a memory of the US. In the converse experiment, Rescorla (1974) paired an auditory CS with a mild shock US. The rats were then exposed to a series of stronger shocks. At test, the tone alone elicited a stronger fear CR, as if it had been paired with the stronger US even though it had not. Again, the result is most congruent with an S–O association. Although aversive Pavlovian conditioning appears to be dominated by S–O associations, learning is not exclusively S–O. Within the same experiment, Rescorla showed that first- but not second-order associations were altered by an inflation procedure (Table 2). It is unclear why two different types of associations are formed. Potentially, the CS tends to become associated with the most salient aspect of the US. In first-order conditioning, the shock US is likely to be the most salient feature. In second-order conditioning, the stimulus serving as the “US” may be less salient than the emotional reaction it generates.

Table 2.

Inflation design and results in first- and second-order conditioning

Training/test	Inflation group	Control group
First-order conditioning	Tone→mild shock	Tone→mild shock
Second-order conditioning	Light→tone	Light→tone
Inflation	Strong shocks alone	Context exposure
First-order test	Tone→strong CR	Tone→weak CR
Second-order test	Light→weak CR	Light→weak CR

For simplicity, standard counterbalancing was omitted from the table (data based on Rescorla 1974). CR, Conditional response.

An even richer associative network underlies appetitive conditioning. In a typical devaluation experiment, a CS is paired with an appetitive US, often a food substance. Learning is shown when the subjects approach the location of the US (the food port) when the CS is presented (but before food delivery). In some instances, if the CS is visual and localizable, the subject will approach the CS itself (i.e., sign tracking). Next, the value of the US is reduced either by selective satiation or by pairing its consumption with nausea (induced by lithium chloride [LiCl] injection) to form a taste aversion. Both of these treatments will result in complete rejection of the food. Postconditioning devaluation of a food US, by selective satiation (Holland and Rescorla 1975) or by taste aversion (Holland and Straub 1979), has been shown to reduce food port or CS (sign tracking) approach CRs in a probe test. This finding, replicated many times over in both humans (Gottfried et al. 2003; Bray et al. 2008) and rodents (Holland 1981; Colwill and Motzkin 1994), suggests that subjects mentally recall the devalued US when presented with the CS (Table 3). US revaluation can also turn aversive CRs appetitive. Normally, a CS predicting intraoral infusion of unpleasant high-sodium chloride solution will elicit escape-type CRs, but if the animal is put into a salt-appetite state the aversive CR will turn appetitive, that is, the animal will sign track to the CS (Robinson and Berridge 2013). In further support of the S–O account, presentation of an appetitive Pavlovian CS will bias instrumental action selection toward those actions earning the exact same outcome as predicted by the CS (Kruse et al. 1983; Colwill and Motzkin 1994). Because in these experiments the CS has never been directly paired with the instrumental action (no opportunity for S–R association), it is the cognitive expectation of the outcome elicited by the CS that explains this selective Pavlovian instrumental transfer (PIT) effect. In all of these cases, the evidence suggests that US identity controls the CR and is, therefore, encoded in the associative structure guiding Pavlovian conditioned response.

Table 3.

Devaluation design and results

	Devaluation group	Control group
Phase 1	Tone→food	Tone→food
Phase 2	Food→LiCl  Food is rejected	Food/LiCl unpaired Food not rejected
Test	Tone→Ø  Weak CR	Tone→Ø Strong CR

First-order appetitive Pavlovian associations are sensitive to devaluation of the US. LiCl, Lithium chloride; CR, conditional response.

The above evidence argues that a simple reflexive S–R associative structure cannot fully explain Pavlovian conditioning responding and that a cognitive, S–O associative structure can control such behavior. This, however, does not suggest that an S–O association is the exclusive association formed during Pavlovian conditioning. Indeed, it has been suggested that both associations develop and that they compete, or perhaps interact, to control conditioned responding (Holland and Rescorla 1975). Although satiety and taste-aversion devaluation produces complete rejection of the food, it does not often completely attenuate Pavlovian CRs, suggesting that some aspect of this responding may be driven by an S–R associative structure (or a less detailed S–O structure; see Dayan and Berridge 2014). Moreover, there are instances in which CRs are not sensitive to US devaluation. For example, if the basolateral amygdala (BLA) has been lesioned, rats will acquire a food-port approach CR, but this response will be insensitive to US devaluation (Hatfield et al. 1996), suggesting it is controlled by an S–R associative structure (more on this later). Which associative structure dominates behavioral control depends on a variety of factors including the CS form, type of pairing, CR form, and the requirement for detailed outcome discrimination.

The Representation of Outcomes

It has been long recognized that stimuli, conditioned or otherwise, consist of many elements. Within an S–O framework, the CS may, therefore, become linked to one or more elements of US. This concept was formalized by Jerzy Konorski (1948, 1967) and later adapted by Anthony Dickinson (Dickinson and Balleine 2002). These and other investigators (Wagner and Brandon 2001; Delamater 2012; Dayan and Berridge 2014) have suggested that the extent to which a CS becomes associated with some or all properties of the US determines its influence over behavior. Take, for example, a grape-flavored sucrose solution US. This stimulus has very specific identifying features, for example, its grape taste, as well as features that may be more general (i.e., overlapping with other USs) including its fluidic and caloric properties and its general appetitive nature. Evidence suggests that the sensitivity of CRs to devaluation is mediated by CS retrieval of the identity-specific (e.g., specific taste) features of the US. This was deduced in a clever experiment in which rats were trained that one of two stimuli predicted one of two food pellets identical in all ways except for their specific flavor (Zener and McCurdy 1939; Holland and Rescorla 1975). After training, one of the food pellets was devalued. When the CS predicting the devalued food pellet was presented, rats showed attenuated food-port approach CRs, but when the other CS was presented rats continued to respond as normal, even though this CS predicted a reward that was very similar to the one that had been devalued. The aforementioned selective PIT effect also provides evidence of encoding of more specific features of the US in the S–O association (Table 4). Moreover, that an animal given eyeblink conditioning displays the blink CR in only the eye on which the US was applied (Betts et al. 1996) supports the encoding of specific location information by the CS.

Table 4.

Pavlovian instrumental transfer (PIT) design and results

	Training	Result
Selective PIT
Pavlovian conditioning	Tone→food 1 Noise→food 2	Tone→strong CR Noise→strong CR
Instrumental conditioning	Response 1→food 1 Response 2→food 2	Acquire both independent actions
PIT test	ITI, tone, noise Response 1→Ø Response 2→Ø	ITI: response 1 ≅ response 2 Tone: response 1 > response 2/response 1 > ITI press Noise: response 1 < response 2/response 2 > ITI press
General PIT
Pavlovian conditioning	Tone→food 1 Noise→Ø	Tone→strong CR Noise→no CR
Instrumental conditioning	Response→food 2	Acquire instrumental action
PIT test	ITI, tone, noise Response→Ø	Tone press > ITI press Noise press ≅ ITI press
Devaluation	Food→LiCl	Food is rejected
PIT test	ITI, tone, noise Response→Ø	Tone press > ITI press Noise press ≅ ITI press Tone approach CR ≅ ITI approach CR Noise approach CR ≅ ITI approach CR

An appetitive conditioned stimulus (CS) can both bias the selection of instrumental action (outcome-specific PIT) by way of generating a detailed representation of the paired unconditioned stimulus (US), and can invigorate the performance of a nonselective range of instrumental actions by way of the CS acquiring general motivational properties. Counterbalancing is not represented. CR, Conditional response; LiCl, lithium chloride; ITI, intertrial interval.

There is ample evidence that more than just the identity-specific information can be encoded during Pavlovian conditioning. In PIT, a Pavlovian relationship is first trained and then, subsequently, an instrumental action is trained. In the critical transfer test, the Pavlovian stimulus is presented while the subject performs the previously trained instrumental action and the effect of the CS on the instrumental action is observed. The first example of PIT was by Estes and Skinner (1941), who found that a tone previously paired with shock could suppress a rat's lever pressing for food. When the instrumental action and the Pavlovian relationship are trained with different rewarding outcomes, the task is referred to as general PIT because it assumes that it is the general motivational properties of the CS that are transferred allowing the CS to invigorate a nonselective range of reward-seeking behaviors (Balleine 1994; Corbit et al. 2007). These data support formation of an associative link between a CS and the general appetitive properties of the US. The idea that a Pavlovian CS provides a motivational influence over instrumental action is referred to as incentive motivation (Bolles and Zeigler 1967; Rescorla and Solomon 1967).

A similar conclusion is reached by exploring conditioned reinforcement effects in which an appetitive CS can serve to reinforce a new instrumental association in the absence of any US (Rescorla and Solomon 1967; Williams 1994). Both the general PIT and conditioned reinforcement phenomena suggest that the CS has taken on (or has access to) the general motivational value of the US. Interestingly, neither general PIT (Rescorla 1994; Holland 2004) nor conditioned reinforcement (Parkinson et al. 2005) are sensitive to US devaluation, suggesting that these behavioral responses are not mediated by a representation of the identity-specific details of the US. That general PIT (Balleine 1994) is sensitive to changes in motivational (e.g., hunger/thirst) state, suggests that some general features of the US important for determining its current biological significance (e.g., fluidic or caloric properties) are encoded in the Pavlovian associative structure that guides this form of CR (Balleine 1994).

These experiments provide evidence that many different features of the US can be encoded in the S–O associative structure guiding Pavlovian conditioning responding. These features may each have a different node that can be activated by external presentation of the US itself or by a CS (Delamater and Oakeshott 2007), or may exist in a single hierarchical representation of the US (Dayan and Berridge 2014). In either case, the level of detail accessed by the CS is determined by a variety of conditioning factors. In one interesting example of this (Vandercar and Schneiderman 1967), rabbits were conditioned that a tone predicted an eye shock and both heart rate and eye blink CRs were measured. If rabbits encoded the details of the shock US, they would be expected to show the very specific eye blink CR. If they encoded the general aversive nature of the US, the tone should elicit an increase in heart rate (fear CR). Results showed that the tone elevated both heart rate and the eye blink CRs, but only if it predicted the shock with a short latency. If the tone predicted the shock with a longer latency, only the heart rate CR was elevated, suggesting that the tone only had access to a fairly undetailed US representation. As mentioned above, general PIT also relies on a relatively undetailed US representation (i.e., insensitive to devaluation) (Balleine 1994), but this very same CS will also elicit food-port approach CRs that do require a detailed US representation (i.e., sensitive to devaluation).

This multifaceted conditioned responding suggests that multiple forms of learning may occur during Pavlovian conditioning. These different learning forms may be differentially recruited based on the nature of the CS–US predictive relationship. Wagner has proposed a computational model of conditioning that is based on the idea that both the specific information and the general motivational or emotional aspects of the US can develop their own associative links with elements of the CS and that these two different types of associative links form with different temporal dynamics (Wagner and Brandon 1989). Perhaps more importantly, this suggests that Pavlovian conditioning may use multifaceted neural mechanisms with different types of Pavlovian associations requiring different neural circuitry. We discuss this below in more detail (see section on appetitive conditioning).

CR Deliberateness

Although Pavlovian conditioned responding can involve a cognitive expectation of a predicted rewarding or aversive stimulus, these are not deliberative actions intended to facilitate consumption or avoidance of the US. Holland discovered this by manipulating the CS–US contingency. He paired light with food delivery, a preparation in which rats quickly learn to approach the food-delivery port on presentation of the light. In this experiment, however, CR performance during the light (but before food delivery) would omit the US. Rats acquired and maintained an approach CR during the light to a similar degree as a yoked-control group for which there was no response contingency, even though this resulted in a considerable loss of available food (Holland 1979a). This and related results provide the critical distinction between Pavlovian and instrumental responding, which is sensitive to such contingency changes.

THE TOPOGRAPHY OF THE CONDITIONED RESPONSE

In discussing S–O versus S–R associations above, we pointed out that contrary to the prediction of S–R theory that the CR and the UR should be similar to each other, they are often different. This is strikingly the case when tolerance-producing drugs are used as the US. The development of tolerance can largely be accounted for by the development of a CR that is antagonistic to the UR (Siegel 1991). With drugs that produce sensitization rather than tolerance (Robinson and Berridge 1993), the CR does resemble the UR and the summation of two similar responses results causes the sensitization. The finding that drug CRs can be similar or different is scientifically unsatisfying because an a priori prediction is difficult. Eikelboom and Stewart (1982) proposed a resolution to the issue by saying that the CR always resembles the UR, but that drugs have both a direct effect and also activate compensatory responses. They asserted that the UR was really the compensatory response and was, therefore, the same as the CR. However, one is still left in the position in which a priori it is unknown whether the UR to the drug is compensatory or not until one determines the direction of the CR. This is only part of the complexity. A perplexing example of additional complexity occurs when insulin is the US and changes in blood sugar are measured as a CR. Both hyper- (Siegel 1975b) and hypoglycemia (Woods et al. 1969) have been reported by different investigators, with the critical determinant of the CR's direction being the shape of the context used as a CS (Flaherty and Becker 1984).

The finding that CS form can sometimes dictate the form of the CR also occurs with straightforward appetitive conditioning. Holland (1977) discovered that a tone paired with food caused a “head-jerk” reaction, whereas a light paired with the same US caused a rearing response. This happens even when the two stimuli are conditioned in compound and then tested as elements. Further, if the tone and light are paired in a second-order conditioning procedure, the second-order CS still causes the CS-specific response even though it was paired with the response to the other CS. This topography of the CS-related CR seems to be related to the initial orienting response produced by these stimuli. Light presentation produces rearing, whereas tone produces a head jerk. These responses rapidly habituate but return if the CS is consistently paired with food. Interestingly, these CS-determined CRs do not occur if the same CSs are paired with a shock US. In that case, the CR is always freezing (Holland 1979b).

Fear conditioning provides another striking example of how CRs and URs are often unrelated. Electric footshock, the most common US in fear conditioning, produces an activity burst, but the CR to stimuli paired with the footshock is a freezing response (Fanselow 1980a, 1982). Freezing is never produced by the shock itself. For example, if a rat is placed into a chamber and given a shock immediately on placement in the chamber no conditioning occurs and no freezing occurs (Fanselow 1986b). Clearly, shock does not produce a freezing UR. Importantly, this lack of conditioning with immediate shock has been shown with fear-potentiated startle, fear-induced analgesia, and inhibitory avoidance (Fanselow et al. 1994; Kiernan et al. 1995). The reason no conditioning occurs to the context is because the animal requires some minimal time to process the context before it can serve as a CS. These unique aspects of context conditioning are reviewed elsewhere (see Fanselow 2010).

Given the absence of a single set of rules that can effectively specify the relationship between the behavioral topographies elicited by CS and US, we need to look elsewhere to understand just what a CR will look like. That understanding comes from putting Pavlovian conditioning in a broader functional perspective.

EVOLUTIONARY FUNCTION OF PAVLOVIAN CONDITIONING

Conditioning as Adaptation

Obviously, to the extent we can anticipate future events, we will have the opportunity to behave more adaptively. If Pavlovian learning is how we learn relationships between stimuli then this type of learning should allow us to anticipate and alter our behavior to help us either exploit or defend against significant future events. Salivating to stimuli that predict food allows the process of digestion to begin coincident with consumption. Indeed, Pavlov showed that the salivatory CR to a CS paired with food depended on the type of food. The type of saliva was one that aided in digestion of a meat US but helped dilute an acid US. Freezing in response to danger protects against visually guided predators. A conditioned compensatory CR, coming in advance of a drug, helps mitigate the deviation from homeostasis caused by the substance. Consistent with this, Pavlovian learning proceeds best if the CS occurs shortly before the US (Fanselow 2010). The exact temporal scale varies with the type of conditioning. For eyeblink conditioning, the ideal interval between CS onset and US onset is measured in tenths of seconds; in taste aversion, it is in the tens of minutes and sometimes hours. But both are better learned when CS precedes US. The differences in time scale also makes sense from a functional perspective; dust in a wind blast assaults our eyes much more rapidly than a toxin in food assaults our gut.

Explanations in terms of adaptive function often take the form of logical but unsubstantiatable “just-so stories.” At their best, metrics of adaptability are rules of thumb—obtaining more calories with less effort must be adaptive. However, Pavlovian conditioning is one of the few areas in biology in which there is direct experimental evidence of biological fitness. In an experiment with male blue gouramis, Hollis and colleagues (1989) paired a blue light CS with the opportunity to see, but not interact with a female US. Over the course of training, males acquired courting responses to the blue light. The critical test was when all fish were presented with the light and then the barrier separating the males and females was removed. Several days later, the number of offspring was counted and the paired males produced several orders of magnitude more fry than those for which the CS and US had been unpaired. This is a direct confirmation that Pavlovian conditioning enhances reproductive success! Matthews and colleagues (2007) found similar results in male quail, in which conditioning leads to increased sperm production and an increased number of fertilized eggs. It should also be noted that these experiments build on a literature that shows that virtually every aspect of reproduction from hormonal responses (Graham and Desjardins 1980) to sexual performance (Zamble et al. 1985) and to attraction (Domjan et al. 1988) are significantly influenced by Pavlovian conditioning.

Functional Behavior Systems Approach to CR Topography

When we recognize that CRs have biological utility, then the topography of the CR must be one that is functional in that context. Bolles's recognition that Pavlovian fear conditioning activated defensive behavioral systems predicts that defensive behaviors such as freezing should be CRs (Bolles 1970). Functional systems are typically organized in a temporal sequence. To obtain food, we must decide to forage and then search for food. Once found, the food must be procured and only then can consumption ensue (Collier et al. 1972). Each of these phases of feeding requires completely different behaviors. Timberlake suggested that a CS for food should produce CRs that are appropriate for a particular phase depending on the temporal relationship between CS and US. When the CS–US interval is short the CR will be a consummatory response (e.g., salivation); when it is long, it will be general search behavior (e.g., approach to the food port) (Timberlake 1994). Similar sequences happen in sexual conditioning (Domjan 1994).

Defense is also organized in distinct phases along a predatory imminence scale that is anchored by safety at one end and predator attack at the other (Fanselow and Lester 1988; Fanselow 1994). Again different behaviors are appropriate at different points on the continuum. The rat forages at night to reduce the possibility of encountering a predator, but if a predator is encountered it freezes to reduce the likelihood of attack. However, if the predator makes contact the rat stops freezing and makes vigorous attempts at escape. We have suggested that in fear conditioning the CR is one step lower than the response to the US (Fanselow 1989). A shock, which models painful contact with the predator, produces as a UR a vigorous activity burst. However, a CS paired with shock produces freezing. When a rat lives in a context and receives very infrequent presentations of shock, rather than freeze it adjusts its meal patterns. This approach is called a functional behavior systems approach and it offers considerable power in explaining the CR–UR relationship (Timberlake and Fanselow 1994).

A MODERN DEFINITION OF PAVLOVIAN CONDITIONING

The common definition of Pavlovian conditioning, that via repeated pairings of a neutral stimulus with a stimulus that elicits a reflex the neutral stimulus acquires the ability to elicit that the reflex, is neither accurate nor reflective of the richness of Pavlovian conditioning. Rather, Pavlovian conditioning is the way we learn about dependent relationships between stimuli. As Bolles and Fanselow (1980) stated, “the heart of Pavlovian conditioning … is the change in meaning of the CS; a once neutral cue becomes significant for the animal because it serves as a signal for the US.” CRs are not limited to replicas of a reflex elicited by the US, but are functional sets of behaviors that facilitate adaptive responding in the face of that US. Our modern definition of Pavlovian conditioning is “the process whereby experience with a conditional relationship between stimuli bestows these stimuli with the ability to promote adaptive behavior patterns that did not occur before the experience.” A CR is any response that can be directly attributed to that conditional relationship.

In the above definition, Pavlovian conditioning is considered a process and not a mechanism. Conditioning is embedded in the neural systems that evolved for very different functions (e.g., defense, reproduction, feeding). There is no Pavlovian learning system per se; rather, because of the adaptive value of anticipating events, Pavlovian conditioning appears to have evolved independently within each of these systems. At a process level, each type of Pavlovian conditioning has general similarities. For example, CSs condition better when they precede the US. The best predictors gain associative strength at the cost of other potential predictors. Process models such as Rescorla–Wagner and Pearce–Hall apply very generally. But there are specific differences. Eyeblink, fear, and taste conditioning have their own timeframes and tolerate different delays between CS and US. Some types of learning, such as taste aversion and fear conditioning, are exceedingly rapidly learned, perhaps because a one-time mistake in these domains has dire evolutionary consequences. However, Pavlovian conditioning of specific motor behaviors are often slow, perhaps because to be effective they must be highly refined and well timed. This becomes clear when one focuses on the brain circuits that support conditioning. There is little overlap in the circuitry of functionally distinct classes of conditioning. Thus, at a mechanistic level each type of conditioning needs to be considered on its own. In the remainder of this review, we provide a bit more detail about two such examples: fear conditioning and appetitive conditioning.

IN-DEPTH LOOK AT TWO FORMS OF CONDITIONING

Fear Conditioning

The laboratory study of fear conditioning began with Watson and Rayner's famous “Little Albert” demonstration in which a young child learned to fear an initially attractive white rat that was paired with a disturbing loud noise (Watson and Rayner 1920). During much of the last century, fear conditioning was used as a way to examine learned motivation in rats (e.g., Miller 1948). Much of this work used fear CSs to motivate avoidance responses. However, the avoidance literature led to few advances in our understanding of fear per se. Furthermore, the rules of reinforcement seemed to depend more on the particular avoidance response investigated than any general reinforcement process (Bolles et al. 1966). Indeed, rats often seemed incapable of learning avoidance responses, even when those responses occurred contiguous with the putative reinforcement (Bolles 1969). All of this changed when Bolles (1970) argued that what happens during fear learning is an activation of a defensive behavior system that functioned to limit behaviors to those that evolved to protect animals from danger, particularly predation. His species-specific defense reaction theory focused research on fear as an investigation of defensive behavior and that refocusing spurred on the detailed understanding of fear we have today.

The use of fear conditioning as a tool for understanding Pavlovian conditioning dramatically increased when Annau and Kamin (1961) developed a convenient metric for fear learning. They used Estes and Skinner's (1941) conditioned suppression task but simply suggested that one could quantify suppression as a ratio between CS and pre-CS responding. Conditioned suppression in rats, using the Annau–Kamin suppression ratio, along with eyeblink conditioning in rabbits (Gormezano and More 1964; Thompson 1988), dominated Pavlovian conditioning research for the next 20 years. Following experiments conducted in Bolles’ laboratory (Fanselow and Bolles 1979; Bouton and Bolles 1980; Sigmundi et al. 1980), freezing gradually replaced suppression as the dominant way of assessing fear learning (Anagnostaras et al. 2010).

Why do rodents freeze in fear conditioning experiments? This follows directly from the recognition of fear as the activation of the functional behavior system serving defense. Fear conditioning activates one particular phase of defense, the postencounter phase when a predator has been detected, but is not on the verge of contact (Fanselow and Lester 1988). Freezing is effective at this point for two reasons: (1) stationary prey are more difficult to detect than moving prey, and (2) for many predators, the releasing stimulus for attack is movement. Several other things go on while the rodent freezes, heart rate changes, blood pressure increases, and breathing becomes shallow and rapid. Pain sensitivity is also decreased. If freezing fails to avoid attack the rat will burst into a protean frenzy akin to panic. Through freezing, the rat readies itself for such an activity burst and this preparation can be measured as a potentiated startle response to a loud noise. All of these responses can and have been used to measure conditional fear. Such measures have identified a critical descending circuit for fear learning.

The Descending Fear Circuit

The BLA complex is the hub of the fear circuit (Fanselow and LeDoux 1999). This frontotemporal cortical region consists of the lateral, basolateral, and basomedial nuclei and receives input from the thalamus and from other cortical regions (Swanson and Petrovich 1998). Importantly, both CS and US information converges on single neurons within the region (Romanski et al. 1993) promoting N-methyl-d-aspartate receptor (NMDA)-dependent synaptic plasticity (Miserendino et al. 1990; Fanselow and Kim 1994; Maren and Fanselow 1995). The resulting long-term potentiation (LTP) is what gives the CS the ability to activate defensive behaviors. Indeed, BLA plasticity is sufficient to produce fear learning. Using a mouse with a global knockout of the Creb (adenosine 3′,5′-monophosphate response element-binding protein), which is needed for LTP, Han and colleagues (2007) found they could rescue fear conditioning by replacing Creb in the BLA. This should not be taken to mean that plasticity in other regions is not normally involved in fear conditioning; blocking RNA synthesis, also needed for LTP in regions upstream of the BLA also attenuates fear conditioning (Parsons et al. 2006; Helmstetter et al. 2008). At least some of this upstream plasticity depends on ascending BLA input (Maren et al. 2001; Talk et al. 2004).

Three glutamatergic outputs from the BLA drive fear responding. One consists of projections to the nearby striatal-like central nucleus (CeN) and another is to clusters of GABAergic cells lying in the capsule between the BLA and CeN (Paré et al. 2004). These intercalated cells (ITC) in turn project to the CeN. In addition, the BLA projects to several of the bed nuclei of the stria terminalis (BNST). Output from the BNST drives sustained fear responses, while the CeN drives more short-lived responses. So, for example, fear responses to long CSs and contextual cues depend more on the BNST, and those to discrete, brief CSs depend more on the CeN (Davis et al. 1997; Waddell et al. 2006).

The medial portion of the CeN sends GABAergic projections to the regions responsible for individual fear behaviors, such as the periaqueductal gray (vPAG) (freezing, analgesia, vocalization), hypothalamus (hormonal responses and hypertension), dorsal motor nucleus of the vagus (heart rate), and nucleus reticularis pontis caudalis (potentiated startle) (Davis 1989). GABAergic circuitry within the CeN regulates this output (Haubensak et al. 2010). For example, the lateral nucleus does not project to the medial region of CeN that contains projections neurons. Rather, it activates cells in the lateral portion of CeN. This region consists of opposing fear on and off cells that contact the medial CeN's projection neurons.

The medial CeN generates two important CRs, freezing and analgesia, by projecting to the ventral portion of the vPAG (Fanselow 1991). Within the vPAG the freezing and analgesic CRs have different neurochemical coding as the analgesia, but not freezing, is blocked by opioid antagonists (Helmstetter and Fanselow 1987). The vPAG in turn projects to the rostral ventral medulla and then to motor neurons in the ventral horn of the spinal cord to produce freezing and to the dorsal horn of the spinal cord to produce analgesia by inhibiting ascending pain information (Morgan et al. 2008).

Negative Feedback and Error Correction

Given that the analgesia-producing descending opioid circuitry is engaged as a fear CR, as the CR builds, the reinforcing ability of a painful US will be reduced. Thus, endogenous opioids provide critical negative feedback regulation of fear learning that is responsible for Rescorla–Wagner type effects. The earliest evidence for this comes from the finding that the opioid antagonist naloxone prevents a phenomenon called preference for signaled shock (Fanselow 1979). If given a choice between two environments that deliver identical shock except that in one the shock is preceded by a CS and the other in which it is unsignaled, rats chose the signaled shock environment (Lockard 1963). This occurs because the tone overshadows context conditioning in the signaled case, but in the unsignaled situation all the associative strength goes to the context (Fanselow 1980b). Thus, the rat is choosing to go to the least frightening context. Naloxone, by antagonizing the negative feedback mechanism, prevents this from happening. Naloxone also prevents conventional overshadowing of a light CS by a tone CS (Zelikowsky and Fanselow 2010). Kamin blocking is also prevented by systemic (Fanselow and Bolles 1979) and intra-PAG administration of opioid antagonists (Cole and McNally 2007).

The negative feedback regulation of fear is perhaps most clearly illustrated by the effects of opioid antagonists on simple learning curves. The Rescorla–Wagner model predicts that the asymptote of learning should depend on US intensity (Rescorla and Wagner 1972) and the empirical evidence for this is unequivocal (Annau and Kamin 1961; Young and Fanselow 1992). Functionally, this makes sense; a mild threat should not produce overwhelming fear, but a significant threat should. When opioid antagonists are given during learning asymptotic levels become high and undifferentiated by shock intensity (Fanselow 1981; Young and Fanselow 1992). Indeed, a shock intensity that is normally barely able to condition detectable levels of fear will, under the antagonist, result in the same asymptote as a maximally effective US.

Opioid antagonists, to some extent, also prevent changes in associative strength that are caused by negative ΔV values such as in extinction (McNally et al. 2004a) and overexpectation (McNally et al. 2004b). Thus, in fear conditioning, an opioid negative feedback circuit accounts for the same wealth of phenomena as the Rescorla–Wagner model. In strong support of this notion, there are neurons in both the vPAG and BLA that respond as if they encode the critical error signal (i.e., ΔV) (Johansen et al. 2010).

Mechanistic Models

Knowledge of this circuit should allow us to advance from process models, such as Rescorla–Wagner and Pearce–Hall, to mechanistic models that incorporate what we know about the circuitry and synaptic plasticity. Toward this goal, Krasne and colleagues (2011) have proposed a computational model that uses LTP rules at afferents onto BLA excitatory and inhibitory neurons from the prefrontal cortex, thalamus, and hippocampus that drive both freezing and a negative regulation of fear learning at the PAG. The model predicts the phenomena anticipated by process models such as blocking and extinction. However, the model also predicts some of the phenomena that have been difficult to accommodate in process models, notably renewal of extinguished fear. The model suggests that this is accomplished by potentiation of inhibitory neurons some of which encode conjunctions of context, CS, and extinction. Such conjunctive information is communicated to the BLA as connections between the BLA, prefrontal cortex (extinction processing), and hippocampus (context processing) are necessary for renewal (Orsini et al. 2011). The model requires that extinction promotes LTP of these neurons, and extinction is, indeed, blocked by intra-amygdala application of NMDA-antagonists (Falls et al. 1992). Finally, the BLA contains neurons whose increased activity during extinction coincides with the reduction of activity in neurons that were potentiated by fear acquisition (Herry et al. 2008). Indeed, extinction in a novel context recruits a unique population of context-dependent neurons in the BLA (Orsini et al. 2013).

Appetitive Conditioning

Appetitive Pavlovian conditioning controls a large majority of our reward-related behavior and is most often studied the laboratory with a discrete tone or light CS and food US in hungry rodents. The CS can then come to elicit a variety of CRs. These include specific consummatory reactions (mouth movements related to consummation of the specific reward) (Grill and Norgren 1978), conditioned approach to the food source (i.e., goal approach or “goal tracking”), and, if the CS is visual and localizable (Cleland and Davey 1983), conditioned approach to the stimulus itself (i.e., “sign tracking” or autoshaping) (Brown and Jenkins 1968; Boakes 1977). All of these CRs are sensitive to posttraining revaluation of the US, suggesting that, at least to some extent (see Holland 2008), they are guided by a cognitive image of the specific predicted reward (Holland and Rescorla 1975; Cleland and Davey 1982; Berridge 1991). Food-predictive CSs can also serve to reinforce instrumental behavior (i.e., conditioned reinforcement), invigorate a nonselective range of instrumental actions (i.e., general PIT) and induce conditioned locomotor activation (Estes 1948; Rescorla and Solomon 1967; Lovibond 1983; Dickinson and Dawson 1987). All of these conditioned behaviors are insensitive to US devaluation (Rescorla 1994; Holland 2004; Parkinson et al. 2005), suggesting that they are not guided by retrieval of an identity-specific mental representation of the reward. Rather these Pavlovian “incentive motivational” effects of the US result because the CS acquires general motivational value (Konorski 1967; Dickinson and Dawson 1987) via a connection with the more general (e.g., caloric or fluidic properties). These are not mutually exclusive; the same food-predictive CS can both induce goal approach CRs and invigorate instrumental activity (PIT). Moreover, although goal- and CS-approach conditioning responses are sensitive to US devaluation, these effects are smaller than the complete rejection of the devalued food that results from such procedures. This suggests that Pavlovian conditioning can engender multiple associative processes and may, therefore, engage a variety of levels of neural processing. Below, we consider the neural mechanisms responsible for the development of appetitive Pavlovian CRs and evaluate how these differ depending on the form of the response (and presumably also the form of learning).

Dopamine and Reward-Prediction Error

The activity of dopamine neurons in the ventral tegmental area (VTA) and substantia nigra (SNc) displays many properties of a Rescorla–Wagner/temporal difference reward-prediction error signal. Unexpected reward delivery results in a phasic increase in dopamine cell activity, but when the reward is expected based on the presence of a CS, it no longer elicits such activity; rather the unexpected presentation of the CS induces the phasic response (Schultz 2002). If a conditioned reward is unexpectedly delivered, it elicits a phasic increase in dopamine cell activity (Hollerman and Schultz 1998). Both omission of an expected reward or presentation of an aversive event will induce a phasic pause in tonic dopamine cell activity (Tobler et al. 2003; Ungless et al. 2004). These seminal findings have been replicated in dopaminergic cells identified by optogenetic “phototagging” (Cohen et al. 2012). Moreover, prediction error-like phasic dopamine release has been detected in striatal terminal regions (Day et al. 2007; Roitman et al. 2008; Brown et al. 2011; Wassum et al. 2012; Hart et al. 2014).

Dopamine has also been causally linked to reward-prediction error. A pretrained CS will not only block acquisition of the CR to a novel stimulus, but will also prevent the burst firing in dopamine neurons that would typically occur during learning (Waelti et al. 2001). Such blocking will be prevented if a reward-prediction error-like dopamine signal is artificially induced at the time of reward delivery with optogenetic activation of VTA dopamine cells (Steinberg et al. 2013). These and related studies demonstrate that phasic dopamine release can convey a short-latency, phasic reward signal indicating the difference between actual and predicted rewards important for driving learning.

That phasic mesolimbic dopamine release acts as a reward-prediction error signal selectively when the CS becomes a “motivational magnet” to elicit sign tracking (Berridge and Robinson 2003) suggests that phasic dopamine, at least in the nucleus accombens (NAc), may have a role beyond a passive learning process. In support of this, phasic mesolimbic dopamine release is both necessary for and tracks the ability of a food-paired stimulus to PIT (Wassum et al. 2011a, 2013). Moreover, enhancing the activity of dopamine release specifically in the NAc will enhance both PIT (Wyvell and Berridge 2000) and sign tracking (Peciña and Berridge 2013). Phasic mesolimbic dopamine is, therefore, also involved in the incentive motivational impact of rewards and related stimuli that contribute to the online performance of CRs (for further review, see Berridge 2007).

The above evidence suggests that phasic mesolimbic dopamine release may serve a dual role in Pavlovian conditioned responding, involved in both reward prediction error-mediated acquisition of motivational value to reward-predictive stimuli and the online contribution of this Pavlovian incentive motivation to conditioned responding and reward seeking. This latter response-invigorating function is consistent with the physiological effects of phasic dopamine on striatal projection neurons (SPNs). These GABAergic SPNs can be divided into two projection systems: a direct path to the basal ganglia output nuclei and an indirect path to these output nuclei. Direct pathway activation triggers behavior by disinhibiting motor control areas (Deniau and Chevalier 1985; Freeze et al. 2013; Goldberg et al. 2013). These direct pathway SPNs selectively express the D1, but not D2, dopamine receptors (Surmeier et al. 2007) that have a high affinity for phasic (rather than tonic) dopamine release (Corbit and Janak 2010). Because of coupling to G_αs/olf G-proteins that stimulate cyclic AMP (cAMP) and the activity of protein kinase A (PKA), D1 receptor activation leads to increased SPN excitability (for review, see Gerfen and Surmeier 2011). Therefore, high-concentration dopamine surges activate D1Rs in the direct path, increasing the excitability of these SPNs and, thereby, transiently disinhibiting motor output nuclei.

Circuitry for General Motivational Value

The mesolimbic and nigrastriatal dopamine pathways are only part of the appetitive Pavlovian conditioning circuit. Indeed, dopamine release can be modulated at both the cell body and terminal fields by varied inputs. The CeN is a large component of this circuit. This region projects to both the VTA and SNc (Gonzales and Chesselet 1990; Fudge and Haber 2000), which innervate the NAc and dorsal striatum, respectively, and these projections can have an indirect excitatory effect on a subpopulation of dopamine neurons through inhibition of local GABAergic interneurons (Rouillard and Freeman 1995; Chuhma et al. 2011). The CeN is required to learn from (negative) reward-prediction error (Holland and Gallagher 1993; Holland et al. 2001; Haney et al. 2010) and CeN-SNc-projecting neurons will become activated after learning by reward-prediction error, suggesting that communication between these structures relates to reward-prediction error-mediated learning (Lee et al. 2010). An intact CeN is also required for the acquisition of conditioned-orienting (Gallagher et al. 1990) and sign-tracking responding (Parkinson et al. 2000), but is not required for more specific consummatory CRs (Chang et al. 2012). Similarly, the CeN is required for general, but not outcome-specific PIT (Hall et al. 2001; Holland and Gallagher 2003; Corbit and Balleine 2005). These data suggest that the CeN is required for the acquisition of general motivational value to a CS, but not for acquisition or use of a detailed S–O association. In further support of this hypothesis, rats with CeN lesions are both able to acquire Pavlovian conditioned food port (i.e., goal) approach responding and to flexibly adjust such responding to the current value of the US (Hatfield et al. 1996).

In addition to the CeN, the lateral habenula (LHb) may also play a role in appetitive Pavlovian conditioning by influencing phasic striatal dopamine signaling. The LHb projects to both the VTA and SNc (Herkenham and Nauta 1979), and stimulation of this structure inhibits the activity of dopamine neurons in these regions (Christoph et al. 1986; Ji and Shepard 2007; Matsumoto and Hikosaka 2007). Interestingly, LHb neurons show a firing pattern opposite to a reward-prediction error signal (Matsumoto and Hikosaka 2007), suggesting a potential inhibitory influence over dopamine-mediated reward-prediction error signals. Of course, the circuitry surrounding this influence is more complex than this (for review, see Hikosaka 2010).

Circuitry of Outcome Representations

Pretraining lesions of the orbitofrontal cortex (OFC), BLA, NAc shell and core, and mediodorsal thalamus (MD) do not prevent the acquisition of Pavlovian goal approach responding, but do render this behavior insensitive to US devaluation. Because sensitivity to US devaluation requires a rather detailed reward representation, these regions comprise a circuitry important for the encoding of details in the acquired S–O association and/or the use of this information to guide conditioned responding.

Evidence suggests that the OFC is uniquely important for the acquisition and integration of information about the specific identifying features of US for both Pavlovian learning and performance. Pre- and posttraining OFC lesions result in Pavlovian-conditioned responding that is insensitive to devaluation (Pickens et al. 2003; Ostlund and Balleine 2007). The OFC is not required for choosing between valued and devalued rewards when they are present, supporting a primary role in the use of a specific reward representation. Moreover, OFC lesions abolish outcome-specific PIT (Ostlund and Balleine 2007), such that reward-predicative stimuli are unable to retrieve a detailed cognitive representation of the US and are, therefore, unable to bias action selection. Similar findings with US devaluation procedures have been found in humans (Gottfried et al. 2003) and in nonhuman primates (Murray et al. 2007; for further review, see McDannald et al. 2014a).

OFC neurons can fire in response to appetitive CSs and in anticipation of the predicted reward (Schoenbaum et al. 2003) and this encoding depends on BLA input. The converse is also true; associative encoding in BLA neurons relies on OFC function (Saddoris et al. 2005). The OFC and BLA therefore function in a circuit vital for encoding and retrieving reward-specific representations in Pavlovian S–O associations. Indeed, as with the OFC, pretraining BLA lesions result in CRs that are insensitive to US devaluation (Hatfield et al. 1996). The BLA is also vital for specific PIT (Corbit and Balleine 2005). Recent evidence suggests that rapid glutamate signaling within the BLA mediates this effect and encodes specific reward representations (Malvaez et al. 2015). The BLA's role in representing outcome-specific information is limited to motivationally significant events. An intact BLA is not required to represent the outcome-specific aspects of neutral events (Dwyer and Killcross 2006). Correlates of “valueless” reward representations have been identified in the OFC (McDannald et al. 2014b), suggesting that the BLA may incorporate value in to outcome-specific representations sent from the OFC. Indeed, the BLA is required for attaching motivational significance to rewards themselves (Parkes and Balleine 2013), an effect that relies on μ-opioid receptor activation (Wassum et al. 2009, 2011b).

The BLA and CeN make distinct functional contributions to Pavlovian conditioning. The BLA and CeN are arranged partly in series and much work from aversive Pavlovian conditioning proposes their serial function (Fendt and Fanselow 1999; LeDoux 2000). However, the CeN and BLA each possess independent input and output that allow them to also act in parallel (Balleine and Killcross 2006).

The BLA receives excitatory projections from the MD (van Vulpen and Verwer 1989), which itself is required for Pavlovian conditioning responses to be modified on the basis of posttraining changes in the US value (Mitchell et al. 2007; Izquierdo and Murray 2010). Although, pretraining MD lesions disrupt the sensitivity of Pavlovian CRs to US devaluation, posttraining lesions do not. The MD is, however, recruited posttraining when previous associations need to be suppressed to allow new associations to be formed (Pickens 2008). Posttraining MD lesions will also disrupt outcome-specific PIT (Ostlund and Balleine 2007).

Last, the taste-processing cortices are a prime candidate for US representations, at least for food USs. Indeed, food-paired CSs can activate the gustatory region of the insular cortex (GC) in both rodents (Dardou et al. 2006, 2007) and humans (Veldhuizen et al. 2007; Small et al. 2008). Importantly, these CSs activate the very same neuronal ensembles in the GC that were activated by the food US itself, providing a neural substrate of CS retrieval of a reward representation (Saddoris et al. 2009).

It is, perhaps, not surprising that all of the above structures either directly or indirectly make connections with the striatum. Both the NAc core and shell are required for Pavlovian CRs to be sensitive to outcome devaluation (Singh et al. 2010). Interestingly, as in the amygdala, an NAc dissociation exists in the encoded Pavlovian associations driving PIT. The NAc core is required for the general, but not specific component of PIT, whereas the opposite is true of the NAc shell. A similar dissociation exists in the dorsolateral (DLS) and dorsomedial striatum (DMS), respectively (Corbit and Janak 2007). Although the contribution of the DLS may be more vital for the instrumental component of PIT, evidence does suggest that the posterior DMS is critical for the formation of S–O (Corbit and Janak 2010). If, as mentioned, these striatal structures influence motor output during Pavlovian conditioning, these regions are likely key integration sites where both specific (OFC, BLA, MD, GC) and general (CeN and maybe LHb) information is integrated to influence Pavlovian conditioned responding. How this is achieved is still a matter of intense interest. Of course, the circuitry described here is not complete. The ventral pallidum (Smith et al. 2009), anterior cingulate cortex (Cardinal et al. 2003), and hippocampus (Ito et al. 2005; Gilboa et al. 2014), to name a few, have all been implicated in Pavlovian conditioning and more work is needed to fully delineate the entire circuit.

CONCLUSIONS

Pavlovian conditioning is the process underlying how the brain represents relationships between environmental stimuli. The circuits that serve conditioning are information-processing circuits that correct their errors in prediction and allocate associative strength to the best predictors of significant events. Formation of those associations depends on conditional relationships between the relevant stimuli and is not a result of simple pairing. Through those associations, CSs can exert behavioral change either directly by promoting specific behaviors (S–R associations), or indirectly, by activating representations of the events they predict (S–O associations). In the latter, CSs can activate a general representation of the hedonic/motivational aspects of a US and by so doing promote or inhibit alternate classes of behavior. But they can also activate a representation of the specific sensory aspects of the US, through which they promote behaviors directed at obtaining, or avoiding, specific outcomes. Thus, Pavlovian stimuli are crucial to many, if not most, of the behaviors in which vertebrate animals partake.

Pavlovian conditioning is a functional process. By function we mean that CS-elicited behaviors directly impact biological fitness. By process we mean that a single circuit or mechanism does not mediate conditioning. Rather, the mechanisms of conditioning are imbedded in circuits that serve specific adaptive functions such as feeding, reproduction, and defense. However, likely through convergent evolution, there is considerable generality in the rules that govern conditioning. Thus, there is considerable power and generality in models of this process such as those of Rescorla and Wagner (1972) and Pearce and Hall (1980). To the extent that we know the specifics of the circuit, there appears to be a lot of generality of how the circuits are wired, although the specifics vary. One commonality seems to be a negative feedback loop by which the CS activates a circuit that dampens the reinforcing US input. This feedback regulates the amount of learning, serving an adaptive function of keeping the level of conditioning within a maximally functional range. Perhaps Rescorla (1988) said it best when he stated, “Pavlovian conditioning, it is not what you think.”