Behavioral and neurobiological mechanisms of pavlovian and instrumental extinction learning

Abstract

This article reviews the behavioral neuroscience of extinction, the phenomenon in which a behavior that has been acquired through Pavlovian or instrumental (operant) learning decreases in strength when the outcome that reinforced it is removed. Behavioral research indicates that neither Pavlovian nor operant extinction depends substantially on erasure of the original learning but instead depends on new inhibitory learning that is primarily expressed in the context in which it is learned, as exemplified by the renewal effect. Although the nature of the inhibition may differ in Pavlovian and operant extinction, in either case the decline in responding may depend on both generalization decrement and the correction of prediction error. At the neural level, Pavlovian extinction requires a tripartite neural circuit involving the amygdala, prefrontal cortex, and hippocampus. Synaptic plasticity in the amygdala is essential for extinction learning, and prefrontal cortical inhibition of amygdala neurons encoding fear memories is involved in extinction retrieval. Hippocampal-prefrontal circuits mediate fear relapse phenomena, including renewal. Instrumental extinction involves distinct ensembles in corticostriatal, striatopallidal, and striatohypothalamic circuits as well as their thalamic returns for inhibitory (extinction) and excitatory (renewal and other relapse phenomena) control over operant responding. The field has made significant progress in recent decades, although a fully integrated biobehavioral understanding still awaits.

CLINICAL HIGHLIGHTS.

This article reviews behavioral neuroscience research on extinction, the phenomenon in which learned behaviors are weakened or eliminated by removing the motivationally significant outcome that made them possible. Extinction is a representative process of behavior change, and it is used in cognitive behavioral treatments and therapies. However, extinction does not typically erase the original learning, and extinguished behavior may be vulnerable to relapse. The review provides insights into the behavioral and neural mechanisms of extinction in Pavlovian and instrumental learning that may help facilitate the development of treatments that can further maintain behavior change.

1. INTRODUCTION AND OVERVIEW

The aim of this article is to summarize the behavioral and neural processes that are involved in extinction, the decrease in responding that occurs in either Pavlovian or instrumental (operant) learning when the reinforcer or unconditioned stimulus (US) is no longer presented. The word “extinction” can refer to the result just described or to the procedure in which the reinforcer or US is withdrawn. Extinction has become an intensive focus in behavioral neuroscience over the last few decades for a number of reasons. For one, as a phenomenon of behavior, it is functionally important because it allows a learned behavior to change and adjust as the environment changes. It is perhaps the most fundamental process of behavior change that is studied in the field. Increasingly, it has been identified as a process that underlies many examples of behavior change that are created by psychological or behavioral therapies. For example, in exposure therapy, clinical psychologists aim to reduce anxiety or drug cravings that are evoked by trigger cues that have been associated with emotional trauma or drugs (respectively) by presenting those cues without the significant events. The result is a reduction in a learned behavior, extinction, that is designed to make life better for the client.

In addition to its functional importance, extinction has interested learning theorists because it highlights two paradoxes about learning and behavior. First, although extinction looks like a process in which a learned behavior is simply erased from an organism’s behavioral repertoire, even the earliest work by Pavlov (1) made it clear that an “extinguished” behavior can return or recover, or relapse, spontaneously over time (“spontaneous recovery”). Thus the original learning is at least partly intact in the brain or memory system even though the behavior has gone away. There is a difference between knowledge and behavior, or what learning theorists call learning and performance. The list of recovery effects that support the idea that extinction is not merely an erasure process has grown thanks to behavioral research on extinction that began in the 1970s and continues to this day (e.g., Ref. 2). Those recovery effects (see TABLE 2 description and sect. 2.1) underscore the fact that extinction does not depend on memory erasure but instead depends at least partly on new learning that is often characterized as inhibition. This new inhibitory learning cancels or competes with the original “excitatory” learning that created the behavior in the first place but still leaves its potential to be expressed in behavior at least partly intact.

Table 2.

Recovery or “relapse” effects that occur after extinction (or other forms of retroactive interference) in Pavlovian and instrumental learning

Phenomenon	Description
Renewal	Recovery of extinguished behavior that occurs when the context is changed after extinction. Most often studied when the subject is returned to the original context of conditioning after extinction in a second context (ABA renewal), but it can also occur when the animal is removed from the extinction context and tested in another “neutral” context (ABC renewal; AAB renewal).
Spontaneous recovery	Recovery of extinguished behavior that occurs when the conditioned stimulus (CS) or instrumental response is tested after time has passed following the conclusion of extinction.
Reinstatement	Recovery of extinguished behavior that occurs when the subject is exposed to the US or reinforcer after extinction. Can be controlled by contextual conditioning produced when the Outcome is presented, hence the phenomenon is often strongest in the context in which the Outcome has occurred. Can also occur if the US or reinforcer is itself part of the “context” of acquisition.
Reacquisition	Recovery of responding that occurs when the CS is paired with the US again (Pavlovian), or the response is paired with the reinforcer again (instrumental), after extinction. Often rapid, especially when cues in the background renew conditioned performance (as above). Can be slow when the background cues continue to retrieve extinction.
Resurgence	Recovery of an extinguished instrumental behavior that occurs when extinction is introduced for a second instrumental behavior that has been reinforced to replace it. Almost exclusively studied in instrumental/operant learning.

The second theoretical paradox of extinction is that the rate at which the response goes away (extinguishes) can depend in a rather subtle way on how the behavior was first conditioned. In the so-called partial reinforcement extinction effect (PREE), a behavior that has been reinforced on only some occasions (“partial reinforcement”) is slower to extinguish than a response that has always been reinforced (“continuous reinforcement”). Although folk intuition and early theories of learning could lead one to assume that a behavior that is always reinforced should be strong and especially persistent, the continuously reinforced response is the one that goes away more quickly. The PREE led to an intensive research effort through the 1950s and 1960s that sought to identify the factors that create extinction and behavioral persistence.

Neuroscientists began their focus on extinction beginning in the 1990s and 2000s. The focus was enabled by significant progress that had been made in the 1980s and 1990s in understanding the neural bases of conditioning, and in particular, the neuroscience of fear or threat conditioning (e.g., Refs. 3–6). The work on fear conditioning had major implications for understanding anxiety disorders (e.g., Ref. 7). The study of extinction, a procedure that can reduce that fear, allowed brain scientists to make contact with possible treatments and cures. Given the progress that had been made in understanding the neural basis of conditioned fear, it was only natural to study the processes that could undo it.

One reason that research on the neuroscience of extinction has been successful is that it has built thoughtfully on the prior work at the behavioral level of analysis. This is particularly true of the study of the neurobiology of Pavlovian extinction (e.g., Refs. 8, 9), which has been heavily informed by the relevant behavioral research as well as research on the neurobiology of Pavlovian acquisition processes. The study of the neurobiology of instrumental extinction followed a somewhat different path. It instead largely developed out of early demonstrations of some of the “relapse” phenomena described below (sect. 2.1). There was less initial attention to behavioral research on instrumental extinction or to neurobiological research on instrumental acquisition, both of which became a research focus somewhat later (e.g., Refs. 10–12). Our own view is that behavioral theory provides a systematic framework that gives meaning to the analysis of neural implementations of extinction (see Ref. 13). The present article therefore respects both the behavioral and neural levels of analysis by reviewing research on each of them. We will begin by first considering extinction at the behavioral level. We will consider extinction as it is understood in Pavlovian and instrumental learning (TABLE 1) separately but a clear parallel (with some revealing differences) will emerge. We will then consider the neurobiology of extinction as it is understood in Pavlovian and instrumental learning.

Table 1.

Some behavioral terms and definitions

	Pavlovian Learning	Instrumental (Operant) Learning
		Free Operant	Discriminated Operant
Examples	Tone-shock
	Tone-pellet	Lever press-pellet	Stimulus: Lever press-pellet
Terms	Conditioned Stimulus-Unconditioned Stimulus	Response – reinforcer	Stimulus: Response – reinforcer
	Stimulus-Outcome	Response-outcome	Stimulus: Response-outcome
Abbreviations	CS-US	R-O	S: R-O
	S-O
Acquisition	Conditioned response (CR; e.g., fear, freezing) becomes elicited by the CS	Response (R) increases	Response occurs in S
Extinction	CR declines when the CS is repeatedly presented without the US	R declines when R occurs without O	R occurrence in S declines when it repeatedly occurs in S without O

2. BEHAVIORAL ANALYSIS OF EXTINCTION

As noted above, behavioral research on extinction since the 1970s has underscored the fact that extinction is not permanent but can be undone by a number of experimental manipulations that cause the extinguished response to recover: sometimes referred to as “relapse effects.” The study of these effects has had a major influence on the contemporary understanding of extinction. In what follows, we will describe them, discuss their generality, and then discuss behavioral mechanisms (explanations) of extinction that are consistent with them, including what is thought to be learned in extinction and the theoretical mechanisms that cause it to occur. In the 1970s through the 1990s, the focus of most behavioral research on extinction was extinction in Pavlovian learning, where the perspective that emerged was that extinction depends on the development of an inhibitory process that is expressed primarily in the “context” in which extinction is learned. Subsequent and complementary work in instrumental learning suggests a number of parallels and differences as well.

2.1. Behavior Readily Recovers after Extinction

2.1.1. Pavlovian learning

Several core relapse effects have been identified and studied over the years. They are listed and defined in TABLE 2. The most fundamental of them may be the renewal effect. In this phenomenon, the experimenter manipulates the context in which different parts of the experiment are conducted. Context is usually defined as the room or apparatus in which conditioning or extinction are conducted and learned. In rats and mice, contexts are usually boxes or chambers that are made different by varying as many cues in them as possible (visual cues, odors, textures of the floors, spatial locations in the laboratory, etc.); we will expand the definition of context by considering other types in sect. 2.3. In the most commonly studied form of renewal, conditioning (CS-US pairings) is first conducted in Context A, extinction (CS presentations alone) is conducted in Context B, and once the conditioned response is eliminated, the CS is tested in Context A. The return to Context A can produce a robust recovery (“renewal”) of responding and has been demonstrated in virtually every preparation used to study Pavlovian conditioning. For example, FIGURE 1 illustrates (14) ABA renewal (where each letter denotes the context used for conditioning, extinction, and testing) in both fear conditioning (tone-shock conditioning) and appetitive conditioning (tone-food conditioning); the phenomenon has also been demonstrated in taste aversion learning (e.g., Ref. 15), pigeon autoshaping (16), and several forms of human associative learning (e.g., Ref. 17). A second form of renewal (ABC renewal) occurs when conditioning occurs in Context A, extinction in Context B, and renewal testing occurs in a third context (that has not been associated with either conditioning or extinction). Responding recovers in Context C. A third form of renewal, AAB renewal, is one that occurs when conditioning and extinction are first conducted in Context A and then the test occurs in a second context (B). Here again, the response recovers during the test. Although ABC and AAB renewal are generally assumed to be weaker than ABA renewal (but see, e.g., Ref. 18), the fact that they occur at all indicates that it is not necessary to return to the original conditioning context for the extinguished response to return. Removal of the CS from the context of extinction is sufficient. Bouton and others (19–21) have interpreted such findings to suggest that extinction depends on some form of inhibition that is expressed primarily in the context in which extinction occurs. Thus ABC and AAB renewal effects occur because the response is liberated from an inhibitory process that operates in the extinction context. Moreover, recent neurobiological work (discussed in sect. 3.3.2) suggests that renewal might also involve active suppression of the inhibition learned during extinction. Interestingly, in contrast to extinction, the Pavlovian conditioned response itself often transfers well from the training context to new contexts; in several conditioning preparations, when CS-US pairings occur in Context A, the CS elicits equivalent responding in Context A and Context B, a different context (e.g., Refs. 19, 20; see FIGURE 1 and Ref. 21 for further review). The fact that extinction is more context dependent than conditioning creates an imbalance or asymmetry that is consistent with the ABC and AAB renewal effects and may be a reason why relapse can be easy to observe.

FIGURE 1.

ABA renewal after the extinction of Pavlovian fear conditioning (top) and appetitive conditioning (bottom). In either case, conditioned stimulus (CS)-unconditioned stimulus (US) pairings occurred in Context A (not shown), extinction occurred in Context A (Ext-A), or Context B (Ext-B) (left), and then testing occurred in Context A (right). Top: ABA renewal in conditioned suppression, the fear conditioning method in which a long-duration (e.g., 60 s) CS is paired with footshock, and the “fear” response is indexed by the CS suppression of a baseline of lever-pressing for food; y-axis: the suppression ratio, the number of lever press responses during the CS divided by the number of responses in the CS plus the number in a similar period before the CS. Lower scores indicate more conditioned fear. Extinction (left) occurred at the same rate regardless of whether it occurred in the conditioning context (A) or the other context (B). Renewal in Group Ext-B was strong (right), but fear did not recover to the level of that in a group that received no extinction (NE). [From Bouton and King (19).] Bottom: ABA renewal in appetitive conditioning. Here the behavioral measure is the extent to which an auditory CS associated with a food-pellet US evoked head-jerking behavior. Extinction (left) again occurred at the same rate regardless of context. Extinction is more context-specific than conditioning [From Bouton and Peck (20)]. [Figure reprinted from Bouton (14) with permission from the publisher.]

Another well-known relapse effect is the spontaneous recovery effect that was first reported by Pavlov (1) and we noted above. There, an extinguished response recovers “spontaneously” when time is allowed to pass after extinction has eliminated the response. Spontaneous recovery is widely known (it is described in most introductory psychology textbooks) and accepted as a result that belies the idea that extinction is caused by erasure of the original learning. Ironically, there was little research that attempted to explain it until relatively recently (for one exception, see Ref. 22; for reviews see Refs. 23, 24). Pavlov himself thought that spontaneous recovery suggested that the excitatory conditioned reflex had been inhibited and that inhibition was more “labile” than excitation. A more recent perspective (e.g., Ref. 25) is that spontaneous recovery is a renewal effect that occurs when the CS is tested outside the temporal context in which extinction has been learned. Thus, just as renewal occurs when the physical context is changed after extinction, spontaneous recovery occurs when the temporal context changes. The idea that spontaneous recovery and renewal are due to a common process is consistent with evidence that reminder cues associated with the extinction treatment can attenuate either spontaneous recovery or renewal when they are presented immediately before the test (e.g., Refs. 23, 26, 27). In addition, the idea that the passage of time might change the temporal context is also consistent with other research and indeed a research literature suggesting that many kinds of stimuli can play the role of context (e.g., Ref. 11; see sect. 2.3).

A third well-known relapse effect is reinstatement. In this phenomenon, which was also first reported by Pavlov (1), presentation of the US after extinction can cause the extinguished response to recover to the CS. Reinstatement has been demonstrated in fear conditioning (e.g., Refs. 28–31) and appetitive conditioning (e.g., Refs. 20, 32), although it has been difficult to produce in eyeblink conditioning (e.g., Ref. 33) and its status in taste aversion conditioning is controversial (e.g., Refs. 34, 35). It is another demonstration of the importance of context in extinction: in both fear conditioning and appetitive conditioning, the reinstating US must be presented in the context where testing will occur in order to observe the effect (e.g., Refs. 19, 20, 28, 29). That is, if the US is presented in a different, irrelevant context, no reinstatement is observed in the test context. Studies in fear conditioning suggest that the strength of reinstatement is also correlated with (and can be predicted from) an independent measure of contextual conditioning (19, 28). Such findings imply that reinstatement depends on the context being associated with the US again, and perhaps because such contextual conditioning was part of the “contextual” background that prevailed during the original conditioning, it enables a form of the ABA renewal effect in which a fear or anxiety state evoked by the background functions as Context A (see Refs. 36, 37, for further discussion).

A fourth relapse effect is rapid reacquisition. Here, the conditioned response can rapidly return when the CS is paired with the US again after extinction (e.g., Ref. 33, 38, 39). Rapid reacquisition is another well-known effect that supports the conclusion that some remnant of conditioning persists after extinction, although how that remnant actually translates into the phenomenon is often unexplained, and rapid relearning is not as ubiquitous as is often assumed. For example, early experiments thought to support rapid reacquisition (e.g., Ref. 40) began reconditioning after time had passed after extinction and thus confounded rapid relearning (a faster reacquisition than acquisition curve) with spontaneous recovery. Moreover, reacquisition can be slower than original conditioning after fear conditioning (e.g., Ref. 41, 42) and taste aversion conditioning (43, 44). The reason may be that these forms of conditioning require few original conditioning trials, so that the animal has not had the opportunity to learn that recent CS-US pairings are part of the “context” of other conditioning trials (as opposed to extinction). If the conditions in reacquisition are similar to those during extinction, reacquisition will be slow, but if they are similar to those of acquisition, reacquisition will be fast (e.g., Ref. 38). The results are thus consistent with a contextual account in which recent CS-US trials may be part of the “context” that controls responding to the CS.

The research on renewal, spontaneous recovery, reinstatement, and rapid reacquisition all suggests that extinction depends on something other than erasure or unlearning. Although the effects do not rule out a role for some unlearning during extinction (e.g., Ref. 45; but see Ref. 12), they strongly suggest that extinction must depend at least partly (or even largely) on an active new learning process. One approach has been to think that the CS enters a new, inhibitory association with the US during extinction (25, 46–49). If the original learning is retained, then the CS has two available associations (or “memories”) after extinction, one corresponding to acquisition (e.g., an excitatory CS-US association) and another corresponding to extinction (e.g., an inhibitory CS-“no US” association). Activation of the inhibitory association would cancel a learned response activated by the excitatory one. The availability of two associations makes the current “meaning” of the CS ambiguous, and like an ambiguous word, the current meaning is determined by the current context (25). One illustration of how the system might work is presented in FIGURE 2 (50, 51), where excitatory and inhibitory CS-US associations are present, and the inhibitory one is modulated by the extinction context. Recall that an inhibitory association learned in extinction appears more dependent on the context for its activation or expression than does the excitatory one. Thus removal from the extinction context can be sufficient for the extinguished response to recover.

FIGURE 2.

A cartoon depicting what is learned in Pavlovian extinction. During extinction, a new inhibitory association is learned between the conditioned stimulus (CS) and unconditioned stimulus (US) (blocked red line). The excitatory CS-US association learned during conditioning (green arrow) remains intact; it can activate the representation of the US, causing conditioned responding (not shown). However, its effect is canceled by activation of the inhibitory CS-US association. The inhibitory association is activated by the extinction context (green arrow). Outside the extinction context, the inhibitory association is not enabled, and response recovery occurs. The extinction context thus controls responding by selecting the CS’s inhibitory association with the US.

2.1.2. Instrumental learning

A parallel literature that has focused on extinction after instrumental conditioning (hereafter “instrumental extinction” or “operant extinction”) suggests a picture that is broadly consistent with the Pavlovian findings. All of the relapse effects listed in TABLE 2 have been demonstrated. The renewal effect is well known. The ABA, ABC, and AAB forms of renewal have all been demonstrated in procedures where rats learn to lever press for food-pellet reinforcers (e.g., Refs. 18, 52) and in avoidance learning where they make a shuttling response to avoid presentation of footshock (53). As before, ABC and AAB renewal effects suggest that removal from the extinction context is sufficient to cause response recovery and that extinction learning is more context specific than the original acquisition. However, unlike Pavlovian conditioning, where responding to a CS often transfers perfectly to a new context (e.g., FIGURE 1), a variety of instrumental responses reinforced with food pellets (e.g., lever press, chain pull, and nose poke) appear to weaken when the context is changed (52, 54, 55). This is also true when the response has been trained as a discriminated operant, i.e., one that is only reinforced in the presence of a specific, relatively brief (e.g., 30-s) discriminative stimulus (56). [Renewal effects also occur in the discriminated operant situation, (e.g., Refs. 57, 58).] There are occasional reports that operant responding can transfer without decrement across contexts (for example, see Ref. 59 data presented in FIGURE 3). In addition, most important, the fact that ABC and AAB renewal still occur when responding is attenuated after the first context switch suggests that extinction is nonetheless still more context dependent than acquisition, again suggesting an imbalance that may make relapse possible after extinction.

FIGURE 3.

ABA renewal of responding in an operant conditioning setting. Rats learned to press an “active” lever for an intravenous mix of heroin and cocaine. Extinction: A: responding on the active lever was then extinguished over daily extinction sessions. Inset: responding in groups that received extinction in the same context or a different context from the one in which they had lever pressed for drug. B: responding on the inactive lever. Test: active and inactive lever responding in a Control group tested in the extinction context; a Novel group that was tested in a new context; and a Renewal group that was tested in the original acquisition context after extinction in a different context. Note that responding in this group was at a level comparable to level at the start of extinction. [From Crombag and Shaham (59).]

A substantial amount of research on relapse after instrumental extinction is now conducted with drug self-administration procedures, where rats perform instrumental responses such as lever pressing to earn drug rewards (see Ref. 60, 61 for two reviews). ABA renewal has been widely demonstrated after extinction with rats responding for intravenous heroin (e.g., Ref. 62), cocaine (e.g., Ref. 63), heroin and cocaine combined (e.g., Ref. 59), or orally administered alcohol (e.g., Ref. 64–66). ABA renewal has become an important animal model for investigating the neuroscience of relapse (see sect. 4.1). Spontaneous recovery is also known to occur after the extinction of drug self-administration (e.g., Refs. 67, 68), just as it does after extinction in instrumental learning with conventional food reinforcers (e.g., Ref. 69). There is also a large literature on reinstatement of extinguished drug self-administration (e.g., Refs. 70–72), where the response recovers after extinction when the drug is presented in a manner not contingent on behavior. The assumption is often that exposure to the drug activates neural systems that drive the response again, although there is probably little reason to think that the behavioral processes identified with conventional food-pellet reinforcers, mentioned next, do not also play an important role. A second form of “reinstatement” studied in drug self-administration is a recovery of responding that occurs if the subject is exposed to a stressor (like footshock) after extinction has occurred (e.g., Ref. 73). Interestingly, although shock exposure causes the recovery of extinguished drug seeking, it has no such effect on extinguished food-seeking (e.g., Refs. 74, 75). One explanation is that stress at the time of testing may renew an extinguished response if stress was part of the original “context” of acquisition to begin with (76). That is, drug reinforcers such as cocaine can activate the hypothalamic-pituitary-adrenal axis, perhaps creating an interoceptive state of stress when the animal receives it during acquisition. It thus becomes part of the “context” of acquisition (see sect. 2.3) and causes ABA renewal after extinction has been conducted without the stressor.

Reinforcer-produced reinstatement has been extensively studied with more traditional food-seeking methods. Here, the response returns when food reinforcers are presented freely after instrumental extinction has occurred (e.g., Refs. 77–81). Presentation of the reinforcer is not effective if free reinforcers have also been delivered during the extinction phase (e.g., Ref. 77, 80, 81). Two main explanations have been discussed. First, in typical operant training methods, the organism is reinforced for making the response again soon after receiving the previous reinforcer. Thus recent reinforcers are part of the background “context” in which responding has been reinforced, and reintroducing them after extinction reintroduces a stimulus that has been associated with or sets the occasion for the response. The explanation is consistent with the fact that free reinforcers in extinction reduce the reinstatement effect, presumably by uncoupling the reinforcer presentation from a reinforced response. Second, the reinstating reinforcers can also be associated with the context when they are presented again. As we saw in Pavlovian conditioning, presenting the reinforcer in a different context (or giving the animal massive extinguishing exposure to the context) can weaken the instrumental reinstatement effect (82).

Rapid reacquisition also occurs when the response produces a pellet reinforcer again after extinction has occurred (e.g., Refs. 83–85). Such a result may occur, as in reinstatement, because the animal has previously learned to make the response soon after receiving a reinforcer during training, and their presentations again during reacquisition reintroduce a cue that sets the occasion for the response. Interestingly, occasional free reinforcers delivered during extinction, or even widely spaced response-contingent reinforcers during extinction, can slow rapid reacquisition down (85). They may do so by associating the reinforcer (or response-reinforcer pairings) with both conditioning and extinction, rather than conditioning exclusively.

A final instrumental relapse effect that occurs after extinction is resurgence (TABLE 2). Here, in sessions when a target operant response (R1) is being extinguished, a new response (R2) is introduced and reinforced to replace it. Then, when R2 is itself extinguished, the animal may make the R1 response again (it resurges) (e.g., Ref. 86). The phenomenon has been demonstrated under many conditions, including those in which R1 is initially reinforced with drugs or alcohol (e.g., Refs. 87–89). Thus resurgence is another potential model of relapse to drug taking. Most of the evidence is consistent with the idea that extinguishing R2 in the test phase changes the context in which R1 was extinguished and thus causes a renewal of R1 responding. Most importantly, R2’s reinforcer is removed (90). Therefore, resurgence can be seen as an ABC renewal effect in which the response recovers when the conditions prevailing during extinction change. Consistent with this view, resurgence is weakened when treatments make the final test situation more similar to the conditions of R1’s extinction. As one example, Bouton et al. (91) first reinforced R1 with a distinctive food outcome (O1) and then reinforced R2 with a different food outcome (O2) while R1 was extinguished. R1 resurged in the final test if R2 was extinguished but not if O2 was presented not contingent on responding at the rate it had been earned during the response elimination phase. Importantly, similar presentations of O1 did not have this effect. The results have been replicated and extended with further work in rats and humans (92–94). As a second example, exposures to periods of R2 extinction during the treatment (R1 elimination) phase weakens resurgence in the final test (e.g., Refs. 92, 94–96). Both of the treatments just mentioned would make the conditions of testing more similar to the conditions that were present when R1 was extinguished. Although an alternative, quantitative model of resurgence has been proposed (97), the most recent extension of that model has included a role for context (96). A consensus is thus building for a contextual explanation of the effect. For more detailed discussion of the challenge to the other models, see Ref. 98.

2.2. Extinction is a Representative Form of Retroactive Interference

Extinction is not the only procedure or treatment that results in a change in learned behavior. However, it is interesting to note that other research suggests that the behavioral principles derived through studies of extinction may apply to other paradigms in which a performance learned in one phase is suppressed or inhibited by new learning that is arranged in a second phase. What we know about extinction informs our understanding of several other behavior-change phenomena.

2.2.1. Pavlovian learning

Bouton (36) and others (99–101) have emphasized the idea that extinction is a representative example of retroactive interference, where new learning that occurs during a second phase interferes with performance that was learned in an earlier one. Consistent with this idea, several other forms of Pavlovian retroactive interference are similarly sensitive to context and recovery (relapse) effects. For example, in counterconditioning, a CS is first associated with one US (e.g., footshock) and then a qualitatively different US (e.g., food pellet) in a second phase. In Phase 2, a food-related response replaces the shock-related (fear) conditioned response. (The phases can also be conducted in the reverse order, that is, food to shock.) Counterconditioning is sensitive to renewal (e.g., Refs. 102, 103), spontaneous recovery (Ref. 104 cf. Ref. 105), and reinstatement (106). Thus the Phase 2 performance suppresses the Phase 1 performance in a context-specific way. Interestingly, renewal and spontaneous recovery occur regardless of whether shock conditioning precedes food conditioning or food conditioning precedes shock conditioning (reinstatement has yet to be tested food to shock). Other forms of Pavlovian retroactive interference, such as discrimination reversal learning, are also sensitive to renewal and spontaneous recovery effects (e.g., Ref. 107).

Another notable, although less well known, retroactive interference effect is inhibition with reinforcement (e.g., Ref. 108). Here, over a series of CS-US pairings, conditioned responding may reach a peak and then decline over trials as the CS and US continue to be paired. It is as if continued CS-US pairings cause a form of behavioral inhibition to emerge. The phenomenon is well known in fear conditioning (e.g., Ref. 109). For example, it is widely observed in studies using the conditioned suppression method, where fear of a CS is indexed by the CS’s ability to suppress an ongoing instrumental baseline (e.g., lever pressing) that is reinforced with food (e.g., Refs. 110–113). Interestingly, if the context is changed after inhibition with reinforcement has developed, fear of the CS increases or returns (Ref. 114; see also Ref. 115). Across individual subjects, the strength of the fear increase after the context switch correlates with the degree of inhibition that has developed (114). Thus it is the inhibition that is lost with the context change. Based on this and further results, Bouton and colleagues (114) have argued that the effect is due to the animal learning to adapt or cope with its fear over CS-US pairings, a type of retroactive inhibition that, like extinction, appears to interfere with first-learned performance in a context-specific way.

An argument can be made that proactive interference effects, where the focus is on interference with Phase 2 responding by Phase 1 learning, follow similar rules. For instance, in latent inhibition, simple exposure to the CS without pairings with a US can retard subsequent conditioning when the CS is then paired with a US. This effect is known to be sensitive to context change (e.g., Ref. 116), and importantly, renewal of inhibited performance has been demonstrated after conditioning when the animal is returned to the preexposure context (e.g., Refs. 42, 117, 118). Thus latent inhibition may occur because of retrieval of conflicting information between phases (see Ref. 119). Although the different interference procedures differ from extinction in nontrivial ways (e.g., counterconditioning, but not extinction, involves exposure to a new motivationally significant event in Phase 2), common principles do appear to apply.

One rule of thumb that has emerged in studies of retroactive interference is that the second-learned association is often more context dependent than the first. We have already noted this in extinction: As illustrated in FIGURE 1, when the context is switched after fear or appetitive conditioning, the conditioned response may transfer well across contexts, and yet extinction is more context specific. Is this due to the fact that extinction is second-learned or the fact that it is a form of inhibitory learning? We found that inhibition acquired in the “conditioned inhibition” paradigm (where CS A is paired with a US but occurs without a US when CS X is added to A– X becomes a conditioned inhibitor) also transfers well across contexts (51, 120). Nelson (121) went on to confirm the importance of the second-learned rule: in his experiments, excitatory conditioning (tone-food pairings) transferred perfectly across contexts, unless the tone had previously been trained as a conditioned inhibitor. Inhibition conditioned to a CS likewise transferred across contexts unless the CS had first been trained as a conditioned excitor (through initial tone-food pairings). Thus, regardless of whether an association was excitatory or inhibitory, the second-learned one was context specific and the first-learned one was not. Consistent with this idea, excitatory conditioning can be relatively context specific if it is preceded by nonreinforced preexposure to the CS (122). In Pavlovian retroactive interference paradigms, like extinction, the second thing learned about the CS appears to interfere with the first thing learned in a context-specific way.

2.2.2. Instrumental learning

The idea that extinction is a representative form of a general interference process is also consistent with studies of instrumental learning (11). For example, discrimination reversal learning, where the animal is reinforced for making an instrumental response to stimulus X but not Y in Phase 1 and then Y but not X in Phase 2, is sensitive to both renewal (e.g., Refs. 123–126) and recovery effects over time (123, 126, 127). Such findings again suggest that the first-learning remains and that the second-learning is sensitive to context, just as in extinction.

Recent research has studied punishment, where instrumental behavior is suppressed when a noxious event (like footshock) is presented contingent on the response (128, 129) (see Ref. 130 for review). Although footshock is a salient event that will engage neural processes that are different from those engaged in extinction (as above), punishment is like extinction in being impermanent and highly sensitive to the context (128, 129). For example, a punished response can renew in either Context A or Context C after it has been punished in Context B (128). Spontaneous recovery (131, 132) and reinstatement (133) have also been observed after punishment. The possible similarity of punishment and extinction encourages a broad view of interference and contextual control. It is worth noting that renewal also occurs after “negative punishment,” also known as differential reinforcement of other behavior (DRO) or omission training. Here the instrumental response can be suppressed by making it prevent (rather than produce) presentations of the reinforcer in Phase 2 (134–136) (see also Ref. 137). The ABA renewal effect can be equally strong when the instrumental response is eliminated by either extinction or DRO (136).

Other results are consistent with the general perspective on interference. Rescorla (see Ref. 12) studied the effects of replacing one appetitive reinforcer (e.g., food pellet) with another one (e.g., sucrose liquid) in a second operant training phase. When either the first or second reinforcer is separately associated with LiCl, which makes the animal sick and conditions a taste aversion to that reinforcer, the instrumental response was equally weakened (when tested in extinction), suggesting that the first response-reinforcer association is remembered and is as strong as the second response-reinforcer association after the interference treatment. Interestingly, when time elapses after the second phase of training, the response tends to strengthen (138, 139), suggesting to Rescorla that the second-phase treatment somehow inhibits the first, which can then spontaneously recover over time.

Still another example of retroactive interference may be an instrumental response’s conversion from a “goal-directed action” to a “habit” following extended repetition or practice (e.g., Refs. 140, 141). Early in training, animals perform instrumental behaviors because they have learned that the response leads to an outcome they value; we know this because if we condition a taste aversion (for example) to the outcome, the rat will choose not to make the response when the response is tested in extinction (e.g., Ref. 54, 140, 142–144). In contrast, after extended training or practice, the response can become automatic and habitual and is not affected by such devaluation of the reinforcer (54, 140, 144). Although there is a tendency to think that, once acquired, habits are permanent and persistent, several environmental manipulations can return a habit to its original status as a goal-directed action (145, 146). One of those manipulations may be changing the context (147, 148). For example, if rats receive a modest amount of lever-press training in Context A (to establish the response as a goal-directed action there) and then more extensive training in Context B (to convert it to a habit there), the behavior is demonstrably a habit in Context B but renews to action when it is tested in Context A (the original action context) or in a third context, Context C (148). Consistent with the rule of thumb described above, the first-thing learned (goal-directed action) transfers well across contexts, while the second thing learned (habit) does not. In normally functioning animals, habit learning may not be permanent but (like extinction) may interfere with goal-direction in a context-specific way.

All of these phenomena are consistent with the view that a new instrumental learning experience does not destroy information learned about an instrumental behavior at an earlier point in time. The latter may return to performance with various manipulations of the context. From a clinical perspective, the fact that habits can reconvert to action status may suggest hope for recovery from maladaptive habits such as smoking or drug taking but less optimism about the permanence of adaptive, healthy ones, such as eating well and getting exercise. More generally, the fact that first-learned behavior can recover so easily regardless of interference treatments may be one reason why behavior change is difficult to sustain (149).

2.3. There Are Many Kinds of Context

2.3.1. Pavlovian learning

As noted earlier, most research on the effects of context have manipulated the background physical environment in which conditioning treatments occur. However, in reviewing the various relapse effects above (sect. 2.1), we made liberal use of the idea that many different kinds of “background” cues can actually function as the context. There is good evidence supporting this view. For example, Davidson and colleagues (e.g., Refs. 150–152) have shown that the interoceptive state created by food deprivation (“hunger”) can provide a context for Pavlovian responding. The idea may have important implications for a learning-theory understanding of appetite and food consumption. In addition, the literature on state-dependent learning or state-dependent retention supports the idea that interoceptive states created by drugs can function as interoceptive contexts as well. For example, if fear extinction is conducted while the animal is under the influence of a benzodiazepine tranquilizer (e.g., Ref. 153) or alcohol (e.g., Ref. 154, 155), fear can renew when the animal is tested outside the context produced by the drug. Such findings extend the analysis of the contextual control of extinction to the possible interoceptive contexts provided by drugs. However, they also provide an explanation of why individuals may learn to abuse or misuse anxiolytic medications: if the drug is taken to reduce anxiety, it will paradoxically protect the individual from beneficial extinction that could occur through natural exposure to the cues that elicit anxiety, thus perpetuating a vicious cycle.

We also noted previously that the passage of time may theoretically bring about a change in temporal context. That is, spontaneous recovery was considered a renewal effect that occurs when the CS is tested in a new temporal context. As noted earlier, renewal caused by changing the physical context and that caused by changing the temporal one can both be attenuated by retrieval cues (e.g., Refs. 23, 27). Moreover, the passage of time can control responding to a Pavlovian CS when it explicitly signals the CS’s reinforcement or nonreinforcement. A number of experiments have demonstrated that the time elapsed since the last conditioning trial can be used as a cue predicting whether the next CS will be paired with a food pellet or not (156, 157). Although there are many ways to conceptualize time as a stimulus (e.g., Refs. 158, 159), one idea that accounts for details of the data is that the passage of time involves a cascade of hypothetical stimulus elements that can be associated with reinforcement and nonreinforcement, just as different physical contexts would (Ref. 157, see also Refs. 160–164).

Other experiments suggest that recent events the animal has experienced can serve as contexts. We already noted that recent presentations of a US or reinforcer can provide a kind of context. When a conditioning phase is followed by an extinction phase, the memory of a recent reinforcer (or perhaps even its lingering taste in the mouth) can be viewed as part of the conditioning context. This is one reason that presentations of the reinforcer after extinction can reinstate an extinguished behavior (e.g., Refs. 77, 80). In experiments that manipulated this type of context (165), USs were presented during the intertrial intervals of either conditioning or extinction in different groups of subjects. At the end of several such alternating phases, presentation of the US cued conditioned responding when it had been featured during the conditioning (rather than extinction) but suppressed responding when it had been featured in the extinction (rather than conditioning). There is little question that recent USs can provide part of the context that influences Pavlovian conditioned responding.

2.3.2. Instrumental learning

Recent work on instrumental learning also supports a broad view of context (11). For example, Schepers and Bouton (166) studied the effects of food deprivation as a context. Rats learned to lever press for food pellets while they were satiated, and then the response was extinguished during sessions in which the rats were food deprived. When they were then tested in the satiated state, the response renewed, consistent with a contextual, rather than a purely motivating, role for food deprivation. The results may be relevant to dieters who learn to inhibit their food-seeking while hungry, only to find it difficult to resist seeking food when returned to a satiety state. In other experiments, Schepers and Bouton (76) explored stress as a context. Rats learned to lever press for food pellets in sessions that followed exposures to a stressor that changed daily; the response was subsequently extinguished in the absence of stress. When tested immediately after a stressor (including stressors that had not been directly associated with lever-press training), the response renewed if (and only if) stress had initially been associated with lever press training. In addition to expanding the definition of context, the results may help explain why stress (or negative affect) might cause an inhibited response, such as smoking or drug-taking, to relapse after treatment.

Other recent research has tested the effect of “social context,” the presence or absence of a conspecific, in controlling extinction of an operant response. Browning and Shahan (167) reinforced lever pressing in rats while a second rat was present in an adjacent chamber. They then extinguished responding in the absence of the second rat. When tested again with the adjacent rat, the extinguished response renewed. Interestingly, the second rat could also provide a context for extinction; when reinforcement occurred without the second rat, and extinction occurred in its presence, responding renewed when tests occurred without the rat. Weiss et al. (168) reported complementary results with cocaine self-administration. Once again, a rat in an adjacent chamber would cause extinguished cocaine seeking to renew if it had been present during the original phase in which cocaine-seeking had been trained. There was also evidence that different individual rats serve as effective contexts (or discriminative stimuli) signaling cocaine self-administration or extinction. The authors suggested that the findings are relevant in understanding the effects of peer groups and social cues in controlling drug-taking in the human’s natural world.

We already discussed the idea that recent reinforcer presentations can serve as contexts when we considered reinstatement (where reinforcer presentation after extinction renews responding) and resurgence (where reinforcer presentations during extinction of a response can cue response inhibition) in sect. 2.1.2. One can relatedly ask whether neutral cues that are occasionally presented during the extinction phase can reduce resurgence when they are also presented during a response recovery test [as they do for Pavlovian spontaneous recovery (23, 26), renewal (27), and reinstatement (169)]. A cue presented occasionally in extinction can indeed attenuate an instrumental renewal effect if it is presented during testing (170–172). Such a cue may be less likely to slow rapid reacquisition (172) or attenuate resurgence (173). However, Trask (173) demonstrated that the cue can reduce resurgence if it is also associated with the reinforcer during the response-elimination phase. She argued that pairing the cue with the reinforcer helps maintain attention to the cue, as implied by theories of attention and conditioning (e.g., Ref. 174, 175). Attention to neutral cues might otherwise be reduced in active sessions that are full of behaviors and reinforcers, as is true of a reacquisition phase or the response-elimination phase of the resurgence paradigm.

A last type of “context” is one provided by recent behaviors. Several laboratories have recently studied behavior chains, sequences of behaviors that need to be executed to acquire a final goal or reinforcer (e.g., Ref. 176–178). Many natural behaviors probably occur in chains; for example, junk food or a drug must be purchased or procured before it can be consumed. In a discriminated behavior chain (e.g., see Refs. 179, 180 for reviews), one discriminative stimulus (S1) sets the occasion for an initial response (R1) which then terminates S1 and initiates a second stimulus (S2) which now sets the occasion for a second response (R2). R2 then finally produces the reinforcing outcome (S1-R1-S2-R2-O). A variety of evidence supports the view that, among the many things the animal learns when it learns the chain, it learns to associate R1 and R2. Because of that, R1 becomes part of the “context” controlling R2. As one example, Thrailkill et al. (181) trained the chain and then extinguished R2 on its own (by presenting S2 and allowing R2 to occur without the reinforcer). Then, when S2-R2 was tested again after S1-R1 (i.e., in the original S1-R1-S2-R2 chain), R2 responding was renewed. Critically, presentations of S1 without the opportunity to make R1 did not cause renewal of R2. In addition, extinction of R1 before the chain renewal test prevents the renewal of R2 (182). Thus making one response provides a kind of context for the next response. Interestingly, the contextual control that R1 has over R2 seems to reduce the extent to which R2 is under the influence of the apparatus context, as if the two types of “context” compete (181).

2.4. Behavioral Mechanisms Underlying Extinction and Contextual Control

Behavioral research on extinction, and the context’s apparent influence on it, has often tested various theoretical mechanisms that might underlie the results reviewed above (particularly sects. 2.1 and 2.2). To summarize the different mechanisms that have been proposed, FIGURE 4 illustrates, in cartoon form, various hypothetical associations that could underlie ABA renewal after extinction in Pavlovian conditioning (FIGURE 4, A and B) and instrumental conditioning (FIGURE 4, C and D). In either case, FIGURE 4, A and C, and FIGURE 4, C and D, respectively, show associations in Context A (the conditioning context) and Context B (the extinction context). Notice that FIGURE 4B expands on the conceptualization of Pavlovian extinction provided in FIGURE 2. The inhibitory CS-US association learned during extinction is further assumed to generalize across contexts to some extent when the CS is returned to Context A (FIGURE 4A). FIGURE 4, C and D, illustrates analogous associations that might operate in operant learning. Each numbered link thus represents a mechanism that might contribute to the empirical result of extinction performance in Context B and renewed performance in Context A.

FIGURE 4.

Associative structures that could in principle control responding during ABA renewal in Pavlovian conditioning (A and B) and instrumental learning (C and D). In either case, the extinction Context B could conceivably inhibit performance through any of the associations shown, and the conditioning Context A could activate performance through any of the associations shown (see text for details).

2.4.1. Pavlovian learning

Early research on the contextual control of Pavlovian extinction focused on the possible role of direct associations between the context and the US (links 1 and 4 in FIGURE 4). This is because those are the mechanisms that models of conditioning emphasize (e.g., Ref. 48, 49, 174, 183–185). For example, the seminal Rescorla-Wagner model (183) explicitly emphasized a role for context in any conditioning experiment and simply treated the context as a second CS that was always presented together (in “compound”) with the target CS. When applied to the ABA renewal effect, the Rescorla-Wagner model (and the other, like-minded models) would emphasize the fact that Context A (as well as the CS) can become associated with the US during conditioning (FIGURE 4, link 1) and that Context B might become a conditioned inhibitor (FIGURE 4, link 4) during the CS’s extinction there. Excitation in A and inhibition in B would add to or subtract from the extent to which the CS itself activated the US representation. (This is a way of representing the models’ assumption that the associative strength of a CS and context summate to produce performance.) Thus, in extinction, the context’s inhibitory association with the US (link 4) would suppress performance to the CS, but release from that inhibition and a return to the excitatory context-US association in Context A (link 1) would contribute to ABA renewal there. It is worth noting that the Rescorla-Wagner model would not propose that the CS enters into an inhibitory association with the US, even though that association is represented in the figure.

Despite the Rescorla-Wagner model’s importance as an early stimulus for research on context, its emphasis on direct context-US associations (FIGURE 4, links 1 and 4) did not stand up well to experimental tests (see Refs. 2, 36, 186 for reviews]. With typical conditioning and extinction procedures (cf. Ref. 187), demonstrable context-US associations like those represented by links 1 and 4 were neither necessary nor sufficient for the context to affect responding to the CS. For example, various tests failed to reveal any evidence of excitatory context-US associations in Context A or inhibitory context-US associations in Context B in situations where strong ABA renewal was observed (e.g., Refs. 19, 20, 188). In addition, as noted earlier, a number of experiments found that responding to the CS transferred very well and almost perfectly from Context A to B when extinction began there (see FIGURE 1). This was not expected if context-US associations had been acquired during conditioning in A and were summating with the CS there. Finally, creating strong and demonstrable context-US associations after conditioning by presenting the US on its own in the context repeatedly failed to increase (or in any way affect) responding to the CS (28, 189). Context-US associations did affect responding to the CS if responding to the CS was under the influence of extinction, as in reinstatement, discussed above. However, an extinguished CS was affected by such contextual conditioning, and a nonextinguished CS was not (28, 189). The results overall questioned the idea that the contexts’ role in ABA renewal experiments was a simple result of their direct associations with the US.

It is worth noting that the typical temporal properties of a context might make it different from the typical CS. Contextual cues do not turn on and off the same time the CS does, as compounded CSs do in most compound conditioning experiments. Instead, the animal is usually in the context for a period of time before the first CS occurs, and there are long intervals of exposure to the context between subsequent presentations of the CS (conditioning trials). Long, continuous exposure to the context could reduce its salience (e.g., Ref. 190) and create conditions that encourage the development of occasion setting (e.g., Refs. 191–193). In occasion setting, a stimulus is thought to control performance to a second (target) stimulus in a way that is independent of its direct association with the US. As an illustration, in FIGURE 4, link 2, Context A inhibits the inhibitory CS-US association that was learned in extinction. This turns on responding to the CS via “positive occasion setting” and is part of what allows the animal to respond again to the CS in Context A. (For evidence that occasion setters modulate a CS’s inhibitory association, see Refs. 51, 194, 195.) In link 5, Context B activates the CS’s inhibitory association learned in extinction and thus suppresses responding through “negative occasion setting.” When studied with discrete CSs, occasion setting seems to develop when presentation of the occasion setter precedes the target (e.g., Ref. 192) or is much less salient (e.g., less loud or less bright) than the target (e.g., Ref. 196). These features are arguably true of contexts. In addition, contexts have a number of properties consistent with the occasion setting mechanism (see Ref. 186 for one recent review). Thus, as discussed in sect. 2.1.1, when an extinction context controls extinction performance (as particularly implied by ABC and AAB renewal), it may be activating the “memory” of extinction, or signaling that the CS is not now associated with the US. Occasion setting instantiates the idea that the context disambiguates the current meaning of the CS (e.g., Ref. 25).

Another mechanism that is potentially involved in extinction and its contextual control is represented by FIGURE 4, link 3, where the context becomes associated with the CS. Such associations play an important role in the “Sometimes Opponent Process” or “SOP” models proposed by Wagner (49, 184, 185), where the learning of a context-CS association allows the context to activate a representation of the CS in short-term memory. According to the model, priming the CS in short-term memory this way will decrease the extent to which it can be further processed when it is presented the next time. For example, Wagner (49) proposed that the context will activate the CS representation to a secondary state of activation (“A2”) that would reduce its ability to go into the more primary state (“A1”) when it is presented next. Going into A1 is required for learning about, and perhaps responding to, the CS. Repeated exposure to the CS, and its increasing association with the context, habituates processing of the CS by preventing it from going to A1. Thus, during a series of extinction trials, the CS would be increasingly associated with the context, and the context would increasingly suppress performance to the CS. In addition, under at least some conditions, a context change could then cause renewed responding to the CS by freeing the CS from this suppression. The idea may be useful in explaining ABC renewal effects that occur when the reinforcement histories of Contexts B and C are controlled (i.e., when another CS has been extinguished in the context where responding to a target CS is renewed, e.g., Refs. 16, 197–199; see also Ref. 200).

The mechanism in FIGURE 4, link 3, is widely acknowledged as a possible explanation of latent inhibition, where preexposure to the CS would allow it to be associated with the context, making it less available for learning when it is subsequently paired with the US there (the context is activating it to A2). The idea is consistent with the well-known finding, noted earlier, that latent inhibition is reduced if the context is changed between preexposure and conditioning (e.g., Refs. 15, 116, 118). However, there are other explanations (e.g., Refs. 36, 119). In addition, a longstanding challenge is that habituation to a stimulus often transfers well across contexts (e.g., Refs. 116, 201), raising questions about whether context change actually liberates the CS from suppressed processing (but see Ref. 202 for further analysis). Perhaps most important, we already noted that renewal effects occur in counterconditioning, where the CS is associated with a new US (e.g., food after it has been associated with shock) instead of being presented alone in Phase 2. In Phase 2 of a counterconditioning design, the CS increasingly elicits performance corresponding to the new US, and is thus clearly being processed. However, when it is returned to the original context or tested in a third context, renewal of Phase 1 performance occurs (102, 103). This is more than mere liberation of suppressed CS processing. The counterconditioning result is more readily explained by the idea that Contexts A and B might set the occasion for CS-US1 and CS-US2 (FIGURE 4, link 2), respectively.

We note that although FIGURE 4, link 6, in which the context has an inhibitory association with the CS, is included in FIGURE 4 for completeness, to our knowledge it has never been explored experimentally.

A final possible mechanism underlying the contextual control of Pavlovian extinction that is not represented in FIGURE 4 is configural conditioning. Pearce (203, 204), for example, has argued that the animal learns to associate whole configurations or combinations of cues with the US in conditioning, rather than separately associating each element of a compound with the US. Some version of a configural view is necessary to explain certain conditioning phenomena (such as negative patterning, where animals learn to respond to single elements, A+ and B+ but not their combination, AB-). In addition, it has gone some distance in accounting for a number of novel conditioning effects, including many that have seemed consistent with occasion setting (e.g., see Ref. 192). In configural theory terms, ABA renewal could be explained by allowing the Context B and CS combination to enter into an inhibitory association with the US and the Context A and CS combination to enter into an excitatory one. However, one result that is hard for it to handle is the asymmetry we have noted above: Although conditioned responding to the CS can transfer from Context A to B very well after conditioning (FIGURE 1), extinction does not transfer well when the CS is switched back from Context B to A. That is, one observes a loss of extinction performance: the renewal effect occurs. According to existing configural theory, the fact that there is so little decrement in responding when the CS is first presented in the new context at the start of extinction would require an equally small decrement in the effects of extinction when the CS is returned to the original context. This is a problem for configural theory in its present form (e.g., Ref. 205). The occasion-setting mechanism avoids the problem because positive occasion setters (FIGURE 4, link 5) do not influence responding to a CS until the CS has developed some inhibition in extinction (e.g., Ref. 195).

2.4.2. Instrumental learning

Related research has studied analogous mechanisms of contextual control in instrumental extinction (FIGURE 4, C and D). (To keep things simple, the cartoons represent a free-operant situation in which the response is emitted freely and reinforced and extinguished without a discriminative stimulus.) Once again, there is a possible role of direct associations between the contexts and the reinforcing outcome; in the ABA renewal design, Context A might be directly associated with the reinforcer (link 7), and Context B might acquire an inhibitory association with it during extinction (link 10). These context-reinforcer associations could excite or inhibit the instrumental response through mechanisms of Pavlovian-instrumental transfer, the phenomenon in which Pavlovian cues influence instrumental responding when they are presented with them (e.g., Refs. 206, 207). Pavlovian associations between a context and a reinforcer have been shown to influence an instrumental response (e.g., Ref. 208), confirming that link 7 at least can influence performance. However, the instrumental renewal effect does not depend on them. Todd (18) studied ABA, ABC, and AAB renewal with a procedure that controlled A’s and B’s direct associations with the reinforcer. For instance, in a study of ABA renewal (see TABLE 3), one response (R1) was first reinforced in Context A during sessions that alternated with ones in which a second response (R2) was reinforced in Context B. Then, during extinction, R1 was switched and extinguished in B while R2 was switched and extinguished in A. Notice that the treatment guaranteed that both contexts were equally associated with both reinforcement and extinction (links 7 and 10, respectively). Yet, renewal was observed when the responses were returned to their original contexts after extinction (R1 in A and R2 in B). Bouton and Schepers (128) reported analogous results in punishment (bottom, TABLE 3). There, rats were first reinforced for R1 in A and R2 in B. Then, in a punishment phase, the responses were switched to the opposite context and punished there (the response was associated with footshock); this suppressed responding. Yet, responding again renewed when the Rs were returned to their original contexts. Something other than the contexts’ direct associations with reinforcers or punishers must have been in control. In both extinction (18) and punishment (128), rats can learn to inhibit a specific response in a specific context. The results are not captured by a mechanism like the one represented in links 7 and 10.

Table 3.

Designs of experiments that demonstrated ABA renewal after extinction and punishment when each context’s association with reinforcement and nonreinforcement or punishment was controlled

Phase
Acquisition	Retroactive Interference Treatment	Test
Extinction
A: R1-food	A: R2–	A: R1, R2
B: R2-food	B: R1–	B: R1, R2
Punishment
A: R1-food	A: R2-food + shock	A: R1, R2
B: R2-food	B: R1-food + shock	B: R1, R2

A and B refer to contexts; – refers to extinction. Both experimental designs were within-subject designs; thus every subject received the treatments shown in each phase in an intermixed fashion. Bold indicates the response with the higher level during the test. [See Todd (18) and Bouton and Schepers (95).]

The alternative mechanism encouraged by the Pavlovian extinction research (sect. 2.4.1) is occasion setting by the contexts (FIGURE 4, links 8 and 11). It is worth noting that “occasion setting” is the term originally used by Skinner (209) when he considered the effects of discriminative stimuli on operant responses. Such stimuli are thought to enable a response (rather than directly elicit it), much as a canvas enables (but does not merely elicit) an artist’s brush strokes (210). At a theoretical level, Context A might inhibit the inhibitory response-outcome association (link 8), while Context B might excite it (link 11), analogous to what we saw in Pavlovian extinction. There is evidence that contexts can work in a “hierarchical” way. For example, in an experiment illustrated in TABLE 4, Trask and Bouton (211) reinforced R1 with one outcome (O1) in Context A and a second outcome (O2) in Context B. At the same time, R2 was reinforced with O2 in Context A and O1 in Context B. One of the reinforcers (O2) was then paired with illness induced by LiCl injection to condition a taste aversion (and devalue it) during sessions conducted in both the contexts. During subsequent extinction tests, there was less R1 than R2 in A and less R2 than R1 in B, as if the animals understood that the contexts signaled the specific relations between the responses and outcomes trained in them (see also Ref. 212). Thus contexts can control instrumental responding in a way that cannot be reduced to their simple and direct associations with the reinforcer or response.

Table 4.

Design of an experiment demonstrating occasion setting of instrumental responding by the context

Phase
Acquisition	Reinforcer Devaluation	Test
A: R1-O1, R2-O2	A: O2-Illness	A: R1, R2
B: R1-O2, R2-O1	B: O2-Illness	B: R1, R2

A and B refer to different contexts; R1 and R2 refer to different responses; O1 and O2 are different food pellets (grain and sucrose). illness was created by lithium chloride injection. This is a within-subject design; thus, every subject received the treatments shown in each phase in an intermixed fashion. Bold indicates the response with the higher level during the test. [See Trask and Bouton (211).]

However, there are reasons to think that the occasion-setting mechanism might not apply as readily when rats are not given as much explicit training that contrasts two responses, two outcomes, and two contexts. For example, Todd (18) tested a classic characteristic of occasion setters that is known in the Pavlovian situation. He asked whether a context controlling the extinction of one instrumental response (e.g., chain pulling) can “transfer” its effect and also inhibit performance of a second response (e.g., lever pressing) that has been trained in a similar occasion-setting way in other contexts (see Refs. 213, 214) for analogous tests in a Pavlovian paradigm). No such transfer occurred: the extinction context’s inhibition of one response was specific to that response and had no discernible impact on the other response (see also Ref. 215). Notice that this result is also implied by the findings of Todd (18) and Bouton and Schepers (128) described above. Contextual control of instrumental extinction and punishment can involve learning something quite specific about the specific instrumental response.

This leaves the third alternative sketched in FIGURE 4, C and D, which represent the idea that the context might directly excite the response in Context A (link 9) and/or directly inhibit it in Context B (link 12). In extinction (FIGURE 4D), the emphasis is thus on a direct inhibitory association between the context and the response, as if the context simply shuts it down. Rescorla (e.g., Ref. 12) has argued for such a process in extinction, and Colwill (216) has provided evidence that an S- in a paradigm in which an S- was associated with not reinforcing an operant response that was otherwise reinforced inhibited responding this way. In extinction itself, some findings reported by Bouton et al. (215) seem relevant. In several experiments using conventional stimulus control methods, they found that the rat must receive direct nonreinforcement of the response for instrumental extinction to be learned. For example, in one experiment rats learned both to press a lever (R1) and pull a chain (R2, counterbalanced) in order to receive a food-pellet reinforcer in the same S. They then received trials in which one of the responses (R1) was extinguished during S, but the other (R2) received no treatment. Subsequent tests of R1 and R2 in S revealed that R1 was strongly depressed, but R2 was not affected. In addition, simple nonreinforced presentation of S without the opportunity to make a response had no impact on either R1 or R2, a result that was consistent with clinical studies that have found that the Pavlovian extinction of cues associated with drugs may often have little influence on instrumental drug-taking itself (e.g., Ref. 217). In another experiment, when a lever-press response was reinforced with different outcomes in different Ss (S1:R-O1, S2:R-O2), its extinction in one stimulus also suppressed its performance in the other S. Thus, to a large extent, the inhibition of the response learned in extinction seemed to ignore the response’s association with a specific outcome (there was a trend toward less suppression in the second stimulus, but it was not statistically significant). In a preliminary way, the result suggests that the context can directly inhibit R, instead of modulating the R-O association, in extinction.

The idea that animals learn direct associations between a context and a response is also consistent with other studies of the contextual control of instrumental behavior. In contrast to the cross-context transfer of conditioned responding that has been observed in Pavlovian conditioning (see FIGURE 1), several experiments have established that a context switch after instrumental conditioning can weaken the instrumental response (e.g., Refs. 52, 94, 149; but see, e.g., Ref. 59). The finding is potentially explained by any of the links in FIGURE 4C. However, after controlling for direct context-reinforcer associations (link 7), Thrailkill and Bouton (54) tested for the influence of link 9. In one experiment, rats received extended training of lever-pressing, so that the response had become habitual, as described earlier (e.g., Refs. 140, 141). Thus, when the reinforcer was devalued by pairing it with LiCl, the rats continued to lever press as if it were a habit (e.g., Refs. 141, 218). As usual, the response was weakened when the context was changed. In contrast, when the response received less training, it was a goal-directed action rather than a habit: rats that had an aversion conditioned to the reinforcer suppressed their lever pressing relative to control rats that did not. Crucially, the size of this reinforcer devaluation effect was not reduced by changing the context, suggesting that the animal’s knowledge (and use) of the response-outcome association to determine performance was the same in either context. The results thus suggested that the context did not control the R-O relation but instead controlled the direct evocation of R (as in link 9). Notice that occasion setting represented by link 8 could once again not play a role until the response has received extinction. As described in sect. 2.2.2, Steinfeld and Bouton (147, 148) recently confirmed that habit learning is more affected by context change than action learning (R-O) is. The direct evocation of R (rather than R-O) by the acquisition context is the complement of the direct inhibition of R by the extinction context that results suggest the animal might learn in extinction (link 12).

2.4.3. Integration

Extinction is highly context-specific in either Pavlovian or instrumental learning, and we have just seen that the contextual control can be accommodated by several theoretical mechanisms. Despite the many parallels between Pavlovian and operant extinction, it is interesting that they may be controlled by different behavioral processes. In Pavlovian extinction, the extinction context often seems to operate as a negative occasion setter, hierarchically signaling that the CS will not be followed by the US or activating an extinction memory or association (as illustrated in FIGURE 4, link 5). In contrast, in instrumental extinction, the extinction context often seems to enter into a direct inhibitory association with R, as if it directly suppresses the response. The possibility that a direct inhibitory association between the context and a Pavlovian response might similarly suppress that response is challenged by several other results. For example, in instrumental extinction, extinguishing R in one S can transfer and suppress R in a second S (215, 219) but analogously extinguishing a Pavlovian response in one CS does not transfer and influence the same response in a second CS except under different and restricted conditions (220) (see Ref. 215 for more discussion). The difference may be consistent with the intuition that Pavlovian learning mechanisms allow the organism to learn about stimuli in the environment whereas instrumental learning mechanisms are focused on adjusting behavior. Additional research relevant to this distinction is discussed in sect. 2.5.3.

2.5. Causes of Extinction

Another important theoretical question is, what are the crucial events or processes that make extinction happen? Here we consider three contemporary ideas.

2.5.1. Discrimination of reinforcement rate

One possibility is that the animal eventually learns that the rate of reinforcement is lower in extinction than it was during conditioning and adjusts its responding accordingly.

2.5.1.1. Pavlovian conditioning

Gallistel and Gibbon (221) have argued that the animal continually decides whether or not to respond in extinction by comparing the current rate of reinforcement in the CS with its memory of the rate that prevailed during conditioning. Because rate is the reciprocal of time, the animal is thought to compute a ratio between the amount of time that has accumulated in the CS during extinction and the amount of time accumulated in the CS between USs during conditioning. When the ratio exceeds a threshold, the animal stops responding.

This approach has been investigated in several laboratories with almost uniformly negative results [see Harris (222) for a recent review]. For example, a number of authors have tested the idea that, if the cumulative duration of exposure to the CS is what counts in extinction, many short-duration CSs in extinction should produce the same suppression of behavior as fewer long-duration CSs if the total cumulative exposure time to the CS is matched. In fact, extinction with many short CS exposures is often much more effective than extinction with fewer long exposures (223–224). In addition, when extinction curves are plotted as a function of the number of extinction trials (as opposed to accumulating time in the trials), results with short- and longer-duration CSs superimpose, suggesting that extinction occurs as the number of trials, and not the amount of time in the CS, accumulates (225).

The rate-discrimination hypothesis provides an especially interesting account of the PREE, the phenomenon in which conditioning with partial reinforcement schedules (in which nonreinforced trials are intermixed with reinforced trials) creates a slower loss of responding in extinction than conditioning with a continuous reinforcement schedule (in which every trial is reinforced). According to the rate-discrimination view (221), subjects exposed to partial reinforcement have learned to expect the US after more accumulated time in the CS, and it therefore takes more CS time in extinction to exceed the threshold of accumulated extinction time/expected time to each US. In contrast, a more traditional approach has suggested that partially reinforced subjects have learned to expect the US after more trials than continuously reinforced subjects have. It therefore takes more trials to stop generalizing from conditioning to extinction (e.g., Ref. 227–229).

Contrary to the rate discrimination hypothesis, the PREE still occurs when partially and continuously reinforced CSs predict the US after the same amount of CS time (230–234). For example, rats that received a 10-s CS reinforced on half its presentations (accumulated CS time of 20 s) extinguished more slowly than a continuously reinforced group that received every 20-s CS presentation reinforced (230, 234). When extinction curves of PRF and CRF groups are plotted as a function of the expected number of trials expected to reach reinforcement, the curves can superimpose (230); Chan and Harris (232) demonstrated this result when they compared continuous-reinforcement conditions with partial reinforcement conditions in which one in three or one in five CS presentations were reinforced (rats took ∼3 times and 5 times the number of trials to stop responding in these CSs, respectively). Bouton et al. (230) created conditions that made it possible to further separate time-discrimination and trial-discrimination accounts (e.g., Ref. 227–229). Rats that had every fourth 10-s CS reinforced extinguished more slowly over a series of alternating 10-s and 30-s extinction trials than rats that had received every 10-s CS reinforced. This PREE was still observed when extinction responding was plotted as a function of time units over which the US should have been expected (every 40 s for the partially reinforced group but every 10 s for the continuously reinforced group). In contrast, the PREE disappeared when extinction responding was plotted as a function of the trials over which the US should have been expected (every fourth trial for the partially reinforced group and every trial for the continuously reinforced group). Ultimately, the PREE, and extinction generally, is better captured by trial-based theories (e.g., Refs. 222, 227, 228; see also Ref. 235 for a review of the older literature).

2.5.1.2. Instrumental learning

There has been relatively little corresponding discussion of the discrimination between reinforcement rate in the analysis of instrumental extinction (but see Ref. 236). However, it is worth noting that the trial-based discrimination account noted above (e.g., Ref. 227–229) emerged from extinction studies using instrumental learning procedures. Mowrer and Jones (229) showed, for example, that in a free-operant situation, a PREE disappeared if performance in continuous and partially reinforced groups was plotted as a function of the number of responses (“response units”) that had been reinforced during initial training (cf. Refs. 230–232).

2.5.2. Generalization decrement

The foregoing suggests that the animal stops responding in extinction when it stops generalizing between the conditions that prevailed in conditioning and those that prevail in extinction (e.g., Refs. 227, 228, 237). This idea has been especially useful in explaining the PREE. It is interesting to note that a generalization decrement theory of extinction does not assume that any destruction of the original learning in extinction, or any new learning at all. It is also worth noting that it is a kind of context theory; the reduction of performance in extinction is viewed as a kind of discrimination between two contexts. As such, it is consistent with the spirit of the view that extinction results from behavioral control by the background context (as above). Recent models of extinction from the reinforcement learning perspective (e.g., Ref. 164) have also made a case for a context discrimination process. For example, Redish et al. (238) suggested that in extinction the organism discriminates “states” in which the response is reinforced and those in which it is not, and Gershman et al. (239) similarly proposed that the organism actively clusters or discriminates trials that are reinforced and those that are not (see also Ref. 240). Some results have confirmed at least one unique prediction from this approach (241).

It is worth noting that there is good reason to think that extinction involves new learning, rather than only a passive failure to generalize. For instance, nonreinforcement of a food CS elicits measurable frustration, and this can be associated with stimuli present in the environment (e.g., Ref. 242). Nonreinforcement of the CS in the related conditioned inhibition paradigm (in which a CS is nonreinforced in the presence of a second stimulus and that second stimulus acquires purely inhibitory properties) also generates measurable new learning in the form of conditioned inhibition. There is also evidence suggesting new learning in extinction in studies of the renewal effect. For example, either ABC renewal or AAB renewal (see earlier discussion) implies that the extinction context acquires an ability to modulate (suppress) performance to the CS release from that suppression is sufficient to produce renewal. Such observations suggest that the animal does not stop responding in extinction merely because it fails to generalize. Instead, it appears to have learned that the CS means “no-US” in the extinction context (e.g., FIGURE 4, link 5), or in the case of instrumental extinction, to stop performing the response in the extinction context (link 12).

2.5.3. Prediction error

2.5.3.1. Pavlovian conditioning

Pavlovian conditioning theories have often emphasized the role of prediction error in the acquisition and extinction of conditioned responding. In the influential Rescorla-Wagner model (183), when the US is weaker than that predicted by the CSs present on a trial, the associative strengths of all CSs present will decrease to correct the prediction error. Conversely, when the US is greater than what is predicted, associative strengths will be incremented. The concepts of prediction error and error correction have been utilized in some form by nearly every subsequent model of Pavlovian learning (e.g., Refs. 48, 49, 174, 243) as well as the neuroscience of conditioning and extinction (e.g., Refs. 244, 245, and see below) and the computational science of reinforcement learning (e.g., Refs. 164, 246, 247).

Several findings have supported a role for prediction error in Pavlovian extinction. In the “concurrent excitor” experiment (e.g., Refs. 248–250), when two CSs are separately conditioned (A+ and B+), more associative loss occurs to them when the two are compounded and extinguished together (AB-) than when they are extinguished on their own (A- or B-). The associative strengths of the compounded A and B are thought to summate, and thus create a greater prediction error. Second, in the “protection from extinction” experiment, if A+ is conditioned (as before), but then compounded with an inhibitory CS B (one that signals “no US”) in the compound extinction phase, then CS B “protects” A from extinction, as if the inhibitory B cancels the prediction of the US and thus eliminates prediction error (e.g., Ref. 251). Third, in the “overexpectation” experiment (e.g., Refs. 252–254), A and B are separately conditioned (A+ and B+), but in the next phase they are combined and paired with the US (AB+). Even though A and B are still reinforced on every trial, associative loss is observed to both. Theoretically, their summed strengths now overpredict the US, and they undergo associative decrement to correct the error. Together, these effects all suggest that what is important is not whether a US occurs or not, but whether there is a discrepancy between what is predicted and what occurs.

One issue concerning the prediction error phenomena just described is that the extent to which the US is predicted by the compound is confounded with the level of responding it elicits. That is, two compounded excitors (as in the concurrent excitor experiment) can produce more responding than an excitor alone, and a compound of excitor and inhibitor (in the protection from extinction experiment) will elicit less responding than an excitor presented alone. Rescorla (e.g., Ref. 12) suggested that if extinction involves learning to inhibit the response, then more responding in extinction can lead to more successful response-inhibition learning. However, the response-level and prediction-level mechanisms were separated in a study with pigeon autoshaping that he later reported (248). Rescorla compared the effects of adding a diffuse excitor (e.g., a noise paired with food) or a diffuse positive occasion setter (e.g., a houselight associated with reinforcement of another keylight CS) on the extinction of key pecking to a target keylight CS. In Rescorla’s preparation, when the excitor was compounded with the keylight CS during extinction, it failed to increase pecking to the CS, although it presumably increased the bird’s expectation of the US. In contrast, when the occasion setter was compounded with the extinguishing keylight, it augmented keypecking without perhaps increasing the direct expectancy of the US. The excitor, but not the occasion setter, produced more extinction of the keylight than keylight extinction alone. Thus, in at least this example of Pavlovian extinction, error created by what the CSs predict plays a role that can be separated from error that might be represented in the level of the response.

2.5.3.2. Instrumental learning

Prediction error phenomena have also been studied in discriminated operant situations, where a response is reinforced in the presence of an S, but not in its absence. Several investigators have reported an operant version of the concurrent excitor experiment (67, 68, 249, 255). For example, in several studies, rats were first reinforced for lever-pressing in three individuals (AR+, BR+, CR+). After some extinction of AR-, BR-, and CR-, the response was further extinguished in one stimulus (AR-) and in a compound of the other two (BCR-). Separate tests of responding in A and B revealed more response loss in B, the S that was extinguished together with another S. Lattal and Nakajima (Ref. 253, Experiment 3) similarly reported an instrumental version of the overexpectation paradigm. In that experiment, rats first received lever-press training with AR+, BR+, CR+; the response was not reinforced in a fourth S (DR-). The rats then received further reinforcement of the response in the compounds AB and CD. Final tests indicated that a larger response decrement had accrued to AR than CR. Theoretically, the decrement occurred because A and B had again combined to create overexpectation of the reinforcer in the compound training phase.

As before, prediction error results can be interpreted in two ways. In discriminated operant situations, the animal can learn about either the instrumental response or what the discriminative stimulus predicts (or both). From an error correction perspective, this may again imply the two sources of error described above: 1) the two Ss may overpredict a reinforcer in the Pavlovian sense, and 2) R may also be higher than what the current reinforcer supports, creating what might be called “response error” (256). Response error might cause a direct adjustment of the strength of the response by allowing the animal to learn that the response no longer yields the reinforcer (e.g., FIGURE 4, link 11) or an inhibitory S-R connection (link 12). The two possibilities have been separated, in a preliminary way, in recent experiments (256). Their designs are sketched in TABLE 5. Rats first received training in which two instrumental responses (R1 and R2) were each reinforced in two discriminative stimuli (AR1+, BR1+, CR2+, DR2+). In a concurrent excitor experiment, after some extinction of the individual stimuli, rats received extinction trials with the compound ABC. (They were allowed to perform either R1 or R2 on these trials.) In an overexpectation experiment, rats received reinforced trials with the compound ABC+; here, either response was reinforced in ABC. In both experiments, the ABC compound presumably caused great Pavlovian expectation of the reinforcer, and thus great stimulus prediction error, but this was the same for stimulus A, B, or C. However, in the ABC compound, R1 was evoked by two Ss (A and B), whereas R2 was evoked by only one S (C). Hence, there was more R1 than R2 responding in the compound phase, creating more potential “response error” for R1. Consistent with that possibility, tests in both experiments revealed that the compound treatment had a greater negative effect on R1 responding (tested in B) than R2 responding (tested in C). The results confirm, in an instrumental learning setting, that the level of responding is important in instrumental extinction and can be separated from stimulus error. The findings complement other results suggesting the animal must emit the operant response in extinction to create extinction; under typical conditions, mere Pavlovian extinction of S will not do (215).

Table 5.

Designs of experiments suggesting a role for response error in instrumental extinction

Phase
Training	(Extinction)	Compound	Test
Concurrent Excitor Experiment
A: R1+R2-	A: R1-R2-
B: R1+R2-	B: R1-R2-	ABC: R1-R2-	B: R1+R2-
C: R2+R1-	C: R2-R1-		C: R2+R1-
D: R2+R1-	D: R2-R1-
Overexpectation Experiment
A: R1+R2-
B: R1+R2-		ABC: R1+R2+	B: R1-R2-
C: R2+R1-			C: R2-R1-
D: R2+R1-

Both experiments were within-subject, so all subjects received the treatments shown. A, B, C, and D, different discriminative stimuli (counterbalanced); R1 and R2, lever press or chain pull responses (counterbalanced); +, reinforced; -, nonreinforced. During testing in both experiments, R1 was lower in B than R2 was in C. [See Bouton et al. (256).]

2.6. Summary

The main causes of extinction in Pavlovian and instrumental learning are arguably generalization decrement and prediction error. In generalization decrement, the animal essentially learns to discriminate the conditions of acquisition and extinction or merely stops responding when it stops generalizing from acquisition to extinction. It appears to do this based on the accumulation in memory of a number of nonreinforced trials (e.g., Ref. 222). This process explains the PREE, which occurs because nonreinforced trials delivered in acquisition help acquisition generalize better to extinction. In prediction error, preliminary evidence in Pavlovian extinction (248) suggests that the animal’s overprediction of the US plays the crucial role. Presumably, that overprediction is corrected by acquisition of a mechanism sketched in FIGURE 4B, which the evidence suggests may be negative occasion setting (link 5). In instrumental extinction, the evidence to date contrastingly emphasizes what can be termed response error (256). Instead of the US overprediction of the reinforcer, it is the high level of unreinforced responding in extinction that causes the animal to learn to inhibit the response (FIGURE 4, links 11 or 12). The results of Rescorla (248) and Bouton et al. (256) begin to suggest that stimulus error and response error may be dissociated in Pavlovian and instrumental extinction: Pavlovian extinction is driven by the former and not the latter, and instrumental extinction is driven by the latter. This conclusion, however preliminary, is once again consistent with the idea that Pavlovian learning provides a mechanism for learning about stimuli in the environment whereas instrumental learning provides a mechanism for learning about behavior.

3. PAVLOVIAN EXTINCTION: NEUROBIOLOGICAL MECHANISMS AND CIRCUITS

The neurobiological mechanisms underlying the encoding and retrieval of Pavlovian extinction memories are of paramount interest given the essential role for these processes in emotional regulation and a host of behavioral therapies. Consequently, recent years have witnessed an explosion of research into the neural basis of extinction learning and memory, particularly in Pavlovian fear conditioning paradigms. This work has established important new insights into the brain mechanisms of extinction, including the identification of key cellular and circuit mechanisms involved in this phenomenon. In the following sections, we will review the neural mechanisms of Pavlovian extinction. This discussion will primarily focus on neural circuits involved in the extinction of conditioned fear, which has undergone the most extensive neurobiological analysis to date (8, 9, 257–261). Pharmacological and genetic studies that inform the underlying mechanisms and neural circuitry of extinction will be discussed where appropriate; this literature has been exhaustively reviewed in other outlets (262–266).

3.1. Neural Circuits for Extinction Learning

Decades of research exploring the neural mechanisms for Pavlovian fear conditioning have uncovered the fundamental neural circuits underlying this form of learning (3–6, 267) (FIGURE 5). Sensory information concerning unimodal CSs, such as acoustic tones, reaches the amygdala by way of thalamic and cortical afferents, including the auditory thalamus [e.g., medial geniculate nucleus (MGN)] and primary auditory cortex. Information concerning footshock USs reaches the amygdala by way of somatosensory thalamic relays (e.g., posterior intralaminar nucleus) as well as the brainstem [e.g., parabrachial nucleus and periaqueductal gray (PAG)]. Multimodal contextual information reaches the amygdala via projections from the hippocampus and ventral subiculum. Within this circuit, the basolateral complex of the amygdala (BLA); including the lateral, basolateral, and basomedial nuclei of the amygdala—a locus for the convergence of CS, US, and contextual information—essential for the both the acquisition and expression of conditioned fear. Projections from the BLA to the central nucleus of the amygdala (CEA) drive the expression of conditioned fear responses via a diversity of CEA efferents, including the PAG (freezing), paraventricular nucleus (PVN) of the hypothalamus (glucocorticoid release), and rostral ventrolateral medulla (changes in heart rate and blood pressure). The discovery of this fundamental circuit for fear conditioning has been foundational for understanding the neural mechanisms underlying extinction learning.

FIGURE 5.

Neural circuits mediating Pavlovian fear conditioning, extinction retrieval, and renewal. Left: auditory conditioned stimuli (CS) are processed by the thalamic medial geniculate nucleus (MG) and auditory cortex (AC) en route to the basolateral nucleus of the amygdala (BLA). This information is associated with the footshock unconditioned stimulus (US), which is conveyed by the parabrachial (PB) nucleus and other relays, in the amygdala. Contextual information is processed in the hippocampus (HPC) and medial prefrontal cortex, which are interconnected by the nucleus reuniens (RE). HPC projections and MG nucleus projections to the BLA are involved in the expression of fear conditioned responses (CRs) to auditory and contextual stimuli, respectively. Projections from the central nucleus of the amygdala (CeA) to the ventrolateral midbrain periaqueductal gray (PAG) are involved in the freezing CR. The dorsolateral PAG mediates activity burst unconditioned responses (UR) to the shock US. Middle: the retrieval of extinction memories is associated with infralimbic (IL) cortical inhibition of BLA neurons involved in CR production to auditory and contextual stimuli. IL projections to the RE may suppress retrieval of contextual fear memories. Right: presentation of an extinguished CS outside the extinction context causes a return in the freezing CR, and this is mediated by HPC-mediated excitation of the fear CR, as well as inhibition of IL neurons involved in suppression of the CR.

3.1.1. Basolateral amygdala is essential for extinction learning

The first evidence suggesting a role for the amygdala in the extinction of Pavlovian fear conditioning came from electrophysiological work showing that amygdala neurons exhibit decreases in CS-evoked firing during extinction that parallel decreases in conditional responding (268–270). However, because amygdala lesions produce severe impairments in the expression of fear CRs, a causal role for the amygdala in extinction learning was difficult to establish. This barrier was overcome by Davis and colleagues (271), who first showed that intra-amygdala infusions of APV, an antagonist of the N-methyl-d-aspartate (NMDA) receptor subtype of glutamate receptor, impaired the acquisition of extinction in a fear-potentiated startle paradigm. Later work has confirmed this finding, implicating GluN2B-containing NMDA receptors, in particular, in extinction learning (272–276). Importantly, NMDA receptor antagonists prevent extinction learning when infused into the BLA, and not CEA (277). This pattern of results has also been replicated with infusions of the GABA_A agonist muscimol into the BLA, which also block the extinction of contextual fear (272, 278, 279). The role for the amygdala in extinction is not limited to aversive conditioning procedures, insofar as BLA lesions impair the extinction of appetitive incentive value (280) and NMDA antagonists in the amygdala impair extinction of conditioned place preferences (281) and appetitive approach in pigeons (282).

Importantly, infusion of antagonists of the AMPA subtype of glutamate receptors into the BLA or CEA does not impair the acquisition of extinction despite abolishing expression of the fear CR (277, 283). This latter finding suggests that the expression of a fear CR is not required for extinction learning, which has theoretical implications for how we understand the associative basis of extinction; extinction is not likely to result from an inhibitory S-R association (as above) (284). That said, Krupa and Thompson (285) have reported that inactivation of the facial motor nuclei block CR and UR production and prevent extinction of an eyeblink CR in rabbits. This result is consistent with the view that extinction results in an inhibitory S-R association. However, later work showed that red nucleus inactivation, which also prevents CR production, does not impair extinction (286). Because motor (but not red) nucleus inactivation causes facial paralysis during extinction training, retrieval testing in a drug-free state would presumably yield renewal of the eyeblink CR in the absence of paralysis. Renewal in the nonparalyzed state (i.e., a change in context) might therefore account for the apparent extinction deficit observed when eyeblink extinction is performed under motor nucleus inactivation.

3.1.2. Infralimbic cortex regulates the consolidation of extinction memory

Although the BLA mediates the acquisition of extinction, neither BLA neuronal activity nor NMDA receptor activation is required during the consolidation interval (278). In contrast, impairments in extinction consolidation have been reported with pharmacological manipulations that inhibit neuronal activity or synaptic plasticity in the infralimbic division (IL) of the medial prefrontal cortex (mPFC) (278, 287–290).

Considerable work on fear extinction has suggested an important role for the mPFC in extinction learning (291–293). Specifically, electrolytic lesions placed in the infralimbic (IL) region of the mPFC were found to increase freezing to an extinguished CS during a retrieval test 24 h after extinction. In contrast, the lesions did not affect freezing to the CS during fear conditioning and they did not affect conditional responding during a single session of extinction training itself (294, 295). The localization of mPFC lesions to IL is important insofar as large lesions of the mPFC that encompass both the prelimbic (PL) and infralimbic cortexes fail to produce extinction deficits (296–298). That said, a role for IL in extinction learning is now well established and is supported by convergent evidence using a variety of methodologies and tasks.

The finding that IL lesions spare conditional freezing during both fear conditioning and extinction suggest that it mediates the consolidation and/or retrieval of the inhibitory extinction memory, rather than extinction learning per se. This possibility was first suggested by a report that found that systemic administration of CPP, a competitive NMDA receptor antagonist, before extinction training reduced extinction retrieval 24 h later without affecting conditioned freezing or response suppression during the extinction session (299). Subsequent work found that intra-IL infusions of NMDA receptor antagonists impaired extinction retrieval when made immediately after extinction training or before the retrieval test but had no effect when infued before extinction (272, 287, 290). Similar results have been observed after reversible inactivation of the IL with muscimol (278, 300). This contrasts with the effects of local infusion of NMDA receptor antagonists or muscimol into the BLA, which impair the acquisition but not consolidation or retrieval of extinction (272, 290). Interestingly, once extinction has been learned, the acquisition of a second extinction experience (“reextinction”) requires NMDA receptor activation in the IL, rather than the BL (272). By way of comparison, pharmacological inactivation of the prelimbic cortex (PL) only affects fear expression but does not affect extinction learning or retrieval (301–301). Collectively, these results suggest that synaptic plasticity in the IL is critical for the consolidation and retrieval of long-term extinction memory.

3.1.3. Reciprocal interactions between the mPFC and amygdala influence encoding of extinction memories

Given the important role for BLA and IL neurons in the acquisition and consolidation of fear extinction, it is not surprising that recent work implicates connectivity between these two areas in the regulation of fear (303) and extinction learning. In an interesting series of experiments, Senn and colleagues (304) used a within-subjects procedure to show that projections from BLA to the mPFC influence extinction learning. Optogenetic inhibition of IL-projecting neurons in the BLA yielded impaired extinction, whereas inhibiting PL-projecting neurons in BLA had the opposite effect (and neither manipulation affected within-session decreases in freezing to either CS during extinction training) (304). This suggests that the balance of activity in mPFC-projecting BLA neurons can influence long-term extinction performance.

More recent work has shown a critical role for IL projections to the BLA in extinction learning. For example, optogenetic inhibition of either IL principal cells (305) or IL terminals in BLA (306, 307) impairs the retention of extinction (again without affecting within-session extinction). Within the BLA, the basomedial nucleus (BM) appears to be a particularly important locus for prefrontal inhibitory control of conditioned freezing (306). Optogenetic excitation of IL terminals in BM during extinction training reduced conditioned freezing both during extinction and during a light-free retention test 24 h later. The retention facilitation was specific to activation of the IL-BM projection, insofar as bulk excitation of BM neurons reduced freezing acutely but did not produce enhancements in extinction retention (306). Interestingly, recent work shows that extinction training in mice alters the excitability of IL neurons projecting to the BLA (measured in brain slices ex vivo), and chemogenetic inhibition of these neurons impairs extinction retention (308).

3.1.4. Bottom-up prediction error signals mediate extinction learning

At a theoretical level, the establishment of extinction learning requires that animals detect a change in the prevailing conditions. During extinction training, the omission of an expected US would be expected to generate a negative prediction error (see sect. 2.5.3) and initiate the neural events that ultimately lead to extinction-related plasticity and reductions in conditional responding. In the last several decades, considerable work has revealed discrete neural circuits and mechanisms by which prediction errors influence learning and memory, particularly in the reward system (309). More recently, the neural circuits responsible for generating prediction errors during the extinction of conditioned fear have come under study (244, 245).

During the course of fear conditioning, it has long been appreciated that the acquisition of analgesic CRs play a role in limiting the impact of future nociceptive USs in a manner consistent with the Rescorla-Wagner model, for example (310). Considerable work indicates that this negative feedback mechanism requires opioid receptors in the midbrain PAG (311). Given the importance of PAG opioid receptors in US processing and creating negative feedback during fear conditioning, it is likely that this system also contributes to fear extinction learning. Consistent with this, McNally and colleagues have found that both systemic (312) and intra-PAG (313) administration of naloxone, a µ-opioid receptor antagonist, impair the acquisition but not expression of fear extinction. These effects are dose dependent and are specific to naloxone infusions into the ventrolateral (vlPAG) but not dorsolateral (dlPAG) PAG (313). Consistent with a role in negative prediction error, these same manipulations prevent the overexpectation of fear (314–316). The rate of extinction learning can also be enhanced by increasing endogenous opioid neurotransmission in the vlPAG (317). Interestingly, more recent work has revealed that the vlPAG is also involved in the extinction of conditioned suppression of instrumental lever pressing (318), a fear CR that can be expressed independently of freezing (319).

In addition to generating a negative prediction error, the omission of an expected aversive US generates an affective outcome: relief (47, 320). This unexpected positive outcome might be expected to activate dopaminergic systems (321) that are activated by the experience of unexpected food rewards, for example (309, 322, 323). Indeed, work in Drosophila has shown that extinction induces plasticity in dopaminergic neurons that oppose the activity of neurons representing aversive memory (324). Consistent with the role for dopaminergic neurons in extinction, it has been reported that the extinction of conditioned fear in rodents is impaired by systemic or intracranial infusions of dopamine (DA) D2 receptor antagonists in either IL or the nucleus accumbens (Acb) (325, 326). Moreover, recent work has shown that dopaminergic neurons in the ventral tegmental area (VTA), a region critical for signaling reward prediction errors, are activated by US omission early in extinction training (327). Early extinction of conditioned fear is also associated with CS-induced increases in Acb DA levels (measured with in vivo voltammetry), and these increases in DA are highest during the post-CS period concomitant with the expected US (328). Optogenetic inhibition of VTA neurons during the period of US omission (but not during the ITI) impairs both acquisition and subsequent retention of extinction of conditioned freezing in mice (327, 329). Optogenetic activation of the same neurons during US omission facilitates extinction learning and retention (327). Projections from the VTA to the Acb appear to be critical for extinction learning, insofar as optogenetic inactivation of VTA terminals in NAc impair the acquisition and retention of fear extinction in mice (329). Moreover, chemogenetic activation the VTA→Acb pathway during extinction learning enhances extinction retention and reduces fear renewal in rats (330); similar results have been observed after activation of the BLA→Acb pathway (331). Increasing DA levels with systemic l-DOPA after extinction have also been reported to reduce spontaneous recovery and reinstatement of fear in both mice and humans (332, 333). Collectively these results suggest that the midbrain PAG and VTA are critical for extinction learning by generating prediction errors that influence downstream circuits involved in regulating conditioned fear.

3.2. Synaptic Mechanisms for Extinction Learning

It has long been appreciated that Pavlovian conditioning phenomena are likely to be mediated by cellular processes in brain regions where CS and US information converge (334). In support of this idea, decades of research have now revealed an essential neural circuit for Pavlovian fear conditioning centered on the amygdala (5, 267, 335–337). Anatomically, the amygdala is a locus for convergence of multisensory information including auditory CSs and shock USs, for example (338–340). Much of this sensory convergence occurs in the lateral (LA), basolateral (BL), and basomedial (BM) nuclei, collectively termed the basolateral complex of the amygdala or BLA. The BLA is generally indispensable for the acquisition and expression of Pavlovian fear conditioning (341, 342), and the BL in particular may have an important role in active defensive responses (343–345). Projections from the BLA to the central nucleus of the amygdala (CEA) are engaged by fear conditioning (346, 347) and are necessary for the generation of conditioned fear responses, such as freezing behavior (347, 348). Recently, multisynaptic projections from the CEA to the BLA have also been implicated in fear conditioning (349), suggesting that information flow in this circuit may be more reciprocal than previously appreciated. Ultimately, Pavlovian fear conditioning results in increases in CS-evoked neuronal firing in both the BLA and CEA; learning-related changes in the activity of amygdala neurons are believed to drive conditioned fear responses at various downstream loci.

Defining the neuronal mechanisms underlying fear conditioning is a critical starting point for understanding the synaptic mechanisms for extinction learning. Indeed, considerable work provides strong evidence that both fear conditioning and extinction are mediated by synaptic plasticity in the BLA, including long-term potentiation (LTP) (350–352). LTP in the amygdala has been described both in vivo (351–355) and in vitro (356). Similar to other brain regions, LTP in the amygdala is mediated by NMDA receptors that confer Hebbian associativity (354, 357, 358). The Hebbian property of amygdaloid LTP is ideally suited to permit strong US inputs onto BLA neurons to promote plasticity at weaker CS inputs that converge upon the same neurons (350, 352, 359, 360). Consistent with this idea, Pavlovian fear conditioning induces synaptic potentiation at auditory inputs onto BLA neurons (361–365) and optogenetic manipulation of synaptic strength in this pathway influences the expression of conditioned fear in behaving mice (366). Hence, there is considerable evidence that Pavlovian fear conditioning is mediated, at least in part, by associative LTP in the amygdala.

3.2.1. Synaptic depotentiation of conditioning-induced plasticity in the amygdala

Early views of extinction, and indeed many theoretical models (183), posit that extinction involves “unlearning” the conditioning memory: a process that might result from a reversion of the neurobiological changes established during conditioning. By this view, extinction procedures might be associated with synaptic plasticity in the amygdala (FIGURE 6). For example, there is some evidence that extinction is associated with a reversal or “depotentiation” of conditioning-induced plasticity (367). Consistent with this view, it has been shown that extinction reverses biochemical changes, such as increased Akt (a serine/threonine protein kinase) phosphorylation in the BLA, that result from fear conditioning (368, 369). Interestingly, these effects are prevented by inhibitors of calcineurin, a phosphatase that is increased after extinction. In addition, low-frequency electrical stimulation of BLA afferents increases calcineurin levels, reduces levels of phosphorylated Akt, and decreases the expression of fear-potentiated startle in a manner similar to extinction training (368, 369). However, in these studies animals were presented with CS tests shortly before the extinction or low-frequency electrical stimulation procedures. These CS test trials might have reactivated the conditioning memory, rendering them sensitive to reconsolidation blockade by either the subsequent extinction trials or low-frequency stimulation (370–372).

FIGURE 6.

Synaptic plasticity models for fear conditioning and extinction. Left: long-term potentiation (LTP) in axons conveying conditioned stimulus (CS) information to the basolateral amygdala (BLA) principal neurons (PN) results in the expression of a fear conditioned response. Right: extinction may be supported by several different synaptic changes including depotentiation of the LTP induced during conditioning, recruitment of LTP at CS afferents on amygdala inhibitory interneurons (IN), or LTP at inhibitory interneuron synapses on amygdala PNs (LTPi). All three forms of plasticity would be expected to reduce conditional responding, but only LTP (interneuron) and inhibitory LTP would allow for recovery of the fear CR after extinction.

Electrophysiological studies have also suggested that extinction reverses conditioning-related increases in amygdala synaptic transmission (367). After fear conditioning, synaptic potentials are increased at excitatory synapses in the LA, which is presumably caused by a “behavioral” induction of LTP (361, 362, 365, 367). However, after extinction training these synaptic responses are similar to those in either untrained controls or controls that had received the CS and US unpaired (367). This suggests that extinction procedures may depotentiate conditioning-induced increases in synaptic potentials in the LA. Consistent with this, extinction training reduces the expression of AMPA receptors in the LA, effectively reverting increases in AMPA receptor expression that are observed after conditioning or LTP induction. In addition, extinction training prevents synaptic depotentiation in amygdala brain slices (367). Synaptic depotentiation in vitro and the extinction of conditioned fear in behaving animals is also prevented by cellular manipulations that block AMPA receptor endocytosis, a process that reduces synaptic strength (367, 373). Together, these studies are consistent with the view that extinction may reduce conditioned fear at least to some extent by reversing conditioning-induced increases in BLA synaptic transmission.

3.2.2. Potentiating inhibitory synaptic transmission in the amygdala

Depotentiating synapses that have undergone LTP during fear conditioning would be a parsimonious mechanism for extinction learning. However, this sort of mechanism cannot explain the recovery of extinguished CRs (e.g., renewal, spontaneous recovery, reinstatement, rapid reacquisition) that occurs under a very large number of circumstances (see sect. 2.1). As reviewed above, the fact that conditional responses can return after extinction suggests that extinction procedures result in new learning (e.g., inhibition or an inhibitory memory) that interferes with the excitatory conditioning memory thereby suppressing conditional responding. By this view, extinction must be encoded in a way that preserves the synaptic plasticity associated with conditioning.

One possibility is that extinction learning potentiates excitatory synapses on inhibitory interneurons in the BLA (FIGURE 6), thereby sparing LTP at auditory afferents, for example. Consistent with this, LTP has been described at both cortical and thalamic excitatory synapses on amygdala interneurons (374–376), although only the latter is NMDA receptor dependent. This observation is particularly important, because extinction learning is prevented by NMDA receptor antagonism in the amygdala (271–273, 277). Together, these data suggest that NMDA receptor-dependent LTP in the BLA is required for extinction. Presumably this plasticity is localized to thalamic afferents that terminate on BLA interneurons. Extinction-induced, incremental synaptic plasticity on local interneurons would result in increased feed-forward inhibition of BLA principal neurons to limit the expression of conditioning-induced plasticity. The NMDA receptor dependence of extinction learning has had important implications for developing pharmacological manipulations to facilitate extinction learning. For example, systemic or intra-BLA administration of d-cycloserine (DCS), an amino acid that potentiates NMDAR function, facilitates extinction learning and retention (377–381). Translation of this work to humans has shown some promise (382, 383), although DCS has not proven effective for patients with phobic disorders in many studies (384, 385).

Another locus of inhibition underlying extinction might involve potentiation of synaptic inputs onto islands of inhibitory intercalated cells (ITCs) that are interspersed between the BLA and CEA (386) (FIGURE 6). These GABAergic interneurons serve as an inhibitory interface between the BLA and CEA and limit the excitatory influence of the BLA on the CEA, consequently suppressing the generation of learned fear responses (387–389). Excitatory glutamatergic afferents from BLA projection neurons terminate on ITC neurons and exhibit NMDA receptor-dependent LTP (390). Importantly, extinction training induces LTP at BLA terminals on inhibitory ITCs, which may be a mechanism for limiting CEA-mediated fear responses after extinction (387, 391–394). In addition to excitatory BLA afferents, ITC cells also receive infralimbic and thalamic projections, and plasticity at these terminals might also be involved in both fear conditioning and extinction (386, 395). This may be one mechanism by which IL mediates extinction learning and memory. The induction of LTP at excitatory inputs onto inhibitory interneurons, either within the BLA or among ITC clusters, is a putative mechanism for suppressing the expression of conditioned fear responses after extinction.

An alternative mechanism for increasing the inhibition of BLA principal cells might involve potentiating transmission at GABAergic afferents on these cells, a form of synaptic plasticity termed “inhibitory” LTP (LTPi) (FIGURE 6). For example, high-frequency stimulation of LA neurons evokes an NMDA receptor-dependent LTPi at GABAergic synapses terminating on BLA principal neurons (396). It is not known, however, whether extinction learning generates LTPi in the amygdala to suppress fear. Nonetheless, structural changes in inhibitory synapses in the BLA have been found to occur after extinction learning. For example, extinction greatly increased the size of inhibitory synaptic terminals terminating on excitatory neurons recruited by fear conditioning (397, 398). Extinction training also increases levels of inhibitory GABA_A receptors in the BLA, as well as the GABA-clustering protein gephyrin (399). Moreover, viral-mediated knockdown of inhibitory synapses in the BLA disrupts LTP at subicular afferents and selectively impairs extinction learning (400). Pharmacological manipulations that perturb synaptic plasticity among ITCs also yield extinction impairments (401). Collectively, these data suggest that extinction potentiates inhibitory synaptic transmission to dampen the excitation of BLA principal neurons supporting fear memories, possibly through the induction of LTPi at inhibitory synapses in the BLA. Of course, a critical question is how these inhibitory synapses are in turn modulated to allow for the recovery of extinguished fear, during renewal, spontaneous recovery, and reinstatement, for example.

3.2.3. Cellular consolidation of extinction in the amygdala

After fear conditioning, posttraining inhibition of cellular cascades initiated by NMDA receptor activation (e.g., PKA/MAPK activation, protein synthesis, and gene transcription) impairs the consolidation of fear memory (352, 402). Considerable evidence indicates that extinction also recruits these signaling cascades in BLA neurons during the consolidation of extinction memories. For example, extinction induces immediate early gene expression in the BLA and inhibitors of ERK/MAPK infused into the BLA impair the consolidation of extinction (403, 404). Similarly, inhibitors of protein synthesis or gene transcription infused into the BLA before extinction training also impair the formation of long-term extinction (405). Impaired extinction consolidation is not observed when protein synthesis is inhibited in the medial geniculate nucleus (406), which is the primary relay for auditory information to the forebrain. Importantly, the cellular mechanisms underlying the consolidation of extinction in the BLA are different from those underlying reconsolidation of the original fear memory, for example (274, 407–409). Interestingly, the induction of ERK/MAPK in the PFC and BLA depends on opioid receptors in the vlPAG (410). This suggests that extinction involves cellular and synaptic changes in a broad network of interconnected brain regions.

3.2.4. Extinction-induced synaptic plasticity in the medial prefrontal cortex

Consistent with a role for the IL in the acquisition of extinction, electrophysiological studies have revealed that extinction induces synaptic plasticity in both medial thalamic and hippocampal afferents to the mPFC (411–413), as well as changes in the intrinsic excitability of IL neurons (414, 415). For example, extinction of fear in rats increases AMPA/NMDA ratios in IL principal neurons, a change that is consistent with the induction of synaptic LTP in the IL (415). Increased AMPA currents were associated with insertion of GluA2 containing AMPA receptors, a change that could be prevented with metabotropic glutamate receptor antagonists. NMDA receptor antagonists in the IL also impair the consolidation of extinction memory (290) and reduce burst firing of IL neurons that is required for successful extinction retention (287, 416). Moreover, adolescent rats that normally exhibit impaired fear extinction (417, 418) show reduced IL synaptic plasticity (419).

Other factors that modulate IL synaptic plasticity have also been found to influence extinction learning. For example, infusion of noradrenergic antagonists into the IL impairs extinction learning and disrupts extinction-induced changes in IL plasticity (420). Extinction learning also depends on brain-derived neurotrophic factor (BDNF), a plasticity promoting protein, in hippocampal terminals in the IL (421–424). Postextinction inhibition of histone acetyltransferases in IL also promotes extinction memory and synaptic plasticity in the IL (425). Together, these studies suggest that synaptic plasticity in the IL has an important role in the acquisition of extinction.

3.3. Neural Circuits for Extinction Retrieval

Fear conditioning and extinction yield distributed plasticity throughout the forebrain that is expressed in behavior through the coordinated activity of neural circuits. As previously described, extinction learning yields cellular and synaptic changes that augment, not erase, the neurobiological changes that support conditioning. In the last two decades, considerable progress has been made in defining the neural circuits that coordinate the expression of these synaptic changes, including circuits that permit the contextual control of fear memories after extinction (FIGURE 5). In line with the loci of conditioning-and extinction-related synaptic plasticity, core elements of the circuits involved in extinction retrieval include the medial prefrontal cortex, amygdala, and hippocampus.

3.3.1. Infralimbic cortical projections to amygdala suppress conditional fear

In addition to playing a role in the formation of extinction memories, considerable work indicates that the IL has an important role in retrieving extinction memories once formed. The canonical finding supporting this view came from an early single-unit recording study showing that neuronal activity in the IL is correlated with extinction performance during a retrieval test in rats. Specifically, Milad and Quirk (426) found that after fear conditioning and extinction, an extinguished auditory CS drove increases in short-latency spike firing in the IL during testing. The magnitude of CS-evoked spike firing was correlated with the degree to which freezing was suppressed during the retrieval test, such that animals that showed the lowest freezing exhibited the highest CS-evoked firing in IL. Importantly, IL neurons only became responsive to the CS after extinction training: they were relatively quiescent to nonextinguished CSs (which produced high levels of freezing behavior). Consistent with a role for IL in extinction retrieval, pharmacological inhibition of the IL causes a return of conditional fear responses after extinction. For example, infusions of muscimol (a GABA_A agonist) into the IL increase conditioned freezing after the extinction of either contextual (302) or auditory (427) fear conditioning.

The observation that the IL is engaged during the retrieval of extinction memory suggests that is pivotal in suppressing conditioned fear responses (9, 260, 428) (FIGURE 5). This observation motivated years of subsequent work in both rats and humans, because it suggested an anatomical locus for the inhibitory processes underlying extinction-based behavioral therapies, including exposure therapy. In humans, for example, neuroimaging work has shown that retrieving extinction increases regional cerebral blood flow in the subgenual anterior cingulate cortex, an IL analogue (429–431). Similarly, extinction retrieval increases the expression Fos, a cellular marker of neuronal activity, in the IL in rats (432–435). However, how does increased activity in the IL result in fear suppression during extinction retrieval? Functional circuit tracing studies have shown that IL neurons projecting to the amygdala are strongly activated (as indexed by Fos expression) after extinction retrieval (434). Moreover, work with a novel transgenic rat that expresses a fluorescent reporter in the dendrites of active neurons revealed that BLA neurons engaged by extinction retrieval are preferentially innervated by IL axons (436). This suggests that axonal projections from the IL to the amygdala constitute a critical pathway for the inhibition of fear when extinction is expressed behaviorally. Consistent with this view, the IL has strong projections to the BLA, as well as to a group of interneurons neighboring the BLA (the intercalated cells or ITC) that are positioned to inhibit outflow from the BLA (388, 389) [although some reports fail to observe IL-ITC synaptic connections (395, 437)] Importantly, pharmacological excitation of the IL increases Fos expression in ITC cells (438) and electrical stimulation of IL yields strong inhibition of BLA outflow to the CEA (439). The IL also projects to local inhibitory interneurons in the BLA, and these synaptic inputs produce strong inhibition in the BLA after extinction (440). Hence, there are multiple mechanisms for prefrontal cortical inhibition of conditioned fear responses, including freezing behavior, after extinction.

More recent studies, however, are equivocal about the role of IL in extinction retrieval. In one experiment, optogenetic inhibition of IL principal cells in rats was found to impair the acquisition, but not retrieval, of extinction (305). A similar pattern of results was observed in a different study that optogenetically inhibited IL axonal terminals in the BLA of mice (307). In contrast to these reports, other studies have found that optogenetic inhibition of IL does indeed inhibit extinction retrieval after fear conditioning (441). Moreover, optogenetic activation of the IL can restore extinction performance under conditions that promote relapse after extinction of either fear conditioning (441) or appetitive conditioning (442). The reasons for the disparities in this literature are not yet clear, but differences in the specific cell types targeted in each study might be a critical factor.

Recent single-unit recording work has also presented a more complicated picture of the role for IL in regulating conditional responding after extinction. For example, it has been reported that CS-evoked single-unit activity in IL correlates with fear expression, rather than fear inhibition, under some circumstances after extinction (416). In one case, inbred mice that show poor extinction learning and retrieval exhibited robust CS-evoked spike firing in the IL during a retrieval test in which they freeze at high levels (443). Of course, it is important to consider that investigators typically normalize CS-evoked firing to pre-CS spontaneous firing rates to more easily compare changes in firing across populations of neurons with different firing rates. Fear expression (and inhibition) may be better represented by the spontaneous activity of mPFC neurons. For example, Giustino and colleagues (444) have reported that the expression of fear to auditory CSs is associated with a suppression of spontaneous activity in the mPFC, particularly in the IL. Decreases in the spontaneous firing of IL neurons have also been reported to occur immediately following fear conditioning, a period of time when fear expression is high and extinction learning is compromised (445–447). Conversely, reductions in fear associated with an external inhibitor (a novel cue added to a fear CS) increase spike firing in IL relative to PL (444). In sum, the balance of spike firing in the PL and IL, rather than absolute firing rates in either area, may be a key factor regulating the expression conditioned fear responses. Consistent with this idea, simultaneous recordings of single-unit activity from the PL and IL reveal that reciprocal patterns of firing in PL and IL correlate highly with freezing behavior: firing rate ratios that favor the PL are associated with high fear states and renewal of extinguished fear, whereas ratios that favor IL are associated with low fear states and extinction retrieval (444). These reciprocal patterns of mPFC activity are also manifest in CS-evoked Fos expression (433) and single-unit firing (448) during low- and high-fear states associated with extinction retrieval and fear renewal, respectively.

The IL also has a role in the retrieval of extinction memory in appetitive conditioning tasks. In a seminal report, rats first received pairings of an auditory CS and delivery of a food pellet US to establish conditioned magazine (food cup) approach (449). After conditioning, the approach response was extinguished. Pretraining excitotoxic lesions of the IL did not produce deficits in either conditioned approach or extinction of the approach response. However, IL lesions increased spontaneous recovery of conditioned approach 24 h after extinction training. IL lesions also increase renewal of conditioned approach (449) as well as conditioned freezing (451). The finding that IL lesions increase spontaneous recovery and renewal of both aversive and appetitive CRs suggests that the IL has a general role in inhibiting context-inappropriate CRs.

This idea has been borne out in an elegant series of experiments that demonstrate that the IL is required for the context-dependent expression of inhibition to an extinguished appetitive CS (452). Using a novel behavioral procedure, Laurent and colleagues (452) submitted rats to an appetitive Pavlovian conditioning procedure in which two different auditory CSs (clicker or tone) were paired with one of two outcomes (food or sucrose); each stimulus subsequently was then extinguished in a distinct context. After extinction, the animals underwent instrumental training in which a different action (left or right lever press) earned each outcome; instrumental training was conducted in both extinction contexts. During a transfer test (which was nonreinforced), each extinguished CS was presented in either the context in which it was extinguished or in the alternate context; the animals were given the opportunity to press either lever during the test. In this situation, presentation of the extinguished CS in the context in which it was extinguished suppressed instrumental responding on the action that earned the same outcome and increased responding on the alternate action. Interestingly, pharmacogenetic inhibition of neuronal activity in the IL reversed this response pattern and eliminated the context-dependent suppression of instrumental action produced by the extinguished CS. This suggests that the IL is not only involved in the suppression of conditioned freezing to an extinguished fear CS but has a broader role in mediating context-dependent inhibition. In this case, it is likely that mPFC projections to the dorsal and ventral striatum, which influence instrumental behavior, are critical (453). Of course, the function of the IL is not limited to behavioral inhibition alone. For example, there are neuronal correlates of both instrumental reward seeking and its inhibition in the IL and PL in appetitive conditioning tasks (454). The role of IL in extinction of instrumental behavior is reviewed in sect. 4.1.1.

3.3.2. Hippocampal circuits mediate the context dependence of extinction

It is apparent from both behavioral and neurobiological work that Pavlovian conditioning and extinction yield (at least) two competing memories or associations, a CS-US association that promotes conditional responding and an inhibitory CS-“no US” memory or association that suppresses that response (FIGURES 2 and 4). Neurobiological studies suggest that these associations, at least in the case of fear conditioning and extinction, may reside side-by-side and take the form of distinct synaptic changes among microcircuits in the amygdala. Of course, this begs the question of how the expression of conditioned fear is regulated after extinction given that animals will suppress conditioned fear responses to an extinguished CS in the extinction context after extinction yet will show a return of conditional responding to that CS under other circumstances. This implies that there are neural mechanisms that allow the context-dependent expression of fear memories after extinction (FIGURE 5).

Decades of work have demonstrated a critical role for the hippocampus in context processing (258, 455, 456). The role of the hippocampus in context processing is complex, and many factors influence the degree to which a particular task recruits and relies upon the hippocampus. It appears to be particularly important in situations that require rapid acquisition of contextual representations, such as in contextual fear conditioning wherein animals have only minutes to acquire a contextual representation before footshock is delivered. However, even in this situation, animals can acquire contextual fear conditioning after hippocampal lesions (457–459). What has become clear over the last several decades is that the method by which brain function is disrupted can have a critical role in determining the nature and pattern of deficits observed after the manipulation. For example, animals with permanent (e.g., neurotoxic or electrolytic) pretraining lesions of the hippocampus acquire contextual fear conditioning normally (457–459), whereas those with posttraining lesions exhibit massive deficits, at least for recent context memories (458, 460–462). However, the nature of the context processing deficits observed with reversible hippocampal manipulations is quite different. For example, optogenetic inhibition of the dentate gyrus disrupts the acquisition, but not retrieval, of contextual fear conditioning in mice (463, 464). Of course, behavioral paradigms, procedures, and response measures also influence the contextual demands of a task. As will become clear, these issues are particularly germane to appreciating the role of the hippocampus in the context dependence of extinction.

The first study of the role for the hippocampal system in the context-dependence of extinction was performed by Wilson and colleagues (465) who made pretraining lesions of the fornix before testing rats for spontaneous recovery, ABA renewal, and reinstatement in a conditioned suppression procedure. Although the lesions impaired reinstatement (suggesting they impaired context conditioning, see sect. 2.1.1), they did not affect either renewal or spontaneous recovery. Pretraining lesions of the dorsal hippocampus (DH) also spare renewal of conditioned suppression (466). Dorsal hippocampal lesions or inactivation also spare ABA renewal of appetitive conditioned responses (467), although DH inactivation impairs spontaneous recovery of conditioned magazine approach (468). In contrast to these results, others have reported that lesions of the dorsal hippocampus (both electrolytic lesions of the entire DH or selective neurotoxic lesions of CA1 or CA3), fornix, or entorhinal cortex, made either pretraining or postextinction, impair the renewal of conditioned freezing (469–471). The robust impairments in renewal observed in these studies were observed in both ABA and AAB renewal procedures, suggesting that deficits in context fear alone could not account for the pattern of results. (Context fear acquired during fear conditioning in Context A could influence test performance in the ABA, but not the AAB, procedure.) It has been suggested that the nature of retrieval test procedures (e.g., a single long-duration CS versus multiple discrete CSs) influences the effect of hippocampal lesions on renewal (472). Another possibility is that in conditioned suppression studies animals receive several days of context exposure (in the conditioning context) before conditioning to establish lever pressing. This extensive context exposure in animals with pretraining lesions might allow nonhippocampal systems to mediate the context-dependence of extinction (459, 473).

To mitigate the recruitment of extra-hippocampal systems that compensate for hippocampal damage, Maren and colleagues have examined the effects of pharmacological inactivation of the hippocampus on the renewal of conditioned freezing after the extinction of auditory fear conditioning. This work has shown that infusions of muscimol into the dorsal or ventral hippocampus (VH) before retrieval testing impairs renewal (474, 475), without affecting context discrimination (476) or freezing to a nonextinguished CS (475). Interestingly, renewal impairments after DH inactivation are stronger with AAB and ABC, compared with ABA, renewal procedures (474). This suggests that hippocampal inactivation impairs renewal by affecting the context-dependent excitatory or inhibitory memories (477) rather than the direct influence of the conditioning context. Consistent with a role for the hippocampus in renewal, work in both rats and humans has revealed increased neuronal activity in the hippocampus (indexed by Fos or regional cerebral blood flow, respectively) during renewal (ABA and ABC) of conditioned fear responses after extinction (433, 435, 478–482).

Neurons in the BLA exhibit context-dependent activity (434, 483, 484), and this is abolished by DH inactivation (485). This suggests that hippocampal projections to the amygdala may be involved in regulating the renewal of conditioned freezing. Consistent with this possibility, ABC renewal increases Fos expression in VH neurons that project to the BLA (434, 481), and BLA neurons that are active during ABA renewal receive axonal input from the ventral hippocampus (436). Consistent with this, disconnecting the VH and BLA using asymmetric lesions (434) prevents ABC renewal. Interestingly, optogenetic inhibition of CEA-projecting neurons in the VH of mice also impairs ABA renewal, whereas inhibition of BLA-projecting neurons impaired the expression of context freezing (486). This suggests that VH projections to both BLA and CEA are involved in the contextual control of extinguished fear.

In addition to projections to the amygdala, VH projections to the mPFC are also involved in renewal. For example, both ABA and ABC renewal increase Fos expression in VH neurons projecting to the mPFC (434, 481, 482, 487) and disconnecting the VH and mPFC with asymmetric lesions yields impairments in renewal of conditioned freezing (434). Given that the IL has been implicated in the suppression of conditioned fear after extinction, it is interesting that VH neurons projecting to IL are recruited during ABC fear renewal (482). The increased excitatory outflow from the VH to the IL might be expected to increase, rather than decrease, the inhibition of fear to an extinguished CS. However, it has recently been shown that chemogenetic inhibition of VH neurons projecting to IL reduces ABA renewal, while chemogenetic excitation of these neurons causes increased freezing to an extinguished CS in the extinction context (427). This suggests that VH projections inhibit IL principal neurons, thereby decreasing IL-mediated inhibition of BLA to increase freezing and promote renewal. Interestingly, Marek and colleagues (427) found that VH neurons exert a strong inhibitory influence on IL principal cells by driving parvalbumin (PV)-positive interneurons. Consistent with this, pharmacogenetic activation of IL-projecting VH neurons recruits PV interneurons and increases conditioned freezing to an extinguished CS (427). An inhibitory influence of the VH on the PL has also been implicated in fear expression (488), and chemogenetic excitation of this pathway also impairs the renewal of conditioned freezing after extinction (489). Collectively, this work reveals that hippocampal projections to the mPFC and amygdala are critical for the contextual control of extinction, and oscillatory synchrony in this network is associated with extinction retrieval (490, 491). Notably, these pathways have also been implicated in renewal of appetitive (drug-seeking) responses after extinction (492, 493) (see sect. 4.1.2). Together, these data suggest that there are not only circuits that suppress conditional fear after extinction but also those that drive fear renewal by facilitating the expression of fear that has been extinguished, in part by suppressing inhibitory circuits brought online by extinction training.

3.3.3. Prefrontal projections to midline thalamus mediate retrieval suppression

Considerable work suggests that mPFC projections to the amygdala influence the expression of fear, including suppressing conditioned freezing after extinction. Another route by which the mPFC might influence the expression of fear and extinction memories is through projections to the hippocampus. Recent work suggests that representations of both fear and extinction memories are encoded in hippocampal ensembles (494, 495), and the mPFC has been postulated to influence hippocampal memory retrieval processes (496). The mPFC, however, has no direct projections to the hippocampus but can reach it via disynaptic projections through the entorhinal cortex or the nucleus reuniens (RE), a ventral midline thalamic nucleus. The RE has attracted considerable attention, because it plays an important role in coordinating oscillatory activity in the hippocampus and mPFC and is critical for many forms of learning and memory that require these brain areas (481, 497, 498). Inactivation of the RE impairs spatial working memory (499, 500) and the expression of contextual fear memories (501), for example. This raises the possibility that the RE serves as a relay by which the mPFC influences memory retrieval operations by the hippocampus. Interestingly, work in humans suggests that mPFC might interact with the hippocampus to suppress context-inappropriate or unwanted memories (496, 502), a process termed retrieval suppression (503). After extinction, animals and humans might thus suppress not only the memory of the excitatory CS-US association encoded by the amygdala (504) but also the episodic memory of the conditioning experience encoded in the hippocampus (494) that might also lead to fear expression.

Recently, Ramanathan and colleagues (505) explored this possibility by examining the consequences of pharmacologically inactivating the RE during extinction training or retrieval after Pavlovian fear conditioning in rats. Muscimol infusions into the RE increased conditional freezing to an extinguished auditory CS and impaired both the acquisition and expression of extinction; extinction retrieval impairments were not due to state-dependent generalization decrements (505). Importantly, RE inactivation did not affect freezing to a nonextinguished CS or baseline freezing behavior (501). To explore whether the role for RE in extinction was directed by mPFC inputs, Ramanathan and colleagues used an intersectional chemogenetic approach to selectively silence mPFC neurons projecting to the RE. These experiments showed that inactivating RE-projecting neurons in the mPFC also produced deficits in both the acquisition and expression of fear extinction (505). Interestingly, none of these manipulations influenced the renewal of conditioned freezing to an extinguished CS in the conditioning context. This work suggests that mPFC projections to the RE play a critical role in extinction learning and memory, specifically by inhibiting the expression of conditioned freezing to an extinguished CS in the extinction context. Further work is required to understand whether the RE inhibits the expression of fear via projections to the hippocampus or amygdala (504), but the bulk of evidence suggests that the RE-hippocampus projection will be involved.

3.3.4. Sex, development, and stress effects on extinction

The neural and behavioral mechanisms underlying fear extinction show considerable variation across development and are also influenced by sex. One of the most striking developmental differences in extinction, at least in rodents, is the failure of young animals [postnatal day (PND) 17] to show relapse phenomena, including renewal, reinstatement and spontaneous recovery (507, 508). These phenomena begin to emerge almost 1 wk later in development, at PND24, resulting in the adult extinction phenotype. An important implication of these observations is that the extinction of fear in juvenile rats seems to involve the erasure of fear memory, rather than the formation of a competing inhibitory memory as occurs in adults (509). Interestingly, the erasure phenotype is only observed in male rats, insofar as juvenile PND17 females show renewal, reinstatement, and spontaneous recovery (510).

The malleability of fear memory in juvenile animals is due to the developmental trajectory of two structures critical for extinction learning, the mPFC and amygdala. Specifically, it has been shown that mPFC inactivation does not affect extinction in PND17 rats, whereas it impairs extinction learning in PND24 rats (511). Because the mPFC is critical for the acquisition of inhibitory extinction memories, this implies that juvenile rats do not have the neural circuit architecture of adult animals that allows for parallel fear and extinction memories. In addition, developmental differences in hippocampal function influence contextual learning processes (512, 513), and this would be expected to affect the context dependency of extinction. Moreover, in juvenile rats, inhibitory neurons in the amygdala lack perineuronal nets, which limit synaptic plasticity in adult animals. As a consequence, it is possible that the cellular organization of the juvenile amygdala promotes synaptic plasticity mechanisms that lead to depotentiation of synapses encoding fear memory. Consistent with this, enzymatic digestion of perineuronal nets in the amygdala of adult animals yields a juvenile phenotype in which extinction fails to show renewal or spontaneous recovery (514). Clearly, the developmental trajectory of the mPFC, hippocampus, and amygdala has a powerful influence on the nature of extinction learning and memory.

Another key factor influencing the neural and behavioral mechanisms of extinction is sex. There are prominent sex differences in both fear conditioning (512) and extinction (515), although the nature of these sex differences depends on several factors including the use of signaled or unsignaled footshock and the hormonal status of the animals (516). For example, sex differences are observed in conditioned freezing to contextual stimuli but not to discrete CSs; male rats show greater levels of contextual freezing than female rats (512). In addition, adult female rats in proestrus exhibit both more rapid extinction and greater extinction retrieval than either female rats in diestrus or male rats (516, 517); interestingly, this pattern is reversed in adolescent rats (518). Menstrual cycle in human females also influences extinction learning and recall (519). The cycle-dependent variations in extinction implies that it is modulated by gonadal steroids. Consistent with this, extinction in adult female rats is impaired by ovariectomy (515) and restored by exogenous estrogen administration (515, 521). In addition to sex differences in extinction encoding and retrieval, there are also sex differences in renewal (522). Not surprisingly, sex difference in extinction learning and memory have been linked to the differential recruitment of the prefrontal cortex, hippocampus, and amygdala in male and female rats (516). Moreover, there is considerable evidence indicating that gonadal steroids and their metabolites (523) can have direct influences on neural circuits regulating learning, memory, and emotion (524, 525).

Extinction learning and memory are not only influenced by genetic and ontological factors but also an array of environmental factors, some of which have already been discussed. For example, stress can have a potent influence both on extinction learning, as well as relapse (526). Stress effects on extinction have been demonstrated with both acute and chronic stressors. For example, when extinction trials occur within minutes to hours of conditioning, there is a marked attenuation of extinction retrieval later (447). This so-called “immediate extinction deficit” can be attenuated by systemic or intra-amygdala infusions of propranolol, a β-adrenoceptor antagonist (445, 527), suggesting that it might result from the stress of the immediately preceding fear conditioning experience (but see Ref. 528). Recent evidence suggests that stress activates the locus coeruleus, which in turn drives BLA neurons regulating IL activity (529). The mPFC is a highly stress-susceptible brain region, and considerable work indicates that stress effects on extinction are mediated by changes in IL structure and function (530, 531). Stress is also a major driver of relapse of extinguished CRs, and these effects occur in both appetitive and aversive tasks and involve hippocampal-prefrontal circuits (532, 533). However, as noted in sect. 2.1.2, stress only yields relapse of instrumental responding if conditioning was conducted under the influence of the stressor. In other words, stress is most likely to cause relapse of extinguished responding when the stressor recapitulates an interoceptive state similar to that present during conditioning.

4. INSTRUMENTAL EXTINCTION: NEUROBIOLOGICAL MECHANISMS AND CIRCUITS

Our discussion now turns to instrumental extinction, where a behavior (rather than a stimulus) is first associated with a significant outcome and then the outcome is withdrawn. During instrumental extinction, the high level of responding that is no longer reinforced initiates context-dependent learning processes inhibiting the performance of that specific response in that context while leaving much of the specific knowledge that was encoded during instrumental training intact. This learning is profitably viewed as a direct inhibitory association between the context where extinction occurs and the response that previously earned the outcome (sect. 2.4.2). The inhibitory association is symmetrical to the direct evocation of responding by contexts where responding was previously reinforced and which can contribute to ABA renewal.

4.1. Neural Circuits for Extinction and Renewal

The available evidence is that these symmetrical, response-dependent learning processes depend, at least in part, on PL, IL, hippocampus, and cortical-ventral striatopallidal-thalamic circuits involving the nucleus accumbens shell (AcbSh) and nucleus accumbens core (AcbC) (FIGURE 7). The classic view of these circuits, derived from studies of Pavlovian conditioning, is that IL inhibits or suppresses responding and so contributes to the learning and expression of extinction whereas PL initiates responding during renewal and different forms of response restoration (453, 534). However, although there is strong evidence for dichotomies in PL/IL control over Pavlovian extinction (see sect. 3.3), there is less so in the case of instrumental extinction. This difference between Pavlovian and instrumental extinction may be broadly consistent with the fact that there are important differences in the learning processes controlling these forms of extinction learning (sect. 2.4). Moreover, during instrumental learning, the brain regions and circuits involved are likely subsumed under the broader roles of these regions in cognitive and behavioral flexibility (535, 536), response strategy switching (537), inhibitory action control (538, 539), as well as cognitive and behavioral control of actions versus habits (540–542).

FIGURE 7.

Neural circuits mediating extinction, extinction retrieval, and renewal in instrumental learning. Confirmed causal connectivity is indicated by unbroken lines, unconfirmed connectivity by broken lines. The symmetrical inhibitory (left) and excitatory (right) influences of context depend, at least in part, on neuronal ensembles distributed across the prelimbic (PL) and infralimbic (IL) prefrontal cortex. During extinction learning and extinction retrieval, these ensembles converge with glutamatergic inputs from basolateral amygdala (BLA), ventral hippocampus (HPC), and paraventricular thalamus (PVT) to interface with D1 receptor expressing spiny projection neurons in the nucleus accumbens shell (AcbSh) that preferentially provide inhibitory GABAergic synaptic input to lateral hypothalamic (LH) GABA neurons. During ABA renewal, neuronal ensembles distributed across PL and IL converge with glutamatergic inputs from ventral HPC and PVT to interface with D1 receptor expressing spiny projection neurons in the AcbSh that preferentially project to ventral tegmental area (VTA) [directly and indirectly via ventral pallidum (VP)] and LH. VP also projects to subthalamic nucleus (STN) and VTA. Lateral VTA dopamine neurons mediate renewal via an ascending dopaminergic signal to the striatum and possibly cortex. LH is the only known site of convergence between the extinction and renewal circuits, with the same LH GABAergic neurons receiving monosynaptic inhibitory GABAergic input from both extinction (AcbSh) and renewal (VP) circuits. OFC, orbitofrontal cortex; AcbC, nucleus accumbens core.

The following sections review the neurobiological mechanisms and circuits for extinction, renewal, and reinstatement of instrumental responding. Pharmacological studies have identified numerous neurotransmitters and neuropeptides as important for the extinction of instrumental responding (543), and these act largely within the circuits reviewed below. However, as noted in sect. 1, most of our knowledge about the mechanisms for instrumental extinction comes from studies of the extinction and relapse of instrumental responding for drugs of abuse, and these studies have typically focused on the recovery effects as models of relapse. More has thus been learned about the mechanisms for response restoration after extinction than the mechanisms of extinction learning itself, so this remains an important area for future work. The focus on relapse is understandable given the fundamental problem it poses for the treatment of addictive behavior. Indeed, as noted previously, renewal is remarkably robust across a variety of different drug reinforcers, including alcohol (65, 66, 544), cocaine (545, 546), heroin (547), nicotine (548), and methamphetamine (549).

The focus on drugs of abuse and relapse processes has created a certain separateness to this literature that introduces several caveats on interpretation, however. First of all, drug reinforcers can induce neural circuit adaptations that are often specific for different pharmacological classes of reinforcers. Therefore, the extent to which inconsistencies in the literature are due to drug-specific neuroadaptations remains uncertain. Second, the terminology used in this literature often differs from that used in the behavioral learning literature. To be precise, as well as consistent with the latter, we use the term “renewal” to refer to restoration of extinguished responding that occurs with a change in context after extinction training. We accept the term “reinstatement” to refer to the restoration of extinguished responding provoked by the noncontingent presentations of the reinforcer itself (as above, which we call “priming” here) but also contingent or noncontingent presentations of reinforcer-associated stimuli, or noncontingent presentations of stressors. It is worth noting that most, if not all, studies of reinstatement by presentations of discrete, drug-associated cues (“cue-induced reinstatement”) involve the response-dependent presentation of these cues during the test (i.e., R→S), and these cues typically do not undergo any extinction training. The response-dependent contingency gives a unique role to conditioned reinforcement and is thus unlike the noncontingent presentations of the reinforcer (priming) or stressors. As we show below (sect. 4.3), recognizing this role for conditioned reinforcement, a distinct learning process, goes some way to resolving inconsistencies in the literature on the brain mechanisms for instrumental extinction and response restoration. Third, whereas behavioral analyses of extinction often use multiple responses and reinforcers to isolate learning mechanisms, behavioral analyses in studies involving brain manipulations are often simpler (e.g., single response, single outcome). This can make direct comparison between these two literatures somewhat difficult. Therefore, despite the significant knowledge reviewed in the following sections, more work is needed to link the neuroscience literature with the existing behavioral literature. Finally, most of the studies reviewed in this section have used an ABA design to study renewal and AAA designs to study extinction. Therefore, the extent to which these findings generalize to other forms of renewal (AAB, ABC) remains unknown.

4.1.1. Prefrontal cortex

One of the most influential frameworks for the brain mechanisms for extinction of instrumental responding states that PL is implicated in restoration of responding after extinction and IL in the learning and expression of extinction (453, 550). According to this framework, a PL→AcbC→ventral pallidum (VP) pathway is obligatory for reinstatement whereas an IL→AcbSh pathway is obligatory for extinction. This framework has proved useful in guiding work in the field. Moreover, the opposing roles for PL and IL are consistent with broader roles for these regions in the extinction and reinstatement of other forms of learning including Pavlovian learning (see sect. 3.3) and models of prefrontal function (531).

In the case of instrumental learning, much of the evidence for this dichotomy rests on the contrasting effects of manipulations of PL and IL on contextual control over responding after extinction of drug-seeking. For example, McFarland and Kalivas (551) demonstrated that inactivation of PL via the GABA agonists baclofen/muscimol (B/M) or infusions of the dopamine receptor antagonist fluphenazine prevented reinstatement of cocaine-seeking provoked by noncontingent exposure to cocaine. This role for PL is robust and has been observed for cue-induced reinstatement of cocaine-seeking (552), stress-induced reinstatement of cocaine (553), or high-fat food (554) seeking, cue-induced reinstatement of methamphetamine (555) or heroin (556) seeking, among others. Importantly, it extends to ABA renewal of cocaine (546) and alcohol (558) seeking. It has also been observed for ABA renewal based on a food reinforcer (557) (see below).

If IL is contrastingly critical for extinction learning, then it follows that manipulations of IL should affect extinction learning. The first evidence for this was provided by Peters and colleagues (532) who reported that reversible inactivation of IL prevented expression of extinction of cocaine-seeking, a role that depended on interactions with AcbSh (see below). In an interesting series of experiments, Lalumiere and colleagues provided response-dependent optogenetic inhibition of IL neurons during extinction of instrumental cocaine self-administration. They found that inhibition of these neurons during a brief time window (up to 20 s) following emission of nonreinforced lever presses could modestly impair extinction learning, as shown by both increased responding during extinction training and by increased responding during a subsequent test for cue-induced reinstatement of cocaine seeking. However, reinstatement by cocaine-priming was unaffected (559). This finding that IL activity in the time immediately following a nonreinforced lever press contributes to extinction is important and is consistent with the behavioral demonstration that direct nonreinforcement of the response is necessary for instrumental extinction to be learned (91). However, the relatively modest impact of this manipulation on both extinction learning and subsequent response restoration suggests that other regions and mechanisms are also important. Nonetheless, this role for IL in learning the extinction of cocaine seeking depends on glutamatergic and beta-adrenergic signaling (560) and may be linked to D2 dopamine receptors (561). Extinction learning can be prevented by NMDA receptor antagonism in IL (562) and, conversely, enhanced by positive allosteric modulation of IL AMPA receptor function (563). Importantly, these pharmacological enhancements may deepen extinction learning, making it more resistant to cue-induced reinstatement (67, 255).

If IL is critical to suppression of instrumental responding after extinction, then augmenting activity in IL might prevent renewal and other forms of response recovery. There is a relatively small literature examining this possibility, but IL chemogenetic excitation with the hM3Dq DREADD (564) or IL optogenetic excitation with stable-step function opsins (565) each reduce cue-induced reinstatement of cocaine seeking. Moreover, selective optogenetic excitation of IL neurons that project to the nucleus accumbens reduces return to cocaine self-administration after 1 or 15 days of forced abstinence whereby animals remain in their home cage and do not receive any extinction training (566). Consistent with the notion that an IL-based extinction circuit actively competes with a PL-based relapse circuit to suppress neural circuits promoting renewal or reinstatement, these pro-extinction effects of IL manipulations reduce the reinstating effects of the reinforcer (534, 564) as well as the reinstating effects of brain manipulations [e.g., dopamine and AMPA manipulations in CNQX into AcbSh (563)].

A role for IL in inhibition of instrumental responding after extinction is not limited to studies of cocaine seeking. There is some evidence based on extinction of responding for a food reinforcer (558). Eddy and colleagues (558) used an explicit ABB v ABA experimental design to study the role of PL and IL in expression of extinction and renewal. After lever press training on a VI schedule for a food reinforcer in Context A, rats were extinguished in Context B before testing in Context B (ABB) and Context A (ABA). Rats received microinjections of the GABA agonists B/M into PL or IL before these tests. Consistent with a role for PL in renewal, reversible inactivation of PL reduced responding on ABA test but had no effect on ABB test. In contrast, reversible inactivation of IL increased responding in the extinction context (ABB) but also decreased responding in the training context (ABA). Whereas the increase in responding in the extinction context is consistent with a role for IL in inhibiting instrumental behavior after extinction, the decreased ABA renewal is not. Furthermore, others have not reported the same effect of PL and IL manipulation on renewal (567) and expression of extinction (568, 569). For example, reversible inactivation of PL does not prevent ABA renewal of heroin seeking (567) despite PL contributing to forms of reinstatement (e.g., cue, stress) of heroin seeking (556, 570). Mendoza and colleagues (568) reported no effect of IL B/M infusions on the expression of instrumental extinction in rats trained to lever press for liquid sucrose, and Warren and colleagues (569) could detect no effect in rats trained to respond for food pellets. Although Warren et al. and Mendoza et al. (568) used a single context design (AAA), and Eddy et al. (558) used an ABA design, this design variation may not be critical (557). Still others have reported opposing or no effects of IL manipulations on extinction of drug seeking. For example, IL inactivation attenuates cue- and drug-induced reinstatement of heroin-seeking (556), cue-induced reinstatement of methamphetamine-seeking (555), and ABA renewal of heroin-seeking (567) and has no effect on the expression of extinction of alcohol seeking (557).

The reasons for these discrepancies are unclear. It is worth noting that many variables differ across these experiments. These include the reinforcement schedule used to train responding (ratio vs. interval schedule), the amount and duration of extinction training, the presence versus absence of discrete cues during self-administration and/or extinction, the explicit use of context switch between training and extinction, etc. Reinforcer-specific neuroadaptations may bias circuits for behavioral control during extinction of instrumental responding. Strong evidence for this possibility comes from studies of cocaine self-administration. Cocaine self-administration, even for as little as 1 wk, can cause profound alterations in glutamate neurotransmission in prefrontal cortex, amygdala, nucleus accumbens and elsewhere (571), and these drug-specific neuroadaptations could well determine the specific circuit basis of extinction. For example, cocaine self-administration generates AMPA-receptor silent glutamatergic synapses at both IL (572) and amygdala (573) synapses after the termination of drug self-administration. These synapses are unsilenced across the course of abstinence, which contributes to an incubation of responding. Given this extensive synaptic remodeling by drugs of abuse, it is perhaps not that surprising that the specific mechanisms for behavioral control by extinction training can differ across different reinforcers. Yet, however important these procedural variations may be, they seem inadequate to explain the diversity of findings in the literature. In extinction, the instrumental outcome (food, cocaine, heroin, alcohol) is always absent and the same behavior (e.g., lever pressing) is being extinguished.

Findings from neuronal ensemble manipulation studies by Hope and colleagues (567, 569, 574–576) provide insights into these discrepancies and go some way to resolving much of the inconsistencies in the literature. These studies take advantage of the Fos-LacZ transgenic rat that expresses beta-galactosidase under the control of the c-Fos promoter. When the prodrug Daun02 is applied to Fos-LacZ rats, beta-galactosidase catalyzes Daun02 into the cytotoxic daunorubicin and triggers apoptosis only in activated (i.e., Fos expressing) neurons (577). For example, Warren and colleagues (569) used an AAA design to assess expression of the activity marker c-Fos during retrieval of original training versus extinction memory for food pellet responding. Retrieval of either training or extinction were associated with Fos expression in both PL and IL and Fos was expressed in similar PFC cell types under both conditions. Next, they used the Dauno2 inactivation method in Fos-LacZ transgenic rats to selectively inactivate neuronal ensembles in vmPFC (incorporating mostly IL) that were activated by retrieval of lever press training (i.e., after 0 days extinction) versus those activated by retrieval of recent extinction (i.e., after 2 days of extinction training). They showed that selectively disrupting the vmPFC neuronal ensemble recruited during recall of training decreased later lever pressing whereas selectively disrupting the vmPFC neuronal ensemble recruited during recall of extinction increased later lever pressing. Therefore, nonreinforced instrumental responding could be reduced or increased depending on whether the ensembles recruited during retrieval of the training versus extinction memories were inactivated. Moreover, global reversible IL inactivation via infusions of GABA agonists had no significant effect. Similar findings have been reported for extinction and ABA renewal for heroin seeking (567) and discriminative control over instrumental responding for saccharin + glucose (575), among others.

These findings show that separate neuronal ensembles encoding instrumental extinction versus renewal and other forms of relapse can be colocated in the same prefrontal regions (e.g., IL). They highlight the need for a better understanding of the organization of these ensembles, and in particular their connectivity with striatal circuits that control operant behavior. PFC neurons project extensively throughout the striatum, with clear differences in patterns of innervation not mapping readily onto anatomical boundaries there (578–580). Moreover, within an individual corticostriatal pathway, such as the IL→AcbSh pathway, there are important differences. For example, different IL neurons target distinct AcbSh compartments that, in turn, target distinct hypothalamic, pallidal, and thalamic regions (581) with diverse functions. Ensembles and their circuits, rather than regions, are likely the relevant unit of analysis.

4.1.2. Hippocampus

The dorsal and ventral hippocampus serve important roles in renewal of instrumental responding. Hippocampus, especially CA3, is recruited as shown by c-Fos expression during ABA renewal of cocaine seeking (582). Fuchs and colleagues (546) reported that reversible inactivation of dorsal hippocampus (DH) via infusions of tetrodotoxin prevented ABA renewal of cocaine seeking. Luo and colleagues (583) likewise reported that dorsal hippocampal inactivation, targeting the CA3 subregion, prevented ABA renewal of cocaine seeking. The actions of dopamine at D1 receptors (584) and glutamate at mGluR₁ (585) as well as NR2B expressing NMDA receptors (586) have all been shown to be important for these DH contributions to ABA renewal. The role of DH in ABA renewal of extinguished responding for other reinforcers remains to be determined. The DH contribution depends on basolateral amygdala (BLA) (587), although the actual nature of this interaction remains unknown. It also depends on a circuit reaching the ventral tegmental area (VTA) via lateral septum (LS). Luo and colleagues used trans-synaptic tracing to identify a CA3→LS→VTA circuit and then showed that reversible inactivation of CA3, or disconnection of the LS→VTA pathway, prevented ABA renewal of cocaine seeking.

In ventral hippocampus, both ventral CA1 (vCA1) and ventral subiculum (vSUB) serve important roles in ABA renewal. ABA renewal of heroin seeking is associated with vCA1→IL but not vCA1→PL glutamatergic projections and is associated with synaptosomal GluA2 expression in the IL, impaired basal synaptic transmission, and facilitation of long-term depression (LTD) in the vCA1→IL pathway (493). Theta-burst stimulation of vSUB induces recovery of extinguished cocaine seeking (588), and reversible vSUB inactivation prevents ABA renewal of cocaine (589) or heroin (492) seeking. This role extends to ABA renewal following the suppression of alcohol seeking by punishment (590) (see below). As noted above (sect. 2.2.2) and below (sect. 4.2), punishment shares similar contextual control to instrumental extinction (128), and these findings suggest that vSUB is a common locus for the contextual control over instrumental responding regardless of how behavioral suppression is achieved. The circuitry for this vSUB contribution to renewal and extinction of instrumental responding is poorly understood but involves projections to AcbSh (see below) and PL (591).

4.1.3. Basolateral amygdala

The BLA has a role in both the extinction and renewal of instrumental responding. Studies of c-Fos expression show that the BLA is consistently recruited during ABA renewal across a variety of reinforcers including alcohol (65, 582), sucrose (592), or cocaine (63). The role of BLA in renewal of alcohol seeking is upstream of dopamine actions at D1 dopamine receptors because systemic administrations of a D1 dopamine receptor antagonist do not prevent recruitment of BLA but do prevent renewal (65), but is dependent on dopamine actions in orbitofrontal cortex in renewal of cocaine seeking (593). ABA renewal of cocaine seeking is prevented by reversible BLA inactivation by baclofen/muscimol (546) and ABA renewal of alcohol seeking by microinjections of a mu opioid receptor antagonist (594). The BLA is also important for both the acquisition (595) and expression (596) of extinction of instrumental responding for natural and drug reinforcers. For example, McLaughlin and Floresco (595) showed that reversible inactivation of the BLA during extinction of instrumental responding based on a food reinforcer impaired extinction learning. Millan and McNally (596) reported a similar finding, but for the expression of extinction of alcohol seeking. They showed that instrumental responding that had been reduced via extinction could be restored by pharmacological disconnection of BLA from AcbSh. These findings implicate BLA in the process that reduces or inhibits instrumental responding during extinction.

The BLA cell type(s), the nature of any learning-related plasticity in these BLA neurons and their relationship to extinction and renewal, as well as the precise circuitry in which BLA sits to control instrumental extinction, all require further study. Pharmacological disconnections show serial interactions between DH and BLA in ABA renewal of cocaine-seeking (587). The same disconnections of BLA from orbitofrontal (597) and dorsomedial prefrontal cortex (587) also reduce ABA renewal of cocaine seeking, but ipsilateral disconnection has the same effect, suggesting complex interactions between BLA and these cortical regions in renewal. Optogenetic inhibition of either the BLA→AcbC or BLA→PL prefrontal cortex pathways reduces cue-induced reinstatement of cocaine seeking (598), but whether this extends to renewal is not known. With respect to BLA involvement in extinction of instrumental responding, projections to the AcbSh are important (596) and this could be linked to a more general role of BLA in suppressing or inhibiting appetitive behavior (599), but the roles of other BLA projections remain to be determined.

The dual role for BLA in renewal and extinction of instrumental responding is linked to compartmentalization of BLA circuits. There is significant anatomical and functional segregation of BLA output circuits (600–602), and these have been shown to influence extinction and reinstatement in Pavlovian fear conditioning preparations (304, 483). Because BLA output pathways are heavily collateralized (603), these effects can be difficult to study, but there is segregation along the rostral-caudal axis of BLA. Rostral and caudal parts of BLA target different ventral striatal compartments (604, 605) and pharmacological manipulations of rostral and caudal BLA have different effects on instrumental behavior (595, 606, 607).

4.1.4. Nucleus accumbens

In vivo electrophysiological recordings during instrumental behavior identify robust activity changes in Acb neurons related to initiation and execution of the instrumental response as well as delivery and consumption of the earned liquid or food reward (608–610). These changes are observed in a minority (typically 30%) of neurons. In general, Acb neurons show phasic excitations in the period immediately before lever press (1 s), and phasic excitations or phasic inhibitions after execution of the response (611–618). These are not secondary to general motor activity or arousal because they are specific to the drug-self administration setting (e.g., they are not observed on a treadmill). Indeed, anticipatory responses (excitations or inhibitions) before initiation of an instrumental response are linked to initiation or termination of distinct behaviors, with different subsets of Acb neurons putatively coding for different behavioral segments (e.g., orienting, posture shift, approach) (610, 615, 616). Post lever-press responses (excitations and inhibitions) have been linked to receipt and ingestion of the reinforcer, a finding supported by studies using Pavlovian tasks (619, 620), but they can also be dissociated from the direct pharmacological impact of the reinforcer (618).

Several general principles about Acb activity during instrumental responding emerge from this literature. First, the same patterns of phasic activity changes are observed regardless of the type of reinforcer earned. The same anticipatory and post responding changes are observed for lever presses yielding sucrose, food pellets, cocaine, heroin, and alcohol (611, 614, 621, 622), although there is some evidence for a fourth class of neuron (that shows phasic excitations both before and after lever press) recruited during instrumental responding for cocaine (611). Second, despite showing the same phasic activity changes, different nonoverlapping Acb ensembles are recruited during seeking and receipt of different rewards (e.g., heroin versus cocaine-seeking, food versus cocaine seeking, water versus ethanol seeking) (611, 612, 621, 622). Third, these changes are not secondary to the reinforcement schedule in effect because they have been observed across different reinforcement schedules (615, 623), suggesting that they are only weakly related to specific instrumental contingencies. Finally, although most studies have recorded from the Acb core (AcbC), there is little difference in responses across AcbC and the Acb shell (AcbSh) subregions.

The origin and nature of the synaptic inputs driving these changes in Acb single unit activity remain poorly understood. Current models emphasize the role of prefrontal cortical inputs and corticostriatal pathways in initiating instrumental responses. However, although individual PFC (prelimbic and infralimbic) and Acb neurons show similar and highly correlated patterns of discharges during instrumental behavior, for the majority of PFC-Acb neuron pairs, Acb activity precedes PFC activity (624). This pattern implicates more complex and indirect interactions between these regions in the initiation of instrumental behaviors.

Studies of dopamine release in the Acb during instrumental tasks using fast scan cyclic voltammetry (FSCV) are in strong agreement with single unit findings, identifying pre lever-press DA transients typically occurring within 10 s before a lever press and also within 2 s after a lever press (625–627). This pattern of dopamine transients has been observed to lever presses that yield cocaine or sucrose and there are similar DA transients between drug (cocaine) versus natural (food, water) rewards (628). Studies with intravenous cocaine identify a third class of DA transients, spontaneous transients, during self-administration that are not linked to specific behavior or stimuli (626, 627). Individual lever presses tend to occur at the peak of the lever-press DA transient (629) and electrically stimulating DA release initiates lever pressing (625). These transients are observed in both AcbC and AcbSh, however larger lever-press transients have been observed in AcbSh versus AcbC (630). These findings, unsurprisingly, implicate DA in both the anticipatory excitatory responses and the postresponse activity changes during instrumental behavior. However, anticipatory excitations survive D1 and D2 receptor antagonism (616) and whereas Acb single units show reinforcer selectivity, DA release during instrumental tasks does not (628).

The impact of extinction on these in vivo activity changes is poorly understood. Moreover, due to constraints in identifying units across multiple sessions, studies have typically focused on changes occurring in a single session of extinction. These methodological limitations have been addressed in more recent studies using optical approaches and reviewed below in sect. 4.1.7. Nonetheless, both Acb single-unit (612, 615) and FSCV studies of DA release (627) during extinction yield similar conclusions. During extinction of cocaine self-administration, anticipatory excitatory Acb responses and anticipatory DA release are maintained intact through a single extinction session, even though lever pressing itself declines. In contrast, Acb responses (phasic excitations or inhibitions after responding) and DA release post lever pressing are reduced across extinction training and can be restored via noncontingent presentation of cocaine or cocaine-associated stimuli (612, 615, 627). Interestingly, these findings with extinction of intravenous cocaine self-administration contrast with extinction of instrumental responding for water, where all activity changes are reduced across the course of a single session of extinction training and are restored via noncontingent presentations of a water-associated cue and water itself (631).

4.1.4.1. Nucleus accumbens shell

Like BLA and PFC, AcbSh shows robust activation during ABA renewal of alcohol (65), sucrose (592) or cocaine (63, 632) seeking. This depends on D1 dopamine receptors (65) and the AcbSh neurons activated by exposure to the renewal context cause renewal because selective lesioning of these activated neurons prevents renewal (632). Moreover, this recruitment is shared with ABA renewal of alcohol seeking after punishment-imposed suppression of responding (633). Reversible inactivation of AcbSh via infusions of GABA agonists attenuates context-induced reinstatement of cocaine (634) but not alcohol seeking (64).

Dopamine 1 receptor and dopamine 2 receptor expressing spiny projection neurons (SPNs) are recruited during ABA renewal of methamphetamine or cocaine seeking (549, 632). Systemic administration of D1 or D2 receptor antagonists prevents ABA renewal of cocaine seeking (545) and systemic administrations of a D1 receptor antagonist prevents this renewal of alcohol seeking (65). Microinjection of a D1 antagonist into the AcbSh prevents ABA renewal of heroin and alcohol seeking (635, 636). The neuropharmacology of AcbSh in ABA renewal is complex and depends on glutamate acting at mGluR_2/3 (547, 637) and AMPA receptors (585), endogenous opioids acting at the mu-opioid receptor (MOR) (638) as well as the actions of cocaine and amphetamine-regulated transcript (CART) (639) among others.

Again, like the BLA and PFC, a complementary set of findings implicates AcbSh in extinction of instrumental responding. Extinction training has potent effects on key elements of dopaminergic and glutamatergic neurotransmission in the AcbSh. Of note, extinction of cocaine self-administration reverses self-administration-induced deficits in tyrosine hydroxylase (Th) and NR1 NMDA receptor subunit levels and also increases levels of the GluR1 and GluR2/3 AMPA receptor subunits (640–642). In individual subjects, the magnitude of response suppression by extinction is positively correlated with the extent of GluR1 and GluR2/3 upregulation. Extinction of instrumental behavior was necessary for the changes in Th, GluR1, and NR1 levels; however, simple passive exposure to the self-administration context absent the opportunity to lever press was sufficient to upregulate GluR2/3 expression. Moreover, Self and colleagues (643) showed that the upregulation of GluR1 and GluR2 AMPA receptor subunits was essential to instrumental extinction learning because viral-mediated overexpression of these subunits reduced the amount of extinction training needed to achieve an extinction criterion and also reduced the later propensity to relapse during stress-induced reinstatement. Other studies have confirmed and extended this role. For example, reversible inactivation of AcbSh prevents the expression of extinction (i.e., causes recovery) of cocaine (534, 634) and alcohol (644, 645) seeking. This role for AcbSh in the expression of instrumental extinction depends on glutamate actions at AMPA receptors and projections from the BLA (596). These findings have encouraged attempts to therapeutically target Acb. In rodents, deep brain stimulation of AcbSh can promote extinction of instrumental responding for cocaine and also attenuate cocaine priming or cue-induced reinstatement (646, 647), although the underlying mechanisms are poorly understood (648). Similar attempts have been made in humans, with some case reports that deep brain electrical stimulation of Acb can have positive therapeutic effects against heroin and alcohol use in humans (648, 650). However, this evidence rests on a handful of cases and should be treated with caution.

The findings reviewed above strongly implicate glutamatergic inputs to AcbSh in both extinction and renewal of instrumental responding. AcbSh receives glutamatergic inputs from PFC, vSUB, thalamus, and BLA, among other regions. The existing evidence implicates each of these inputs in either extinction, ABA renewal, or both of instrumental responding. Pharmacological disconnection of the IL→AcbSh pathway prevents expression of the extinction of cocaine seeking (534) whereas selective excitation of this projection using a retrograde chemogenetic approach suppresses cue-induced reinstatement of cocaine seeking (564), strongly supporting a role for this pathway in the inhibition of instrumental responding. However, the IL→AcbSh pathway is also recruited during ABA renewal of heroin seeking and reversible vmPFC inactivation, selective lesioning of vmPFC-activated corticostriatal neurons, or pharmacological disconnection of vmPFC→AcbSh reduce this ABA renewal (567, 651).

vSUB neurons projecting to the AcbSh are recruited during ABA renewal of heroin (590) and alcohol seeking (590). Pharmacological disconnection of the vSUB→AcbSh pathway prevents renewal of heroin seeking (591) and chemogenetic disconnection of this pathway also prevents ABA renewal after the punishment of alcohol seeking (591).

Paraventricular thalamus (PVT) provides extensive glutamatergic inputs to AcbSh (579, 652, 653), and these interact anatomically and functionally with both dopamine terminals (653) as well D1 and D2 SPNs (655, 656). PVT contributes to various forms of relapse of instrumental responding (for review see Ref. 656), including cue-induced reinstatement of alcohol (657–659) and cocaine seeking (661). Importantly, PVT lesions abolish ABA renewal of alcohol seeking (662) and the PVT→AcbSh pathway is strongly recruited during this renewal (662). However, as with the other glutamatergic inputs to AcbSh, there are also mixed findings. For example, reversible inactivation of PVT can potentiate cue-induced reinstatement of instrumental responding for food (663) and chemogenetic excitation of PVT can reduce instrumental responding for heroin (664).

Inputs to AcbSh from caudal BLA (cBLA) are important for expression of extinction of drug seeking and preventing relapse. For example, the cBLA→AcbSh pathway is recruited during expression of extinction of alcohol seeking (662), and pharmacological disconnection of the cBLA→AcbSh pathway prevents the expression of extinction and causes a restoration of extinguished alcohol seeking (596). The cBLA→AcbSh pathway may exert a more general role in regulating the impact of drug-associated stimuli on alcohol seeking (599).

Many of the inconsistencies regarding the role of AcbSh and its glutamatergic inputs in extinction and renewal of instrumental responding may be resolved by recognizing the significant compartmentalization and segregation of both AcbSh afferents and efferents (665–667). Corticostriatal and amygdalostriatal terminals overlap in AcbSh, and these inputs are segregated from both vSUB and PVT inputs (665–668). Within AcbSh, there are differences between the dorsal (AcbShDm) and ventral (AcbShV) regions of the medial AcbSh, which have segregated afferents and efferents (581, 667). The anatomical segregation is supported by functional segregation of dorsal from ventral medial AcbSh in extinction versus renewal of instrumental responding. For example, the dual role for AcbSh in extinction versus ABA renewal can be linked, respectively, to segregated channels in ventral (AcbShV) versus dorsal medial AcbSh (AcbShDm) (670), and studies using deep brain stimulation have confirmed this opposing role (671). These findings are supported by other reports of functional differences between AcbShV and AcbShDm in motivation and reward (672, 673).

Recent studies have begun to address these issues (674). AcbSh has significant projections to ventral tegmental area (VTA), lateral hypothalamus (LH), and ventral pallidum (VP), among other regions. The AcbSh projections to these regions emerge predominantly from D1 SPNs (620, 674), although they differ in the degree of segregation. AcbSh D1 SPN projections to VTA collateralize extensively to the ventral pallidum (676) whereas those projecting to the LH are largely separate from VTA/VP projections (675). These separate D1 SPN output pathways serve complementary roles in extinction and renewal. The AcbSh D1→VTA pathway is necessary for ABA renewal of alcohol seeking because optogenetic inhibition of this pathway prevents it renewal (675). Interestingly, inhibition of this pathway has no effect on the acquisition or maintenance of nonextinguished instrumental responding, suggesting that it may depend on extinction-induced plasticity in this circuit. Whether this role extends to renewal of instrumental behaviors supported by other reinforcers is unknown, but a general role in renewal seems likely given the general role for VTA in renewal (see below). In contrast, the AcbSh D1→LH projection is necessary for expression of extinction of instrumental responding. Optogenetic inhibition of AcbSh terminals in the LH prevents the expression of extinction (i.e., causes recovery) of alcohol seeking (674) provided alcohol seeking has been acquired in the same context. Conversely, optogenetic excitation of this pathway can mimic the effects of extinction to prevent response recovery when it is otherwise likely (675). However, this is only observed after extinction training, again implying the possible existence of extinction-related plasticity in this pathway.

4.1.4.2. Nucleus accumbens core

The AcbC has a well-documented role in a variety of forms of reinstatement (cue, priming, stress) of extinguished instrumental responding for drug rewards (550, 677, 678). This role is due to glutamatergic inputs from the PL, because optogenetic inhibition or pharmacological disconnection of the PL→AcbC pathway prevents these forms of reinstatement (679, 680). However, the role of AcbC and its PL inputs in the renewal effect remains unclear. Studies of neural activation have detected no significant increase in AcbC c-Fos expression during ABA renewal of alcohol or cocaine seeking (63, 65). Neither AcbC D1 dopamine receptor antagonism nor mGluR_2/3 agonism affects ABA renewal of heroin seeking (547, 635), but AcbC D1 antagonism does reduce cue-induced reinstatement of heroin seeking. In contrast, Fuchs et al. (634) showed that reversible inactivation of the AcbC via GABA agonist infusions reduced ABA renewal of cocaine seeking. Likewise, Chaudhri and colleagues (636) reported that D1 receptor antagonism of AcbC reduced ABA renewal of ethanol seeking. Finally, Stankeviciute and colleagues (681) reported that ABA renewal of cocaine seeking was associated with a transient increase in SPN spine head diameters that could permit rapid transient synaptic potentiation. These contrasting findings are not simply due to differences in the drug reinforcer used because Cruz and colleagues could find no evidence for a role of AcbC in ABA renewal of cocaine seeking using the Daun02 inactivation method (632). However, the contrasting findings may be due to the specific nature of the testing procedures in these experiments, in particular the presence of response-contingent stimuli (see sect. 4.3).

AcbC undergoes a variety of changes linked to extinction of instrumental responding for drug rewards, with the strongest evidence derived from studies of glutamatergic plasticity following extinction of cocaine-seeking. Extinction of cocaine-seeking promotes a general loss of LTP and LTD at PFC inputs to AcbC (682, 683) that are linked to upregulation of proteins (e.g., Homer 1 b/c) controlling the expression, clustering, and internalization of glutamate receptors, notably mGluR5. These extinction-related changes could act to prevent reinstatement because viral-mediated overexpression of Homer1b/c in AcbC of rats trained to self-administer cocaine mimicked the effects of behavioral extinction training and reduced LTD at PFC → AcbC synapses as well as cue-induced reinstatement (682). Interestingly, these effects were specific to extinction of cocaine, not food reinforcement. In addition, there are transient changes in AcbC glutamatergic neurotransmission linked to successful expression of extinction of cocaine seeking. For example, in mice tested for expression of extinction of cocaine seeking, there is transient synaptic potentiation (i.e., elevation of AMPA:NMDA ratios) selectively in AcbC D2 SPNs (684). In contrast, cue-induced reinstatement of extinguished cocaine-seeking is associated with transient synaptic potentiation selectively in AcbC D2 SPNs (684).

These findings suggest a “binary engram” in AcbC D1 versus D2 SPNs controlling expression of extinction versus restoration of cocaine seeking. This is in contrast to the AcbSh, where contextual control over extinction versus renewal is linked to D1 SPNs located in different output circuits (675). Again, it will be of interest to examine the generality of these findings.

4.1.5. Ventral pallidum

VP serves a key role in renewal and reinstatement of instrumental responding, including reinstatement of instrumental responding based on cocaine (551, 685–687), heroin (556) and alcohol (638). For example, microinjections of mu-opioid receptor antagonists (686), pharmacological inactivation (551, 685), or chemogenetic silencing (688) of VP prevent drug priming, cue-induced, or cue + prime reinstatement of cocaine seeking in rats. VP is also important for ABA renewal of alcohol seeking because it is prevented by chemogenetic silencing of VP (689, 690) or VP microinjections of MOR antagonists (638).

VP consists of heterogeneous populations of GABAergic, glutamatergic, and cholinergic releasing neurons. These neurons also express various peptides and neurotransmitters including calretinin, enkephalin, dynorphin calbindin, parvalbumin, neuropeptide Y, or somatostatin (691) that can be dynamically regulated by exposure to drugs of abuse. Cell-type specific manipulations of VP during renewal show that Gad1 and parvalbumin neurons, but not vGlut2 neurons, are recruited during ABA renewal of alcohol-seeking and chemogenetic inhibition of either Gad1 and parvalbumin neurons attenuates renewal (690).

VP receives projections from numerous regions, including cortex, AcbC and AcbSh, PVT, subthalamic nucleus, VTA, and BLA. These projections differentially target the ventromedial (VPvm) and dorsolateral (VPdl) regions of VP (578, 692, 693). A variety of lines of evidence implicate projections from AcbC to VP, which preferentially target VPdl, in cue-induced and priming-induced reinstatement of instrumental responding for drug rewards. For example, pharmacological disconnection (551) or optogenetic silencing (694, 695) of the AcbC→VP pathway prevents both types of reinstatement for cocaine or alcohol seeking. Indeed, pallidal projections from AcbC are more important than nigral projections for this reinstatement (695). The AcbC projection uses both GABA and neurotensin (687) and is regulated by endogenous opioids acting at the MOR (696). This evidence for a role of VP in multiple forms of reinstatement, and the roles for striatopallidal projections in other aspects of instrumental behavior (697, 698), are consistent with the widely-held view that cortical-striatal-pallidal connectivity is a final common pathway for reinstatement of drug seeking (550). However, this AcbC projection is not obligatory for renewal. Although an AcbC→VP pathway is recruited during ABA renewal of instrumental responding for alcohol (699), optogenetic silencing of the AcbC→VP pathway does not reduce renewal (694).

The VP outputs necessary for renewal, and some forms of reinstatement, are increasingly well understood. VPvm projects to VTA and LH (692, 700–702) whereas VPdl projects to subthalamic nucleus (STN) and substantia nigra (692, 701, 702). Both of these output pathways are important for renewal and reinstatement of instrumental responding. A VP→VTA pathway is important because VP neurons projecting to VTA are recruited during renewal of alcohol seeking (690) or cue-induced reinstatement of cocaine seeking (688); chemogenetic disconnection of this pathway prevents both effects. This pathway depends on VP parvalbumin neurons because renewal is reduced by cell-type selective disconnection of PV neurons from VTA (690). A VP→LH pathway is necessary for renewal because VP neurons projecting to LH are recruited during ABA renewal of alcohol seeking (690). Within this pathway, Gad1 neurons projecting to the LH mediate renewal (690). Importantly, these VP Gad1 neurons converge monosynaptically onto the same LH Gad1 neurons that receive monosynaptic input from AcbSh D1 neurons and contribute to extinction of instrumental responding. This provides a critical, and to date the only known, location of cellular convergence between the circuits controlling expression of extinction and those controlling expression of renewal of instrumental responding. This cellular convergence is important. The convergence of distinct striatopallidal circuits for extinction and renewal onto the same LH Gad1 neurons provides strong support for mechanisms that involve competition between extinction and acquisition memories in the control of behavior after extinction. This cellular convergence also provides an obvious target for therapeutic development to prevent relapse.

Finally, a VP→STN pathway is also important because VP neurons projecting to STN are recruited during renewal and chemogenetic disconnection of the VP→STN pathway reduces ABA renewal. This role may be a general one in instrumental responding because STN manipulations reduce cocaine as well as alcohol self-administration behavior (703–705) and reduce drug priming reinstatement of methamphetamine seeking (706).

4.1.6. Hypothalamus

The tuberal (i.e., dorsomedial, perifornical, and lateral) hypothalamus contributes to both extinction and renewal, as well as other forms of response recovery after instrumental extinction. LH neurons are recruited during ABA renewal of sucrose (592), alcohol (66), and cocaine (64) seeking. Reversible inactivation of LH via GABA agonist infusions reduces ABA renewal of alcohol seeking (670). This role is shared with ABA renewal following the punishment of alcohol seeking (632).

AcbSh (670) and VP (690) are important sources of inputs to LH during renewal of alcohol seeking. Some studies suggest that inputs from AcbShV are selectively recruited during renewal whereas those from AcbShDm are selectively recruited for extinction (645, 670). Moreover, this recruitment of an AcbShV→LH pathway during renewal is shared with ABA renewal following punishment of alcohol seeking (631). However, optogenetic manipulation of these inputs has failed to find any evidence for a causal role in renewal (675).

Instead, as noted above, AcbSh inputs to LH are critical for suppressing instrumental responding after extinction (675). These AcbSh inputs provide monosynaptic inhibitory input to LH Gad1 neurons (620, 675) that underpin the role of an AcbSh→LH pathway in inhibiting instrumental responding after extinction. VP inputs to LH, on the other hand, are critical for ABA renewal. As reviewed above, VP neurons projecting to LH are recruited during ABA renewal of alcohol-seeking and chemogenetic inhibition of this projection reduces ABA renewal (690). This is linked to VP^Gad1 neurons because chemogenetic disconnection of VP Gad1 neurons from LH reduces renewal (690). Interestingly, these AcbSh and VP inputs provide monosynaptic GABAergic inputs to the same LH^Gad1 neurons (690). Given the fundamental role for LH^Gad1 neurons in appetitive behaviors more broadly, including approach and consummatory behaviors (707, 708), this is good evidence that convergence of AcbSh→LH and VP→LH pathways onto the same LH^Gad1 neurons controls the competition between contextual control over extinction versus renewal of instrumental responding.

The roles of other hypothalamic inputs in extinction and renewal are poorly understood. Lateral septum (LS) is one input. There is significant interconnectivity between AcbSh, notably its rostral dorsomedial region, and the LS (669). Pharmacological disconnection implicates an LS→LH pathway in expression of Pavlovian cocaine conditioned place preference and the LS→VTA pathway in ABA renewal of cocaine seeking (709). However, the distinction between LS→VTA and LS→LH pathways and whether these are separate or collaterals of common LS neurons is not known. Finally, the roles of other inputs to LH, especially from BNST (710) and BLA (711) in renewal or other forms of instrumental relapse remain unknown.

Second, the LH outputs for renewal remain poorly understood. The VTA is an obvious target. LH provides extensive inputs to VTA (707, 712–714), and retrograde tracing studies confirm that projections are recruited during cue-induced reinstatement of cocaine seeking (715). These inputs are derived from numerous LH cell classes, including orexin (716), GABA (712), and glutamate (712) neurons, that have direct and indirect effects on VTA dopamine neuron activity as well as pronounced effects on appetitive motivation and behavior (707, 710, 717). However, other relevant LH targets include the lateral habenula (718), ventral pallidum, and PVT. LH neurons provide extensive projections to PVT, and PVT is well implicated in the renewal of instrumental responding (657). Importantly, because PVT in turn projects to AcbSh, it functions as a thalamic return in an AcbSh→LH→PVT→AcbSh loop (581). Indeed, expression of extinction of alcohol seeking was associated with activity of IL inputs to hypothalamus and, in turn, a hypothalamus→PVT pathway (719). This projection into PVT was derived from LH orexin/prodynorphin neurons, and the opioid (prodynorphin) content of these neurons (720) was important for expression of extinction (719).

4.1.7. Ventral tegmental area and substantia nigra

The VTA and SN are targets of many of the projections described previously and contain dopamine neurons that are likely critical to understanding extinction, renewal, and some forms of reinstatement of instrumental responding. Multiple inputs to VTA are recruited during the renewal or reinstatement of instrumental responding. AcbSh→VTA inputs are recruited during ABA renewal of alcohol-seeking (675) and cue-induced reinstatement of cocaine seeking (715). These inputs are causal because their optogenetic inhibition prevents renewal. VP→VTA inputs are also important for both renewal of alcohol seeking and cue-induced reinstatement of cocaine seeking because chemogenetic disconnection prevents both effects (688, 689). Finally, an LS→VTA pathway has been implicated in ABA renewal of cocaine seeking because pharmacological disconnection of this pathway prevents renewal (583).

There is a rich and diverse pattern of connectivity between these inputs and different VTA cell types (712–714, 721–726). AcbSh projections to VTA emerge predominantly from D1 expressing SPNs and provide direct inhibitory GABAergic inputs to both VTA dopamine and GABAergic neurons (675, 721, 726). VP projections to VTA emerge from parvalbumin expressing neurons and have complex direct and indirect effects on VTA dopamine and GABAergic neurons (723, 727). This targeting of different types of cells in the VTA yields diverse effects, including direct and indirect effects of AcbSh inputs on VTA dopamine versus GABA neurons achieved via different GABA receptors and effects on VTA GABAergic neurons that vary across the medial to lateral VTA. It is likely that AcbSh inhibition of VTA GABAergic neurons contributes to ABA renewal because renewal of alcohol seeking is prevented by chemogenetic excitation of VTA Gad1 neurons (675). Studies of cue-induced reinstatement of cocaine seeking support a local disinhibitory action for the VP→VTA pathway because dopamine neurons are necessary for reinstatement, but this was not due to direct input to dopamine neurons from VP terminals (688).

Regardless of whether they are recruited directly or indirectly via disinhibition, VTA dopamine neurons are necessary for renewal (728). Chemogenetic inhibition of these neurons prevents ABA renewal of alcohol-seeking. Interestingly, this role is linked specifically to dopamine neurons in the lateral VTA. Fiber photometry studies indicate that lateral but not medial VTA dopamine neurons show calcium transients during renewal (728), and optogenetic inhibition of lateral but not medial VTA dopamine neurons prevents renewal (728). This stands in marked contrast to activity profiles and role of VTA dopamine neurons in reacquisition. During reacquisition, both medial and lateral VTA dopamine neurons show robust calcium transients around instrumental behaviors earning the alcohol reward and optogenetic inhibition of either medial or lateral VTA dopamine neurons impairs instrumental responding during reacquisition (728).

These VTA mechanisms for extinction, renewal, and reinstatement are important to understand because they mediate the key role for ascending dopamine pathways in renewal and different forms of reinstatement. They may also underpin learning of instrumental extinction. Indeed, recent findings using dLight fiber photometry show that acquisition of instrumental responding for alcohol is associated with robust dopamine binding across the ventral striatum (AcbShM, AcbC, and AcbShL) (728). This binding is reduced across extinction training but is restored across the ventral striatum during both renewal and reacquisition (728). The reduction in dopamine binding during extinction stands in marked contrast to self-administration and this reduction may play a critical role in extinction learning (signaling response error or a generalization decrement between training and extinction). However, it is worth noting that a role of ascending dopamine projections in renewal, reinstatement, and extinction also extends beyond the striatum. For example, infusions of a D1 receptor antagonist into the orbitofrontal cortex (OFC) reduce ABA renewal of cocaine seeking (592). Interestingly, pharmacological disconnection of the dopamine neurotransmission in OFC from BLA also reduces this renewal, implying that a VTA→OFC→BLA circuit underpins at least some of the dopamine involvement in renewal (592). In addition, dopamine actions in PL contribute to for cue-induced reinstatement of heroin seeking (570) whereas dopamine actions in IL contribute to extinction of a cocaine-associated cue (561).

4.2. Extinction of Instrumental Aversive Learning

The vast majority of work on the neurobiology of instrumental extinction has focused on extinction, renewal, and reinstatement of instrumental appetitive learning, as just described. Far less attention has been paid to the mechanisms of instrumental aversive learning. Here we focus on findings from two preparations: avoidance learning and punishment.

In active avoidance experiments, rats receive CS–shock pairings, but shock can be avoided by making a specified response (e.g., stepping onto a platform located in the chamber, running to another location, etc.). This avoidance behavior can then be extinguished via presentations of the CS without shock but with the avoidance response still available. Under these conditions, rats may initially learn several competing and complementary Pavlovian (context–shock; context–CS; CS–shock) and instrumental [e.g., action (step onto platform)–no shock; action–safety cue] associations that control behavior. They then likewise learn multiple new associations during extinction. Unsurprisingly, given the probable involvement of both Pavlovian and instrumental contingencies, extinction of active avoidance learning recruits neural circuits overlapping with both Pavlovian fear extinction and extinction of instrumental appetitive learning. Notably, the PFC is critical to extinction of active avoidance. Active avoidance extinction learning is attenuated via reversible inactivation of IL (729) so that IL inactivation before extinction training does not affect within-session extinction of avoidance but does impair the long-term retention of this extinction learning. This role for IL depends on multiple downstream targets, as extinction of active avoidance learning is associated with activity in both an IL→BLA pathway and an IL→Acb pathway (730). However, it is important to note that extinction of active avoidance is as strongly associated with activity in a PL→Acb pathway. Multiple corticoamygdala and corticostriatal pathways are necessary for expression of active avoidance (731), and in turn, multiple corticoamygdala and corticostriatal pathways are recruited by extinction to inhibit these active avoidance behaviors and/or the Pavlovian fear that underpins them. Extinction of avoidance learning, like Pavlovian fear extinction, also depends interactions between vHIP and PFC. Specifically, this extinction is associated with increased activity in both vHIP→PL and vHIP→IL pathways (732). BDNF is a key molecule in these pathways because extinction of avoidance learning increased BDNF expression in vHIP neurons and knockdown of BDNF in vHIP impaired retention of this extinction learning (732). Much remains to be learned about the mechanisms for extinction of avoidance learning. One attractive possibility is that PL→BLA and PL→Acb pathways mediate Pavlovian fear (PL→BLA) and instrumental avoidance responses (PL→Acb), respectively. Extinction would then recruit PL→BLA and PL/IL→Acb pathways with vHIP →PL and IL pathways enabling switching between the various Pavlovian and instrumental associations controlling behavior in these tasks.

Punishment is also instrumental aversive learning. In a typical punishment experiment, subjects are first trained to respond for a reward (e.g., lever press→sucrose pellet) and then this responding is punished via imposition of a new contingency on that responding (e.g., lever press→shock). Under these conditions, animals (including humans) form aversive R (lever press)→O (shock) associations and inhibitory context-response associations. As reviewed in sect. 2.2.2, punishment can be viewed as a form of interference similar to extinction learning. Consistent with this, studies of the brain mechanisms for renewal after punishment show some overlap with those of renewal of instrumental appetitive learning after extinction (131). For example, as noted in part previously, ABA renewal of punished responding based on cocaine self-administration is associated with activation of PL, IL, and anterior insular cortex, dorsomedial and dorsolateral striatum, BLA, vSUB, PVT, LHb, and substantia nigra (733). ABA renewal of punished responding based on alcohol self-administration is associated with activation of OFC, DLS, BLA, AcbC, LH, and vSUB (589, 632). Thus, recruitment of BLA is common across ABA renewal of extinguished responding and ABA renewal of punished responding, with other instances of overlap are observed in PL, IL, AcbC, PVT, and vSUB.

Manipulation studies largely converge on a similar conclusion. ABA renewal of punished responding for alcohol, like ABA renewal of extinguished responding, depends on vSUB and its projections to AcbSh (590), D1 receptors in the AcbSh and AcbC (734) and also on the LH (633). The overlap is perhaps not that surprising as ABA renewal following punishment or extinction both involve the restoration of appetitive instrumental behavior. The notable exception is BLA. Whereas c-Fos studies of ABA renewal of punished or extinguished responding based on cocaine or alcohol reinforcement each show selective increases in BLA c-Fos expression during renewal, reversible inactivation of BLA via baclofen/muscimol infusions has opposite effects on ABA renewal: It increases renewal of punished responding but decreases renewal of extinguished responding (735). The reasons for this discrepancy remain unclear but could be due to the fact that punishment involves stronger conflict between appetitive and aversive motivation than does extinction. There is evidence for separable coding of appetitive and aversive motivation by distinct BLA principal neurons (346, 602, 736, 737).

4.3. Sex and Individual Differences in Instrumental Extinction

The nature and role of individual differences in instrumental learning and its extinction have received less attention than Pavlovian conditioning. When sex has been examined, it appears to have a greater impact on instrumental aversive learning than instrumental appetitive learning and can often be attributed to differences in performance rather than learning.

Females outperform males in active avoidance learning, especially when the requirements of the avoidance response increase (e.g., two-way active avoidance) but males may outperform females in passive avoidance learning (738). This effect is due, at least in part, to differences in behavioral responses to footshock, with males tending to express passive freezing behavior (which impedes active avoidance learning) whereas females tend to express active locomotor behaviors (which is necessary for active avoidance learning) (738, 739). Studies of sex differences in punishment learning reveal similar complexities. For example, female rats can be more sensitive to punishment than male rats when the punisher is delivered under a probabilistic reinforcement schedule but may not be otherwise (740) (for review see Ref. 741), but these differences between sexes can be smaller than other individual differences between animals (742). There is less compelling evidence for consistent sex differences in instrumental appetitive learning, either in the older (743–745) or more recent literature (738, 740, 741). In addition, when such differences have been reported they can often be attributed to differences in performance (e.g., locomotor activity) rather than differences in the underlying learning mechanisms (738, 743–745). Other individual difference variables may also be important. For example, Saunders et al. (746) reported differences in the strength of ABA renewal of extinguished cocaine-seeking that depended on whether rats were predisposed to express appetitive learning via approach to signals for reward (i.e., sign tracking) versus approach to the location of reward delivery (i.e., goal tracking), with goal-trackers especially sensitive to renewal.

Therefore, this remains an important area for further work. It will be important to isolate the critical learning processes or specific performance factors that may underpin any differences observed, their generality across schedules of reinforcement, and the relationships to other schedule-generated behaviors. Regardless, the behavioral studies reviewed in sect. 2 providing key advances in understanding the nature of instrumental extinction learning have tended to use both male and female subjects, so there is currently little evidence that questions the generality of these mechanisms.

4.4. Placing the Neural Circuits for Instrumental Extinction in a Broader Framework for Instrumental Learning and Decision-Making

Unlike the study of the neurobiology of Pavlovian extinction, which grew directly out of research on Pavlovian learning, the study of the neurobiology of instrumental extinction has proceeded largely independently of study of the behavioral mechanisms of instrumental learning. Indeed, the instrumental extinction literature has made contact with the Pavlovian conditioning literature more frequently than it has the instrumental learning literature. How the mechanisms for instrumental extinction, renewal, and reinstatement relate more broadly to our understanding of the neurobiology of instrumental learning has received relatively little attention. However, we will try to provide some of that here.

As well noted by Balleine and colleagues (747–749), the acquisition and use of instrumental knowledge has been linked to parallel and partially segregated cortical-striatal thalamic-cortical circuits (FIGURE 8A). These circuits mediate different forms of behavioral control during instrumental learning (747–749). Pavlovian and other forms of elementary control of reward and motivation are traditionally linked to ventral striatum. More complex value-based decision-making requiring integration of this elementary information with goal-directed action (i.e., R–O) depends on a medial PFC (PL)→dorsomedial striatum (DMS)→substantia nigra reticulata (SN)→mediodorsal thalamus (MD)→cortex feedback loop (“action” panel). In addition, habitual behavior controlled by internal states or environmental stimuli (i.e., S–R) is thought to depend on sensorimotor cortices (SM)→dorsolateral striatum (DLS)→globus pallidus→posterior thalamus (PO)→cortex feedback loop (“habit” panel). These loops interface with, and are in turn influenced by, complex patterns of ascending dopaminergic inputs, so they are neither serial nor independent (749).

FIGURE 8.

A: parallel and partially segregated cortical-striatal thalamic-cortical circuits allow different forms of behavioral control during instrumental learning. Contextual control over instrumental behavior after extinction is embedded in a “selection” loop involving AcbSh. Reinstatement of extinguished instrumental behavior is embedded in an ‘invigoration’ loop involving AcbC. Separate loops involving dorsal striatum determine goal-directed action (dorsomedial striatum [DMS]) and habitual (dorsolateral striatum [DLS]) response control. AcbSh, nucleus accumbens shell; AcbC, nucleus accumbens core; IL, infralimbic cortex; LH, lateral hypothalamus; MD, mediodorsal thalamus; PL, prelimbic cortex; PO, posterior thalamus; PVT, paraventricular thalamus; SM, somatosensory cortex; SN, substantia nigra; VTA, ventral tegmental area. B: within the “selection loop,” AcbSh D1→LH projection mediates response inhibition in extinction (rose color), whereas AcbSh D1→VPm→LH/VTA mediates response restoration during renewal (green color). The VPm→VTA pathway emerges from VP parvalbumin (PV) neurons whereas the VP→LH pathway emerges from VP Gad1 neurons. The extinction and renewal pathways converge onto the same LH Gad1 neurons. These same selection circuits interface with hypothalamic and midbrain behavioral control columns to mediate the response-generative effects of nonreinforcement.

Ventral striatopallidal circuits underpin the control of instrumental behavior by environmental stimuli (747, 750). Instrumental extinction is embedded, at least in part, within these ventral striatopallidal circuits. Within AcbSh (FIGURE 8B), there are at least two anatomically and functionally distinct circuits. The first involves AcbSh→LH and mediates response inhibition in extinction. The second involves AcbSh→VP→LH/VTA and mediates response restoration during renewal. These circuits emerge from separate populations of AcbSh D1 neurons and form largely segregated pathways, but they converge with each other at least at the level of Gad1 neurons in the LH. This convergence on the same LH Gad1 neurons, identified both functionally and physiologically, may underpin behavioral competition between contextual control over extinction versus renewal. These circuits, in turn, have at least two thalamic returns to ventral striatum and cortex. The first is via the PVT, which provides inputs to both AcbSh and IL and is essential for both extinction and renewal. The second is via mediodorsal thalamus (MD) (698, 751), which provides inputs to PL and dorsomedial striatum. The role of this MD thalamic return in extinction and renewal is unknown, but it provides an obvious interface with dorsal striatal goal-directed action systems that could generate the response-specific inhibition and initiation that characterize extinction and renewal of instrumental responding.

These AcbSh-based circuits undermine (AcbSh→LH) or restore (AcbSh→VP/VTA) performance of specific instrumental responses after extinction. Their roles could be viewed in terms of inhibiting or augmenting the substrates for instrumental responding (752–754). Hippocampal inputs, which serve a key role in gating cortical and thalamic information flow through the AcbSh (755–758), are important in permitting contextual selection between these AcbSh-based circuits that promote versus inhibit instrumental responding (FIGURE 7). Cortical, thalamic, and amygdala glutamatergic inputs target different AcbSh compartments. The precise function of these inputs in promoting versus inhibiting instrumental responding after extinction will be dictated by their specific profiles of connectivity with the distinct AcbSh neuronal ensembles contributing to these pathways, and these profiles differ significantly within individual inputs. For example, caudal versus rostral BLA, dorsal versus ventral PL, and anterior versus posterior PVT each reach distinct compartments of AcbSh. Given this compartmentalization of inputs and the functional as well as anatomical segregation of AcbSh outputs, it is perhaps not surprising that relatively nonselective manipulations of them yield inconsistent and conflicting findings, whereas more selective manipulations of neuronal ensembles or specific projections have been more consistent. These AcbSh circuits underpin the selection of specific instrumental behaviors and are shared with other instances of selective modulation of instrumental behavior (e.g., specific Pavlovian–instrumental transfer) (698, 759–761). They are at least partly separate from dorsal striatal circuits for goal-directed versus habitual behavioral control, allowing at least some preservation of specific dorsal striatal action, stimulus, outcome, and contingency knowledge encoded during instrumental learning.

Reinstatement of instrumental responding that is provoked by drug-associated stimuli, stressors, or priming with the drug itself is associated with a separate cortico-ventral striatal-pallidal- circuit involving AcbC→VPdl→VTA/SNc (550, 551, 685, 762) (“Invigoration” panel of FIGURE 8A). This AcbC-based circuit is linked to a thalamic return to both ventral striatum and prefrontal cortex, in part via the MD and ventromedial (VM) thalamus. This AcbC-based circuit mediates the nonselective invigoration of instrumental responding and is shared with other instances of nonselective modulation by stimuli, rewards and appetitive states such as general Pavlovian–instrumental transfer (698, 759–761).

The functional segregation of AcbSh (selection) from AcbC (invigoration) based circuits is important (752). The impact of various reinstating events (stressors such as footshock, drug-associated stimuli cues) on extinguished instrumental performance depends on the context in which they are presented (see sect. 2.1.2). AcbSh based circuits mediate this influence of context to “set the stage” for the impact of these reinstating events, at least in part via their well described control feedforward spiraling dopamine circuitry (see below) and via the PVT. This view of the roles of AcbSh and AcbC circuits in instrumental extinction, renewal, and reinstatement explains many inconsistencies in the literature regarding whether and when AcbC circuits are important for restoration of extinguished instrumental responding. For example, studies that have identified a role for AcbC in renewal often employ discrete response-contingent drug-associated cues (e.g., tone + light paired with drug infusion) during training. These cues are rarely presented and extinguished during the instrumental extinction phase, but testing conditions often restore these response-dependent cues. Therefore, these procedures involve conditioned reinforcement and general Pavlovian-instrumental transfer: both of which are well known to depend on AcbC (323, 747, 761, 763). It is unsurprising, then, that there is compelling evidence for AcbC involvement in renewal procedures that also involve presentation of response-contingent reinstating events.

These ventral selection and invigoration loops are neither independent of each other nor independent of the dorsal loops. Rather, dynamic interactions between parallel striatal circuits, including feedback and feedforward dopaminergic control (749), underpin the complexity and flexibility of instrumental learning and behavior (745, 762). The key role for AcbSh in contextual control over extinction versus renewal is important because AcbSh can influence dopamine neurotransmission across the rest of the striatum via its projections to VTA (749) and so can broadly undermine (extinction) or enable (renewal) instrumental performance via changes in dopamine. It follows that the brain mechanisms for extinction and renewal of instrumental responding are likely much more complex than the findings reviewed here suggest, and they are likely distributed across the basal ganglia and its major inputs. This idea is supported by recent striatum-wide mapping of cellular activity (765). Moreover, although the literature has focused extensively on the roles of PL and IL in instrumental extinction and renewal, several other cortical (e.g., rostral agranular insular; orbitofrontal) (546, 597, 731, 766) and striatal (dorsolateral) (549) regions are implicated in extinction or restoration of instrumental responding, and these regions mediate distinct aspects of instrumental learning and behavior (e.g., coding and retrieval of action or outcome value).

This view of the mechanisms of instrumental extinction, renewal and reinstatement provides insight into other poorly understood aspects of instrumental extinction. The purpose of instrumental extinction is the elimination of behavior. Yet, operant extinction has predictable and well documented behavioral “side effects” in both laboratory animals and humans. These include temporary increases in the behavior that is being extinguished (i.e., extinction burst), a return or relapse to other behaviors that had been extinguished (resurgence), and increased frequency of “survival” behaviors such as aggression (767–769) and drinking (770) (for review see Refs. 771, 772). Indeed, up to 40% of people receiving instrumental extinction-based treatment interventions show one or more of these behavioral side effects (769). These response-generative effects of operant extinction are robust and can be problematic for the practical application of extinction-based therapies. They have traditionally been linked to the aversive, frustrative effects of reward omission (773, 774) (but see sect. 2.2.2 for analysis of resurgence). It is noteworthy that the ventral striatopallidal circuits controlling extinction of instrumental responding converge on and influence hypothalamic and brainstem behavioral control columns involving LH, lateral habenula, VTA, pedunculopontine, periaqueductal gray (645, 775, 776) areas that mediate these fundamental survival behaviors (777, 778) (FIGURE 8B). These behavioral control columns are recruited during instrumental extinction or reward omission (64, 66, 592, 779–781), in an AcbSh-dependent manner (645), and this recruitment could underpin the response generative side effects of instrumental extinction.

5. CONCLUSIONS

Extinction is extensively studied today at both the behavioral and neurobiological levels. This is because it is a fundamental and perhaps representative behavior-change phenomenon and because it is often thought to underlie behavioral therapies designed to reduce or eliminate unwanted behaviors and behavior problems.

Research at both the behavioral and neurobiological levels is consistent in pointing to several themes. Perhaps the most important is that extinction is rarely the destruction of the original learning, but instead involves new learning that recruits new learning and memory circuits in the brain. At the behavioral level, we know that extinction is not unlearning because there are so many recovery or “relapse” effects, perhaps the most fundamental of which is renewal. Consistent with this, the research on the neurobiology of extinction in Pavlovian learning suggests it requires a tripartite neural circuit involving the amygdala, prefrontal cortex, and hippocampus. Synaptic plasticity in the amygdala is essential for extinction learning, and prefrontal cortical inhibition of amygdala neurons encoding fear memories is involved in fear retrieval. Hippocampal-prefrontal circuits mediate fear relapse phenomena, including renewal. In instrumental learning, extinction and response recovery effects involve prefrontal and hippocampal circuits interfacing with striatopallidal circuits for inhibitory (extinction) and excitatory (renewal and other “relapse” effects) control over operant responding.

Behavioral work also suggests many parallels between extinction as it is understood in Pavlovian and instrumental learning. In both, we find all the basic recovery effects, that extinction is a representative form of interference, and that the role of context is played by a wide range of stimuli. However, research on the behavior-theoretical structures that underlie extinction, and the behavioral processes that engage them, suggests differences between Pavlovian and instrumental extinction. Consistent with the idea that Pavlovian learning mainly endows the organism with processes that allow it to learn about stimuli in the environment, performance after extinction often depends on the context selecting between existing excitatory and inhibitory memories or associations between the CS and the US, and extinction is caused by generalization decrement and a form of error correction in which the learning process corrects what the CS predicts. In contrast, but consistent with the idea that instrumental learning gives the organism a set of processes that allow it to change its voluntary behavior, extinction may occur because the organism learns to suppress or inhibit the previously-reinforced behavior in the extinction context and respond to error correction that detects and corrects the level of the response.

These ideas are further consistent with the biological processes that underlie extinction. There are clear parallels in the brain mechanisms for Pavlovian and instrumental extinction (see FIGURES 6 and 7). At neural circuit levels, both forms of extinction depend on prefrontal mechanisms for response or behavioral selection (i.e., initiation or termination of behavior), hippocampal mechanisms endowing contextual control over this selection, as well as thalamic feedback mechanisms allowing both bottom-up and top-down control. At the cellular level, both forms for extinction can also share similar forms of synaptic plasticity. There are also obvious and important differences in the underlying biological processes that respect fundamental differences between brain mechanisms for learning about stimuli in the environment (Pavlovian) versus learning about behavior (instrumental).

The field has made remarkable progress. Nonetheless, we remain far from an integrated, coherent understanding of extinction. Extinction may seem behaviorally simple, but as we have shown, this simplicity belies significant and important complexities in what is learned and how this learning affects behavior. In general, we know more about the biological mechanisms controlling behavior after extinction [e.g., selecting between existing excitatory and inhibitory memories (Pavlovian) or responses (instrumental)] than we do about the biological root causes of extinction learning: generalization decrement, error correction, and response inhibition. For instrumental extinction, progress toward an integrated understanding has been limited by differences in terminology, by the use of simpler behavioral designs, and by a focus on a single form of contextual control. Moving forward, the designs and approaches advocated here provide powerful behavioral tools for neuroscientists to isolate the distinct and complex mechanisms of extinction.

GRANTS

Manuscript preparation for this work was supported by National Institutes of Health Grants R01-DA-033123 (to M.E.B.) and R01-MH-117852 and R01-MH-065961 (to S.M.) as well as National Health and Medical Research Council National Health Grants GNT1164514 and GNT1138062) and Australian Research Council Grant DP170100075 (to G.P.M).

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

The authors jointly wrote the manuscript, with M.E.B. leading on sects. 1, 2, and 5, S.M. on sect. 3, and G.P.M. on sect. 4; M.E.B., S.M., and G.P.M. edited and revised manuscript; M.E.B., S.M., and G.P.M. approved final version of manuscript.