Behavioral and Neurobiological Mechanisms of Extinction in Pavlovian and Instrumental Learning

Abstract

This article reviews research on the behavioral and neural mechanisms of extinction as it is represented in both Pavlovian and instrumental learning. In Pavlovian extinction, repeated presentation of a signal without its reinforcer weakens behavior evoked by the signal; in instrumental extinction, repeated occurrence of a voluntary action without its reinforcer weakens the strength of the action. In either case, contemporary research at both the behavioral and neural levels of analysis has been guided by a set of extinction principles that were first generated by research conducted at the behavioral level. The review discusses these principles and illustrates how they have informed the study of both Pavlovian and instrumental extinction. It shows that behavioral and neurobiological research efforts have been tightly linked and that their results are readily integrated. Pavlovian and instrumental extinction are also controlled by compatible behavioral and neural processes. Since many behavioral effects observed in extinction can be multiply determined, we suggest that the current close connection between behavioral-level and neural-level analyses will need to continue.

Behavioral research over the last few decades has made significant progress in uncovering the mechanisms that underlie extinction, the behavioral phenomenon in which learned behavior decreases in strength or frequency when the event that reinforced it is removed (e.g., Delamater & Westbrook, this issue). Extinction is important, in part because it is one of the most basic of all behavioral change effects, and in part because it is thought to be involved in many clinical treatments that are designed to get rid of unwanted learned behaviors, thoughts, and emotions (e.g., Craske, Kircanski, Zelikowsky, Mystkowski, Chowdhury, & Baker, 2008; Craske, Liao, Brown, & Vervliet, 2012). Since the 1970s, extinction has been extensively studied in Pavlovian conditioning, where responding to a conditioned stimulus (CS) that has been associated with an unconditioned stimulus (US) decreases when the CS is then presented repeatedly alone. The results of this literature, along with the larger behavioral research literature of which it is a part, support a set of principles of extinction that are summarized in Table 1. These principles have gone on to shape further research on extinction at both the behavioral and neurobiological levels of analysis. The purpose of the present article is to discuss these principles and explore how they have facilitated progress in behavioral and neurobiological research on Pavlovian extinction and a “newer” frontier concerning the extinction of instrumental learning, where the focus is on voluntary behaviors that are controlled by their consequences (reinforcers). We also suggest that continued success at elucidating the neurobiological mechanisms of Pavlovian and instrumental extinction will require continued research at the behavioral level.

Table 1.

Summary principles of extinction

Extinction is not the same as erasure Responding can return or “relapse” through spontaneous recovery, renewal, reinstatement, rapid reacquisition, resurgence The context plays a fundamental role in extinction “context” can be provided by exteroceptive background cues as well as interoceptive cues such as drug state, hormonal state, mood state, deprivation state, recent events, expectation of events, and time Extinction is at least partly a context-specific form of inhibitory learning Performance declines in extinction because of (1.) generalization decrement and (2.) the correction of prediction error Extinction is a retroactive interference paradigm that shares many features with other “interference paradigms” involving retroactive and/or proactive interference

Summary extinction principles

The first extinction principle supported by behavioral research is perhaps the most widely recognized today: Although behavior goes away in extinction, Extinction is not the same as erasure. Learning theorists have long recognized that there is a difference between behavior on the one hand and the organism’s knowledge on the other. Extinction is a good example of the so-called “learning-performance distinction:” Although performance is at a zero level at the end of extinction, the original learning is still retained in long-term memory and the brain. Pavlov’s early demonstrations (1927) of spontaneous recovery were the first to support this idea: If time is allowed to elapse after extinction, responding can return. The list of related recovery effects was expanded in the 1970s and 1980s (e.g., see Bouton, 1988, for an early review). In renewal, extinguished responding returns when the CS is removed from the extinction context and tested in another context (e.g., Bouton & Bolles, 1979a; Bouton & King, 1983; Bouton & Ricker, 1994; Laborda, Witnauer, & Miller, 2011). In reinstatement, behavior recovers if the unconditioned stimulus (US) is presented again after extinction (e.g., Rescorla & Heth, 1975). And in rapid reacquisition, responding can return to the CS quickly if CS-US pairings are resumed after extinction (e.g., Napier, Macrae, & Kehoe, 1992). Since the 1980s and early 1990s, these recovery phenomena have been seen as potential models of relapse after extinction (e.g., Bouton, 1988, 2002; Bouton & Swartzentruber, 1991; Laborda, McConnell, & Miller, 2011). Because they do not occur unless the CS was associated with the US in the original conditioning phase, they each indicate that at least part of the knowledge that was acquired during conditioning must survive extinction. Some comparisons of extinguished and nonextinguished CSs have even suggested that extinction can leave the strength of the original CS-US association more or less “fully preserved” (Rescorla, 1996; see also Delamater, 1996).

A second group of extinction principles follows directly from the evidence supporting the first. At the same time the various relapse phenomena indicate that extinction is not erasure, they also demonstrate that The context plays a fundamental role in extinction. Renewal, which indicates that extinction performance can be lost when the context is changed, is the most direct demonstration of this principle. However, all of the other relapse effects are also arguably context effects (e.g., Bouton, 1988, 1993, 2004). For example, reinstatement occurs at least in part because the reinstating US presentations condition the context, and this contextual conditioning is the trigger that causes responding to return to the CS. The supporting evidence includes the fact that reinstatement is typically not observed unless the US is presented in the context where the CS will be tested (e.g., Bouton, 1984; Bouton & Bolles, 1979b; Bouton & Peck, 1989), and the strength of reinstatement correlates with measures of contextual conditioning (Bouton, 1984; Bouton & King, 1983). The other relapse effects are also context effects, because “context” is provided by many different kinds of stimuli. In typical experiments on renewal, the contexts are provided by the conditioning chambers in which conditioning and/or extinction are conducted. However, interoceptive cues can play the role of context, as they do in state-dependent extinction, where extinction performance is shown to be specific to the context created by a drug administered during extinction (e.g., Bouton, Kenney, & Rosengard, 1990; Cunningham, 1979; Hart, Harris, & Westbrook, 2009; Lattal, 2007). Furthermore, recent USs and recent CS-US pairings are part of the context of conditioning and can cause extinguished responding to return—as they can in reinstatement and rapid reacquisition (e.g., Baker, Steinwald, & Bouton, 1991; Bouton, Woods, & Pineño, 2004). Spontaneous recovery can also be conceptualized as the renewal effect that occurs when the extinguished CS is tested in a new temporal context (e.g., Bouton, 1988; Brooks & Bouton, 1993). Temporal cues can clearly disambiguate the current meaning of the CS; rats can use the temporal context provided by the current inter-trial interval to signal whether or not the next CS will be reinforced (e.g., Bouton & Hendrix, 2011).

Perhaps a deeper point about the context’s role in extinction is that extinction depends at least partly on a context-specific form of inhibitory learning. This idea is most clearly consistent with the so-called ABC and AAB forms of the renewal effect. In these situations, extinguished responding returns when conditioning, extinction, and testing (respectively) occur in Contexts A, B, and C or in A, A, and B. The common feature of these forms of renewal is that the return of the response does not depend on return to the original context of conditioning (as in the so-called ABA renewal effect); mere removal from the extinction context is sufficient. This fact suggests that the response is actively inhibited in the context of extinction. In principle, such inhibition can take any of several forms. The simplest form is the one suggested by many models of associative learning, including the Rescorla-Wagner model (1972): Extinction with the CS in a neutral Context B, for example, could make the associative strength of the context become negative (see also Pearce & Hall, 1980; Wagner, 1981). The context would essentially predict “no US.” Although such direct inhibition could occur in extinction, evidence of it is rare and usually lacking (e.g., see Bouton, 1993, for one review). Behavioral research on Pavlovian extinction has instead supported the form of inhibition known as “negative occasion setting.” Here, the context of extinction serves as a hierarchical information cue that signals that the CS will not be paired with the US (e.g., Holland, 1992; Schmajuk, Lamoureux, & Holland, 1998). In a way, it sets the occasion for the CS’s meaning. We will suggest a third form of inhibition when we consider the context’s role in instrumental extinction: The context may directly inhibit the instrumental response (e.g., Rescorla, 1993, 1997). We should add that the effects of contexts on extinction are not exclusively inhibitory. They can have “excitatory” influences, too, as in reinstatement, when the recent association of the context and the US triggers responding to the CS, and in ABA renewal, where testing in the context of conditioning typically causes stronger renewal than testing in a neutral context (ABC renewal; e.g., Harris, Jones, Bailey, & Westbrook, 2000).

The next summary principle of extinction is that behavior weakens in extinction in part because of (1.) generalization decrement and (2.) new learning that is driven by prediction error. Generalization decrement occurs in extinction because the stimulus conditions change when extinction begins. For instance, simple omission of the US can remove some of the stimulus support for responding; when the US is still occasionally presented during extinction, but without a contingent relationship with the CS, the loss of responding is slowed (e.g., Frey & Butler, 1977; Rescorla & Cunningham, 1977; Spence, 1966). Although such a result is consistent with several behavioral mechanisms, one is that presenting the US in extinction makes the stimulus conditions of extinction more similar to those of conditioning. Generalization decrement was emphasized in a generation of extinction theories that were developed in the 1950s and 1960s (e.g., Amsel, 1967; Capaldi, 1967). It was the main mechanism that explained the partial reinforcement extinction effect (PREE), in which responding declines more slowly over extinction trials in subjects that have not been reinforced every time they made the response (e.g., Mackintosh, 1974). For example, according to sequential theory (e.g., Capaldi, 1967), the PREE occurs because the subject has been reinforced for responding in the presence of a memory of not being rewarded, which makes responding persist over more nonrewarded extinction trials. In frustration theory (e.g., Amsel, 1967), responding is prolonged because the animal has learned to respond in the presence of frustration cues that are present in extinction. Notice that, consistent with the first summary extinction principle, generalization decrement does not imply that extinction erases the original learning.

As noted above, however, phenomena like ABC and AAB renewal further suggest that extinction is also controlled in part by new (context-dependent) inhibitory learning. According to modern conceptions of the learning process, new learning is driven by “prediction error,” a behavioral mechanism that is embodied in the Rescorla-Wagner model (e.g., Rescorla & Wagner, 1972) and many of the models of conditioning that followed it (e.g., Pearce & Hall, 1980; Wagner, 1981). According to this view, the degree of associative change that occurs on any conditioning or extinction trial is governed by the difference between what is predicted by all the cues present on the trial and the US that actually occurs. The discrepancy between what is predicted and what occurs is the prediction error. To examine the idea, a number of experiments have explored the effects of compounding other CSs with a target CS during extinction. When the target is compounded with a separate predictor of the US, the US prediction is especially strong and responding to the target is especially decremented (e.g., Leung, Reeks, & Westbrook, 2012; Rescorla, 2000, 2006; Wagner, 1969). Conversely, when the target CS is compounded with a CS that actively predicts “no US” (that is, a conditioned inhibitor), prediction error is reduced and there is less associative consequence of extinction trials (e.g., Rescorla, 2003; Soltysik et al., 1983). We should note that presenting a target CS in compound with other CSs during extinction can introduce complications. For example, compounding the target CS with two excitors may be less effective than compounding it with one excitor (McConnell, Miguez, & Miller, 2013), and there may be increased generalization decrement when a CS extinguished in compound with other stimuli is later tested alone (Urcelay, Lipatova, & Miller, 2009; Vervliet, Vansteenwegen, Hermans, & Eelen, 2007). Perhaps the most striking evidence for the role of prediction error in extinction, however, is the “overexpectation effect,” in which two CSs are separately associated with the US and then presented in a compound that is paired with the US. Even though the compound is reinforced on each trial, the compounded CSs summate to predict a greater US than the one that occurs, and therefore undergo some extinction (see Kremer 1978; Lattal & Nakajima 1998; Rescorla, 2007). This sort of result has had a profound effect on how we understand the conditions that drive new learning in extinction. Interestingly, just like other extinction phenomena, the decrement in conditioned responding caused by overexpectation is subject to spontaneous recovery as well as renewal (e.g., Rescorla, 2006; 2007)

The last principle of extinction in Table 1 is that the extinction paradigm (in which a CS is presented without the US after previous CS-US pairings) is just one of several interference paradigms that all follow similar principles (Bouton, 1993). In interference paradigms, the animal learns conflicting information about a CS in different phases of an experiment, and these hinder one another. In extinction, the focal interference is retroactive; Phase 2 learning interferes with performance from the first. In other paradigms, such as discrimination reversal learning and counter conditioning (in which a CS is paired with USs of different emotional valences in Phases 1 and 2), there is a mixture of both retroactive and proactive interference. And in still other paradigms, such as latent inhibition, the focus is primarily on proactive interference. Although theories of the various interference paradigms have often focused on interference at the level of storage and learning, there is evidence to suggest that interference often occurs at the level of performance output (see Bouton, 1993). Here again, there is a difference between knowledge and performance. As in extinction, physical context and temporal context (time) have similar influence on these paradigms. Miller and Escobar (2002) have expanded this point of view, for example, to also include situations in which a given US is associated with different predictors in different phases.

Neurobiological mechanisms of Pavlovian extinction

Advances in our understanding of extinction from a behavioral point of view contributed directly to a new interest in the underlying neurobiological mechanisms that began to crystallize and mature in the 2000s (e.g., Delamater, 2004; Myers & Davis, 2002). In general, the neuroscience literature can be divided into several areas that emphasize one of the major principles just described. Perhaps the largest area is one investigating the neural circuits and synaptic changes that support new learning in extinction. Although there is evidence that some of the synaptic changes that occur during conditioning can be reversed by extinction (Kim et al., 2007; Lin, Yeh, Lu, & Gean, 2003; see also Delamater, 2012), many findings strongly support the idea that new learning occurs, and that like conditioning (as well as other forms of learning), extinction depends on NMDA receptor activity (for reviews, see Davis, 2011; Orsini & Maren, 2012; Quirk & Mueller, 2008). Many studies have now shown that blocking this receptor immediately before or after extinction training blocks fear extinction (e.g., Burgos-Robles, Vidal-Gonzalez, Santini, & Quirk, 2007; Falls, Miserendino, & Davis, 1992; Santini, Muller, & Quirk, 2001), indicating that the NMDA receptor is necessary for both encoding and consolidation of extinction. Indeed, NMDA receptors in the basolateral amygdala (BLA; Falls et al., 1992) are necessary during extinction training, whereas those in the infralimbic area of the medial prefrontal cortex (IL mPFC; Burgos-Robles et al., 2007) are needed immediately following training. In a complementary way, enhancing NMDA activity has been shown to facilitate fear extinction. Modulating NMDA activity has thus become an important therapeutic target for clinical scientists seeking to optimize extinction-based therapies (see below for a more detailed discussion).

Interactions between the BLA and IL mPFC are now thought to embody the inhibitory mechanism that controls fear responding after extinction. As the behavioral research makes clear, however, the context plays a major role in modulating its expression. A corresponding literature has focused on understanding the role of context, and has implicated the workings of the hippocampus, a region known to be critically involved in processing contextual stimuli and forming representations of the context. The work of Maren and colleagues (Corcoran & Maren, 2001, 2004; Hobin, Ji, & Maren, 2006; Orsini, Kim, Knapska, & Maren, 2011; see also Orsini & Maren, 2012) has emphasized hippocampal involvement in renewal (but see Campese & Delamater, 2013; Frohardt, Guarraci, & Bouton, 2000; Wilson, Brooks, & Bouton, 1995; Zelikowsky, Pham, & Fanselow, 2012). Using several different manipulations, they have demonstrated that hippocampal inactivation prior to renewal testing prevents extinguished fear from returning. Orsini et al. (2011) have extended those findings by showing that disrupting communication between the hippocampus and BLA by severing direct or indirect pathways (via the PFC) abolishes renewal. Although there is far less research examining hippocampal involvement in other relapse effects, data from our laboratory has shown that it plays a role in reinstatement after fear extinction (Frohardt et al., 2000; Wilson et al., 1995), although its role after appetitive extinction is less certain (Fox & Holland, 1998).

Another area focuses on the role of prediction error and how it is encoded in the brain. Recent work in fear extinction implicates the actions of endogenous opioids in the detection of negative prediction errors. McNally and colleagues (McNally, Pigg, & Weidemann, 2004a; McNally & Westbrook, 2003) have demonstrated such action in several experiments in which they blocked opioid activity with naloxone. Rats injected with naloxone immediately prior to (but not after) extinction showed dose-dependent deficits in within-session extinction that remained on later tests. Even more convincingly, blocking opioid receptors prevented the loss of responding due to overexpectation when two CSs were reinforced in compound (McNally, Pigg, & Weidemann, 2004b). Prediction errors appear to be encoded by other brain systems as well. Studies of appetitive learning have implicated the activity of midbrain dopamine neurons. These neurons fire in response to surprising rewards and are suppressed when expected rewards are withheld (Hollerman & Schultz, 1998; Waelti, Dickinson, & Schultz, 2001). Thus, dopamine neurons can detect the occurrence, size, and direction of prediction errors. More recent studies have also demonstrated the role of dopamine prediction errors in fear extinction, suggesting a broader role for them than in learning about the absence of positive rewards (Holtzman-Assif, Laurent, & Westbrook, 2010).

As noted above, researchers have identified the NMDA receptor as a critical component of the new learning that occurs in extinction. That discovery has led to investigation of whether enhancing NMDA activity can facilitate or strengthen extinction learning. Most of this work has focused on the drug d-cycloserine (DCS), a partial agonist of the receptor. Prior to its use in fear extinction studies, DCS had been shown to facilitate other forms of learning in animals, including eyeblink conditioning (Thompson & Disterhoft, 1997) and maze learning (Quartermain, Mower, Rafferty, Herting, & Lanthorn, 1994; Pussinen et al., 1997). Walker, Ressler, Lu, and Davis (2002) were the first to show that DCS enhanced fear extinction. In their experiments, animals treated with DCS shortly before extinction showed less fear when tested the following day. Administration of DCS without extinction training had no effect on its own. Other laboratories have since demonstrated similar results with fear extinction (Bouton, Vurbic, & Woods, 2008; Langton & Richardson, 2010; Ledgerwood, Richardson, & Cranney, 2003; Woods & Bouton, 2006) and non-fear paradigms such as conditioned place preference or aversion (Botreau, Paolone, & Stewart, 2006; Myers & Carlezon, 2010; Paolone, Botreau, & Stewart, 2009). Importantly, several of these studies have shown that post-training administration is similarly effective in enhancing extinction on subsequent tests, suggesting that DCS facilitates consolidation of extinction memories. This conclusion is consistent with other findings that the effect of DCS is reduced as the delay between extinction and post-training DCS administration is increased (Ledgerwood et al., 2003).

The generality of these effects across different learning paradigms suggests that DCS may have translational potential when administered alongside extinction-based clinical treatments, and indeed some human studies have shown significant benefits for exposure therapy (e.g., Otto et al., 2010; Ressler et al., 2004; Smits et al., 2013; but see Litz et al, 2012). However, as the behavioral research on extinction makes clear, there are important issues to consider regarding the use of extinction-enhancing drugs in therapy. One is that there is no a priori reason to think that a drug that enhances extinction learning will change the nature of extinction learning qualitatively. Thus, one must ask whether DCS affects extinction’s fundamental context specificity. In our laboratory, DCS facilitates fear extinction but leaves animals vulnerable to renewal (Bouton et al., 2008; Woods & Bouton, 2006); even extinction learning facilitated by DCS is still context-specific. It is more likely that DCS enables extinction learning to progress more quickly or with fewer exposures to the CS; that is, DCS enhances extinction quantitatively rather than qualitatively. In clinical settings, this may translate to achieving treatment goals in fewer therapy sessions. However, a further caveat is that DCS is ineffective with minimal training––some extinction must be learned while the drug is in the system in order for DCS to facilitate it (Bouton et al., 2008; Smits et al., 2013; Weber, Hart, & Richardson, 2007). There is also evidence from animal studies that DCS may actually impair extinction if too little training is given. Lee, Milton, and Everitt (2006) found that DCS-treated animals given a single CS exposure can display more fear than controls on subsequent tests. Based on these studies, it is clear that DCS does not erase fear memories or protect against renewal, and may not decrease (or may even increase) fearful responding under some conditions. Several findings from human studies appear to be consistent with that possibility (Litz et al., 2012; Smits et al., 2013).

Another area of research has responded to the behavioral evidence that extinction does not cause erasure by seeking new ways to interfere with fearful memories more permanently. This research began by focusing on the way that fear memories are stored in the brain. Investigators have long known that presenting fear-conditioned animals with a retrieval cue can make the fear memory temporarily more sensitive to various disruptions (Misanin, Miller & Lewis, 1968; Nader, 2003; Nader, Schafe, & LeDoux, 2000). If performed within a short post-retrieval window, these manipulations result in long-term decreases in fear responding to the CS. Such findings have led Nader and colleagues to propose that fear memories are not necessarily permanent after being consolidated. Rather, the act of retrieving a memory makes it unstable again. Before being reconsolidated, a newly unstable memory can be weakened if the neurobiological processes required for memory storage (or in this case re-storage) are stopped. This idea is mainly supported by experiments in which weakly fear-conditioned rats are given drugs that stop protein synthesis from occurring soon after memory retrieval. When later tested, the rats fail to show any fear (Nader et al., 2000; see Kindt, Soeter, & Vervliet, 2009, and Soeter & Kindt, 2012, for similar findings in humans).

Similar effects have more recently been demonstrated with behavioral procedures that use extinction in place of drugs. Monfils, Cowansage, Klann, and LeDoux (2009) reported that rats given fear extinction shortly after memory retrieval did not show any signs of relapse, including spontaneous recovery, renewal, and reinstatement. On the other hand, rats that were given extinction after a longer post-retrieval delay (i.e., after reconsolidation was presumed to be complete) demonstrated the usual recovery effects. The findings have generated considerable interest because they suggest that under certain conditions extinction procedures may lead to erasure and unlearning. However, studies in this area are still few, and there are several unanswered questions about the behavioral and neurobiological mechanisms that are involved. One important issue, which has also been a longstanding question for the earlier studies using post-retrieval drugs, is whether this retrieval–extinction procedure actually changes (or erases) fear memories or instead makes them less accessible. The absence of fear responding on a test cannot distinguish between these possibilities (e.g., see Lattal & Stafford, 2008). Another question concerns the fact that memory retrieval is typically achieved with a brief nonreinforced presentation of the CS that would otherwise be considered an ordinary extinction trial. It is not clear why extinction trials should reactivate fear memories and allow more extinction trials to cause erasure in the Monfils et al. (2009) paradigm, but context-dependent new extinction learning in most other extinction protocols, although it is becoming apparent that different molecular processes would be involved (see Auber, Tedesco, Jones, Monfils, & Chiamulera, 2013). Also, although analogous findings have been replicated in other Pavlovian conditioning experiments (Flavell, Barber, & Lee, 2011; Schiller, Monfils, Raio, Johnson, LeDoux, & Phelps, 2010), they have not been obtained universally (Chan, Leung, Westbrook, & McNally, 2010; Flavell et al, 2011; Ishii, Matsuzawa, Matsuda, Tomizawa, Sutoh, & Shimizu, 2012; Kindt & Soeter, 2013; Soeter & Kindt, 2011).

Behavioral mechanisms of instrumental extinction

Other advances are being made in understanding the behavioral and neurobiological mechanisms of instrumental extinction. In instrumental (or operant) learning, organisms learn to perform certain behaviors (e.g., pressing a lever or pulling a chain) when a reinforcing event like a food pellet is presented as a consequence of it. In extinction, the behavior declines when the reinforcer is no longer presented. Instrumental extinction is as important to understand as Pavlovian extinction if we want a complete understanding of the phenomenon. Furthermore, it is worth studying in its own right because instrumental behavior is the animal laboratory’s model of voluntary action, choice, and decision making (e.g., Balleine & Ostlund, 2007). Understanding instrumental extinction may thus lead to more direct insight into the inhibition of voluntary behaviors, such as drug taking, overeating, and gambling.

Our laboratory has recently expanded from studying Pavlovian extinction to also studying the behavioral mechanisms that are involved in instrumental extinction. This effort has used our previous research on Pavlovian extinction, and the principles listed in Table 1, as its guide. In general, the findings indicate strong parallels between the principles that govern extinction of Pavlovian and instrumental conditioning. For example, the demonstration of relapse effects (renewal, resurgence, rapid reacquisition, and reinstatement) after instrumental extinction once again indicates that extinction does not erase original learning but instead results in new learning that is at least partly context-dependent.

Relapse after instrumental extinction

Perhaps the most basic of the relapse effects is the renewal effect. All three forms of renewal (ABA, AAB, and ABC) have now been demonstrated after the extinction of instrumental behavior (e.g., Bouton, Todd, Vurbic, & Winterbauer, 2011). For example, Bouton et al. (2011) first trained rats to lever press for food pellets in Context A on a variable-interval 30 s (VI 30 s) schedule (pellets were made available with a 1/30 probability every second). After several sessions of acquisition, half the rats were switched to Context B for extinction (where lever presses no longer resulted in food pellet delivery) and the other half received extinction in Context A. Extinction lasted for four sessions, at which point responding was quite low. In two final sessions, all rats were tested in their extinction context and in the other (renewal) context (order counterbalanced). For rats that underwent extinction in Context B, a return to Context A caused a significant increase in lever press responding (ABA renewal, see also Nakajima, Tanaka, Urushihara, & Imada, 2000). And for rats that received extinction in Context A, a move to Context B also caused responding to increase (AAB renewal; see also Todd, Winterbauer, & Bouton, 2012a). Renewal also occurred in a similar experiment in which acquisition, extinction, and testing occurred in separate contexts (ABC renewal; see also Todd, Winterbauer, & Bouton, 2012b).

Like renewal, resurgence indicates that instrumental extinction does not erase the original learning (see Bouton, Winterbauer, & Todd, 2013, for a review). In resurgence, one operant behavior (e.g., pressing one lever) is first reinforced and then undergoes extinction in a second phase. While the first behavior is in extinction, a second behavior (e.g., pressing a second lever) is reinforced. In a third phase, when the second behavior is then extinguished, the first behavior returns, or “resurges.” For example, Winterbauer and Bouton (2010, Experiment 1) first trained rats to lever press (L1) for food pellets on a VI 30 s schedule. In the next phase L1 was nonreinforced (extinguished) while a second lever (L2) was reinforced for some rats, but presented and nonreinforced for other rats. In the final phase, when both L1 and L2 were presented and nonreinforced, only rats that had L2 reinforced during the previous phase showed resurgence. That is, if while L2 was being reinforced while L1 was being extinguished, responding resurged on L1 when L2 was subsequently extinguished.

Several behavioral mechanisms of resurgence have been suggested (e.g., Leitenberg, Rawson, & Bath, 1970; Leitenberg, Rawson, & Mulick, 1975; Podlesnick & Shahan, 2009; Shahan & Sweeney, 2011). However, we have emphasized that resurgence can be understood as another form of the renewal effect (e.g., Winterbauer & Bouton, 2010). According to this perspective, reinforcement of the second behavior is part of the “context” in which the first behavior is extinguished. Resurgence then occurs when the reinforcer context is removed (see Bouton, Rosengard, Achenbach, Peck, & Brooks, 1993). Thus, resurgence is conceptually similar to the ABC renewal effect (e.g., Bouton et al, 2011). The difference is that in resurgence the animal’s own behavior and/or the reinforcers it produces serves as the context, instead of the physical surroundings.

One prediction of this account of resurgence is that any manipulation that makes the final change in context less detectable should reduce the effect. One way this can be accomplished is by “fading” or “thinning” the rate of reinforcement of the second behavior during extinction of the first behavior. Thus, the first behavior undergoes extinction in the context of infrequent reinforcement, which results in the extinction context being similar to the final test context where both responses no longer produce reinforcement. To test this idea, Winterbauer and Bouton (2012) first trained rats to lever press (L1) on a VI-30 s schedule. Next, during extinction of L1, a second lever (L2) was reinforced with the same schedule throughout this phase (either fixed interval [FI] or random interval [RI] 20 s) for several sessions. For these two groups, when L2 was then extinguished, there was a strong resurgence effect. However, final test performance was much different for two other groups. For these groups, during extinction of L1, the schedule of reinforcement of L2 was gradually thinned from either a FI or RI 20 s schedule to a final schedule of FI or RI 120 s. When L2 was subsequently nonreinforced, the degree of resurgence in these two groups was much less pronounced. Similar results have recently been reported by Sweeney and Shahan (2013). Thus, thinning the reinforcement schedule of L2 results in less resurgence. This finding indicates that resurgence depends upon the rate of reinforcement used during the extinction phase, a result previously demonstrated by Leitenberg at al. (1975).

In the natural world, the “lapses” in extinguished instrumental behaviors like drug taking or binge eating that might be caused by renewal and resurgence usually lead to new pairings of the behavior and the reinforcer (e.g., Bouton, 2000). These new action-reinforcer pairings might then lead to reacquisition of the original behavior. Several experiments from our laboratory and others have shown that reacquisition is also modulated by the context. For example, rats that were not deprived of food (e.g., Todd et al., 2012a, Experiment 1) were first trained to lever press for either sucrose or sweet/fatty pellets in Context A and extinguished in Context B (e.g., Todd et al., 2012a, Experiment 1). After initial renewal testing, rats were then divided into two groups: One group received reacquisition in Context A (the acquisition context) and one group in Context B (the extinction context). During this phase, every fifth press now earned a pellet. Reacquisition of lever pressing was faster in Context A than in Context B. One way to think about this is that the extinction context slowed reacquisition of the original behavior. In a second experiment (Todd et al., 2012a, Experiment 2b), after acquisition and extinction in Context A, reacquisition was faster in Context B than in Context A. In this experiment, rats were quicker to reacquire lever pressing in a context in which no lever pressing had occurred before. This result emphasizes the inhibitory nature of the extinction context as well as the fact that removal from the extinction context is sufficient for recovery to occur. Other forms of context can also influence reacquisition. For example, Woods and Bouton (2007) introduced infrequent response-reinforcer pairings during extinction of lever pressing. This procedure slowed reacquisition relative to a group that received simple extinction. Woods and Bouton (2007) argued that adding the occasional response-reinforcer pairings during extinction increased generalization between extinction and reacquisition. Essentially, the response-pellet pairings allowed them to become part of the “context” associated with nonreinforcement.

Following extinction, non-contingent presentations of the reinforcer can cause reinstatement of instrumental responding (e.g., Baker et al., 1991). Behavioral research suggests that this form of “relapse” operates through at least two possible mechanisms (see Bouton & Swartzentruber, 1991). One possibility is that the reinforcer may serve as a discriminative stimulus, or type of context, that directly supports the instrumental response (e.g., Ostlund & Balleine, 1997). Thus, when the reinforcer is presented, it can set the occasion for responding again. In this way, reinstatement may be a form of ABA renewal; responding increases upon a return to the context of conditioning. A second possibility is that reinstatement occurs because presentation of the reinforcer results in conditioning of the context, which then facilitates responding (e.g., Pearce & Hall, 1979; Baker et al., 1991). As noted earlier, prior research indicates that this mechanism is especially important in Pavlovian reinstatement (e.g., Bouton, 1984; Bouton & Bolles, 1979b; Bouton & King, 1983). And consistent with its role in instrumental learning, reinstatement following instrumental extinction depends on the reinforcer being presented in the context of testing and is weakened by introducing sessions in which the animal is exposed to the context repeatedly between re-presentation of the reinforcer and testing (Baker et al., 1991). Like the other recovery effects, reinstatement highlights the fact that extinction is not erasure. And both the discriminative stimulus and contextual conditioning mechanisms implicate a role for context.

Behavioral mechanisms of the contextual control of instrumental extinction

The finding that removal from the extinction context (AAB, ABC) is sufficient for renewal to occur is especially strong evidence that the extinction context somehow inhibits instrumental responding. As in Pavlovian conditioning, there are several ways this inhibition might operate (see Bouton et al., 2011). As noted earlier, one possibility is that during extinction, the organism learns an inhibitory association between the context and the reinforcer. In this case, the context would acquire inhibitory properties akin to those of a conditioned inhibitor and suppress the representation of the reinforcer (e.g., Polack, Laborda, & Miller, 2011). A second possible mechanism is conceptually similar to the well-accepted model of Pavlovian extinction discussed earlier, in which the context, as an occasion setter, activates an inhibitory association between the CS and US (e.g., Bouton, 1997; Bouton & Ricker, 1994). In instrumental extinction, the context might analogously activate an inhibitory association between the response and the reinforcer. Finally, a third possible mechanism is an inhibitory association between the context and the response. According to this mechanism, the extinction context would directly suppress the instrumental response. Rescorla (1993, 1997) has suggested that this type of association is formed between discriminative stimuli and their responses during extinction, although his experiments did not separate the approach from an occasion setting approach (Bouton, 2004; Rescorla, 1993, p. 335; Rescorla, 1997, p. 249).

Based on recent experiments conducted in our laboratory, Todd (2013) has suggested that instrumental extinction may be best characterized by the inhibitory context-response mechanism. In his experiments, renewal was observed when both the extinction and renewal contexts had equivalent reinforcement histories and associative properties (e.g., Bouton & Ricker, 1994; Campese & Delamater, 2013; Delamater, Campese, & Westbrook, 2009; Harris et al., 2000; Rescorla, 2008). For example, in one experiment (Todd, 2013, Experiment 1), rats were first trained to perform one response (R1, lever press or chain pull, counterbalanced) in one context (A), and the other response (R2) in a different context (B). The method ensured that during the acquisition phase, both contexts were equally associated with reinforcement. Next, R1 underwent extinction in Context B, and R2 underwent extinction in Context A. This symmetrical treatment ensured that both Contexts A and B were also equally associated with nonreinforcement. Finally, R1 and R2 were both tested in their extinction and conditioning contexts. There was a clear renewal effect for both responses: R1 was high in A but low in B, whereas R2 was high in B but low in A. Using analogous designs, Todd (2013) also demonstrated AAB and ABC renewal. Because the contexts were equally associated with conditioning and extinction, their direct associations with the reinforcer did not differ and could not produce the differential responding observed during the renewal test.

While the renewal effects observed by Todd (2013) cannot be explained by differential context-reinforcer associations, they can be explained by either the inhibitory context-response or the occasion-setting mechanisms discussed above. According to the inhibitory context-response hypothesis, either response would be released from its inhibition when tested in the other context. According to the occasion-setting hypothesis, either response would be released from hierarchical inhibitory control by its extinction context. One problem for the occasion-setting account, however, is that the effects of negative occasion setters tend to transfer and influence other suitable targets (e.g., Holland & Coldwell, 1993; Morell & Holland, 1993). In Todd’s experiments, such transfer should have reduced any renewal of R1 (or R2) when it was tested in a context that had been associated with extinction of the other response. However, because transfer of occasion setting is often incomplete, some renewal could have still been observed. To further test for the role of negative occasion setting, Todd (2013, Experiment 4) therefore went on to compare the strength of renewal in a group for which such transfer was possible and a group for which it was not. After rats were trained to perform R1 and R2 in Contexts A and B, one group received extinction of R1 in B and R2 in A. Renewal of R2 was then tested in Context B. If extinction results in the context becoming a negative occasion setter, then the fact that R1 had been extinguished in Context B should reduce the size of any renewal effect of R2 there. To test this, a second group received extinction of R2 in A, but extinction of R1 occurred in a separate context. This group was simply exposed to Context B to ensure it was equally familiar. For this group, Context B had not been trained as a negative occasion setter, and any renewal of R2 there would therefore be strong and undiminished. However, renewal testing revealed that R2 was equally renewed in Context B in the two groups. There was thus no evidence that negative occasion setting had been learned. In fact, the experiment also casts further doubt on the idea that the extinction context enters into a direct inhibitory association with the reinforcer. Such an association could have theoretically reduced renewal of R2 due to inhibition of the reinforcer. Overall, the results are most consistent with the idea that inhibition provided by the extinction context is of the form of a simple and direct inhibitory association between the context and a specific response. The animal simply learns not to make a specific response in a specific context. Other recent research has further supported this hypothesis (Todd, Vurbic, & Bouton, submitted).

Performance of the instrumental response itself appears to have a central role in its extinction. We have previously demonstrated in an ABA renewal design that exposure to the renewal context alone, in the absence of the opportunity to perform the response (i.e., the lever was not inserted into the context), does not weaken the strength of ABA renewal (Bouton et al., 2011, Experiment 4). One way to interpret this finding is that Pavlovian excitation conditioned to the context may not play an important role in ABA renewal (exposure to the context alone should result in extinction of this excitation). However, the finding also suggests that in instrumental extinction, extinction learning may depend on the organism actually performing the response. This notion has been elegantly demonstrated by Rescorla (1997) in experiments that manipulated the overall likelihood of two responses during extinction. Rescorla found that extinction was more complete for the response that had been made more frequently, suggesting that performance of the response itself contributes to extinction. In several experiments, Rescorla (2000, Experiment 2; 2006, Experiment 3) also reported that responding during extinction with two discriminative stimuli presented in compound was higher than responding during a third stimulus extinguished alone. However, extinction training with the compound resulted in greater loss of responding when the individual stimuli were later tested. The greater level of responding created by the compound thus allowed extinction to be “deepened.” Similar effects have been demonstrated more recently in rats responding for cocaine reward (Janak, Bowers, & Corbit, 2012; Kearns, Tunstall, & Weiss, 2012). As before, presenting discriminative stimuli in compound caused more responding during extinction than exposure to a discriminative stimulus alone. But when responding was later tested for spontaneous recovery, it was lower in a stimulus that had been extinguished in compound with a second one, relative to a third stimulus that had been extinguished on its own. We should note that high levels of responding in extinction might also reflect a higher expectation of reinforcement, and thus potentially implicate a role for prediction error in instrumental extinction. We have already seen that similar stimulus compounding effects in Pavlovian extinction have been interpreted in terms of prediction error (e.g., Rescorla, 2000, 2003, 2006; see above). Indeed, Rescorla (2006, Experiment 5), successfully separated the role of prediction error from response level in Pavlovian extinction. However, to our knowledge there has been no separation of these possibilities in an instrumental extinction experiment to date.

Similarities and differences between Pavlovian and instrumental extinction

The preceding discussion demonstrates that the rules that summarize Pavlovian extinction provide a useful framework for studying instrumental extinction. That research suggests that the principles established in Pavlovian extinction often do apply to instrumental extinction. However, the research has also uncovered differences. First, the nature of the inhibition learned in instrumental extinction may be different from the inhibition learned in Pavlovian extinction. As noted above, the instrumental learning data have suggested a role for a direct inhibitory context-response association. In contrast, the Pavlovian data are not consistent with this approach (see Harris et al., 2000), and instead implicate negative occasion-setting by the extinction context. In retrospect, it may not be surprising that the response is more the “focus” in instrumental learning; in Pavlovian conditioning, the animal is mainly learning to react or not to a CS. A second difference is that the strength of the operant response decreases when the context is changed after conditioning (e.g., Bouton, Todd, & León, in press), whereas the Pavlovian response typically seems unaffected (e.g., Rosas, Todd, & Bouton, 2013). Once again, the difference may be consistent with the intuition that the response is the focus of instrumental learning. We would emphasize, however, that operant extinction is still more context-specific than operant conditioning—as implied by ABC and AAB renewal, and perhaps the resurgence effect. Like Pavlovian extinction, instrumental extinction does reflect a context-specific inhibitory effect.

A third difference between instrumental and Pavlovian extinction is that the instrumental situation has more moving parts. For example, the presence of explicit conditioned reinforcers (stimuli associated with reinforcer delivery) are always potentially present. Biobehavioral accounts of addiction and instrumental learning have appropriately emphasized the role of both the discriminative stimulus and conditioned reinforcers in instrumental learning (e.g., Everitt & Robbins, 2005; Flagel, Akil, & Robinson, 2009; Milton & Everitt, 2012). But it is worth noting that the presence and use of conditioned reinforcers in experimental procedures in behavioral pharmacology studies is highly variable, and this can complicate interpretation. For example, extinction sometimes involves presentation of the conditioned reinforcer (Bossert, Liu, Lu, & Shaham, 2004; Crombag & Shaham, 2002) and sometimes not (Fuchs, Evans, Parker, & See, 2004; Schwendt, Reichel, & See, 2012). In relapse tests, the conditioned reinforcer is sometimes response-contingent (Chaudhri, Sahaque, & Janak, 2009; Fuchs et al., 2004) and sometimes it is presented noncontingently (Sutton et al., 2003). Unsystematic presentation of the conditioned reinforcer in extinction or “reinstatement” testing will inconsistently introduce Pavlovian processes in addition to instrumental processes. Unfortunately, the term “reinstatement” has also been imprecisely attached to a mixture of phenomena—noncontingent presentation of the conditioned reinforcer, contingent presentation of the conditioned reinforcer, noncontingent presentation of the reinforcer, or testing in the original conditioning context (which learning theorists would call ABA renewal). Given the complexity of instrumental learning methods and its underlying mechanisms, it would make sense to be as precise as possible with how it is described.

Finally, it is important to note that although generalization decrement is likely to play a role in both Pavlovian and instrumental extinction, its role in the extinction of free-operant responding is necessarily large. This is because, in the typical operant arrangement, the delivery of a reinforcer serves as a cue or discriminative stimulus that directly precedes and sets the occasion for the next lever-press response. When the reinforcer is removed in extinction, the response thus loses a direct source of stimulus support. Consistent with this idea, if reinforcers are presented in extinction but not contingent on behavior, the response is slower to extinguish (e.g., Baker, 1990; Rescorla & Skucy, 1969; Winterbauer & Bouton, 2011). Again, a similar effect occurs in Pavlovian extinction (e.g., Frey & Butler, 1977), and reinforcer presentations can have many effects (e.g., Baker, 1990). But one important function of the reinforcer in free operant methods is that it is a stimulus that directly leads to the next response (e.g., Reid, 1958).

Neurobiological mechanisms of instrumental extinction

Instrumental learning and extinction procedures have recently become important tools for investigating the neurobiology of drug-taking and relapse (e.g., Marchant, Li, & Shaham, 2013). The approach has been stimulated and enriched by behavioral work. For example, many studies have examined the contextual control of extinguished operant behavior reinforced by drugs of abuse (see Bouton, Winterbauer, & Vurbic, 2012 and Millan, Marchant, & McNally, 2011 for reviews). The area has the potential to elucidate neural and behavioral mechanisms that might contribute to drug taking behavior.

Instrumental extinction and relapse of drug reinforced behavior

Renewal of extinguished instrumental behavior has been repeatedly demonstrated when responding is first reinforced with drugs of abuse. Crombag and Shaham (2002) were among the first to demonstrate renewal with a drug self-administration paradigm. In their experiment, rats were first reinforced for lever pressing with intravenous administration of a mixture of cocaine and heroin in Context A. After many sessions of extinction in Context B, responding renewed when testing occurred back in Context A. This form of renewal (ABA) has since been demonstrated using a number of drug reinforcers, including alcohol (e.g., Chaudri et al., 2009; Hamlin, Newby, & McNally, 2007; Zironi, Burattini, Aircardi, & Janak, 2006), heroin alone (e.g., Bossert et al., 2004), and cocaine alone (e.g., Hamlin, Clemens, & McNally, 2008). Interestingly, although the other forms of renewal (AAB and ABC) have now been clearly demonstrated with food-reinforced responses (Bouton et al., 2011; Todd, 2013), these forms have yet to be convincingly demonstrated in drug self-administration. Several experiments have failed to demonstrate AAB renewal (Bossert et al., 2004; Crombag & Shaham, 2002) or ABC renewal (Zironi et al., 2006) of responding for drugs. At this point, it is not clear whether the failure to obtain AAB and ABC renewal with drug reinforcers is due to methodological differences or a more fundamental difference between food-pellet and drug reinforcers.

Other behavioral forms of relapse have been studied in drug self-administration experiments. Resurgence has been demonstrated with responding for alcohol (Podlesnik, Jimenez-Gomez, & Shahan, 2006) and cocaine (Quick, Pyszczynski, Colston, & Shahan, 2011). In these experiments, the instrumental response was extinguished while an alternative response was reinforced with food. When this second response was then extinguished, the original drug-seeking response resurged even though the drug remained absent. Reacquisition has in turn been studied with behaviors that have been reinforced with alcohol (e.g., Perry & McNally, 2012; Willcocks & McNally, 2011, 2013). Like food-reinforced behavior, the reacquisition of alcohol-reinforced behavior is modulated by the context. For example, Willcocks and McNally (2011, Experiment 4) first trained rats to nose-poke for alcoholic beer. Then, extinction of responding was conducted in Context B. During a reacquisition test, when nose-pokes were again reinforced with beer, the latency to the first response was less in Context A than in B. This result is similar to the findings of Todd et al. (2012a) described above, where reacquisition was more rapid when it occurred outside the context of extinction (cf. Willcocks & McNally, 2011, Experiment 3). Reinstatement of drug self-administration by noncontingent presentation of drug reinforcers has also been widely demonstrated. For example, extinguished lever pressing that was previously reinforced with heroin or cocaine is reinstated by noncontingent infusion of the drug (Banks, Sprague, Czoty, & Nader, 2008; Botly, Burton, Rizos, & Fletcher, 2008; de Wit & Stewart, 1981, 1983; Weerts, Kaminski, & Griffiths, 1998). Under some conditions, extinguished drug seeking can also be reinstated by stressors such as food deprivation and unsignaled footshock delivered in the test context (Shalev, Highfiled, Yap, & Shaham, 2000).

Neurobiology of instrumental extinction and relapse

There is also a growing body of research in the drug self-administration area that has started to elucidate the neural mechanisms of instrumental extinction and relapse (for reviews see Bossert, Marchant, Calu, & Shaham, 2013; Crombag, Bossert, Koya, Shaham, 2008; Marchant et al., 2013; Peters, Kalivas, & Quirk, 2009; Willcocks & McNally, 2013). The prefrontal cortex is again critically involved. In particular, two regions have been implicated in controlling instrumental extinction performance––the IL mPFC mentioned in our previous discussion of Pavlovian extinction, and the prelimbic area of the medial prefrontal cortex (PL mPFC). These regions are thought to have opposing roles in extinction of conditioned fear (Laurent & Westbrook, 2009; Peters et al., 2009). There is some evidence that they may have opposing roles in extinction of instrumental behavior as well. Recently, Peters and De Vries (2013) have shown that the IL mPFC is critical for instrumental extinction to be learned. Inactivating NMDA receptors prior to extinction of sucrose seeking disrupted later extinction performance. Moreover, Peters, LaLumiere, and Kalivas (2008) have reported that inactivating the IL mPFC following the extinction of cocaine seeking caused an increase in responding; activating this region just prior to relapse (reinstatement) testing reduced the overall strength of response recovery. These data suggest that the IL mPFC somehow inhibits behavior during extinction (see Peters et al., 2009). However, other studies suggest that these effects may depend on the type of reinforcer, or may instead reflect one of several different roles for this region on extinction and retrieval (Bossert et al, 2011, 2013). For instance, Bossert and colleagues (2011) have reported that inactivating a subset of IL neurons reduced renewal of heroin seeking, an inhibition of responding that is not in agreement with its putative role in mediating extinction performance. Those results are instead consistent with the effects of inactivating the PL mPFC, which is known to reduce relapse after extinction (e.g., McFarland & Kalivas, 2001; Willcocks & McNally, 2013). For example, Willcocks and McNally (2013) first trained rats to nosepoke for alcoholic beer in Context A. Responding was extinguished in Context B before being tested in both Contexts. Inactivation of the PL mPFC reduced renewal (the recovery of responding seen in Context A) but had no effect on the expression of extinction (responding in Context B).

Many other brain structures have been implicated in the extinction and relapse of drug seeking. For example, the medial dorsal region of the tuberal hypothalamus (MDH) is thought to exert inhibitory control over extinguished reward seeking (for a review see Marchant, Millan, & McNally, 2012). Along with the IL mPFC and the MDH, the nucleus accumbens shell has been implicated in the inhibition of responding during extinction (for a review see Millan et al., 2011; Millan & McNally, 2011), while the ventral tegmental area (e.g., Bossert et al., 2004), nucleus accumbens core and shell (e.g., Chaudhri et al., 2009; Fuchs, Ramirez, & Bell, 2008; Millan & McNally, 2012), the basolateral amygdala and lateral hypothalamus (e.g., Hamlin et al., 2008; Hamlin et al., 2007), and dorsal hippocampus (Fuchs, Eaddy, Su, & Bell, 2007) have all been related to relapse (renewal) following extinction.

Effect of DCS on instrumental extinction

The many parallels between Pavlovian and instrumental learning have led investigators to ask whether treatments that enhance Pavlovian extinction can also enhance the extinction of instrumental responding. Much research has focused again on the effects of DCS. When delivered systemically, DCS has been shown to facilitate extinction in animal studies using several different reinforcers, such as food (Leslie & Norwood, 2013; Shaw et al., 2009), alcohol (Vengeliene, Kiefer, & Spanagel, 2008), and cocaine (Nic Dhonnchadha et al., 2010; Thanos, Bermeo, Wang, & Volkow, 2011a; Thanos et al., 2011b). More recently, a similar enhancement has been reported when DCS was infused directly into the IL mPFC (Peters & De Vries, 2013). These studies suggest that DCS may have similar actions across different extinction learning paradigms.

As before, however, one critical issue is whether DCS simply facilitates normal extinction learning or fundamentally changes something about it, such as its context-specificity. As described above, studies of fear extinction have shown that DCS-treated animals remain vulnerable to renewal, indicating that extinction learning remains context specific (Bouton et al., 2008; Woods & Bouton, 2006). Moreover, given the general complexity of instrumental learning and the involvement of Pavlovian associations in supporting instrumental behavior, another issue is whether DCS targets extinction of instrumental associations, Pavlovian associations, or both. We have investigated these questions in a series of experiments with rats lever pressing for food pellets (Vurbic, Gold, & Bouton, 2011). After acquisition in Context A, extinction with DCS was conducted in Context B. Rats were then tested in both contexts to determine whether any enhancement of extinction was specific to the context where extinction was learned (i.e., Context B). In contrast to the results cited above, there was no effect of DCS in either context. We noted that in all of the other studies in which DCS was delivered systemically, animals were presented with discrete reinforcer-paired conditioned stimuli (e.g., buzzer sound with concurrent retraction of the lever; Leslie & Norwood, 2013; Shaw et al., 2009), which may have become conditioned reinforcers. These stimuli were also presented during the extinction and test phases. It is therefore possible that DCS facilitated Pavlovian extinction of the conditioned reinforcers rather than any instrumental association. Consistent with this idea, in an experiment by Thanos et al. (2011b), DCS facilitated extinction of lever pressing for cocaine only in mice presented with cocaine-paired cues during extinction. Mice given extinction without cue exposure showed no such effect. In addition, Torregrossa, Sanchez, and Taylor (2010) obtained a DCS effect on extinguished lever pressing when it was delivered immediately following extinction sessions of the cocaine-paired conditioned reinforcers alone.

Although studies of the systemic DCS administration thus point to the possibility that it primarily affects Pavlovian associations during instrumental extinction, the results with localized brain infusions of DCS complicate the picture. Peters and De Vries (2013) recently showed that DCS delivered directly into the IL mPFC enhanced extinction of instrumental behavior without a conditioned reinforcer being present. And interestingly, Torregrossa et al. (2010) had previously found that direct delivery of DCS to the IL mPFC had no effect on extinction of the conditioned reinforcer without the response. Reconciling the various effects of DCS will be an important step in understanding how it might be used in behavioral therapies for drug addiction. At present, studies on the use of DCS during drug cue exposure in humans have had little success (Kamboj et al, 2011; Kamboj, Joye, Das, Gibson, Morgan, & Curran, 2012; Price et al., 2013; Yoon et al., 2013).

In addition to the growing instrumental literature on the effects of DCS, new work has extended the retrieval-extinction procedure that appears to prevent relapse of extinguished fear (Monfils et al., 2009; Schiller et al., 2010). Using a modified version of the Monfils et al. method, Xue et al. (2012) found that a brief 15-min extinction session followed 10 min later by a longer 180-min extinction session prevented spontaneous recovery, reinstatement and renewal in rats self-administering cocaine or heroin.. Control rats were not given the first session, but were instead given longer 195-min sessions of extinction to equate the total time. The results were partially replicated by Millan, Milligan-Saville, and McNally (2013), who used a similar procedure to prevent renewal in rats responding for alcohol. However, Millan et al. also found the same pattern (i.e., a lack of renewal) in a second experiment in which the order of the retrieval and extinction sessions was reversed. Such a result suggests that the effect may be due to mechanisms other than memory retrieval and disruption of reconsolidation. More research will be necessary to understand the findings.

Conclusion

Our review has illustrated a tight coupling between behavior theory and research on the neural mechanisms of extinction. Indeed, it can be said that the study of neurobiological mechanisms of extinction has stood on the shoulders of research that has investigated its behavioral underpinnings. However, we will close by noting that, in all likelihood, the linkage between these levels of analysis will need to be ongoing. As our review indicates, extinction and the many behavioral effects that relate to it can often and in principle be multiply determined. For example, in instrumental extinction, the extinction context alone might directly inhibit the reinforcer, the response, or activate an inhibitory association between the response and reinforcer (cf. Todd, 2013). Or in the ABA renewal of extinguished Pavlovian or instrumental responding, renewal can follow from several theoretical mechanisms, such as the removal of any of the various forms of inhibition present in Context B, or any of several excitatory influences of Context A (e.g, see Nelson, Sanjuan, Vadillo-Ruiz, Pérez, & León, 2011, for a recent discussion of renewal’s complexity). Without careful supporting behavioral investigation, any neurobiological study can fail to determine the precise effect of a neural manipulation. Any structure involved in extinction (such as the IL mPFC) could in principle be involved in any of several possible inhibitory mechanisms. Its inactivation (for example) could therefore result in increased responding through any of them. Likewise, neural manipulations that influence renewal, such as inactivation of the PL-mPFC, are equally open to varieties of interpretation. An accurate and sophisticated understanding of the neural mechanisms behind extinction and the various lapse and relapse effects will require an equally sophisticated understanding of the behavioral mechanisms. Success at that the neurobiological level of analysis may thus always need support from careful and continued work at the behavioral level of analysis.