Occasion setting, inhibition, and the contextual control of extinction in Pavlovian and instrumental (operant) learning

Abstract

An occasion setter is a stimulus that modulates the ability of another stimulus to control behavior. A rich history of experimental investigation has identified several important properties that define occasion setters and the conditions that give rise to occasion setting. In this paper, we first consider the basic hallmarks of occasion setting in Pavlovian conditioning. We then review research that has examined the mechanisms underlying the crucial role of context in Pavlovian and instrumental extinction. In Pavlovian extinction, evidence suggests that the extinction context can function as a negative occasion setter whose role is to disambiguate the current meaning of the conditioned stimulus; the conditioning context can also function as a positive occasion setter. In operant extinction, in contrast, the extinction context may directly inhibit the response, and the conditioning context can directly excite it. We outline and discuss the key results supporting these distinctions.

Research on extinction in both Pavlovian and operant conditioning has revealed that contextual stimuli are important in determining performance in both paradigms (see Vurbic & Bouton, 2014 for review). Although several extinction phenomena are consistent with this conclusion (e.g., reinstatement, spontaneous recovery, rapid reacquisition, and resurgence), the most straightforward illustration of it is the renewal effect (Bouton & Bolles, 1979a). In Pavlovian conditioning, after the conditioned stimulus (CS) and unconditioned stimulus (US) are paired in one context (typically referred to as Context A), extinction presentations of the CS without the US are given to eliminate conditioned responding. The response to the CS often returns (or is renewed) when the CS is tested outside of the extinction context. Renewal can occur in several forms. In ABA renewal, responding returns in the original conditioning context after extinction has occurred in a second one. In ABC renewal, responding returns in a third context after extinction has occurred in a second one. In AAB renewal, responding returns when the CS is tested in a new context after extinction has occurred in the original conditioning context. Although the ABA design often produces the strongest renewal effect (Thomas, Larsen, & Ayres, 2003), the fact that it occurs in the ABC and AAB designs suggests that a change of context after extinction is sufficient to produce the effect. Importantly, more recent research in operant conditioning suggests that one can observe a very similar pattern there. That is, although an operant response reinforced with an outcome decreases in strength when the outcome is removed, ABA, ABC, and AAB forms of renewal have all been observed (Bouton, Todd, Vurbic, & Winterbauer, 2011; Todd, 2013). As a first approximation, one might claim that both Pavlovian and operant extinction do not erase the original learning, but involve new learning that leaves the CS or the response “ambiguous” and sensitive to the context. This view of extinction is now reasonably well accepted in the field (e.g., Myers & Davis, 2002, 2007; Quirk & Mueller, 2008; Maren, Phan, & Liberzon, 2013; Vervliet, Craske, & Hermans, 2013).

This article is concerned with a more detailed understanding of the behavioral mechanisms engaged in contextual control of Pavlovian and operant extinction. Most research on this problem has suggested that contexts often (and perhaps usually) operate in a way that does not depend on their direct excitatory or inhibitory associations with the US (e.g., Bouton, 2004). In Pavlovian extinction, it has been argued that the contexts instead operate through occasion setting (e.g., Bouton, 1993, 2004). In the present article, after briefly reviewing the main hallmarks of occasion setting, we focus on the contextual control of extinction of Pavlovian and operant responding. Although there are many similarities between these forms of learning, there are differences, too. We suggest that, whereas the suppression of responding that results from Pavlovian extinction involves negative occasion setting by the extinction context, the suppression of responding that occurs in operant extinction may primarily involve a more direct form of inhibition of the response by the context.

Occasion setting in Pavlovian preparations

Several lines of evidence suggest that a CS often elicits the conditioned response through its direct association with the US. For example, Rescorla (1973) paired a light CS with a loud noise US in a conditioned suppression procedure. One group of rats then received habituation trials with the noise US, in which the noise was presented on its own repeatedly (habituating the response to the US). Each group then received a test with presentations of the CS. For the control group (which did not receive US habituation), presentations of the light elicited a strong conditioned response (CR), but this was diminished in the habituated group. Thus, the CR reflected the current value of the US and not the value learned during conditioning (as would be expected if the CS directly elicited the conditioned response; see also Rescorla, 1974). Other work by Holland and Rescorla (1975) showed a reduction in responding to a CS following devaluation of a food US with satiety or after pairing it with high-speed rotation.

Although such research suggests that CSs can control responding by activating a representation of the US, stimuli can also modulate or influence responding to another CS that has its own association with the US. In a most basic way, the modulating stimulus can provide information about whether or not the CS will be followed by the US. Whereas a CS may be better suited to indicate when the US is coming, a modulating stimulus, or occasion setter, indicates whether the US is coming (e.g., Bouton, 1997; Holland, 1992). A typical occasion setting experiment consists of Pavlovian training of a target CS (A) followed by a US (+) only if a feature stimulus (X) is also present (XA+). The US is not presented when the feature is absent (A-). This describes a feature-positive (FP) discrimination; X signals that the A-US relationship will be reinforced (A−, XA+). Situations in which a target CS will not be followed by the US if it is preceded by a feature stimulus (A+, XA−) are called feature-negative (FN) discriminations. According to traditional elemental theories of learning (e.g., Rescorla & Wagner, 1972), these discriminations should result in X acquiring its own excitatory (FP) or inhibitory (FN) association with the US. However, several lines of evidence suggest that occasion setters do not operate through that mechanism.

First, the nature of the target, and not the feature, controls the form of the conditioned response in an occasion setting paradigm. In early experiments, Ross and Holland (1981) used visual (light) and auditory (tone) stimuli to differentiate between the occasion setting and more traditional (e.g., Rescorla-Wagner, 1972) explanations of FP learning. Their method utilized the fact that the form of the CR depends on the modality of the CS in appetitive Pavlovian conditioning (as described by Holland, 1977). If the Rescorla-Wagner explanation is correct, then conditioned responding in a FP discrimination should reflect excitatory learning to X; the form of the CR should be dictated by X, and not A. However, if X modulates the A-US association, then the form of the CR should reflect control by A. One group of rats received FP training in which the US followed a simultaneous presentation of XA and not A presented alone. A second group received similar training with serial presentations of X and A. In a test, rats that received XA in simultaneous compound acquired an excitatory CR to X (i.e., if X was a light, rats reared to the compound, and if X was a tone, rats displayed head-jerking). In contrast, rats that received serial X-A acquired an excitatory CR to A, but not to X (i.e., if A was a light, rats reared to A only when X preceded it, and when A was a tone, rats displayed head-jerking). Rescorla (1985) reported complementary results with pigeon autoshaping: Diffuse noise and light stimuli could serve as modulators that excited or inhibited a keypecking response elicited by the keylight CS, even though they did not elicit pecking on their own. The form of the CR thus suggested that the feature stimulus (X) works by enabling the association between the target (A) and the US. While it is possible that the modulating feature stimulus might evoke an (unmeasured) motivational state that influences responding to a subsequent CS (e.g., Wagner & Brandon, 1989), such a possibility seems unlikely when the other hallmarks of occasion setting (below) are also considered.

A second hallmark of occasion setting is that altering the feature’s association with the US does not influence its ability to modulate responding to a target. Extinction (nonreinforced presentations of the feature alone) does little to change the ability of a positive occasion setter to modulate responding (Ross & Holland, 1981). Analogous findings have been shown in feature-negative discriminations (Holland, 1984), during which animals received reinforced presentations of X (X+) following acquisition of a FN discrimination (e.g., A+, X→A−). Again, changing X’s association with the US did not eliminate its ability to modulate responding to A. Such results suggest that an occasion setter might influence responding to the target through a mechanism that is independent of its direct association with the US.

Third, occasion setters often do not summate to produce larger or smaller CRs when presented in compound with CSs other than the ones with which they have been trained. As predicted by most learning theories, presenting an excitatory or inhibitory CS in compound with a CS that predicts the same US should affect responding (Rescorla, 1969; Rescorla & Coldwell, 1995). In contrast, occasion setters trained in both FP (Holland, 1986) and FN (Holland, 1989b) discriminations do little to excite or inhibit responding to separately-trained CSs. However, features can “transfer” and serve as effective occasion setters for other CSs if they have been trained as targets in similar discriminations (Holland, 1986; Rescorla, 1985). Although this transfer is not always complete (Holland, 1989b), a feature trained in a FP discrimination will modulate the CR to targets of other FP discriminations (but not to targets of separate FN discriminations). The same is true of features trained in FN discriminations. There is also evidence that transfer can occur to extinguished Pavlovian CSs in both FN and FP paradigms (e.g., Swartzentruber & Rescorla, 1994), suggesting that a target may mainly need to have both excitatory and inhibitory relations with the US to be modulated by an occasion setter.

Conditions that permit occasion setting to develop arrange the target stimulus to be more salient than the feature stimulus. For example, as previously noted, early work suggested that serial presentations of feature and target produced more occasion setting than did simultaneous presentations (Ross & Holland, 1981; Ross, 1983). One consequence of serial presentation is that the feature has a trace-conditioning relation with the US, and may thus be a weaker predictor of the US than is the contiguous target. But serial presentation is not necessary to permit occasion setting; as Holland (1989a) demonstrated, occasion setting may also develop in simultaneous compounds if the feature is sufficiently less salient than the target. In his Experiment 1, rats were trained with a simultaneous tone-light compound. Occasion setting to the feature light developed when the target was a relatively loud, 88db noise (and thus sufficiently more salient than the light) but not when the noise was 78db. One consequence of procedures in which the target is more salient than the feature is that the target acquires a relatively strong excitatory association with the US when it is reinforced and can therefore acquire a stronger inhibitory association when it is nonreinforced (e.g., Rescorla, 1988). This should yield better occasion setting according to the view that the occasion setter works primarily by modulating the target’s inhibitory association with the US (Rescorla, 1985; see also Bouton, 1997, 2007; Bouton & Nelson, 1994).

Contextual control of Pavlovian extinction: The role of negative occasion setting

The evidence suggests that Pavlovian extinction results in new learning [that a CS no longer predicts the US (CS-no US); Pearce & Hall, 1980]. This new learning is highly dependent on the context for expression (e.g., Bouton, 1993, 2004). Research in Pavlovian fear and appetitive conditioning documents several sources of response recovery (relapse) after extinction that support this view (see Bouton, 2004; Vurbic & Bouton, 2014 for reviews). As noted earlier, the renewal effect is an especially clear and well-understood illustration of the role of context in controlling performance in extinction. Notice, though, that the basic renewal effect is not necessarily incompatible with familiar accounts of conditioning (e.g., Rescorla & Wagner, 1972): In an ABA renewal design, renewal might occur because Context A might signal the reinforcer (acquire an excitatory association with the US), Context B might signal the absence of the reinforcer (acquire an inhibitory association with the US), or both. The fact that renewal occurs in the ABC and AAB designs emphasizes a role for inhibition in the extinction context (but of course does not rule out a role for excitation in A in the ABA design). But like simple FP and FN discriminations themselves, there is nothing inherent in a description of the renewal procedure that demands an occasion-setting account (e.g., Delamater & Westbrook, 2014). Nonetheless, a classic Rescorla-Wagner account has rarely (if ever) stood up to empirical tests.

It is worth noting that the history of research on contextual control of Pavlovian extinction paralleled, rather than followed, the seminal work on modulation or occasion-setting processes by Holland and Rescorla in the 1980s. However, from the start, several lines of evidence suggested that the contexts were not influencing responding to the CS as simple excitors or inhibitors. Indeed, the research suggested that demonstrable and direct associations between a context and a US were neither necessary nor sufficient for the context to control responding to the CS (see Bouton & Swartzentruber, 1986, Bouton, 1993 for early reviews). This is exemplified by one of the earliest reports (Bouton & Bolles, 1979a). When the CS was nonreinforced in Context B prior to its conditioning in Context A, Context B was found to reduce responding to the CS relative to a condition in which pre-exposures to the CS occurred in another context (see also Bouton & Swartzentruber, 1989; Westbrook, Jones, Bailey, & Harris, 2000; Holmes & Westbrook, 2014). These demonstrations are important, because conditioning models like the Rescorla-Wagner model do not allow inhibitory associations to develop before the CS is conditioned, as prior conditioning is required to generate the necessary prediction error.

In other experiments, Bouton and King (1983; see also Bouton & Swartzentruber, 1989) found no evidence of summation in ABA renewal experiments that included summation tests for Context-US inhibitory associations. That is, there was no observable inhibition of responding to a nonextinguished CS when it was tested in Context B (where another CS had been extinguished). Bouton and Swartzentruber (1986) extended the analysis by investigating summation in a procedure developed to produce a strong inhibitory Context B-US association. In their experiments, training consisted of extended intermixed sessions in which a tone (T) was reinforced in Context A and nonreinforced in Context B (AT+/BT−). Although theories of conditioning predict that this procedure should endow Context B with more inhibition than simple extinction would (because T is repeatedly conditioned in A and returned to B with an ability to create more negative prediction error), Context B still failed summation tests (i.e., Context B did not inhibit responding to other CSs; Experiment 1) as well as retardation tests (Context B was readily conditioned as an excitor; Experiment 2) and supernormal conditioning tests (i.e., conditioning of a new CS in Context B proceeded normally; Experiment 1). Although Context B was repeatedly shown to modulate (suppress) responding to T, there was no evidence to support the hypothesis that it had the properties of a conditioned inhibitor.

Other explanations can account for the lack of conditioned inhibition by a context that demonstrably suppresses conditioned responding. For instance, weaker responding in Context B might be explained by a priming account that assumes that nonreinforced exposure to the CS in Context B results in a Context B-CS association that might decrease the ability of the CS to elicit a response (Hall & Mondragón, 1998; e.g., Wagner, 1978, 1981). However, this account does not anticipate renewal effects after counterconditioning (e.g., Holmes, Leung, & Westbrook, 2016; Peck & Bouton, 1990), where the CS is paired with shock in Context A and then food in Context B (or the reverse). For example, in the shock→food case, appetitive responding replaces fear to the CS in Context B, but fear of the CS is renewed when the CS is returned and tested Context A. Thus, Context B essentially suppresses Phase 1 performance at the same time it occasions Phase 2 performance, suggesting adequate CS processing during Phase 2. Another possibility is that extinction results in conditioned inhibition in the context that is difficult to detect through traditional summation tests because of a decrease in attention to the context during extinction (Larrauri & Schmajuk, 2008). However, we are not aware of any empirical support for this view.

Meanwhile, tests of Context A similarly suggested that demonstrable excitation in A was not necessary in order to observe positive modulation by Context A relative to Context C (Bouton & Swartzentruber, 1986). Bouton and King (1983) assessed whether excitatory Context A-US associations developed in an ABA renewal design using context preference tests, a measure that detects levels of context-elicited fear that were not measured by simple tests of baseline response suppression (e.g., Bouton, 1984; Bouton & King, 1983). Using their method, there was no evidence of excitatory Context A-US associations in any behavioral observations (e.g., freezing or context-preference tests). Moreover, the surprisingly complete transfer of excitatory responding to a CS to a new context (i.e., from Context A to Context B) in both fear-conditioning and appetitive conditioning paradigms (e.g., Bouton & King, 1983; Bouton & Peck, 1989) further suggested that Context A does not enter into a direct excitatory association with the US. This is because, if Context A is associated with the US, and this context-US association summates with the CS-US association (as a Rescorla-Wagner analysis would require), then there should be a response decrement when the CS is first tested in a neutral context. Experiments have also shown that attempts to remove Context A-US excitation (through separate extinction of Context A) do not weaken ABA renewal (Bouton & King, 1983; Bouton & Peck, 1989). Such results may provide a direct analogue to research suggesting that a positive occasion setter is not damaged by extinction of its positive association with the US (Rescorla, 1986).

Other results suggested that strong and demonstrable associations between a context and a US were not sufficient to augment responding to the CS (Bouton, 1984; Bouton & King, 1986). Specifically, when shock USs were presented after fear conditioning, they created strong and demonstrable contextual conditioning as measured by context-preference tests. However, unless the CS had previously been through extinction and was under the influence of inhibition, contextual fear had no demonstrable effect on performance to the CS when it was subsequently tested (Bouton, 1984; Bouton & King, 1986). Whether a conditioned context will augment responding to an extinguished CS depends on the degree to which the CS is otherwise under the influence of extinction during testing, regardless of whether the procedure contained occasional nonreinforced CSs or not (Bouton & King, 1986). Bouton (1984; Experiment 5) further found that when responding to a CS was brought to the same point on the response scale through either conditioning and partial extinction or weak conditioning, demonstrable contextual conditioning specifically modulated the extinguished CS, and not the weakly conditioned CS.

The results demonstrating that context-US associations are neither necessary nor sufficient to affect responding to a CS suggest that contexts may modulate performance by some other mechanism, like occasion setting. Indeed, there are also parallels between contexts and the hallmarks of occasion setters described above. First, regarding response form, it appears that Context A can modulate (turn on) responding to a CS even when it does not elicit a CR on its own. In conditioned suppression experiments, there is little suppression to the context in the form of baseline suppression (Bouton & King, 1983); and in appetitive conditioning studies that measured conditioned head-jerking to an auditory CS (Bouton & Peck, 1989), the context modulated head-jerking to the CS even though there is little evidence that the context itself supports head-jerking. Rescorla (2008) similarly demonstrated that while contexts control the ability of key lights to elicit key pecking in the various forms of renewal (ABA, AAB, and ABC), contexts do not elicit key pecking on their own. Overall, the results suggest that contexts, like occasion setters, modulate the form of the response that is controlled by the CS.

Regarding transfer and summation, the various summation tests of excitation and inhibition in B suggest that the contexts do not transfer and influence responding to other CSs (e.g., Bouton & King, 1983; Bouton & Swartzentruber, 1986, 1989). In a further parallel, recall that occasion setting can transfer to CSs that have been targets of a separately trained occasion setting relation (Davidson & Rescorla, 1986; Lamarre & Holland, 1987). Swartzentruber and Bouton (1988) found that this was also true of contexts. Specifically, they trained a context discrimination in which rats received two CSs (tone and light-off) differentially reinforced in four contexts. Rats received tone-US pairings in Context A, and tone alone presentations in Context B. They received similar treatment of the light-off CS in Contexts C and D. After acquiring this context discrimination, rats demonstrated almost perfect transfer of responding to the light-off CS in Context A. The transfer result, replicated and extended by Swartzentruber (1991), suggested that contexts, like occasion setters, can function to set the occasion for responding to a CS trained in a similar discrimination as the original CS.

Other experiments have tested whether modulation by context can survive separate manipulation of the context’s association with the US. We have already mentioned that extinction of Context A does not weaken ABA renewal (Bouton & King, 1983; Bouton & Peck, 1989), a direct analogue to similar findings in positive occasion setting (Rescorla, 1986). Perhaps inconsistently, reinforcement of Context B, the extinction context, does affect its ability to suppress responding to the CS—conditioning of Context B causes “reinstatement” of responding to the CS when the CS is again tested there (e.g., Bouton, 1984; Bouton & Bolles, 1979b; Bouton & King, 1983; Holmes & Westbrook, 2013; Rescorla, Durlach, & Grau, 1985). However, Rescorla (1985) demonstrated that inhibitory control by a negative feature can likewise be abolished by pairing it with the US; importantly, though, inhibitory control was then restored if the feature was repeatedly nonreinforced. In a similar way, nonreinforced exposures to the reinstatement context abolishes reinstatement, and evidently restores extinction performance in that context (e.g., Bouton & Bolles, 1979b). Note that a conditioning model like the Rescorla-Wagner model would predict that the conditioning of a negative feature or context would permanently destroy its modulatory ability, because it would change the only product of conditioning (namely, associative strength). (Note also that Rescorla’s [1985] finding concerning the injurious effects of reinforcing the negative feature after discrimination training should not be confused with the fact that reinforcing a negative feature during discrimination training can enhance discrimination learning by increasing attention or processing of the feature during training [Rescorla, 1991]).

Swartzentruber (1991) demonstrated that contexts can acquire a modulatory function that competes with modulation performed by a discrete occasion setter. Using a blocking design (e.g., Kamin, 1969), he showed that prior training of a discrete occasion setter blocked contextual control over excitatory responding to a CS when the context and the occasion setter redundantly signaled reinforcement of the CS. (An excitatory stimulus produced no such blocking.) In addition, positive occasion setting by a discrete stimulus was conversely blocked by a context when training followed acquisition of excitatory contextual control. Thus, in Pavlovian discriminations, the context can acquire a modulatory function that is redundant to that of a positive occasion setter.

It is worth noting that contexts usually have a temporal relationship with CSs that may encourage occasion setting by making the context less salient than the CS itself (cf. Holland, 1989a). In most standard conditioning preparations, the onset of the context occurs long before any CS presentation, and there are lengthy intervals of context exposure during the intertrial intervals (ITIs) between successive CS presentations; such exposure to the context should reduce its salience (e.g., Darby & Pearce, 1985). For example, in conditioned suppression (e.g., Bouton, 1984; Bouton & King, 1983, 1986; Bouton & Swartzentruber, 1986, 1989; Rescorla, 1973, 1974), where the CS is often 60-120 s in duration, the first CS is often not presented until at least 20 mins into the session, and ITIs can average about 20 minutes; in appetitive conditioning (e.g., Bouton & Peck, 1989; Brooks & Bouton, 1993), the CS is 10-30 s in duration, with the first trial occurring after about 6 min in the context and ITIs of about 4 min; and in pigeon autoshaping (e.g., Rescorla, 2008), the CS is often 5 s, and the first CS occurs at about 1 min into the session with ITIs of about 1 min. These characteristic temporal relationships may be important, because the development of occasion setting (as opposed to feature excitation or inhibition) is evident with procedures in which target onset is delayed after feature onset by only 10 s or so (e.g., Rescorla, 1985: Ross & Holland, 1981). Further consistent with this perspective, what evidence there is of summation between CS and context is apparently restricted to methods in which CS and context onsets are essentially simultaneous. Miller, Grahame, and Hallam (1990) found evidence of CS-context summation only when CS onset occurred at zero or 5 s into a test session; insertion of a brief delay (30 s) between placement of the rat in the context (conditioning chamber) and presenting the CS abolished that summation. Perhaps also consistent, Polack, Laborda, and Miller (2012) found evidence of contextual inhibition after Pavlovian extinction when they used temporal relationships that could have made the context especially salient. When rats received fear conditioning in Context A and then extinction in Context B, Context B developed inhibition (confirmed with summation or retardation tests) when 30-s CS presentations were separated by extremely short ITIs (i.e., 6 s), and when CS onset was simultaneous with placement in the context during the summation test (as in Miller et al., 1990). It thus seems likely that, as is true of tones and lights in occasion setting experiments (e.g., Ross & Holland, 1981), a context can function as either a CS or an occasion setter depending on the circumstances. However, under ordinary conditions, in which the context precedes (and temporally surrounds or embeds) the CS, occasion-setting mechanisms may well prevail.

Contextual control of operant extinction: A possible role for direct response inhibition

As we noted at the beginning of this article, operant extinction, like Pavlovian extinction, is also strongly influenced by the context. Extinguished operant responding also readily renews when tested outside of the extinction context; similar to Pavlovian extinction, ABA, AAB, and ABC renewal are all evident following operant extinction (Bouton, et al., 2011; Nakajima, Tanaka, Urushihara, & Imada, 2000; Todd, 2013). Given this similarity, one might assume that the context also functions as a negative occasion setter in operant extinction. However, recent research suggests that context does not necessarily function as a negative occasion setter in operant extinction, and may instead inhibit the response more directly.

Todd (2013) assessed the role of negative occasion setting by the context in operant extinction. He first found that ABA renewal was evident when the associative histories of Contexts A and B were matched by training and extinguishing separate responses in each (i.e., a lever press response was reinforced in A, extinguished in B, then tested in A, while a chain pull response was complementarily trained in B, extinguished in A, then tested in B). Subsequent experiments demonstrated that AAB and ABC renewal were also evident following similar procedures that matched reinforcement and extinction histories of the testing contexts. Renewal was thus demonstrated in each case when all context-outcome associations were equated. Such results suggest that differences in associative strength between the contexts (i.e., simple excitatory or inhibitory context-reinforcer associations) are not required to produce the renewal effect. They also begin to question an occasion setting account, because some degree of transfer should have been evident if contexts had become occasion setters; they should have been able to control a second target (response) that had been trained similarly (Holland, 1986; Rescorla, 1985). However, the results could have been consistent with an occasion setting account if that transfer had been only incomplete or partial (Holland & Coldwell, 1993; Morell & Holland, 1993).

To further test a role for occasion setting, Todd (2013, Experiment 4) therefore asked whether one could find any evidence of such transfer (see Table 1 for the experimental design). He first trained rats on two operant responses, R1 and R2 (lever press and chain pull, counterbalanced), in two contexts (A: R1+; B: R2+). Once each response was established, all rats received extinction of R2 in Context A (A: R2−). However, for one group, Group Ext B, R1 was extinguished in Context B (B: R1−), and for a second group, Group Ext C, R1 was extinguished in a third context, Context C (C: R1−). Each group was then tested for R2 responding in both Contexts A and B. In this design, transfer of negative occasion setting predicts that renewal of R2 responding (in Context B) should be weaker in Group Ext B than Group Ext C, because Context B had potentially served as a negative occasion setter for R1 only in that group. However, the results of the experiment were inconsistent with a role for negative occasion setting. Both groups showed robust and equivalent renewal of R2 when tested in Context B. Extinction of R1 in Context B thus did not lead to the ability of Context B to modulate the inhibition of the target (R2) of another negative feature (Context A). (The result also further challenges the possibility that the animals learned that B was a simple inhibitor for the reinforcer, because this would also predict that Group Ext B’s Context B would inhibit R2.) The results instead suggested that the rats had learned to inhibit each specific response in a specific context. To explain the lack of transfer of contextual control across responses, one might argue that there may be less inherent generalization between responses than there is between different auditory and visual targets used in typical occasion setting experiments. However, the similar treatments of the two responses should have encouraged generalization between them via acquired equivalence (e.g., Hall, 1996). A role for acquired equivalence in enabling transfer of occasion setting across targets has been emphasized by Bonardi (1998).

Table 1.

Experimental design of Experiment 4 from Todd (2013).

Group	Acquisition	Extinction	Test 1	Test 2
Ext-B	A:R1+ B:R2+ C: −	A:R2− B:R1− C: −	A:R2−; B:R2−?	A:R2−; C:R2−?
Ext-C		A:R2− B: − C: R1−	A:R2−; B:R2−?	A:R2−; C:R2−?

A, B, and C refer to contexts. R1, R2: lever press and chain pull responses, counterbalanced. +/− : presence or absence of a reinforcer, respectively. − : context exposure without the lever or chain present.

A series of recent experiments has produced more direct evidence suggesting response inhibition in instrumental extinction (Bouton, Trask, & Carranza-Jasso, 2016). An initial experiment showed that making the response in extinction was necessary for effective operant extinction to occur (Bouton et al., 2016, Experiment 1). Rats acquired a discriminated operant response (in which a response was only reinforced in the presence of a discriminative stimulus, S). Groups then received either extinction of the response in S (Group SR), presentations of the S without the opportunity to make the response (Group S; the response manipulandum was removed from the chamber), or exposure to the chamber without the response or S (Control). Next, all groups were tested with the response and the S in extinction. Group SR showed little evidence of responding, whereas groups S and Control showed robust and equivalent responding. Thus, nonreinforced presentations of the S without the opportunity to make the response did not weaken the response (see also Thrailkill & Bouton, 2015a, 2016). This result is not consistent with theories suggesting a role for Pavlovian motivational support of instrumental responding (e.g., two-process theory, e.g. Rescorla & Solomon, 1967), because weakening S-O alone did not affect operant responding. Instead, the rats needed an opportunity to make the response for it to be weakened during extinction. A second experiment (Bouton et al., 2016, Experiment 2) replicated the result and also demonstrated that extinction of one response (e.g., chain pulling) had no discernible impact on a second response (e.g., lever pressing) that had also been reinforced in the same stimulus. Thus, nonreinforcement of the response was necessary to produce extinction, and it suppressed only that specific response.

Another experiment (Bouton, et al., 2016; Experiment 4) asked whether learning to inhibit a response in one stimulus also suppressed it in another stimulus (see Table 2 for the experimental design). If the animal learned a general form of response inhibition, or if that inhibition were controlled by the background context (as suggested above), then one would expect response inhibition to transfer between separate stimuli trained and tested in the same context. Rats first acquired two responses, R1 and R2, each in the presence of two distinct stimuli (e.g., S1R1, S2R1, S3R2, and S4R2). Groups then received one of four different extinction treatments: Group S1R1 received extinction of R1 in the presence of S1, Group S1 received extinction exposures to S1 without the opportunity to make R1 (the manipulandum was removed from the chamber), Group R1 did not receive S1 but could otherwise make R1 responses (nonreinforced), and a control group was placed in the chamber without responses or stimuli. Next, each group received a test of R1 in S2 and of R2 responding in S3. The test results (shown in Figure 1) clearly indicated suppressed R1 during S2 in Group S1R1, and no response inhibition in the S3R2 test. The pattern suggests that Group S1R1 learned to inhibit R1 during extinction, and that the inhibition was also manifest in another discriminative stimulus controlling R1 but not to a stimulus that controlled a different response. These results expand upon Rescorla’s (1993, 1997) earlier work suggesting a role for response inhibition by providing evidence that inhibition of a response can transfer across separate discriminative stimuli.

Table 2.

Experimental design of Experiment 4 from Bouton et al. (2016).

Group	Acquisition	Extinction	Test
S1R1	S1R1+ S2R1+ S3R2+ S4R2+	S1R1−	S2R1−? S3R2−?
S1		S1−
R1		R1−
Control		−

S1, S2, S3, S4: 30-sec tone, click, flashing light, and continuous light Ss, counterbalanced. R1, R2: lever press and chain pull responses, counterbalanced. +/− : presence or absence of a reinforcer, respectively. − : context exposure without the lever or chain present.

Figure 1.

Results from the testing phase of Experiment 4 from Bouton et al. (2016). See text for details.

In a final experiment, Bouton et al. (2016, Experiment 5) examined the nature of the response inhibition further (see Table 3 for the experimental design). Rats first acquired the same response (a lever press) in the presence of S1 and S2. However, each S signaled a different outcome (O), such that R produced O1 (e.g., a grain pellet) in S1, and R produced O2 (e.g., a sucrose pellet) in S2 [i.e., S1-(R-O1) and S2-(R-O2)]. Following acquisition, Group S1R received extinction of R in S1, Group S1 received S1 with R absent, and Group R had R available but did not receive either S. As before, the final control group received exposure to the chamber without the R or S present. Next, groups were tested on R with S1 and S2 in extinction. If animals learn to inhibit the response specifically, then Group S1R should suppress responding during both S1 and S2 in the tests. However, if S signals inhibition of a specific R–O relationship, then Group S1R should suppress responding during S1, but not S2. Only Group S1R demonstrated any effect of extinction in the test, and importantly, the response was inhibited in both S1 and S2. Note that S1 and S2 were from different modalities (auditory and visual, counterbalanced), making simple stimulus generalization a less plausible explanation of transfer. There were no other group differences in the test. The results suggest that inhibition of R in S1 transferred to S2 despite S2’s connection with a different reinforcer and a different S-(R-O) relationship. The pattern, again, is consistent with the hypothesis that the animal learned to inhibit its response during extinction. However, it should be noted that the response was less suppressed in the S2 stimulus than in S1. Thus, while a substantial amount of the inhibition learned during extinction of S1R transferred to a new stimulus that occasioned the same response, inhibition was most complete in the extinguished stimulus (e.g., Rescorla, 1993; see also Bonardi, 1989).

Table 3.

Experimental design of Experiment 5 from Bouton et al. (2016).

Group	Acquisition	Extinction	Test
S1R	S1R-O1 S2R-O2	S1R−	S1R−? S2R−?
S1		S1−
R		R−
Control		−

S1, S2: 30-sec tone and continuous light Ss, counterbalanced. R1, R2: lever press and chain pull response manipulanda, counterbalanced. O1, O2: grain and sucrose pellets, counterbalanced. +/−: presence or absence of a reinforcer, respectively. − : context exposure without the lever or chain present.

Todd, Vurbic, and Bouton (2014) reported evidence that the response inhibition suggested above is indeed controlled by the context. The design of their Experiment 3 is summarized in Table 4. Rats acquired two discriminated operant responses in Context A (S1R1 and S2R2), and two other SR combinations that included the same Rs occasioned by different Ss each in a different context (S3R1 in Context B and S4R2 in Context C). Each SR combination was then extinguished in its respective acquisition context. In subsequent tests for AAB renewal, rats received S1R1 and S2R2 in Contexts A and B, and A and C. If extinction results in inhibition of a specific response in a specific context, then AAB renewal should be weaker for S1R1 than S2R2 in Context B, and weaker for S2R2 than S1R1 in Context C, because of the prior extinction treatments in the two contexts. This prediction was confirmed. In each test, rats suppressed the response that had been previously extinguished in the test context, and renewal was observed when each response was tested outside its extinction context. Therefore, extinction seems to involve learning to suppress a specific response in a specific context, even when response inhibition occurred in the presence of a different S. Context-specific response inhibition may thus play a role in the extinction of operant responding.

Table 4.

Experimental design of Experiment 3 from Todd et al. (2014).

Group	Acquisition	Extinction	Test
AAB	A:S1R1+, S2R2+ B:S3R1+ C:S4R2+	A:S1R1−, S2R2− B:S3R1− C:S4R2−	A:S1R1− vs B:S1R1−? A:S2R2− vs B:S2R2−? and A:S1R1− vs C:S1R1−? A:S2R2− vs C:S2R2−?

A, B, and C refer to contexts. R1, R2: lever press and chain pull responses, counterbalanced. S1, S2, S3, S4: 30-sec tone, click, flashing light, and continuous light Ss, counterbalanced. +/− : presence or absence of a reinforcer, respectively.

Contextual control of the nonextinguished operant response

It is worth noting that, in addition to its role in extinction, the context plays a surprisingly direct role in controlling operant behavior after acquisition. A number of recent experiments have demonstrated that the operant response is reliably weakened when it is tested in a new context (e.g., Bouton et al., 2011; Bouton, Todd, & Léon, 2014; Todd, 2013). Control of operant responding by the acquisition context contrasts with many failures in our hands to observe effects of changing the context after appetitive and aversive Pavlovian learning (e.g., Bouton, Frohardt, Sunsay, Waddell, & Morris, 2008; Bouton & King, 1983; Bouton & Peck, 1989; Bouton & Sunsay, 2001; Bouton & Swartzentruber, 1986; Brooks & Bouton, 1993; Swartzentruber & Bouton, 1986). The relative lack of a context switch effect on Pavlovian acquisition learning appears general across laboratories (Bevins & Ayres, 1992; Grahame, Hallam, Geier, & Miller, 1990; Harris, Jones, Bailey, & Westbrook, 2000; Kaye & Mackintosh, 1990; Leaton, 1974; Lovibond, Preston, & Mackintosh, 1984; Neumann, 2006; Thomas, et al., 2003). For completeness, however, we note that exceptions can be found under certain conditions (see Hall & Mondragón, 1998, for more discussion).

Thrailkill and Bouton (2015b) studied the mechanism through which the context controls the nonextinguished operant response. In their experiments, rats acquired free-operant lever press responding with relatively few (i.e., 90) or relatively many (i.e., 360) response-outcome pairings. Rats then received a reinforcer devaluation treatment [either pairings of the outcome (food pellet) with illness from a lithium chloride injection, or unpaired presentation of the outcome and lithium chloride; Adams, 1982]. Once the animals rejected the pellets when offered, operant responding was tested without the pellet in the acquisition context, Context A, and in a second context, Context B. The results of these tests are presented in Figure 2. Reinforcer devaluation had no effect on lever pressing in rats trained with 360 response-outcome pairings. Even though this group rejected the outcome, their responding was similar to a group that had received unpaired presentations of the outcome and illness. Such insensitivity to reinforcer devaluation suggests that the response had become a habit. As defined by Dickinson (1985), habits are responses under the control of an S-R association, and are thus not immediately sensitive to the value of the outcome (i.e., the outcome does not participate in the prevailing association). In contrast, reinforcer devaluation had a strong effect on responding in the group that received few response-outcome pairings (responding was lower in the paired group than the unpaired group). After 90 response-outcome pairings, responding was sensitive to a change in the value of the outcome, and met the criterion for an action (i.e., a response under control of the outcome representation, or R-O association; Dickinson, 1985). The test in Context B, however, further revealed a decrement in the response in all groups. The status of the outcome (valued or devalued) and the number of response-outcome pairings (many or few) had no effect on the decrement in responding produced by changing the context. Importantly, the size of the devaluation effect after minimal training was the same in each context, suggesting near-perfect transfer of the action (or R-O) component of responding across contexts. Once the R-O component was removed, a portion of what remained of operant responding (i.e., habit) remained sensitive to the context. Therefore, it appears that the S-R, or habitual, component of the operant response is what was context-specific.

Figure 2.

Effect of a context switch on instrumental responding after either 90 response-reinforcer pairings (left) or 360 response-reinforcer pairings (right). After instrumental training, but before the tests shown, the paired groups received separate pairings of the reinforcer with lithium chloride (to condition an aversion to it) and the unpaired groups received the reinforcer and lithium chloride unpaired. The reinforcer devaluation effect at left suggests that the response was an action after 90 reinforcers; the lack of a reinforcer devaluation effect at right suggests that the response was a habit there. Notice that the context switch affected the habit (right). It also affected responding at left, but only that which remained after devaluation (responding made out of habit); it did not affect the size of the reinforcer devaluation effect (the evidence of action). Adapted from Thrailkill and Bouton (2015b).

The results of Thrailkill and Bouton (2015b) suggesting direct control of habit by the context seem consistent with results suggesting direct inhibition of the response in extinction. However, in addition to direct excitatory or inhibitory associations with the response, contexts can also hierarchically control a response-outcome association under some conditions. In an experiment reported by Trask and Bouton (2014; modeled after a discriminated operant experiment reported by Colwill and Rescorla, 1990), rats first learned two response-outcome relations in Context A (R1-O1, R2-O2) and the opposite relations in Context B (R1-O2, R2-O1) during intermixed sessions. Following acquisition, the rats received a devaluation treatment in which O2 was repeatedly paired with lithium chloride. In a subsequent extinction test with the two responses in each context, the rats selectively suppressed the response that produced O2 there (R2 in Context A, R1 in Context B). From these results, it was evident that a context can function to set the occasion for R-O associations; rats displayed knowledge of which response led to which outcome in each context [i.e., a Context-(R-O) association], even when all binary (S-O, S-R, and R-O) associations were equated. Thus, contexts can apparently acquire a hierarchical or modulating function under some conditions (Trask & Bouton, 2014).

Conclusions

Although Pavlovian and operant extinction share many similarities, including a role for learning about the context, there appear to be differences in what is learned in extinction. In particular, negative occasion setting plays an important role in Pavlovian extinction, such that the context functions as a “whether” gate, disambiguating the CS’s current relation with the US (Bouton, 1997). Occasion setting can also play a role in operant learning, which is the paradigm, of course, where Skinner first coined the term (1938). Our recent work, however, suggests that negative occasion setting may not play as significant a role in the extinction of operant behavior, because there is surprisingly little transfer of the context’s control of inhibition of one response to other responses (Bouton et al., 2016; Todd, 2013; see also Todd et al., 2014). Instead, the most parsimonious explanation of the results may be that operant extinction involves direct inhibition of the response by the context. This type of learning may parallel (and functionally oppose) a related tendency for the acquisition context to evoke the response directly (Thrailkill & Bouton, 2015b; Figure 2). Although there is evidence that the context in Pavlovian and operant extinction may play other roles under some conditions, its role in Pavlovian extinction is mainly to determine the current meaning of the CS, whereas its role in operant extinction may mainly be to inhibit the response.

If it is accepted that Pavlovian and operant extinction might differ in the mechanism underlying contextual control, why should occasion setting be more important in Pavlovian than operant extinction? Here we can only speculate. We previously noted that instrumental behaviors can be either goal-directed actions or habits, that is, responses that are influenced (or not) by the current value of the outcome associated with them. To distinguish between actions and habits theoretically, Dickinson and Balleine (e.g., 1993) have suggested an associative-cybernetic perspective that proposes separate memory systems that correspond to each of them. Actions (R-O) are stored in an associative system that represents the response, the outcome, and the associative link between them. Pavlovian (S-O) associations are also represented there. Habits (S-R, or perhaps Context-R) are contrastingly stored in a separate habit memory system, which represents the response, the antecedent stimuli for it, but not the outcome. Although the associative-cybernetic model has yet to be expanded to address extinction, we suggest that a habit’s representation in the habit system might uniquely allow response inhibition to develop there. Thus, response inhibition may be the form of inhibition that develops in the habit system in order to cancel or counteract a habit. That may be a reason why an inhibitory context-response association is relatively unique to the instrumental situation.

Highlights.

• -Context plays a fundamental role in both Pavlovian and instrumental (operant) extinction
• -Contextual occasion setting is important in the Pavlovian case
• -Direct control of the response (e.g., response inhibition) is important in the instrumental case