Resurgence of Integrated Behavioral Units

Abstract

Two experiments with rats examined the dynamics of well-learned response sequences when reinforcement contingencies were changed. Both experiments contained four phases, each of which reinforced a 2-response sequence of lever presses until responding was stable. The contingencies then were shifted to a new reinforced sequence until responding was again stable. Extinction-induced resurgence of previously reinforced, and then extinguished, heterogeneous response sequences was observed in all subjects in both experiments. These sequences were demonstrated to be integrated behavioral units, controlled by processes acting at the level of the entire sequence. Response-level processes were also simultaneously operative. Errors in sequence production were strongly influenced by the terminal, not the initial, response in the currently reinforced sequence, but not by the previously reinforced sequence. These studies demonstrate that sequence-level and response-level processes can operate simultaneously in integrated behavioral units. Resurgence and the development of integrated behavioral units may be dissociated; thus the observation of one does not necessarily imply the other.

In his search for the fundamental units of behavior, Skinner (1935) defined a response unit as a functional class of events subject to control by reinforcing stimuli. A lever press could become a behavioral unit, but so could interresponse times (IRTs) (e.g., Morse, 1966; Platt, 1973; Zeiler, 1977) and integrated sequences of simple responses (e.g., Fetterman & Stubbs, 1982; Grayson & Wasserman, 1979; Hawkes & Shimp, 1975; Reid, Chadwick, Dunham, & Miller, 2001; Schwartz, 1981, 1982, 1986; Shimp, 1976, 1979). Zeiler (1977) identified three different kinds of response units that can be distinguished: formal, conditionable, and theoretical units. The formal response unit is the operational definition of the measured response—that which the experimenter prescribes as prerequisite for reinforcer presentation. Conditionable response units are operants (Skinner, 1938). If a behavior is required for reinforcer presentation and reinforcement increases its probability, then it is a conditionable response unit. Finally, Zeiler (1977) defined theoretical response units as follows:

The term response unit may also be used to refer to something inferred rather than observed directly. A response, a stimulus–response relation, or some cognitive activity, can be postulated to underlie observed performance. Inferred units are being used when it is asserted that organisms learn turning responses, or to approach certain locations in a maze, or interresponse times, or entire sequences of behavior. (pp. 222–223)

Thus, theoretical response units are conditionable units (operants) that answer the question of how reinforcement organizes behavior and what behavior is strengthened. They may include self-imposed behavioral patterns lasting several seconds (Shimp, 1976, 1979) or combinations of simpler operants (Zeiler, 1986a, 1986b).

Identification of formal and conditionable response units is unambiguous. The identification of theoretical response units is more difficult because theoretical units are inferred over repeated observations of behavior rather than observed directly. This study concerns the identification of theoretical response units as integrated response sequences influenced by reinforcement contingencies.

Several different terms have been used to represent the concept of theoretical response units, beginning with Skinner's (1935, 1938) functional response units that may change over time (see also Arbuckle & Lattal, 1988; Reid et al., 2001; Schneider & Morris, 1992). Shimp (1976) used the term functional units of analysis. Schwartz (1981, 1982, 1986) has argued that contingent reinforcement could turn an arbitrary sequence of discrete key pecks into an integrated behavioral unit, which he has also called functional behavioral units or complex behavioral units.

How can these theoretical behavioral units be identified and measured? Four techniques have been used in recent years, although we will argue that not all of these techniques are adequate.

1. One technique of identifying theoretical behavioral units was explained clearly by Zeiler (1977), although it had been implemented earlier (e.g., Findley, 1962; Kelleher, Fry, & Cook, 1964). This technique studies the hypothesized unit directly by first imposing the hypothesized unit as a formal unit, and then demonstrating that it is conditionable. For example, to test the claim that certain reinforcement schedules can lead to the development of IRTs as theoretical behavioral units, one could differentially reinforce a particular IRT and demonstrate that IRT is controlled by reinforcement in the same way as expected for simple responses.

2. Stability and replicability in the temporal pattern of reinforced behavior (over a period of several seconds) may indicate a self-imposed organization of behavior under the control of reinforcement contingencies (e.g., Hawkes & Shimp, 1975; Pisacreta, 1982; Reid et al., 2001; Shimp, 1976, 1979; Terrace, 2001). If one demonstrates that this temporal pattern is also under stimulus control, it would provide even stronger evidence that the behavioral pattern functions as a behavioral unit. For example, Reid et al. (2001) examined the development of integrated behavioral units by observing the development of self-imposed organization of behavior when differential reinforcement was provided for sequences of two lever presses in a free-operant procedure. They concluded that integrated behavioral units consisting of two lever presses had formed by demonstrating that the regularities in IRTs and transition probabilities between levers were stable, replicable, and under the stimulus control produced by demarcating stimuli and by local reinforcement probability.

3. Several researchers have concluded that sequences of two or three lever presses or key pecks had become organized into integrated (theoretical) behavioral units by demonstrating that the generalized matching law was better described by sequence matching than by matching of individual responses (Fetterman & Stubbs, 1982; Reid et al., 2001; Schneider & Davison, 2005; Schneider & Morris, 1992; Schwartz, 1986; Stubbs, Fetterman, & Dreyfus, 1987). The matching relation describes the influence of relative reinforcement rate on relative response rate, but what “response” is controlled by the relative reinforcement rate? Stubbs et al. demonstrated that when behavior becomes organized in such a way that matching of response sequences is observed when matching of individual responses is not, then one can conclude that the response sequence is the functional response unit even if the individual key peck had been the behavioral unit earlier in training.

4. The most common technique of identifying and measuring theoretical behavioral units inherently assumes that reinforcement strengthens behavioral units whether the units are key pecks, lever presses, or integrated sequences of heterogeneous responses (cf. Baum, 2002). A behavioral unit that has been reinforced frequently should have substantial strength as a whole, allowing it to resist disruption by a change in contingency (Schwartz, 1981, 1982). Therefore, the technique involves changing the reinforcement contingencies and measuring whether the putative behavioral unit maintains its “integrity”—maintaining the same behavioral sequence and same temporal structure (Reed, Schachtman, & Hall, 1991). If it does maintain its integrity, yet its frequency of production changes in accordance with the reinforcement contingency, then one can conclude that the pattern has become a theoretical behavioral unit. This technique has been widely implemented in research involving behavioral units (e.g., Grayson & Wasserman, 1979; Neuringer, Kornell, & Olufs, 2001; Reed et al., 1991; Schwartz, 1981, 1982, 1986; Wasserman, Nelson, & Larew, 1980; Zeiler, 1977, 1986b).

Techniques 1 and 4 require the demonstration that the behavioral unit is the same before and after a change in contingency—that it has maintained its integrity. For example, if the change in contingency is a shift to extinction, then “to the extent that a stereotyped sequence has become a behavioral unit, under extinction it should either occur or not, but if it occurs, it should occur in the same stereotyped way that it does during reinforcement conditions” (Schwartz, 1981, p. 34).

These two techniques are problematic because reinforced behavior sequences can occur repetitively in the same order and with the same temporal structure without the sequence achieving the status of an integrated behavioral unit (Reid, Dixon, & Gray, in press). For example, behavioral chains are controlled by reinforcement contingencies, and repetitions of behavioral chains are often highly regular in their order and temporal structure. Rather than being integrated structures, behavioral chains are individual operants linked together by response-produced stimuli. Under the right conditions, a discriminative stimulus can set the occasion for the first response in the chain, and the response-produced stimuli may allow the chain to progress to the end. The important distinction between integrated behavioral units and behavioral chains is the level at which reinforcement is presumed to operate. With behavioral chains, reinforcement is presumed to influence each response of the sequence separately (Catania, 1971; Skinner, 1938). With integrated behavioral units, reinforcement is presumed to operate at the level of the integrated unit, independent of its constituent responses.

Reid et al. (in press) argued that in order to demonstrate that a response sequence has become an integrated behavioral unit, one must demonstrate control by processes acting at the level of the purported unit (which we call “sequence-level processes”) rather than control only by response-level processes (such as all those involved in behavioral chains). An example of such a demonstration was outlined as technique 3 above: pitting sequence matching against response matching and observing that the matching law holds only with sequences, not with individual responses (as done originally by Stubbs et al., 1987, and more comprehensively by Schneider & Davison, 2005). Sequence matching is a good example of a sequence-level process not accounted for by more parsimonious response-level processes. Two examples of response-level processes are the effects of discriminative stimuli on the choice of the next response, and the observation that reinforcement strengthens individual responses in a sequence depending upon their ordinal position in the sequence or their temporal contiguity with reinforcement (Catania, 1971).

Like sequence matching, extinction-induced resurgence appears to be another example of behavior potentially influenced by a sequence-level process. Resurgence is observed only after the resurgent response has been exposed to a history of reinforcement, and then followed by a period of extinction (or very low reinforcement probability) (Cleland, Guerin, Foster, & Temple, 2001; Epstein, 1985; Lieving & Lattal, 2003). Presumably, the resurgent response has been strengthened sufficiently by reinforcement to make it the most probable response following extinction conditions. Prior studies of extinction-induced resurgence have demonstrated resurgence of simple responses, rather than complex sequences that might involve the integration of simpler responses. Nevertheless, to the extent that resurgence implies a history of strengthening by reinforcement, a demonstration of resurgence of entire response sequences may potentially indicate that reinforcement has strengthened response sequences as an integrated behavioral unit. If so, then resurgence procedures may provide a much-needed technique to study the development of integrated behavioral units, not only their identification as units.

The main purpose of Experiment 1 was to assess whether extinction-induced resurgence could reliably demonstrate that well-learned response sequences had become integrated behavioral units. At first glance, resurgence of complete response sequences would appear to be a sequence-level process that cannot be accounted for by response-level processes. Thus, we also asked whether resurgence always, never, or only in certain situations, indicates the existence of behavioral units.

An additional goal of this study concerned the traditional “problem of behavioral units”: As quantitative models of behavior are developed that accurately predict behavior measured as discrete responses (such as the matching law), how can we handle the transition to new integrated behavioral units (such as those observed in sequence matching)? A learning model that assumes that the individual key peck or lever press is the response unit that gains strength through reinforcement will have considerable problems unless it includes some mechanism for changing the nature of the response unit. Presumably, once integrated behavioral units develop, the processes controlling their production are no longer limited to the response-level processes that affected the simpler responses. Sequence-level processes presumably begin to control behavior, and reinforcement is assumed to strengthen the entire sequence as a new integrated behavioral unit. Therefore, our goal was to see if sequence-level processes and response-level processes can simultaneously control behavior, or whether integrated behavioral units, once organized and controlled by reinforcement, preclude control by response-level processes.

Experiment 1

Method

Subjects

Four naïve Wistar female rats, approximately 4 months old at the beginning of the study, were maintained at 85% of their free-feeding weights with continuous access to water in their individual home cages.

Apparatus

Two Med Associates (St. Albans, VT, USA) rat operant chambers measuring 26 cm wide, 32 cm long, and 30 cm high were used. A liquid dispenser (Med Associates ENV-200M), located through a 3-cm circular opening in the center of the control panel 2.5 cm above the floor, provided access to a cup containing 0.01 ml condensed milk diluted 50% with water. The arm of the dispenser remained in the up position until activated by a short pulse to fill the cup, which then remained available for an unlimited time period. The control panel also contained two response levers, located 9 cm to the left and right of the dispenser and 5.5 cm above the floor, and required a force of 0.15 N to operate. A 2.5-cm white 28-V lamp was located 5 cm above each lever. A 28-V houselight was centered near the top of the back wall opposite the control panel. Each operant chamber was located inside a sound-attenuating cubicle with a ventilation fan that masked extraneous noises. In an adjacent room, a computer with Med-PC software and a Med Associates interface controlled each session and stored each event and its time of occurrence.

Procedure

Training

After the reduction in body weights, subjects were trained to press levers by a successive approximations procedure. They then were exposed for three sessions to a discrete-trials procedure that reinforced the completion of two lever presses, without regard to which levers were used. For the next five sessions, reinforcement was contingent upon either heterogeneous sequence (i.e., LR and RL). Homogeneous sequences were unreinforced and immediately began the next trial. Sessions terminated after 30 reinforcers or after 30 min, whichever came first.

Experimental procedure

All experimental sessions contained 50 trials. Each trial began by turning on the lamps above the two levers, and ended either when the subject completed two responses or 1 min elapsed without completing two responses. When the response sequence was correct, access to the milk solution was made available by refilling the cup, and the lamps were turned off for 3 s. If the response sequence was incorrect or if 1 min elapsed without the completion of two responses, all lamps were turned off for a timeout period of 10 s. The houselight remained on at all times except during timeout.

In each experimental phase, each subject was trained to emit a particular two-response sequence. This target sequence was constant throughout all trials of the session. When the stability criteria were achieved, each subject was shifted to the next phase, which required a different target sequence. The order of exposure to these target sequences was counterbalanced across subjects (see Table 1). With heterogeneous target sequences, responding was considered stable within a phase after at least 25 sessions and when subjects were clearly discriminating between the two heterogeneous sequences (i.e., there was no overlap over five consecutive sessions in the numbers of the two heterogeneous sequences). Homogeneous sequences are learned quickly, so we required a minimum of 10 sessions when homogeneous sequences were the targets. Changes in phase always began with the next session, never within a session.

Table 1. Order of phases for each subject in Experiment 1. Each phase required subjects to complete a particular reinforced sequence of left (L) and right (R) lever presses.

Phase:	1st	2nd	3rd	4th
Subject:	Target Sequence:
R21	LR	RL	RR	LL
R22	LR	RL	LL	RR
R23	RL	LR	RR	LL
R24	RL	LR	LL	RR

Results and Discussion

Figure 1 shows the frequencies of each of the four possible response sequences (RR, LR, RL, and LL) for each subject in Phase 1 across blocks of five sessions. In order to more clearly identify differences in the curves, homogeneous and heterogeneous sequences are depicted in separate panels. The left panels depict the frequencies of heterogeneous sequences (RL, LR) and identify the reinforced sequence. The right panels depict the frequencies of the unreinforced homogeneous sequences in the same sessions. The left panels show a clear differentiation between the two heterogeneous sequences as they came under control of the reinforcement contingencies. For all subjects, the frequency of the reinforced sequence systematically increased while the frequency of the unreinforced heterogeneous sequence systematically decreased. In order to encourage the formation of integrated behavioral units consisting of the reinforced sequence, we continued reinforcing this sequence for many more sessions than necessary for clear differentiation between the two heterogeneous sequences.

Fig 1. Each graph shows the average number of each response sequence observed in Phase 1 for each subject in Experiment 1, depicted over blocks of five sessions.

The two heterogeneous sequences are depicted in the left panels, and the homogeneous sequences are depicted in the right panels. Phase 1 reinforced a heterogeneous sequence, depicted in the left panels as LR+ or RL+. Error bars represent one standard error.

The homogeneous sequences depicted in the right panels of Figure 1 were unreinforced errors. In 3 of the 4 subjects, the most common error was to repeat the last response required in the reinforced sequence. For example, Subject R21's LR sequence was reinforced and the most common error was RR. Subjects rarely produced the other erroneous homogeneous sequence.

Figure 2 shows the same analysis for Phase 2 in which the opposite heterogeneous sequence was reinforced. The number of sessions required for subjects to produce this new reinforced sequence more often than the previously reinforced heterogeneous sequence (now undergoing extinction) was substantially greater than that observed in the previous phase for all subjects. Phase 1 required 3–15 sessions to observe clear differentiation in the frequencies of the reinforced and unreinforced heterogeneous sequences. However, Phase 2 often required 35–40 sessions to produce this differentiation. Extinction of the previously reinforced sequence also required many sessions for 3 of the 4 subjects. This slow rate of learning may have been due to the extended training in Phase 1 that encouraged the formation of integrated behavioral units. Consistent with this hypothesis, subject R23 showed the fastest learning in Phase 2 and was exposed to the smallest number of sessions in Phase 1.

Fig 2. Each graph shows the average number of each response sequence observed in Phase 2 for each subject in Experiment 1, depicted over blocks of five sessions.

The two heterogeneous sequences are depicted in the left panels, and the homogeneous sequences are depicted in the right panels. Phase 2 reinforced a heterogeneous sequence, depicted in the left panels as LR+ or RL+. Error bars represent one standard error.

In contrast, the right panels of Figure 2 show an immediate change in the types of homogeneous sequence errors produced. In Phase 1, subjects tended to repeat the last response of the reinforced sequence. Those particular erroneous sequences disappeared abruptly with the transition to the newly reinforced sequence, even though subjects were yet to emit many reinforced sequences. Instead, the most frequent homogeneous sequences were those that repeated the last response required in the new reinforced sequence. This observation has been documented before (e.g., Grayson & Wasserman, 1979), and it is consistent with previous reports that the terminal response in a sequence is more sensitive to changes in reinforcement contingency than are earlier responses in the sequence (e.g., Corbit & Balleine, 2003; Reid, 1994; Reid et al., in press; Williams, 1999; Williams, Ploog, & Bell, 1995). Yet this observation is counterintuitive because it conflicts with predictions based on reinforcement contiguity: The extensive reinforcement (from Phase 1) of the old sequence should have increased its resistance to change, especially of the terminal response most contiguous with reinforcement. As a result, one would expect persistence errors of the terminal response to be observed when the reinforcement contingency was changed. These persistence errors were not observed. Thus, error production did not appear to be affected by resistance to change of the previously learned sequence. Greater sensitivity of the terminal response to changes in reinforcement contingency was a stronger determinant of sequence errors than was the reinforcement history of the prior sequence (see also Reid, 1994; Reid et al., in press).

The left panels of Figure 3 depict all four sequences observed in Phase 3, in which previously reinforced heterogeneous sequences were extinguished and a new homogeneous sequence was reinforced. Recall that Phases 3 and 4 lasted only 10 sessions each because homogenous sequences are learned quickly. Therefore, Figure 3 shows the frequencies of each sequence across individual sessions, rather than across blocks of sessions. We were interested in the changes that occur soon after the transition to the new phase, so the first three sessions are depicted for emphasis. The last three sessions of the condition are also depicted to show the steady-state levels obtained for each sequence. The first three sessions in the left panels show a rapid increase observed in the newly reinforced homogeneous sequence and a rapid decrease in the previously reinforced heterogeneous sequence. This rapid decrease stands in sharp contrast to the very slow extinction curves depicted in Figure 2, which were also heterogeneous sequences. Extinction of heterogeneous sequences had required 35–40 sessions when the new target was a heterogeneous sequence (Phase 2), but only 3–6 sessions were required when the new target was a homogeneous sequence (Phase 3). The steady-state level of responding at the end of the phase was nearly perfect accuracy on the reinforced homogeneous sequence for all subjects. With each subject, the sequence reinforced extensively in Phase 1 occurred at negligible rates in Phase 3. Therefore, there was no indication of extinction-induced resurgence.

Fig 3. The left panels depict all four sequence types observed in Phase 3, and the right panels depict Phase 4, for each subject in Experiment 1.

Each graph shows the frequencies of each sequence across individual sessions, rather than across blocks of sessions. The first three sessions and the last three sessions of each phase are depicted in order to emphasize acquisition and the final steady-state levels of each sequence. Phases 3 and 4 reinforced homogeneous sequences, depicted in each panel as LL+ or RR+. Extinction-induced resurgence of a previously reinforced response sequence is identified in the right panels by arrows.

The results of Phase 4 are depicted in the right panels of Figure 3. In Phase 4, the previously reinforced homogeneous sequence was no longer reinforced, and the other homogeneous sequence was reinforced. In contrast to acquisition in Phase 3, at the start of Phase 4 resurgence of a previously learned heterogeneous sequence substantially retarded learning of the new homogeneous sequence. For all subjects the heterogeneous sequence that had been reinforced in Phase 2 (see Figure 2) showed substantial resurgence in Phase 4 (identified in Figure 3 by arrows), even becoming the dominant sequence for 3 subjects during the first three sessions of the phase. In all subjects, this sequence occurred at frequencies much higher than the currently reinforced sequence. For all subjects resurgence of complex heterogeneous sequences occurred after a 10-day period of extinction (Phase 3) during which the frequency of this sequence was approximately zero. Ironically, the resurgent sequence was the same sequence that had shown such a slow increase in frequency in Phase 2.

Transitions to each new phase included extinction of the previously reinforced response; thus, it is important to demonstrate that the observed increase in sequence production, which we identify as extinction-induced resurgence, was not simply the result of increased extinction-induced variability. A method providing a quantitative comparison of resurgence versus increased variability would be useful as a measure of the relative strength of resurgence. Extinction-induced variability presumably would affect both heterogeneous sequences equally, since their difficulty levels should be equal. Thus, the simplest approach is to compare the number of observations of the two sequences at each of the first few sessions of the phase (when the transient effects of resurgence would be expected to be observed). Figure 4 compares the frequency of occurrence of the resurgent sequence with that of the other heterogeneous sequence, averaged across subjects, for the first three sessions of Phase 4. This average is representative of individual subjects because every subject showed the same effect. We examined the difference between the frequencies of the resurgent sequence and the other sequence using a 2 × 4 (Sequence × Session) mixed analysis of variance (ANOVA) with the first four sessions as a repeated-measure factor across the 4 subjects. Overall, the resurgent sequence occurred significantly more often than the other heterogeneous sequence, F(1,6)  =  25.725, p  =  .002, η²  =  .811. Therefore, resurgence was not an artifact of extinction-induced variability.

Fig 4. The frequency of occurrence of the resurgent sequence is compared to the frequency of the other heterogeneous sequence, averaged across subjects, for each of the first three sessions of Phase 4 in Experiment 1.

Error bars represent one standard error.

We repeatedly observed that the most common error was to repeat the last response required in the new reinforced sequence. The ordinal position of responses within the sequence differentially affected the types of errors produced—clearly a response-level process. Was this response-level process responsible for resurgence? That is, did resurgence depend upon the resurgent sequence sharing the terminal (or initial) response with the newly reinforced sequence? Resurgence was observed in all 4 subjects, yet only 2 subjects (R21 and R24) shared the same terminal response in the resurgent sequence and the currently reinforced sequence. Thus, resurgence would not have been expected with the other 2 subjects, yet it was observed. Furthermore, only 2 subjects (R22 and R23) shared the same initial response in the resurgent sequence and the currently reinforced sequence, yet resurgence was observed in all subjects. The same logic holds with the potential influence of the previously reinforced sequence: only 2 subjects shared the same initial or terminal response in the resurgent sequence and the previously reinforced sequence, yet resurgence of the sequence reinforced in Phase 2 was observed in all subjects. Therefore, we found no evidence that resurgence was due to a response-level process in this experiment. Naturally, Experiment 1 does not rule out the possibility of response-level influence on resurgence in other situations.

Extinction-induced resurgence was observed in all subjects in Phase 4, but why did no subjects show resurgence in Phase 3? Phase 3 provided the same opportunities to observe extinction-induced resurgence that were provided in Phase 4: the opportunity for the recurrence of a previously reinforced behavior (the heterogeneous sequence reinforced extensively in Phase 1) under the condition that reinforcer delivery ceases for a more recently reinforced behavior (the heterogeneous sequence reinforced in Phase 2, which was no longer reinforced in Phase 3) (Cleland et al., 2001; Epstein, 1985). To the extent that resurgence of these sequences implies the existence of integrated behavioral units, it is important to understand why the units would be observed in one phase but not in an apparently similar phase. Understanding the causes of resurgence may elucidate how integrated behavioral units develop. Experiment 2 continued this exploration by asking how resurgence of sequences depends upon the number and order of exposure to the reinforcement conditions of Experiment 1. Experiment 2 repeated the study with a different order of reinforced sequences for each subject, examined which conditions produce extinction-induced resurgence, and examined the implications for the development of integrated behavioral units. Our main concern was not in resurgence, per se, but in what it can teach us about the development of integrated behavioral units. We wanted to establish whether extinction-induced resurgence is, in fact, a sequence-level process, and whether sequence-level and response-level processes can coexist.

Experiment 2

Method

Subjects

Four naïve Wistar female rats, approximately 4 months old at the beginning of the study, were maintained at 85% of their free-feeding weights with continuous access to water in their individual home cages.

Apparatus

The same rat chambers used in Experiment 1 were used in the current experiment with one modification. The liquid dispensers were replaced with standard pellet dispensers (Med Associates, ENV 203M) that delivered 45-mg Noyes (Formula A/I) pellets into an un-illuminated receptacle (Med Associates, ENV-200-R2M), accessible through a 5-cm square opening.

Procedure

The procedures used for training and the four phases of this experiment were the same as those used in Experiment 1 with two exceptions. As illustrated in Table 2, the order of reinforced sequences across phases was different. Experiment 1 did not include transitions between previously reinforced homogeneous sequences and newly reinforced heterogeneous sequences. This type of transition was included in the current experiment. The other difference in procedure concerned the number of sessions in each phase. A stability criterion was in effect for Experiment 1 that allowed different subjects to receive different amounts of exposure to each reinforced sequence in Phases 1 and 2. In the current experiment, the number of sessions in each phase was held constant across subjects. Phases that reinforced heterogeneous sequences lasted 40 sessions, and those reinforcing homogeneous sequences lasted 10 sessions for all subjects.

Table 2. Order of phases for each subject in Experiment 2. Each phase required subjects to complete a particular reinforced sequence of left (L) and right (R) lever presses.

Phase:	1st	2nd	3rd	4th
Subject:	Target Sequence:
A1	RL	LL	RR	LR
A2	RL	RR	LL	LR
A3	LR	LL	RR	RL
A4	LR	RR	LL	RL

Results and Discussion

Figure 5 shows the frequencies of each of the four possible response sequences for each subject in Phase 1 across blocks of five sessions. As in the earlier figures, homogeneous and heterogeneous sequences are depicted in separate panels to more clearly identify differences in the curves. As the left panels show, all subjects differentiated between the reinforced and unreinforced heterogeneous sequences. The right panels depict homogeneous sequences, which were always errors in this phase. In 2 of the subjects (A3 and A4), the differences in frequency between the two homogeneous sequences demonstrated a transient tendency to repeat the last response required in the heterogeneous target sequence. Subjects rarely produced the other erroneous homogeneous sequence. These observations are consistent with the results of Phases 1 and 2 of Experiment 1, but the effects were larger and more enduring in the earlier experiment.

Fig 5. Each graph shows the average number of each response sequence observed in Phase 1 for each subject in Experiment 2, depicted over blocks of five sessions.

The two heterogeneous sequences are depicted in the left panels, and the homogeneous sequences are depicted in the right panels. Phase 1 reinforced a heterogeneous sequence, depicted in the left panels as LR+ or RL+. Error bars represent one standard error.

The left panels of Figure 6 depict the results of Phase 2, in which previously reinforced heterogeneous sequences were extinguished and a new homogeneous sequence was reinforced. Recall that Phases 2 and 3 lasted only 10 sessions because homogenous sequences are learned quickly. Therefore, Figure 6 shows the frequencies of each sequence across individual sessions, rather than across blocks of sessions. All subjects learned the new homogeneous target sequence rapidly, producing this reinforced sequence in nearly every trial by Session 10, ending the phase. The previously reinforced heterogeneous sequence was effectively extinguished.

Fig 6. The left panels depict all four sequence types observed in Phase 2, and the right panels depict Phase 3, for each subject in Experiment 2.

Each graph shows the frequencies of each sequence across individual sessions, rather than across blocks of sessions. The first three sessions and the last three sessions of each phase are depicted in order to emphasize acquisition and the final steady-state levels of each sequence. Phases 2 and 3 reinforced homogeneous sequences, depicted in each panel as LL+ or RR+. Extinction-induced resurgence of a previously reinforced response sequence is identified in the right panels by arrows.

The right panels of Figure 6 depict the results of Phase 3, which provided the first opportunity to replicate the extinction-induced resurgence observed only in Phase 4 of Experiment 1. All subjects showed resurgence of the heterogeneous sequence that had been reinforced in Phase 1. Resurgence in Subject A1 was short-lived, but the other 3 subjects showed stronger, more persistent resurgence. Resurgence was observed in each subject regardless of whether the resurgent sequence shared the same initial or terminal response as the currently reinforced sequence or as the previously reinforced sequence in the prior phase. Therefore, the ordinal position of responses within the sequences was not a causal factor in extinction-induced resurgence.

Once again, it is important to demonstrate that the observed increase in sequence production, which we identify as extinction-induced resurgence, was not simply the result of increased extinction-induced variability. Figure 7 compares the frequency of occurrence of the resurgent sequence with that of the other heterogeneous sequence, averaged across subjects, for the first three sessions of Phase 3. This average is representative of individual subjects because every subject showed the same effect. We examined the difference between the frequencies of the resurgent sequence and the other sequence using a 2 × 4 (Sequence × Session) mixed analysis of variance (ANOVA) with the first four sessions as a repeated-measure factor across the 4 subjects. Overall, the resurgent sequence occurred significantly more often than the other heterogeneous sequence, F(1,6)  =  13.201, p  =  .011, η²  =  .688. Therefore, resurgence was not an artifact of extinction-induced variability.

Fig 7. The frequency of occurrence of the resurgent sequence is compared to the frequency of the other heterogeneous sequence, averaged across subjects, for each of the first three sessions of Phase 3 in Experiment 2.

Error bars represent one standard error.

Figure 8 depicts the results of Phase 4 in which reinforcement was shifted from homogeneous to heterogeneous sequences. The curves in the left panels were similar to the curves in the left panels of Figure 5, in which subjects first learned to differentiate between the two types of heterogeneous sequences. But in Figure 8, for the first 5–10 sessions all 4 subjects produced the unreinforced heterogeneous sequence more often than the currently reinforced sequence. When one value is greater than another, we can ask what process enhanced the larger value, or we could ask what process diminished the smaller value, or both. The higher frequency of the unreinforced heterogeneous sequence may indicate resurgence of the sequence reinforced in Phase 1 (three, not two, phases earlier). For all subjects, this was the same sequence that resurged in Phase 3. Alternatively, or perhaps in addition, the currently reinforced heterogeneous sequence may have begun the phase with low frequencies because it had never before been reinforced and had always produced timeouts. The current procedure does not allow us to separate these two interpretations, so we cannot be certain that resurgence was obtained in this phase.

Fig 8. Each graph shows the average number of each response sequence observed in Phase 4 for each subject in Experiment 2, depicted over blocks of five sessions.

The two heterogeneous sequences are depicted in the left panels, and the homogeneous sequences are depicted in the right panels. Phase 4 reinforced a heterogeneous sequence, depicted in the left panels as LR+ or RL+. Error bars represent one standard error.

Recall that in Phases 1 and 2 of Experiment 1, subjects tended to repeat the last response of the currently reinforced sequence, such that the target sequence LR yielded a substantial number of RR errors. This same effect can be observed in the current study by examination of the right panels of Figure 8 depicting homogeneous sequences. All homogeneous sequences can be considered errors because the only reinforced sequence was heterogeneous. Nevertheless, all subjects produced more of the homogeneous sequence that shared the same response as the terminal response of the currently reinforced heterogeneous sequence, replicating the results of Experiment 1. This elevated level was not simply a carryover effect from the previous phase because the opposite homogeneous sequence had been previously reinforced with subjects A2 and A3. These homogeneous sequences are unlikely to reflect extinction-induced resurgence because these particular sequences had not been reinforced and then extinguished in either Phase 1 or Phase 2 for subjects A1 or A4, even though they had been with subjects A2 and A3. Thus, the simplest explanation is that the most common error in the homogeneous sequences was to repeat the last response required in the currently reinforced sequence, not in the previously reinforced sequence—a response-level process observed in every subject in both experiments.

This experiment was designed to explore further the relationship between resurgence of response sequences and the development of integrated behavioral units. We changed the order of conditions from that of Experiment 1 to see if resurgence would occur in a different phase. Resurgence occurred exclusively in Phase 4 in Experiment 1, but it occurred in Phase 3 of the current study. There was inconclusive evidence that these same sequences may have resurged in Phase 4 as well, even though they were not the most recently reinforced candidate sequences. Resurgence did not appear to be due to any response-level processes in this experiment, thus replicating the results from Experiment 1. Response-level processes were observed: The most common error for all subjects was to repeat the last response required in the currently reinforced sequence, not in the previously reinforced sequence. This counterintuitive observation replicates the types of errors observed in Experiment 1.

General Discussion

All subjects in both experiments produced extinction-induced resurgence of a heterogeneous response sequence. In all subjects in both experiments, the most frequent errors in sequence production were sequences that repeated the last response required in the currently reinforced sequence, but not in the previously reinforced sequence. Thus, error production appeared to be controlled by a response-level process because the ordinal position of responses in the sequence differentially affected the types of errors produced. If the observed resurgence can be demonstrated to be a sequence-level process, then these experiments appear to be the first demonstration that both sequence-level and response-level processes can operate simultaneously within clearly established theoretical behavioral units (not to be confused with traditional molar vs. molecular explanations of behavior on reinforcement schedules, in which theoretical behavioral units are not measured).

Does this resurgence demonstrate that the sequences had become integrated behavioral units? Using the criteria for identifying theoretical behavioral units proposed by Zeiler (1977), the first technique discussed in the introduction, one would conclude that the sequences were, indeed, integrated theoretical behavioral units because when reinforced directly, the resurgent sequences were differentially controlled by reinforcement (they were conditionable units). In addition, they maintained their “integrity” following shifts in reinforcement contingency with each subject in each experiment (technique 4) because out of all possibilities, the resurgent sequences were precisely those that were reinforced two phases earlier.

Because the two techniques above are limited in their ability to distinguish between integrated behavioral units and behavioral chains, Reid et al. (in press) proposed an alternative procedure for identifying integrated behavioral units that provides a systematic method of evaluating the integrity of behavioral units. Their procedure asks: Were sequence-level processes actually involved, or was the behavior pattern (i.e., resurgence) due simply to one or more response-level processes? Their procedure of separating response-level from sequence-level processes provides more precision than the techniques described earlier for identifying integrated behavioral units. It begins with the creation of a decision tree representing the possible decisions made at each point in the response sequence.

Figure 9 depicts the conditional decision tree representing each of the possible sequences for both experiments. Because the sequences consisted of only two responses, only two decision levels were possible (one to select the initial response and one for the terminal response). The tree is composed of three identical subtrees, each representing the choice of pressing the left or right lever. The first decision level, labeled “A”, represents the choice of the initial response in the sequence. This choice is likely to be affected by cues indicating the beginning of the trial, such as prior food delivery or timeout. At the second level, “B” and “C” represent the points at which the terminal response of the sequence is selected. This later decision is not likely to be affected by the same beginning-of-trial cues, but other discriminative stimuli present in the environment (including the subject's previous response and spatial location in the chamber) may influence lever selection. The completed sequences are represented at the bottom of the figure, to be followed by reinforcement or timeout.

Fig 9. This conditional decision tree represents the response-by-response choices as responses are produced to form response sequences.

The letter A represents the choice point for the initial response of the sequence. Letters B and C represent the choice points for the second, terminal response that terminates the two-response sequence.

For each subject in both experiments, the observed extinction-induced resurgence can be represented as a single change at a particular choice point. As an example, consider Subject R21 in Experiment 1 as it was shifted from a reinforced sequence of RR in Phase 3 to a new reinforced sequence LL in Phase 4. Before the new sequence was learned, the subject produced extinction-induced resurgence of the sequence RL, which had been the reinforced sequence in Phase 2. The transition from RR to RL represented a change only in the response produced in the terminal position. In Figure 9, this change is represented by choice point C. That is, resurgence for this rat in this phase is represented as a response-level change at a single choice point, C.

This analysis allows one to answer questions related to the separation of response-level from sequence-level processes. For example: Did resurgence in each condition depend upon whether the extinguished sequence and the resurgent sequence involved a change in the initial versus terminal response position? Resurgence in Experiment 1 involved a change in the terminal position for R21 (RR→RL, represented by choice point C) and R24 (LL→LR, choice point B) and a change in the initial position for R22 (LL→RL, point A) and R23 (RR→LR, point A). Similarly, resurgence observed in Phase 3 of Experiment 2 involved a change in the terminal position for Subjects A2 (RR→RL, point C) and A3 (LL→LR, point B) and a change in the initial position for A1 (LL→RL, point A) and A4 (RR→LR, point A). Therefore, resurgence in both experiments was produced by changes at all three possible choice points, some presumably influenced by beginning-of-trial cues and others not. Therefore, resurgence did not depend upon whether it involved a change in the initial versus terminal response position. Resurgence did not depend upon this response-level process.

If only one or a limited number of possible choice points had been involved, then one might conclude that resurgence was a response-level process, or the result of a limited combination of response-level processes. However, this analysis demonstrates that extinction-based resurgence of these response sequences was a sequence-level process involving changes at every possible choice point. It cannot be accounted for by proposing a more parsimonious combination of response-level processes. We are faced with two possible explanations in each experiment: either (a) subjects repeated a previously reinforced and well-learned response sequence as an integrated behavioral unit (a sequence-level process), or (b) every subject in both experiments produced that one particular series of individual response decisions in which every decision conformed to their previously reinforced sequence, no matter what the response pattern had been in the intervening phase. Clearly, the latter explanation is highly improbable and unparsimonious. Sequence-level processes were clearly operative. We can confidently conclude that these well-learned response sequences had formed integrated theoretical behavioral units.

Sequence-level processes were operative, but response-level processes were simultaneously operative within these integrated behavioral units. Recall that the most frequent errors in sequence production were sequences that repeated the last response required in the currently reinforced sequence (but not in the previously reinforced sequence). Thus, error production appeared to be controlled by a simple response-level process for all subjects. This process appears to be related to the assignment-of-credit problem during acquisition (Staddon & Zhang, 1991), rather than to the strength or persistence of responding in the previous phase. The ordinal position of responses in the sequence differentially affected the types of errors produced. To our knowledge, this is the first demonstration that response-level processes continue to be operative after integrated behavioral units have formed. Thus, reinforcement can work simultaneously at the level of the simple response and at the level of the integrated behavioral unit. This observation provides a potential solution to the traditional “problem” of behavioral units mentioned earlier: As quantitative models of behavior are developed that accurately represent behavior measured as discrete responses, how can we handle the transition to new integrated behavioral units? A learning model that assumes that the individual key peck or lever press is the response unit that gains strength through reinforcement may have considerable problems unless it includes some mechanism for changing the nature of the response unit. The current evidence implies that the processes acting on individual responses continue to operate even when new behavioral units form, and new sequence-level processes can be added to the determinants of the behavior stream. The “problem” of the behavioral unit may not require extensive modification of existing quantitative models of behavior.

Is resurgence related to the strength of the behavioral unit? The notion of a behavioral unit is strongly tied to the assumption that these units gain strength when followed by reinforcement (Baum, 2002). Since the resurgent sequences were integrated behavioral units, we can ask whether resurgence was more likely to be observed following phases that provided more reinforcement for the sequences. Or, were additional factors (other than reinforcement history) involved in the production of resurgence?

In Experiment 1 (see Figure 1) Phase 1 provided extensive reinforcement for a heterogeneous sequence, and each subject produced the reinforced sequence with high accuracy. Therefore, the reinforced sequence should have become an integrated behavioral unit with considerable strength. Indeed, when the contingency was shifted to Phase 2 (see Figure 2), acquisition of the new sequence appeared to be retarded most for the subjects with the longest reinforcement history for the old sequence (Phase 1). Behavioral units with such strength should be highly likely to show resurgence, but no resurgence of these sequences was observed in Phase 3. By comparing the reinforcement histories of the heterogeneous sequences in Phase 1 (Figure 1) to that of Phase 2 (see Figure 2), it seems clear that the strength of the reinforced heterogeneous sequence in Phase 2 would be weaker than in Phase 1. Yet it was this presumably weaker sequence that resurged strongly in Phase 4 with every subject.

Experiment 2 also reinforced a heterogeneous sequence in Phase 1 (see Figure 5) for an extended period to encourage the formation of integrated behavioral units with considerable strength. It was precisely these sequences that resurged in Phase 3 (Figure 6). How can these apparently conflicting results between experiments be reconciled? A richer reinforcement history of a complex behavioral unit does not necessarily lead to greater probability of its resurgence. For example, we obtained resurgence of heterogeneous sequences in both studies, but no resurgence of homogeneous sequences was obtained (even with nearly perfect accuracy during training). Extinction-induced resurgence appears to depend upon additional factors unrelated to the strength of the behavioral unit. Possible examples may include the difficulty of detecting changes in reinforcement contingency, and the difficulty of discriminating between reinforced and unreinforced response sequences. Because such additional factors were not measured in this study, the variables that influence the probability of resurgence remain elusive. For researchers interested in studying resurgence, the current procedure provides a rich opportunity to tease out the particular sets of conditions that produce resurgence.

The development of integrated behavioral units does not necessarily mean that resurgence will be observed, even though it was observed in certain conditions of this study. What about the converse question: Does observed resurgence prove the existence of integrated behavioral units? Not necessarily. Recall that simple behavioral chains can be confused with integrated behavioral units, yet no integration across responses exists in chains. Under the right conditions, resurgence of extended chains of responses might be observed, but that does not mean they are behavioral units (Reid et al., in press). The conditions that would encourage resurgence of behavioral chains are those with strong beginning-of-trial cues and response-produced stimuli, similar to those in the procedures and apparatus used by Schwartz (1981) and Pisacreta (1982). Long chains involving many heterogeneous responses have been repeatedly demonstrated (e.g., Kelleher, 1966), perhaps because behavioral chains do not require integration across constituent responses. Therefore, we would expect future researchers to observe resurgence of response sequences longer and more complex than those capable of being integrated into behavioral units.