Abstract
The present study compared relapse after responding was eliminated by extinction or omission training in rats. In Experiment 1, lever pressing was reinforced with food pellets in Context A and then eliminated with either extinction or omission training in Context B. The response was then tested in Contexts A and B in either the presence or absence of free food pellets delivered on a random time schedule. All rats showed higher responding when tested in Context A than Context B, and there was little evidence that omission training attenuated this ABA renewal effect. Noncontingent pellets increased responding after extinction but not after omission. However, when responding on the last day of response elimination was compared to responding during the test in the response-elimination context, there was some evidence that omission-trained rats showed a small increase in responding even when tested with free pellets. Results of Experiment 2 suggest this increase was not due to differences in the temporal distribution of pellets during elimination and the test, and that the result might be due to mere removal of the omission contingency, but any such effect is small and difficult to detect statistically. The results provide new information about factors generating relapse after omission training.
Keywords: differential reinforcement of other behavior, extinction, omission, reinstatement, relapse, renewal
Extinction is a basic behavioral process that refers to the elimination of a target response when its reinforcing consequence is no longer presented. Extensive basic and applied research in animals and humans suggests that reduction of operant and respondent behaviors during extinction does not reflect an erasure of the original learning. Instead, the evidence suggests that extinction involves learning to inhibit the behavior (see Bouton, 2017, 2019 for reviews). Extinguished responding seems to return (or recover) readily when (a) the response is tested outside of the extinction context (Renewal), (b) the reinforcer is once again presented (Reinstatement), (c) reinforcers for an alternative response are removed (Resurgence), (d) following the passage of time (Spontaneous Recovery), and (e) the reinforcement contingency is reintroduced (Rapid Reacquisition; see Bouton, 2017, for a review). The various recovery effects have been considered possible mechanisms of relapse (e.g., Bouton & Swartzentruber, 1991; Podlesnik & Shahan, 2010). One view is that each demonstrates the fundamental context-specificity of extinction (e.g., Bouton, 2017, 2019).
Insights from extinction research have been useful to researchers and clinicians in the domain of applied behavior analysis. Behavioral treatments are effective for producing initial reductions in problem behavior; however, relapse after treatment is common and a major barrier to treatment efficacy. For example, differential reinforcement of alternative behavior (DRA) or functional communication training (FCT) are effective treatments for problem behavior, but resurgence of problem behavior occurs when reinforcement for the alternative response is disrupted (e.g., Gratz, Wilson, & Glassford, 2019; Hoffman & Falcomata, 2014; Lieving, Hagopian, Long, & O’Connor, 2004; Marstellar & St. Peter, 2014; Neely, Graber, Kunnavatana, & Cantrell, 2019; Volkert, Lerman, Call, & Trosclair-Lasserre, 2009; Wacker et al., 2013). Most research into relapse processes in clinical settings attempts to translate methods developed in studies of extinction learning (e.g., Liddon, Kelley, Rey, Liggett, & Ribeiro, 2018), even though extinction alone is rarely used as a technique to eliminate behavior in clinical settings. Clearly, it is possible that relapse might be influenced by the specific response elimination technique and that different treatments for problem behavior might be differentially susceptible to relapse.
For example, omission training is a response elimination procedure that might provide a more robust form of behavioral suppression than extinction (Uhl & Garcia, 1969). Omission training, also known as differential reinforcement of other behavior (DRO), is a schedule of reinforcement in which reinforcers are delivered contingent on the nonoccurrence of a target response (Reynolds, 1961). During omission training, the organism must learn to withhold, or inhibit, the target response in order to receive reinforcement because reinforcers are delayed or omitted contingent on the occurrence of the target response. Omission is commonly used to treat a wide variety of problem behavior including substance abuse (e.g., contingency management and the therapeutic workplace; Higgins et al., 1991; Silverman et al., 1996), medical noncompliance (e.g., Cuvo, Reagan, Ackerlund, Huckfeldt, & Kelly, 2010; Shabani & Fisher, 2006; Slifer, Avis, & Frutchey, 2008), and problem behavior common in individuals with developmental and intellectual disabilities (e.g., aggression, self-injurious behavior, and disruption; Iannaccone, Hagopian, Javed, Borrero, & Zarcone, 2019). In addition to their common therapeutic use, omission schedules are abundant in everyday social interactions. For example, consuming too many alcoholic drinks at a social outing might result in a partner withholding affection upon returning home—or a previously promised treat might be omitted after a child throws a tantrum at the grocery store (see also Uhl & Sherman, 1971).
There has been a substantial amount of research on the efficacy of omission contingencies at eliminating behavior (for reviews, see Davis, Kurti, Skelly, Redner, White, & Higgins, 2016; Homer & Peterson, 1980; Poling & Ryan, 1982; Weston, Hodges, & Davis, 2018; Whitaker, 1996). However, there has been considerably less attention to evaluating the prevalence and extent of relapse following the end of treatment. Factors that influence relapse after omission may differ from those that influence relapse after extinction. For example, studies have found less spontaneous recovery (Davidson & Walker, 1970; Topping & Ford, 1974; Zeiler, 1971), reinstatement (Harman, 1973; Topping & Larmi, 1973; Uhl, 1973; Uhl & Garcia, 1969; Uhl & Sherman, 1971), and slower reacquisition (Mulick, Leitenberg, & Rawson, 1976; Pacitti & Smith, 1977; Uhl & Garcia, 1969; Uhl & Sherman, 1971) following omission training when compared to extinction. The stronger impact of omission on reinstatement and reacquisition is consistent with the view that reinforcers delivered during omission training may provide an important part of the “context” that controls behavioral inhibition (e.g., Bouton, 2019; Bouton & Schepers, 2014; Trask, Schepers, & Bouton, 2015; Uhl & Garcia, 1969). The stronger impact on spontaneous recovery, however, may be consistent with the possibility that omission creates a more durable form of behavioral suppression.
Nakajima, Urushihara, & Masaki (2002) more recently found a renewal effect following omission training in a two-group experiment with rats. In Group ABA, lever pressing was reinforced with food pellets in one context (A), eliminated with omission training in a second context (B), and tested in the original training context (A) with the omission contingency in effect. Group AAA experienced the same number of acquisition, omission, and test sessions but they all occurred in the same context (A). Nakajima et al. found an increase in response rates during the test condition for Group ABA but no increase in Group AAA. Though this experiment showed that behavior eliminated through omission training is susceptible to ABA renewal, it is possible that omission training nonetheless mitigates renewal when compared to response elimination through extinction.
To our knowledge, only one study has compared renewal after omission training to that after other response elimination techniques (Kearns & Weiss, 2007). The study used a discriminated operant arrangement in which intravenous infusions of cocaine originally reinforced the operant response. Rats first earned cocaine contingent on a nose-poke response in the presence of a discriminative stimulus (S; tone) presented in a distinct context (Context A). In a response elimination phase, rats were assigned to four groups and responses no longer produced cocaine. Group AAA Extinction received the response elimination phase in the same context as training (Context A), while the other three groups (ABA Extinction, ABA DRO, and ABA FT) received the response elimination phase in a second distinct context (Context B). Group ABA Extinction and Group AAA Extinction received presentations of S with the nose poke available but no reinforcers were presented. Group ABA DRO earned food pellets (a new reinforcer) contingent on the omission of the nose-poke response during S, and Group ABA FT received food pellets during S independent of whether the response occurred. All rats were then tested for responding in extinction in Context A during S presentations. An increase (renewal) of responding during S was observed for each ABA group (ABA Extinction, ABA DRO, and ABA FT), but not in Group AAA Extinction. Further, although renewal was similar in Groups ABA DRO and ABA FT, it was significantly less in those groups than in Group ABA Extinction.
Kearns and Weiss’s (2007) results suggest that renewal might be equivalently weakened by either a DRO or an FT procedure. However, the results might depend on the use of one reinforcer during the response acquisition phase and a different reinforcer during the response elimination phase. Because Groups ABA DRO and ABA FT showed similar increases in responding in the S in the test, the authors suggested a role for counterconditioning of the S as a mechanism for the attenuation of renewal (relative to the ABA extinction group). More specifically, they suggested that pairings of food pellets and S could have resulted in the acquisition of food-related conditioned responses that competed directly with cocaine-related responses evoked by S in the test.
The present experiments compared relapse in a free-operant setting after responding was eliminated by extinction or omission training. In Experiment 1, lever pressing was reinforced with food pellets in Context A and then eliminated with either extinction or omission training in Context B. In a test phase, the response was tested in Contexts A and B in either the presence or absence of free food pellets delivered on a random time (RT) schedule. The tests in Context A and presentation of free pellets were expected to cause relapse through ABA renewal and reinstatement, respectively, when the response had undergone extinction. The question was whether susceptibility to these forms of relapse would be different when the response was eliminated by omission. Experiment 1 directly compared relapse following omission with relapse following extinction and Experiment 2 examined another potential source of relapse following omission.
Experiment 1
In Experiment 1, lever pressing was first reinforced with food pellets on a random interval (RI) 30-s schedule in a distinct context (Context A); this was then followed by a response elimination phase in a second context (Context B). For half the rats, response elimination consisted of sessions in which the lever was available but pressing did not result in the reinforcer (Group Extinction). For the remaining rats, the lever was available, but pellets were delivered contingent on the omission of the response for a specified period of time (Group Omission). For Group Omission, a lever press would restart the interval between scheduled pellet deliveries. All rats then received tests in Contexts A and B in a counterbalanced order (see also Bouton, Todd, Vurbic, & Winterbauer, 2011). In each test, the lever was available but had no programmed consequence. Half the rats in Groups Extinction and Omission received noncontingent pellets in the test (Groups Extinction Pellets and Omission Pellets) and half were tested without pellets (Groups Extinction No Pellets and Omission No Pellets). For all rats, we expected responding to increase when tested in Context A (Renewal). Noncontingent pellets were expected to increase extinguished responding (Reinstatement), but to have no effect on responding eliminated by omission training (Topping & Larmi, 1973; Uhl & Sherman, 1971). Finally, we expected that the absence of pellets in the test (No Pellets) might increase responding eliminated by omission (Resurgence) but not extinction (Pacitti & Smith, 1977; Uhl & Garcia, 1969).
Method
Subjects.
Thirty-two female Wistar rats (Charles River, St. Constance, Quebec, Canada), aged 75–90 days at the start of the experiment, were individually housed in a room maintained on a 16–8 light–dark cycle. Experimental sessions were conducted during the light portion of the cycle at the same time each day. Rats were food deprived and maintained at 80% of their free-feeding weights for the duration of the experiment. The 80% free-feeding weight was determined by weighing each rat after a 5-day acclimation period in which food was freely accessible. Rats were allowed ad libitum access to water in their home cages. Weights were maintained by supplemental feeding approximately 30 min postsession.
Apparatus.
The apparatus consisted of two unique sets of four conditioning chambers (model ENV008-VP; Med Associates, Fairfax, VT) located in separate rooms of the laboratory. Each chamber was housed in its own sound attenuation chamber. All boxes measured 31.75 × 24.13 × 29.21 cm (length × width × height). The side walls and ceiling were made of clear acrylic plastic, and the front and rear walls were made of brushed aluminum. A recessed food cup measured 5.1 cm × 5.1 cm and was centered on the front wall approximately 2.5 cm above the grid floor. In both sets of boxes, a retractable lever (model ENV-112CM; Med Associates) was positioned to the left of the food cup. The lever was 4.8 cm wide and was 6.3 cm above the grid floor. It protruded 2.0 cm from the front wall when extended. The chambers were illuminated by 7.5-W incandescent bulbs mounted to the ceiling of the sound-attenuation chamber. Ventilation fans provided background noise of 65 dBA.
The two sets of boxes had unique features that allowed them to serve as different contexts. In one set of boxes, the floor consisted of 0.5-cm-diameter stainless steel floor grids spaced 1.6 cm apart (center-to-center) and mounted parallel to the front wall. The ceiling and a side wall had black horizontal stripes, 3.8 cm wide and 3.8 cm apart. A distinct odor was continuously presented by placing approximately 10 ml of a 4% anise solution (Anise Extract; McCormack, Hunt Valley, MD) in a dish immediately outside the chamber. In the other set of chambers, the floor consisted of alternating stainless steel grids with different diameters (0.5 and 1.3 cm), spaced 1.6 cm apart. The ceiling and the left side wall were covered with dark dots (2 cm in diameter). An odor was provided by approximately 10 ml of an 8% coconut solution (Coconut Extract; McCormack) placed in a dish immediately outside the chamber. These two contexts were counterbalanced across rats as Contexts A and B. Reinforcement consisted of the delivery of a 45-mg food pellet into the food cup (MLab Rodent Tablets; TestDiet, Richmond, IN). The apparatus was controlled by computer equipment located in an adjacent room.
Procedure.
Food restriction began 1 week prior to the beginning of training. Animals were handled each day and maintained at their target body weight with a daily feeding.
Magazine training.
On the first day, the rats were trained to eat from the food magazine in a 30-min session in which approximately 60 pellets were delivered according to an RT 30-s schedule by initiating pellet delivery in a given second with a uniform 1 in 30 probability. One session occurred in Context A and another similar session was conducted in a box from the other set of chambers (Context B), with the order counterbalanced. Levers were retracted during these sessions.
Response training.
Response training began the day following magazine training. One session was conducted each day in Context A for a total of 8 days. The sessions consisted of the insertion of the left lever and then its retraction after 30 min. Lever press responses were reinforced with the delivery of a pellet according to an RI 30 s schedule. A timer determined pellet availability by querying a uniform 1 in 30 probability each second. Once the program indicated a pellet was available, the next press would deliver the pellet and restart the timer. No hand shaping was necessary.
Response elimination.
Response elimination began the day following the last day of response training and was conducted for one session per day in Context B for a total of 8 days. Rats were randomly divided into two groups (n = 16). Half of the rats were assigned to the Omission group and the other half were assigned to the Extinction group. Group Omission received omission training in which food pellets were delivered contingent on the nonoccurrence of lever pressing for a prespecified time interval. A food pellet was delivered every X s if a lever press did not occur; each lever press reset the time interval. If a rat did not lever press at all, a pellet would be delivered every X s. The omission interval (X) was set at 10 s on the 1st day of the phase, 15 s on the 2nd day, 20 s on the 3rd day, 25 s on the 4th day, and 30 s on the remaining 4 days. Group Extinction received extinction training in which the lever was present but no pellets were delivered. All sessions were 30 min in duration.
Test.
All rats were tested for lever pressing in both Contexts A and B on the day following the final response elimination session. Order of testing (Context A first, Context B first) was counterbalanced in each group. Test sessions were 10 min in duration; the lever was present, but presses had no scheduled consequences. Half of the rats in each group were tested with no pellets delivered, and the other half were tested with pellets delivered on an RT 30-s schedule.
Data analysis.
Response rates (responses/min) were analyzed with analysis of variance (ANOVAs) with an alpha level set to .05. The analyses compared groups based on between-subject factors (e.g., Treatment, Pellet), as well as on within-subject factors (e.g., Session). Where useful for interpreting the results, we report effect size (partial eta squared). Where evidence for or against a directional hypothesis is useful in interpreting the results, we report Bayes factors (BF) using a default prior of 0.707 (van Doorn et al., 2019). Statistical analyses were conducted with JASP software (JASP team, 2019).
Results
Response training.
The results of the response training phase are shown in the left panel of Figure 1. Response rate and reinforcer rate in the final session of response training (Session 8) for the individual rats are given in Tables 1 and 2. All rats readily acquired the operant response for food pellets and response rate increased across sessions of the acquisition phase. A Treatment (Omission, Extinction) by Pellet (Pellets, No Pellets) by Session (8) ANOVA found a significant effect of session, F(7, 196) = 103.25, MSE = 11.43, p < .001, and no other significant effects or interactions, largest F(7, 196) = 1.32.
Fig. 1.

Results of Experiment 1. Response rate (responses per minute) for each group plotted across sessions of the response training (Left) and response elimination (Right) phases. Error bars are the standard error of the mean and appropriate for between-group comparisons only.
Table 1:
Rates of lever pressing (responses/min) across phases of Experiment 1
| Training 8 | Response Elimination | Test | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Subject ID | Group | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 8* | Context B | Context A | |
| 12 | Extinction Pellets | 29.43 | 12.30 | 2.87 | 1.60 | 2.20 | 2.80 | 0.87 | 0.50 | 0.73 | 1.00 | 8.70 | 16.00 |
| 16 | Extinction Pellets | 22.10 | 5.60 | 2.70 | 0.30 | 0.30 | 0.27 | 0.17 | 0.53 | 0.33 | 1.20 | 3.70 | 14.60 |
| 112 | Extinction Pellets | 31.10 | 6.33 | 2.57 | 1.07 | 1.33 | 1.00 | 1.47 | 0.37 | 0.27 | 1.50 | 6.70 | 12.30 |
| 116 | Extinction Pellets | 37.93 | 5.33 | 1.00 | 0.57 | 0.30 | 0.30 | 0.30 | 0.47 | 0.40 | 0.20 | 5.50 | 18.60 |
| 117 | Extinction Pellets | 29.83 | 9.57 | 4.30 | 2.40 | 2.03 | 2.53 | 2.67 | 3.73 | 3.90 | 3.00 | 9.00 | 20.50 |
| 121 | Extinction Pellets | 33.77 | 11.03 | 5.10 | 1.80 | 0.53 | 0.47 | 0.40 | 0.40 | 0.33 | 1.10 | 9.00 | 17.40 |
| 127 | Extinction Pellets | 22.80 | 12.30 | 5.57 | 2.33 | 2.50 | 2.00 | 1.03 | 1.10 | 1.30 | 1.00 | 13.30 | 16.80 |
| 131 | Extinction Pellets | 31.73 | 8.47 | 4.07 | 1.13 | 1.40 | 1.70 | 0.67 | 0.67 | 0.63 | 0.30 | 6.60 | 12.40 |
| 13 | Extinction No Pellets | 29.60 | 10.63 | 3.93 | 1.43 | 1.93 | 0.80 | 0.63 | 0.30 | 0.30 | 0.60 | 1.10 | 7.20 |
| 17 | Extinction No Pellets | 22.90 | 5.50 | 4.00 | 2.13 | 2.07 | 1.57 | 0.87 | 0.50 | 1.23 | 0.00 | 0.10 | 7.80 |
| 19 | Extinction No Pellets | 20.87 | 2.93 | 1.80 | 1.77 | 0.87 | 1.13 | 0.73 | 0.63 | 1.27 | 1.90 | 1.90 | 5.20 |
| 113 | Extinction No Pellets | 27.47 | 8.77 | 3.43 | 3.17 | 1.87 | 1.50 | 0.80 | 0.50 | 0.77 | 0.20 | 1.30 | 7.90 |
| 120 | Extinction No Pellets | 30.67 | 16.70 | 11.67 | 4.37 | 0.63 | 0.27 | 0.17 | 0.00 | 0.03 | 2.00 | 0.60 | 12.10 |
| 124 | Extinction No Pellets | 26.10 | 3.93 | 4.60 | 1.17 | 0.63 | 0.47 | 0.27 | 0.27 | 0.17 | 0.40 | 0.20 | 7.90 |
| 126 | Extinction No Pellets | 30.77 | 20.80 | 9.27 | 4.00 | 2.57 | 1.73 | 1.20 | 1.30 | 2.63 | 0.50 | 1.00 | 13.00 |
| 130 | Extinction No Pellets | 23.57 | 5.70 | 3.07 | 3.63 | 2.23 | 4.37 | 1.23 | 0.53 | 0.13 | 1.90 | 0.90 | 18.10 |
| 11 | Omission Pellets | 25.93 | 9.03 | 4.13 | 0.63 | 0.27 | 0.60 | 0.23 | 0.13 | 0.43 | 0.60 | 4.70 | 5.50 |
| 15 | Omission Pellets | 27.10 | 9.43 | 3.53 | 1.77 | 2.70 | 2.50 | 1.73 | 1.20 | 0.47 | 0.30 | 1.10 | 5.00 |
| 111 | Omission Pellets | 22.60 | 11.17 | 3.90 | 1.80 | 2.20 | 1.97 | 1.73 | 1.27 | 2.00 | 0.30 | 3.00 | 3.10 |
| 115 | Omission Pellets | 30.53 | 9.77 | 2.70 | 1.07 | 0.50 | 0.57 | 0.37 | 0.13 | 0.20 | 0.30 | 9.20 | 15.70 |
| 118 | Omission Pellets | 29.67 | 7.67 | 7.00 | 4.87 | 3.17 | 4.23 | 3.27 | 2.47 | 3.03 | 4.50 | 7.60 | 5.80 |
| 122 | Omission Pellets | 23.17 | 11.90 | 3.73 | 1.20 | 0.87 | 0.70 | 0.53 | 1.07 | 0.70 | 0.30 | 0.90 | 10.10 |
| 128 | Omission Pellets | 18.37 | 9.10 | 2.37 | 1.33 | 0.87 | 1.30 | 0.67 | 0.67 | 0.70 | 2.30 | 1.40 | 16.00 |
| 182 | Omission Pellets | 32.87 | 2.30 | 0.83 | 0.43 | 0.40 | 0.20 | 0.17 | 0.17 | 0.40 | 0.80 | 0.10 | 3.00 |
| 14 | Omission No Pellets | 22.77 | 8.23 | 4.37 | 3.07 | 1.50 | 1.67 | 1.33 | 0.90 | 0.50 | 0.30 | 0.50 | 9.60 |
| 18 | Omission No Pellets | 19.13 | 0.43 | 1.43 | 0.13 | 0.20 | 0.07 | 0.10 | 0.00 | 0.00 | 0.60 | 1.90 | 9.80 |
| 110 | Omission No Pellets | 25.87 | 7.87 | 4.70 | 1.27 | 1.03 | 1.13 | 0.77 | 0.83 | 1.00 | 0.80 | 4.00 | 12.70 |
| 114 | Omission No Pellets | 43.70 | 10.03 | 2.57 | 1.23 | 0.80 | 0.63 | 0.67 | 0.73 | 0.10 | 0.90 | 1.60 | 7.20 |
| 119 | Omission No Pellets | 26.67 | 8.70 | 2.80 | 1.83 | 1.33 | 1.37 | 0.80 | 0.30 | 1.03 | 0.10 | 2.20 | 10.10 |
| 123 | Omission No Pellets | 23.90 | 2.83 | 1.10 | 0.60 | 0.30 | 0.77 | 0.30 | 0.47 | 0.23 | 0.20 | 4.40 | 7.90 |
| 125 | Omission No Pellets | 10.90 | 8.03 | 2.57 | 1.00 | 0.67 | 1.17 | 0.67 | 1.10 | 1.57 | 1.10 | 2.40 | 7.80 |
| 129 | Omission No Pellets | 16.43 | 8.60 | 3.00 | 1.30 | 2.37 | 2.23 | 0.90 | 1.10 | 1.13 | 0.00 | 2.90 | 4.60 |
Note. 8* is the response rate from the first 10 min of final (8) session of response elimination.
Table 2:
Rate of reinforcer presentation (reinforcers/min) across phases of Experiment 1
| Training | Response Elimination | Test | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Subject ID | Group | 8 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 8* | Context B | Context A |
| 12 | Extinction Pellets | 1.70 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 2.90 | 2.80 |
| 16 | Extinction Pellets | 1.87 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.60 | 2.30 |
| 112 | Extinction Pellets | 2.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.40 | 1.60 |
| 116 | Extinction Pellets | 1.87 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 3.10 | 2.40 |
| 117 | Extinction Pellets | 2.07 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 2.60 | 1.60 |
| 121 | Extinction Pellets | 2.13 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 2.00 | 2.70 |
| 12V | Extinction Pellets | 2.17 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 2.20 | 1.90 |
| 131 | Extinction Pellets | 1.70 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 2.30 | 2.70 |
| 13 | Extinction No Pellets | 1.67 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| 17 | Extinction No Pellets | 2.17 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| 19 | Extinction No Pellets | 1.77 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| 113 | Extinction No Pellets | 2.03 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| 120 | Extinction No Pellets | 2.07 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| 124 | Extinction No Pellets | 1.87 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| 126 | Extinction No Pellets | 1.90 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| 130 | Extinction No Pellets | 1.53 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| 11 | Omission Pellets | 1.83 | 2.77 | 3.07 | 2.57 | 1.77 | 1.30 | 1.70 | 1.90 | 1.77 | 1.70 | 1.80 | 2.10 |
| 15 | Omission Pellets | 1.43 | 3.43 | 2.73 | 2.87 | 2.27 | 1.90 | 1.93 | 1.80 | 1.87 | 1.90 | 1.60 | 1.60 |
| 111 | Omission Pellets | 1.80 | 3.47 | 3.10 | 2.60 | 2.00 | 1.80 | 1.53 | 1.87 | 1.87 | 1.80 | 2.40 | 1.70 |
| 115 | Omission Pellets | 1.93 | 3.97 | 3.53 | 2.80 | 2.30 | 1.90 | 1.87 | 1.80 | 1.87 | 1.90 | 2.40 | 2.10 |
| 118 | Omission Pellets | 1.80 | 2.70 | 2.23 | 2.00 | 1.40 | 1.00 | 1.03 | 0.93 | 0.63 | 0.30 | 2.00 | 2.60 |
| 122 | Omission Pellets | 2.00 | 2.03 | 2.07 | 2.43 | 2.23 | 1.83 | 1.83 | 1.83 | 1.87 | 1.80 | 2.10 | 2.60 |
| 128 | Omission Pellets | 1.60 | 1.73 | 2.03 | 2.03 | 1.40 | 1.17 | 1.67 | 1.70 | 1.63 | 1.40 | 1.90 | 1.00 |
| 132 | Omission Pellets | 1.73 | 3.20 | 2.47 | 2.63 | 1.83 | 1.40 | 1.80 | 1.90 | 1.83 | 1.80 | 2.50 | 2.50 |
| 14 | Omission No Pellets | 2.20 | 2.10 | 2.43 | 2.40 | 1.70 | 1.70 | 1.77 | 1.90 | 1.87 | 1.80 | 0.00 | 0.00 |
| 18 | Omission No Pellets | 1.80 | 4.03 | 2.20 | 2.10 | 1.63 | 1.53 | 1.57 | 1.73 | 1.73 | 1.80 | 0.00 | 0.00 |
| 110 | Omission No Pellets | 2.27 | 4.80 | 3.37 | 2.33 | 2.03 | 1.67 | 1.77 | 1.83 | 1.70 | 1.80 | 0.00 | 0.00 |
| 114 | Omission No Pellets | 1.97 | 3.17 | 2.83 | 1.90 | 1.70 | 1.60 | 1.70 | 1.90 | 1.83 | 1.90 | 0.00 | 0.00 |
| 119 | Omission No Pellets | 2.10 | 0.57 | 0.50 | 1.53 | 2.13 | 1.93 | 1.97 | 1.97 | 1.97 | 1.90 | 0.00 | 0.00 |
| 123 | Omission No Pellets | 2.17 | 4.17 | 2.07 | 2.40 | 2.10 | 1.73 | 1.90 | 1.87 | 1.93 | 1.90 | 0.00 | 0.00 |
| 125 | Omission No Pellets | 2.13 | 1.33 | 1.57 | 1.77 | 1.83 | 1.60 | 1.63 | 1.60 | 1.20 | 1.60 | 0.00 | 0.00 |
| 129 | Omission No Pellets | 1.73 | 3.77 | 2.87 | 1.63 | 1.60 | 0.77 | 1.53 | 1.77 | 1.93 | 1.90 | 0.00 | 0.00 |
Note. 8* is the response rate from the first 10 min of final (8) session of response elimination.
Response elimination.
The results of the response elimination phase are shown in the right panel of Figure 1. Response rates and reinforcer rates for each session of the response elimination phase are given for individual rats in Tables 1 and 2. All rats reduced their responding over the course of response elimination. There were no differences in rate of response elimination among groups receiving omission training versus extinction. These observations were supported by statistical analysis. A Treatment (Omission, Extinction) by Pellet (Pellets, No Pellets) by Session (8) ANOVA found a significant effect of session, F(7, 196) = 90.69, MSE = 2.45, p < .001, and no other significant effects or interactions, largest F(1, 28) = 1.48, MSE = 9.91.
Test.
The results of the test are shown in Figure 2 (see Tables 1 and 2 for individual data). We compared responding in the test in the response elimination context (Context B) with responding in the acquisition context (Context A). Return to the acquisition context (Context A) resulted in increased responding in all groups—an ABA renewal effect. In Group Omission, 15 out of 16 rats increased their response rate in Context A in comparison to Context B. In Group Extinction, 16 out of 16 rats responded at a higher rate in Context A than in Context B. The renewal effect was not influenced by the method of response elimination (extinction vs. omission training). Group Extinction Pellets made more responses than Group Extinction No Pellets in each test context—a reinstatement effect. Statistical analysis supported these observations. A Treatment (Omission, Extinction) by Pellet (Pellets, No Pellets) by Test Context (A, B) ANOVA revealed significant main effects of pellet, F(1, 28) = 14.87, MSE = 12.12, p < .001, ηp2 = .35, treatment, F(1, 28) = 11.78, p = .002, ηp2 = .30, and test context, F(1, 28) = 95.02, MSE = 8.27, p < .001, ηp2 = .77, as well as a significant pellet by treatment interaction, F(1, 28) = 13.46, p = .001, ηp2 = .32, and a treatment by context interaction, F(1, 28) = 5.15, p = .031, ηp2 = .16. The other interactions did not approach significance, Fs < 1.
Fig. 2.

Results of Experiment 1. Comparison of response rate in the test phase sessions. Groups were tested in each context (A and B) either with noncontingent pellets (Right) or without pellets (Left). Error bars are the standard error of the mean and appropriate for between- group comparisons only.
To decompose the interactions, we conducted separate Pellet by Context ANOVAs on response rates from Groups Omission and Extinction. For Group Omission, there was a significant effect of test context, F(1, 14) = 25.61, MSE = 9.03, p < .001, ηp2 = .65, indicating renewal. The effect of pellet and pellet by test context interaction did not reach significance, Fs < 1. For Group Extinction, there were significant effects of test context, F(1, 14) = 79.52, MSE = 7.50, p < .001, ηp2 = .85, and pellet, F(1, 14) = 34.89, MSE = 9.84, p < .001, ηp2 = .71. The pellet by test context interaction did not approach significance, F < 1.
In order to assess whether the response elimination treatment influenced ABA renewal, we compared responding in Context A (the renewal context) with a Treatment by Pellet ANOVA. There was a significant main effect of treatment, F(1, 28) = 11.41, MSE = 14.96, p = .002, ηp2 = .29. There was also a significant treatment by pellet interaction, F(1, 28) = 6.29, p = .018, ηp2 = .18. The main effect of pellet did not reach significance, F(1, 28) = 4.02, p = .055. Pairwise comparisons indicated that Group Extinction Pellets differed from each of the other groups, Fs (1, 14) ≥ 11.76, MSE = 12.97, p = .004, ηp2 = .50, which did not differ from one another, Fs < 1. Responding in Group Extinction Pellets was higher because of a reinstatement effect produced by presentation of the pellets. However, the similar responding in Groups Extinction No Pellets, Omission Pellets, and Omission No Pellets, provides no support for the possibility that the omission contingency decreased the renewed level of responding that was evident in Context A.
Groups Extinction Pellets and Omission Pellets received noncontingent pellets in the Context A and Context B tests. Individual reinforcer rate data for the tests are shown in Table 2. Each group received a similar rate of noncontingent reinforcers during the tests. A Group (Extinction Pellets, Omission Pellets) ANOVA compared obtained reinforcer rates in the tests and found no significant effects or interaction, Fs < 1.
Comparison of responding in the test in Context B to the end of the response elimination phase.
In order to further understand how responding changed in the response elimination context after the response elimination phase was complete, we compared responding in the test in the response elimination context (Context B) with responding in the first 10 min of the final session of the response elimination phase (in Context B). The mean data are plotted in Figure 3 and the individual-subject response and reinforcer rates are in Tables 1 and 2. There was clear evidence that pellet delivery reinstated responding in Group Extinction Pellets versus Group Extinction No Pellets. However, the pattern of results was perhaps more interesting in the two Omission groups. Groups Omission No Pellets and Omission Pellets were similar in the first 10 min of the final session of response elimination, and the removal of pellets resulted in a resurgence of responding in Group Omission No Pellets. However, there was also an unexpected increase in responding in Group Omission Pellets.
Fig. 3.

Results of Experiment 1. Comparison of response rate in the first 10 min of the final response elimination session (“Last R. E.”) with response rate in the 10-min test session in the same context (Test: Context B). Error bars are the standard error of the mean and appropriate for between-group comparisons only.
Statistical analysis supported the observations above. A Treatment (Omission, Extinction) by Pellets (Pellets, No Pellets) by Session (Final Response elimination, Context B Test) ANOVA revealed significant effects of pellets, F(1, 28) = 20.60, MSE = 3.79, p < .001, ηp2 = .42, and session, F(1, 28) = 45.69, MSE = 2.49, p < .001, ηp2 = .62, as well as significant treatment by pellets, F(1, 28) = 7.87, p = .009, ηp2 = .22, pellets by session, F(1, 28) = 19.00, p < .001, ηp2 = .40, and three-way interactions, F(1, 28) = 15.53, p < .001, ηp2 = .36. The treatment by session interaction did not reach significance, F(1, 28) = 2.01, p = .168, and the treatment effect was not significant, F(1, 28) = 2.59, p = .118. To decompose the interactions, we conducted separate Pellet by Session ANOVAs on response rates from Groups Omission and Extinction. For Group Omission, there was a significant effect of session, F(1, 14) = 12.25, MSE = 3.03, p = .004, ηp2 = .47, indicating some recovery of responding between response elimination and testing, but the pellet effect and pellet by session interaction did not approach significance, largest F = 1.22. For Group Omission No Pellets, 8 of 8 rats increased their responding in the Context B test. Interestingly, 6 of the 8 rats in Group Omission Pellets increased their responding in the test. Planned comparisons using the pooled error term found a significant increase in Groups Omission No Pellets, p = .018 and Omission Pellets, p = .039. For Group Extinction, there were significant effects of pellet, F(1, 14) = 35.04, MSE = 2.92, p < .001, ηp2 = .71, session, F(1, 14) = 40.00, MSE = 2.18, p < .001, ηp2 = .74, and a pellet by session interaction, F(1, 14) = 41.22, p < .001, ηp2 = .75. As is clear, the increase in Group Extinction Pellets was significant, F(1, 7) = 44.00, MSE = 4.02, p < .001, ηp2 = .86, whereas the evidence suggested that responding did not differ in the first 10 min of the final treatment session and the test in Context B in Group Extinction No Pellets, F(1, 7) < 1, BF = 2.93.
Inter-pellet times.
As noted above, there was evidence of a small increase in response rate in the Context B test session in comparison to the final response elimination session in both Omission groups. This result was unexpected in Group Omission Pellets, because reinforcers were presented at the same average rate during the test as they were obtained during the response elimination phase. However, pellet presentations in the two sessions differed in a manner that could have been discriminable. During the response elimination phase, the omission contingency arranged a minimum inter-pellet time of 30 s. However, because the pellet test condition involved an RT 30-s schedule, inter-pellet times were often shorter than 30 s in the test sessions. This is indeed what the rats experienced; the RT 30-s schedule resulted in a higher reinforcer rate in comparison to the first 10 min of the final session of response elimination, F(1, 7) = 6.32, MSE = 0.17, p = .040. To further illustrate, Figure 4 plots the relative frequency distributions of inter-pellet times obtained by Group Omission pellets in the final session of response elimination, in which the omission schedule arranged a 30-s pellet–pellet (and response–pellet) interval (Mean [M] = 35.52 s; Standard error of the mean [SEM] = 1.12), and in the test in Context B, in which the RT 30-s schedule arranged response-independent pellets (M = 27.99 s; SEM = 2.17). Based on the figure, it is clear that the distribution of the inter-pellet intervals differed across the two sessions for Group Omission Pellets.
Fig. 4.

Results of Experiment 1. Relative frequency of obtained inter-pellet times during the final session of response elimination and in the Context B test for Group Omission Pellets.
Discussion
Lever pressing in the omission and extinction groups did not differ during the response elimination phase, a result that might depend on the specific parameters used in omission scheduling (see Uhl & Garcia, 1969). However, the groups differed in several notable ways during the test. In Group Extinction, responding increased when tested in the response training context (Context A), demonstrating ABA renewal. Noncontingent pellets also caused extinguished responding to increase regardless of whether the test occurred in Context B or Context A. That is, the reinstatement effect was similar in Contexts A and B and did not interact with whether the response was experiencing renewal. Context and free pellets had what appeared to be additive effects on responding after extinction (see also Trask & Bouton, 2016).
Responding also increased for the omission group when tested in Context A, with little evidence that omission training had an impact on ABA renewal. The effect of noncontingent pellet deliveries during the test, however, was different for rats following omission training. When compared to Group Extinction, there was no evidence to suggest that noncontingent pellets influenced responding during the test following omission training. The fact that the pellets did not augment responding relative to a no-pellet condition after omission is consistent with analyses that emphasize the role of the discriminative properties of the reinforcer in omission learning (Uhl & Garcia, 1969), as well as the observation that noncontingent reinforcers presented during response elimination through extinction fail to reinstate responding when they are also introduced during later testing (Baker, 1990; Rescorla & Skucy, 1969; Winterbauer & Bouton, 2011).
The results also suggest that Groups Omission Pellets and Omission No Pellets increased their responding during testing in Context B after the response had been eliminated there (Fig. 3). For Group Omission No Pellets, this result is consistent with a resurgence effect which might occur when discriminative control over inhibitory learning by pellets is removed (Bouton & Trask, 2016; Trask, Keim, & Bouton, 2018). According to this view, Group Omission Pellets was expected to remain suppressed. However, this group also appeared to increase its responding modestly. Perhaps, then, the response recovery was due at least partly to removal of the omission contingency, which occurred for both groups. Upon closer inspection, however, the temporal distribution of pellets during the test differed from that arranged by the omission schedule (Fig. 4). This could suggest the interesting possibility that the temporal distribution of the pellets acquired discriminative control over inhibition of responding. Experiment 2 explored this possibility.
Experiment 2
Experiment 2 was designed to test whether the modest increase in responding observed in Group Omission Pellets in the Context B test in Experiment 1 could have been the result of the change in temporal distribution of pellets between the response elimination phase and the test. Rats received response training in Context A and omission training in Context B. They were then divided into two groups and received a single test session in Context B. The groups differed in the temporal distribution of noncontingent pellets received during the test. In Group Variable Time (VT), each rat received the noncontingent pellets according to the temporal spacing of noncontingent pellets experienced by one of the rats in Group Omission Pellets during the test in Context B in Experiment 1. In Group Yoked, each rat received noncontingent pellets according to the exact temporal spacing of pellets that it experienced in the response elimination session that immediately preceded the test. We expected Group VT to increase their responding in the test and replicate the effect observed in Group Omission Pellets in Experiment 1. However, if the temporal distribution of pellets acquires a discriminative function over inhibitory learning in omission training, then we expected responding in Group Yoked to remain suppressed in the test.
Method
Subjects and apparatus.
Sixteen Wistar rats acquired from the same supplier were housed and maintained under conditions identical to those described in Experiment 1. The experiment took place in the same set of operant chambers as in Experiment 1.
Procedure.
Magazine training, response training, and response elimination phases were identical to those for Group Omission in Experiment 1. At the conclusion of the response elimination phase, rats were divided into two groups. On the following day, all the rats received a single 10-min test session with the lever present in Context B. In the test, lever presses were recorded but had no programmed consequence. All rats received noncontingent pellets during the test. Each rat in Group Yoked received pellets according to the exact sequence of inter-pellet intervals that she experienced in the final response elimination session that occurred in the first 10 min of the session on the day before the test. (These were extracted from the computer record.) For Group VT, each rat received noncontingent pellets according to the sequence of inter-pellet intervals experienced in Context B test of Experiment 1 by the rat run in the same operant chamber in Group Omission Pellet.
Results
Response training and elimination.
All rats acquired the operant lever press response in Context A without incident. The individual subject response rate and reinforcer rate data from the training phase are shown in Tables 3 and 4. A Group (Yoked, VT) by Session (8) ANOVA found a significant effect of session, F(7, 98) = 25.05, MSE = 17.79, p < .001. The effects involving the group factor did not approach significance, Fs < 1. Likewise, training with the omission contingency in Context B rapidly suppressed responding in all rats during the response elimination phase. These data are shown in the left panel of Figure 5. A Group by Session ANOVA once again found a significant effect of session, F(7, 98) = 14.23, MSE = 19.96, p < .001, and no significant effects involving the group factor, Fs < 1.
Table 3:
Rate of response (responses/min) across phases of Experiment 2
| Training | Response Elimination | Test | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Subject ID | Group | 8 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 8* | Context B |
| 21 | Yoked | 16.67 | 3.97 | 3.90 | 3.30 | 1.63 | 1.60 | 0.57 | 0.30 | 0.33 | 0.30 | 4.30 |
| 22 | Yoked | 25.17 | 7.10 | 1.87 | 1.03 | 1.43 | 3.47 | 3.00 | 2.13 | 1.00 | 1.40 | 1.10 |
| 25 | Yoked | 34.93 | 1.40 | 1.43 | 0.93 | 0.90 | 0.37 | 1.07 | 0.27 | 0.53 | 0.60 | 1.20 |
| 26 | Yoked | 22.53 | 13.23 | 4.57 | 0.70 | 0.67 | 0.70 | 0.53 | 0.70 | 0.47 | 0.20 | 0.50 |
| 211 | Yoked | 18.77 | 3.07 | 1.67 | 1.10 | 1.53 | 1.13 | 1.43 | 0.53 | 1.07 | 2.00 | 1.50 |
| 212 | Yoked | 34.90 | 13.90 | 1.30 | 0.83 | 0.73 | 1.23 | 0.27 | 0.53 | 0.23 | 0.30 | 0.50 |
| 215 | Yoked | 20.80 | 5.13 | 2.07 | 1.10 | 1.47 | 1.00 | 0.87 | 0.43 | 2.30 | 3.00 | 2.20 |
| 216 | Yoked | 23.27 | 9.47 | 2.53 | 1.27 | 2.20 | 1.80 | 0.37 | 1.07 | 0.83 | 0.40 | 4.30 |
| 23 | VT | 17.83 | 1.57 | 0.43 | 0.40 | 0.57 | 1.93 | 0.40 | 0.73 | 0.47 | 1.30 | 4.80 |
| 21 | VT | 26.87 | 2.23 | 0.43 | 0.57 | 0.33 | 2.27 | 0.97 | 1.43 | 1.13 | 1.70 | 2.30 |
| 27 | VT | 16.47 | 7.10 | 3.57 | 1.43 | 1.97 | 1.10 | 0.30 | 0.93 | 0.50 | 0.70 | 2.60 |
| 28 | VT | 28.33 | 0.60 | 0.23 | 0.30 | 0.13 | 1.27 | 0.53 | 0.33 | 0.27 | 0.40 | 0.40 |
| 29 | VT | 19.30 | 10.73 | 9.30 | 6.43 | 5.00 | 1.63 | 0.77 | 0.80 | 1.67 | 2.40 | 0.80 |
| 210 | VT | 18.00 | 0.67 | 0.67 | 0.57 | 0.30 | 2.80 | 0.57 | 1.13 | 0.57 | 0.40 | 0.10 |
| 213 | VT | 57.00 | 15.63 | 1.37 | 1.33 | 3.50 | 0.97 | 0.53 | 0.77 | 0.67 | 1.10 | 0.00 |
| 214 | VT | 26.27 | 13.57 | 1.67 | 1.23 | 1.07 | 0.57 | 1.67 | 0.83 | 3.70 | 2.70 | 3.90 |
Note. 8* is the response rate from the first 10 min of final (8) session of response elimination.
Table 4:
Rate of reinforcer presentation (reinforcers/min) across phases of Experiment 2
| Training | Response Elimination | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Subject ID | Group | 8 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 8* | Test Context B |
| 21 | Yoked | 1.90 | 4.23 | 2.47 | 1.70 | 1.93 | 1.63 | 1.80 | 1.90 | 1.90 | 1.80 | 1.80 |
| 22 | Yoked | 1.80 | 3.37 | 3.20 | 2.63 | 1.90 | 0.70 | 1.13 | 1.43 | 1.63 | 1.90 | 1.40 |
| 25 | Yoked | 2.00 | 5.50 | 3.40 | 2.63 | 2.03 | 1.83 | 1.77 | 1.90 | 1.87 | 1.90 | 1.80 |
| 26 | Yoked | 1.93 | 1.43 | 2.57 | 2.77 | 2.13 | 1.83 | 1.90 | 1.80 | 1.90 | 1.40 | 1.90 |
| 211 | Yoked | 1.77 | 4.70 | 3.33 | 2.70 | 2.00 | 1.67 | 1.60 | 1.87 | 1.60 | 1.30 | 1.30 |
| 212 | Yoked | 1.67 | 2.57 | 3.57 | 2.77 | 2.17 | 1.57 | 1.93 | 1.87 | 1.90 | 1.80 | 1.90 |
| 215 | Yoked | 1.47 | 4.10 | 3.10 | 2.50 | 1.77 | 1.60 | 1.80 | 1.87 | 1.53 | 1.30 | 1.30 |
| 216 | Yoked | 1.80 | 2.80 | 2.80 | 2.43 | 1.77 | 1.33 | 1.80 | 1.77 | 1.77 | 1.90 | 1.80 |
| 23 | VT | 2.00 | 5.13 | 3.83 | 2.83 | 2.27 | 1.20 | 1.87 | 1.80 | 1.87 | 1.70 | 1.70 |
| 24 | VT | 2.53 | 4.83 | 3.80 | 2.87 | 2.30 | 1.70 | 1.77 | 1.63 | 1.73 | 1.90 | 2.10 |
| 27 | VT | 2.23 | 3.10 | 2.67 | 2.40 | 1.60 | 1.60 | 1.90 | 1.57 | 1.80 | 1.70 | 1.00 |
| 28 | VT | 1.90 | 5.60 | 3.87 | 2.90 | 2.33 | 1.70 | 1.87 | 1.90 | 1.93 | 1.50 | 2.50 |
| 29 | VT | 1.97 | 1.73 | 0.57 | 0.77 | 0.73 | 1.40 | 1.70 | 1.77 | 1.57 | 1.80 | 2.10 |
| 210 | VT | 2.33 | 5.57 | 3.70 | 2.77 | 2.23 | 1.20 | 1.80 | 1.67 | 1.87 | 1.00 | 1.60 |
| 213 | VT | 2.27 | 2.33 | 3.57 | 2.70 | 1.37 | 1.63 | 1.90 | 1.70 | 1.87 | 1.30 | 2.60 |
| 214 | VT | 2.30 | 1.87 | 3.47 | 2.60 | 2.10 | 1.87 | 1.57 | 1.77 | 1.03 | 1.80 | 2.60 |
Note. 8* is the response rate from the first 10 min of final (8) session of response elimination.
Fig. 5.

Results of Experiment 2. Response rate (responses per minute) for each group plotted across sessions during response elimination. Response rates in the 10-min test are compared with those of the first 10 min of the last response elimination session (“Last R. E.”) at right. Error bars are the standard error of the mean and appropriate for between-group comparisons only.
Test.
Of primary interest was the comparison of responding in the first 10 min of the final response elimination session and responding in the test in each group. These results are shown in the right panel of Figure 5. Each group responded modestly more in the test than in the final session of response elimination. However, whether the sequence of inter-pellet intervals matched that received during response elimination did not influence responding in the test. A Group (VT, Yoked) by Session (Last Response elimination, Test) ANOVA did not find a significant effect of session, F(1, 15) = 2.76, MSE = 1.53, p = .118, nor group or interaction, Fs < 1. An increase in response rate was observed in 5 of 8 rats in Group VT, and 4 of 8 rats in Group Yoked. Planned comparisons using the pooled error term of response rate in the first 10 min of the final response elimination session with response rate in the test for each group found no evidence of an increase in Group VT, p = .167, or Group Yoked, p = .422. Reinforcer rates are shown in Table 4 and Figure 6 shows the relative frequency of obtained inter-pellet times during the test session for Groups VT and Yoked. Groups VT and Yoked did not differ in the rate of reinforcers obtained in the first 10 min of the final session of response elimination and the test session, largest F(1, 14) = 3.43, MSE = 0.12, p = .085, BF = 0.75.
Fig. 6.

Results of Experiment 2. Relative frequency of obtained inter-pellet times during the test session for Groups Yoked and VT.
Discussion
Experiment 2 was designed to test the possibility that the response recovery observed in Group Omission Pellet in the Context B test in Experiment 1 was due to the difference in the temporal distribution of pellets experienced between response elimination and testing. As can be seen in Figure 6, we obtained the expected difference in the distribution of inter-pellet times in the two groups during testing. However, results did not replicate the response recovery observed in Experiment 1. Since Group VT was a direct replication of Group Omission Pellets from Experiment 1, the results of the two groups were combined. Here the results of the two experiments continued to suggest a small recovery of responding during testing: An Experiment (1, 2) by Session (Last Response Elimination, Test: Context B) ANOVA found a significant effect of session, F(1, 14) = 4.98, MSE = 3.26, p = .042, ηp2 = .26. While the modest response recovery thus appears to be real, it could not be differentiated from Group Yoked, a group that received noncontingent pellets at exactly the times they had received omission-contingent pellets in the last response elimination session before the test. This means that the Omission Pellets group in Experiment 1 could have merely been sensitive to the removal of the omission contingency, rather than the change in the temporal distribution of the pellets. We note that responding did not differ over the last 2 days (days 7 and 8) of the response elimination phase (F < 1), when the omission contingency was in force, whereas it did increase from day 8 to the test session, when the omission contingency was removed. Thus, mere removal of the negative contingency between behavior and the reinforcer may be sufficient to cause a modest response recovery after a DRO treatment.
General Discussion
The present research was designed to examine the extent to which responding eliminated by extinction or omission training would be differentially sensitive to a renewal effect created by returning the response to the original training context and a reinstatement effect created by presentation of noncontingent reinforcers. In Experiment 1, groups whose responding was eliminated by either omission or extinction both showed higher response rates when tested in the response training context (Context A) than in the response elimination context (Context B), thus demonstrating ABA renewal after either response elimination procedure. There was no evidence that omission training attenuated this ABA renewal effect in that all groups (except the Extinction Pellets group, which was additionally influenced by the reinstating effects of pellet presentation) showed an equivalent recovery of responding in Context A. The results are consistent with previous research suggesting that ABA renewal can occur after omission training (Kearns & Weiss, 2007; Nakajima et al. 2002), although our results differ from Kearns and Weiss’s (2007) findings that omission training (as well as noncontingent reinforcers) mitigated renewal in their procedure. One explanation of the difference in our findings is that the present procedure did not allow for a response competition mechanism that Kearns and Weiss suggested explained their own results. Recall that they had used a discriminated operant procedure in which the target response was reinforced with cocaine during the presence of a tone and then eliminated with a DRO or FT procedure using food reinforcers that was also in effect only in the presence of the tone. The authors suggested that the tone might have evoked new food-related behaviors (e.g., magazine entries) that interfered with the cocaine-related behavior (i.e., the target response).
After extinction, the presentation of noncontingent pellets during the test increased responding regardless of whether it occurred in Context A or B. This was the expected reinstatement effect (e.g., Baker, 1990; Rescorla & Skucy, 1969). We also found more responding in Group Extinction Pellets than Group Omission Pellets in both contexts, indicating that noncontingent pellets caused more response recovery after extinction than omission training. The results are consistent with the idea that pellets were a part of the response elimination context where rats learned to inhibit responding for the omission group but not the extinction group. Therefore, pellets likely had some discriminative control over response inhibition for the omission group. But the opposite effect was true for the extinction group, whose only prior exposure to pellets was in the response training context, where pellets set the occasion for continued responding (e.g., Bouton & Trask, 2016; Thrailkill, Ameden, & Bouton, 2019). The findings and interpretation are consistent with prior research suggesting that behavior eliminated with omission training shows little to no increase during tests with noncontingent reinforcer delivery (Buel, 1975; Harman, 1973; Topping & Larmi, 1973; Uhl, 1973, 1974; Uhl and Garcia, 1969; Uhl & Sherman, 1971).
When we compared response rates during the last day of response elimination to rates during the test in the response elimination context (Context B), we found that Group Omission Pellets and Group Omission No Pellets both showed a small increase in their responding. This effect suggested that both presenting noncontingent pellets and removing the pellets delivered during omission training can cause some response recovery after the present omission procedure. Experiment 2 tested whether the slight, yet unexpected, response increase in Group Omission Pellet was due to the difference in the temporal distribution of pellets delivered during omission training and testing (Fig. 4). We found that both a group that had a new temporal distribution of pellets during testing (Group VT) and a group that had a pellet distribution that was the same as in the last response elimination session (Group Yoked) increased responding slightly during the test. Although that increase was not significant, a significant response increase from the last session of response elimination to the test was still detected when we combined Group Omission Pellets from Experiment 1 and Group VT from Experiment 2, indicating there may be a small and not easily detectable effect. The fact that we did not find a difference between the VT group and a group that received the same pellet distribution as in the final omission training session provides no support for the idea that the rats were responding to a change in the temporal pellet distribution. We therefore tentatively conclude that rats in Experiment 1’s Group Omission Pellets may have increased responding during testing with pellets because of the mere removal of the omission contingency.
Perhaps the most surprising finding in Experiment 1 was that there was no difference in responding between Group Omission Pellets and Group Omission No Pellets during testing. For Group Omission, pellet delivery was a part of the response elimination context, and we thus expected pellets in the test to continue to inhibit responding and the absence of pellets to result in more response recovery (Trask & Bouton, 2016). It is unclear why we did not find a more pronounced resurgence effect in the omission group, but there are several possibilities. First, because of the response cost inherent in the omission contingency, omission training might produce a stronger response inhibition than other response elimination procedures. Unlike extinction, DRA, or noncontingent reinforcement, the omission contingency has an integral negative punishment contingency in which each response further delays reinforcer delivery by resetting the response-pellet interval. The added cost to responding in the omission contingency makes it so that the organism “has something to lose” for responding and might deter full response recovery when conditions are ambiguous or it is unclear whether or not the contingency is in effect. Of course, the fact that omission training left the response no less vulnerable to renewal than after extinction in Experiment 1 suggests we should exercise caution in accepting this possibility. It is also notable that renewal readily occurs even after positive punishment procedures (e.g., Bouton & Schepers, 2015; see Bouton, 2019, for more discussion).
Second, it is important to note that the omission contingency may be difficult to discriminate from other conditions because its consequences are delayed and there is no immediate stimulus change that is contingent on responding. Since the omission contingency is difficult to detect, it might also be difficult to detect when the contingency is lifted. For example, the suspension of reinforcer delivery for not responding could be less discriminable than the suspension of reinforcer delivery for responding (as in typical resurgence procedures, which study response recovery after DRA). This would be consistent with research showing less resurgence after omission training than DRA (e.g., Mulick et al., 1976; Pacitti & Smith, 1977) but inconsistent with studies showing more resurgence after omission training than after DRA (Doughty, da Silva, & Lattal, 2007; Romano & St. Peter, 2016). Another factor that might have contributed to the minimal response recovery we found when we removed pellets after omission training is the fading procedure we used on the first four sessions of omission training. During those sessions, pellet–pellet and response–pellet intervals started at 10 s and increased by 5 s each day until they reached 30 s. Progressively increasing the interval would have exposed rats to “longer than expected” pellet–pellet intervals while the omission contingency was in effect. For example, if rats received pellets about every 10 s by the end of the first omission training session, the 15-s pellet–pellet interval on the 2nd day would have been longer than expected. Perhaps schedule thinning during omission training made the target response more resistant to resurgence because it gave rats prior experience with lengthening pellet– pellet intervals (relative to previous days) while the omission contingency was in force.
Though we did not find evidence that omission training attenuated ABA renewal compared to extinction, it is worth noting that it is difficult to arrange a perfect comparison of renewal after the two procedures. We could assess a pure effect of context in Group Extinction when we tested without food pellets; the fact that there was no change in responding for Group Extinction No Pellet when tested in Context B, relative to the final day of response elimination, suggests that renewal in Context A was due solely to the change in context. However, it is more difficult to conduct a pure assessment after omission training, because the animals receive reinforcers during that training. Thus, if one were to study renewal in the absence of pellets after omission training, the omission group would receive more stimulus change than a comparison extinction group (i.e., both context change and pellet removal). This is a major reason why we included the current tests with noncontingent pellets; in the pellet condition, omission rats were expected to experience context change without pellet change. However, we did not foresee that removal of the omission contingency itself (while retaining the pellets) might also allow some response recovery. Responding in the omission groups was thus apparently influenced by that modest effect as well.
In conclusion, the main finding of the current study is that behavior eliminated by omission training can be resistant to reinstatement created by reinforcer presentation and resurgence created by reinforcer removal, but may be no more resistant to ABA renewal than behavior eliminated by extinction. For applied behavior analysts, this pattern may mean that treatment effects might be durable when DRO treatment is interrupted by errors of commission (the therapist delivering reinforcers at the wrong time) or omission (the therapist missing reinforcer deliveries). However, if treatment is implemented outside of the natural environment (treatment setting, hospital, rehab) extra care should be taken when transitioning back home.
Acknowledgment
This research was supported by National Institutes of Health Grant RO1 DA 033123 to MEB. EAT was supported by National Institutes of Health Grant KO1 DA 044456.
References
- Baker AG (1990). Contextual conditioning during free-operant extinction: Unsignaled, signaled, and backward-signaled noncontingent food. Animal Learning and Behavior, 18, 199–218. 10.3758/BF03205240 [DOI] [Google Scholar]
- Bouton ME (2017). Extinction: Behavioral mechanisms and their implications In Byrne JH, Sweatt D, Menzel R, Eichenbaum H, & Roediger H (Eds.), Learning and memory: A comprehensive reference, 2nd Ed. (Vol. 1, Learning Theory and Behaviour) Oxford: Academic Press; 10.1016/B978-0-12-809324-5.21006-7 [DOI] [Google Scholar]
- Bouton ME (2019). Extinction of instrumental (operant) learning: interference, varieties of context, and mechanisms of contextual control. Psychopharmacology, 236, 7–19. 10.1007/s00213-018-5076-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bouton ME, & Schepers ST (2014). Resurgence of instrumental behavior after an abstinence contingency. Learning & Behavior, 42, 131–143. 10.3758/s13420-013-0130-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bouton ME, & Schepers ST (2015). Renewal after the punishment of free operant behavior. Journal of Experimental Psychology: Animal Learning and Cognition, 41, 81–90. 10.1037/xan0000051 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bouton ME, & Swartzentruber D (1991). Sources of relapse after extinction in Pavlovian and instrumental learning. Clinical Psychology Review, 11, 123–140. 10.1016/0272-7358(91)90091-8 [DOI] [Google Scholar]
- Bouton ME, Todd TP, Vurbic D, & Winterbauer NE (2011). Renewal after the extinction of free operant behavior. Learning & Behavior, 39, 57–67. 10.3758/s13420-011-0018-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bouton ME, & Trask S (2016). Role of the discriminative properties of the reinforcer in resurgence. Learning & Behavior, 44, 137–150. 10.3758/s13420-015-0197-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Buel CL (1975). Investigation of the temporal parameters in omission training with humans in a two-key situation. The Psychological Record, 25, 99–109. 10.1007/bf03394293 [DOI] [Google Scholar]
- Cuvo AJ, Reagan AL, Ackerlund J, Huckfeldt R, & Kelly C (2010). Training children with autism spectrum disorders to be compliant with a physical exam. Research in Autism Spectrum Disorders, 4, 168–185. 10.1016/j.rasd.2009.09.001 [DOI] [Google Scholar]
- Davidson EH, & Walker JL (1970). Resistance to extinction as a function of acquisition and extinction schedules and type of reinforcement. Psychonomic Science, 19, 63–64. 10.3758/BF03335509 [DOI] [Google Scholar]
- Davis DR, Kurti AN, Skelly JM, Redner R, White TJ, & Higgins ST (2016). A review of the literature on contingency management in the treatment of substance use disorders, 2009–2014. Preventive Medicine, 92, 36–46. 10.1016/j.ypmed.2016.08.008 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Doughty AH, da Silva SP, & Lattal KA (2007). Differential resurgence and response elimination. Behavioural Processes, 75, 115–128. 10.1016/j.beproc.2007.02.025 [DOI] [PubMed] [Google Scholar]
- Gratz OH, Wilson AN, & Glassford T (2019). Evaluating the resurgence of problem behavior with three functionally equivalent discriminated operants. The Psychological Record, 69, 117–129. 10.1007/s40732-018-0305-0 [DOI] [Google Scholar]
- Harman RE (1973). Response elimination in concurrent and single operant situations with pigeons. Learning and Motivation, 4, 417–431. 10.1016/0023-9690(73)90007-6 [DOI] [Google Scholar]
- Higgins ST, Delaney DD, Budney AJ, Bickel WK, Hughes JR, Foerg BA, & Fenwick JW (1991). A behavioral approach to achieving initial cocaine abstinence. The American Journal of Psychiatry, 148, 1218–1224. 10.1176/ajp.148.9.1218 [DOI] [PubMed] [Google Scholar]
- Hoffman K, & Falcomata TS (2014). An evaluation of resurgence of appropriate communication in individuals with autism who exhibit severe problem behavior. Journal of Applied Behavior Analysis, 47, 651–656. 10.1002/jaba.144. [DOI] [PubMed] [Google Scholar]
- Homer AL, & Peterson L (1980). Differential reinforcement of other behavior: A preferred response elimination procedure. Behavior Therapy, 11, 449–471. 10.1016/s0005-7894(80)80063-3 [DOI] [Google Scholar]
- Iannaccone JA, Hagopian LP, Javed N, Borrero JC, & Zarcone JR (2019). Rules and statements of reinforcer loss in differential reinforcement of other behavior. Behavior Analysis in Practice. 10.1007/s40617-019-00352-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- JASP Team (2019). JASP (Version 0.10) [Computer software]. https://jasp-stats.org/download/
- Kearns DN, & Weiss SJ (2007). Contextual renewal of cocaine seeking in rats and its attenuation by the conditioned effects of an alternative reinforcer. Drug and Alcohol Dependence, 90, 193–202. 10.1016/j.drugalcdep.2007.03.006 [DOI] [PubMed] [Google Scholar]
- Liddon CJ, Kelley ME, Rey CN, Liggett AP & Ribeiro A (2018). A translational analysis of ABA and ABC renewal of operant behavior. Journal of Applied Behavior Analysis, 51, 819–830. 10.1002/jaba.496 [DOI] [PubMed] [Google Scholar]
- Lieving GA, Hagopian LP, Long ES, & O’Connor J (2004). Response-class hierarchies and resurgence of severe problem behavior. Psychological Record, 54, 621–634. 10.1007/bf03395495 [DOI] [Google Scholar]
- Marsteller TM, & St. Peter CC (2014). Effects of fixed-time reinforcement schedules on resurgence of problem behavior. Journal of Applied Behavior Analysis, 47, 455–469. 10.1002/jaba.134. [DOI] [PubMed] [Google Scholar]
- Mulick JA, Leitenberg H, & Rawson RA (1976). Alternative response training, differential reinforcement of other behavior, and extinction in squirrel monkeys (Saimiri sciureus). Journal of the Experimental Analysis of Behavior, 25, 311–320. 10.1901/jeab.1976.25-311 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Nakajima S, Urushihara K, & Masaki T (2002). Renewal of operant performance formerly eliminated by omission or noncontingency training upon return to the acquisition context. Learning and Motivation, 33, 510–525. 10.1016/S0023-9690(02)00009-7 [DOI] [Google Scholar]
- Neely L, Graber J, Kunnavatana S, & Cantrell K (2019). Impact of language on behavior treatment outcomes. Journal of Applied Behavior Analysis. 10.1002/jaba.626 [DOI] [PubMed] [Google Scholar]
- Pacitti WA, & Smith NF (1977). A direct comparison of four methods for eliminating a response. Learning and Motivation, 8, 229–237. 10.1016/0023-9690(77)90007-8 [DOI] [Google Scholar]
- Podlesnik CA, & Shahan TA (2010). Extinction, relapse, and behavioral momentum. Behavioural Processes, 84, 400–411. 10.1016/j.beproc.2010.02.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Poling A, & Ryan C (1982). Differential reinforcement of other behavior schedules: Therapeutic applications. Behavior Modification, 6, 3–21. 10.1177/01454455820061001 [DOI] [Google Scholar]
- Rescorla RA, & Skucy JC (1969). Effect of response-independent reinforcers during extinction. Journal of Comparative and Physiological Psychology, 67, 381–389. 10.1037/h0026793 [DOI] [Google Scholar]
- Reynolds GS (1961). Behavioral contrast. Journal of the Experimental Analysis of Behavior, 4, 57–71. 10.1901/jeab.1961.4-57 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Romano LM, & St. Peter CC (2016). Omission training results in more resurgence than alternative reinforcement. The Psychological Record, 67, 315–324. 10.1007/s40732-016-0214-z [DOI] [Google Scholar]
- Shabani DB, & Fisher WW (2006). Stimulus fading and differential reinforcement for the treatment of needle phobia in a youth with autism. Journal of Applied Behavior Analysis, 39, 449–452. 10.1901/jaba.2006.30-05 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Silverman K, Higgins ST, Brooner RK, Montoya ID, Cone EJ, Schuster CR, & Preston KL (1996). Sustained cocaine abstinence in methadone maintenance patients through voucher-based reinforcement therapy. Archives of General Psychiatry, 53, 409–415. 10.1001/archpsyc.1996.01830050045007 [DOI] [PubMed] [Google Scholar]
- Slifer KJ, Avis KT, & Frutchey RA (2008). Behavioral intervention to increase compliance with electroencephalographic procedures in children with developmental disabilities. Epilepsy & Behavior, 13, 189–195. 10.1016/j.yebeh.2008.01.013 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Thrailkill EA, Ameden WC, & Bouton ME (2019). Resurgence in humans: Reducing relapse by increasing generalization between treatment and testing. Journal of Experimental Psychology: Animal Learning and Cognition, 45, 338–349. 10.1037/xan0000209 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Topping JS, & Ford TW (1974). Response elimination with DRO and extinction: A within-subject comparison. The Psychological Record, 24, 563–568. 10.1007/BF03394279 [DOI] [Google Scholar]
- Topping JS, & Larmi OK (1973). Response elimination effectiveness of omission and two extinction training procedures. The Psychological Record, 23, 197–202. 10.1007/BF03394155 [DOI] [Google Scholar]
- Trask S, & Bouton ME (2016). Discriminative properties of the reinforcer can be used to attenuate the renewal of extinguished operant behavior. Learning and Behavior, 44, 151–161. 10.3758/s13420-015-0195-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Trask S, Keim CL, & Bouton ME (2018). Factors that encourage generalization from extinction to test reduce resurgence of an extinguished operant response. Journal of the Experimental Analysis of Behavior, 110, 11–23. 10.1002/jeab.446 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Trask S, Schepers ST, & Bouton ME (2015). Context change explains resurgence after the extinction of operant behavior. Revista Mexicana de Analisis de la Conducta (Mexican Journal of Behavior Analysis), 41, 187–210. 10.5514/rmac.v41.i2.63772 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Uhl CN (1973). Eliminating behavior with omission and extinction after varying amounts of training. Animal Learning & Behavior, 1, 237–240. 10.3758/bf03199082 [DOI] [Google Scholar]
- Uhl CN (1974). Response elimination in rats with schedules of omission training, including yoked and response-independent reinforcement comparisons. Learning and Motivation, 5, 511–531. 10.1016/0023-9690(74)90008-3 [DOI] [Google Scholar]
- Uhl CN, & Garcia EE (1969). Comparison of omission with extinction in response elimination in rats. Journal of Comparative and Physiological Psychology, 69, 554–562. 10.1037/h0028243 [DOI] [Google Scholar]
- Uhl CN, & Sherman WO (1971). Comparison of combinations of omission, punishment, and extinction methods in response elimination in rats. Journal of Comparative and Physiological Psychology, 74, 59–65. 10.1037/h0030346 [DOI] [Google Scholar]
- van Doorn J, van den Bergh D, Bohm U, Dablander F, Derks K, Draws T, … Wagenmakers E (2019). The JASP guidelines for conducting and reporting a Bayesian analysis. PsyArXiv. 10.31234/osf.io/yqxfr [DOI] [PMC free article] [PubMed] [Google Scholar]
- Volkert VM, Lerman DC, Call NA, & Trosclair-Lasserre N (2009). An evaluation of resurgence during treatment with functional communication training. Journal of Applied Behavior Analysis, 42, 145–160. 10.1901/jaba.2009.42-145. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wacker DP, Harding JW, Morgan TA, Berg WK, Schieltz KM, Lee JF, & Padilla YC (2013). An evaluation of resurgence during functional communication training. Psychological Record, 63, 3–20. 10.1901/jaba.2009.42-145. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Weston R, Hodges A, & Davis TN (2018). Differential reinforcement of other behaviors to treat challenging behaviors among children with autism: A systematic and quality review. Behavior Modification, 42, 584–609. 10.1177/0145445517743487 [DOI] [PubMed] [Google Scholar]
- Whitaker S (1996). A review of DRO: The influence of the degree of intellectual disability and the frequency of the target behaviour. Journal of Applied Research in Intellectual Disabilities, 9, 61–79. 10.1111/j.1468-3148.1996.tb00098.x [DOI] [Google Scholar]
- Winterbauer NE, & Bouton ME (2011). Mechanisms of resurgence II: Response-contingent reinforcers can reinstate a second extinguished behavior. Learning and Motivation, 42, 154–164. 10.1016/j.lmot.2011.01.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zeiler MD (1971). Eliminating behavior with reinforcement. Journal of the Experimental Analysis of Behavior, 16, 401–405. 10.1901/jeab.1971.16-401 [DOI] [PMC free article] [PubMed] [Google Scholar]
