Separation of Time-Based and Trial-Based Accounts of the Partial Reinforcement Extinction Effect

Abstract

Two appetitive conditioning experiments with rats examined time-based and trial-based accounts of the partial reinforcement extinction effect (PREE). In the PREE, the loss of responding that occurs in extinction is slower when the conditioned stimulus (CS) has been paired with a reinforcer on some of its presentations (partially reinforced) instead of every presentation (continuously reinforced). According to a time-based or “time-accumulation” view (e.g., Gallistel & Gibbon, 2000), the PREE occurs because the organism has learned in partial reinforcement to expect the reinforcer after a larger amount of time has accumulated in the CS over trials. In contrast, according to a trial-based view (e.g., Capaldi, 1967), the PREE occurs because the organism has learned in partial reinforcement to expect the reinforcer after a larger number of CS presentations. Experiment 1 used a procedure that equated partially- and continuously-reinforced groups on their expected times to reinforcement during conditioning. A PREE was still observed. Experiment 2 then used an extinction procedure that allowed time in the CS and the number of trials to accumulate differentially through extinction. The PREE was still evident when responding was examined as a function of expected time units to the reinforcer, but was eliminated when responding was examined as a function of expected trial units to the reinforcer. There was no evidence that the animal responded according to the ratio of time accumulated during the CS in extinction over the time in the CS expected before the reinforcer. The results thus favor a trial-based account over a time-based account of extinction and the PREE.

Behavioral models of Pavlovian learning have traditionally assumed that the conditioning trial, or the occasion on which a conditioned stimulus (CS) is presented, is the basic event that enables associative learning (e.g., Mackintosh, 1975; Pearce, 1994; Pearce & Hall, 1980; Rescorla & Wagner, 1972; Wagner, 1981, 2003; Wagner & Brandon, 1989). Such “trial-based” models differ in many details, but commonly assume that trials on which the CS is paired with the unconditioned stimulus (US) can increase associative strength whereas trials on which the CS occurs without the US can either decrease associative strength or cause inhibition to be learned. The models have been highly successful at explaining and anticipating the outcomes of large number of conditioning experiments (e.g., Pearce & Bouton, 2001). Without supplementation, they only indirectly address the known importance of time and temporal variables in conditioning. An important exception, however, are the models of Wagner (e.g., 1981, 2003; Wagner & Brandon, 1989), which postulate time-linked processes initiated by CS and US presentations that can account for a number of temporal phenomena in classical conditioning (e.g., Todd & Bouton, 2012).

A different approach assumes that timing processes, rather than the incremental effects of trials, are the basis of conditioning and learning (e.g., Balsam, Drew, & Gallistel, 2010; Balsam & Gallistel, 2009; Gallistel, 2012; Gallistel & Gibbon, 2000; Gibbon & Balsam, 1981). According to Gallistel and Gibbon (2000; see also Gibbon & Balsam, 1981), conditioned performance is determined by the animal’s estimation of reinforcement rate, which depends fundamentally on its perception of time. On this view, the animal is sensitive to the amount of time that accumulates over separate CS presentations. During conditioning, conditioned responding emerges when the rate of reinforcement in the CS is judged to be higher than the rate of reinforcement in the background. During extinction, conditioned responding is assumed to stop once the contemporary reinforcement rate in the CS is judged to be lower than it was during conditioning. When developed in more detail, these ideas provide a testable time-based alternative to understanding conditioning and extinction. Because time is assumed to accumulate in the CS over trials, Bouton and Sunsay (2003) referred to this perspective as the “time-accumulation” view (see also Gottlieb, 2004, 2005).

First consider the time-accumulation conceptualization of conditioning. The Gallistel-Gibbon model assumes that the animal compares the rate of reinforcement in the CS and in the interval between trials (the intertrial interval or ITI), an estimate of the base rate of reinforcement. These rates are compared in a ratio of the CS rate over the background rate; when the value of the ratio exceeds a threshold, the animal responds to the CS. Because the rate of reinforcement is the reciprocal of the time between reinforcers, the trial on which the animal first decides to respond is proportional to the ratio of the time accumulated in the ITI (I) over the time accumulated in the CS (T; the so-called I/T ratio). More recent theoretical expositions (e.g., Balsam & Gallistel, 2009; Balsam et al., 2010) still propose this ratio at their core. Although the view immediately captures the fact that increasing the ITI increases the rate of conditioning and that increasing the CS duration decreases it, a number of experiments have produced results that are not consistent with it. For example, groups of subjects with identical I/T ratios have differed in their rates of conditioning depending on the specific values of I and T (Holland, 2000; Lattal, 2009) and in how T accumulating between reinforcers was actually distributed over separate trials (Bouton & Sunsay, 2003; Sunsay, Stetson, & Bouton, 2004). Moreover, the magnitude of the US, and not merely its rate of occurrence, influences the point in conditioning at which the organism decides to respond (Morris & Bouton, 2006). Although trial-based theories emphasize a role for US magnitude in conditioning, the time-accumulation view ignores it. Several empirical analyses have thus created challenges for the time-accumulation account.

The present article is concerned, however, with time-based and trial-based accounts of extinction, the loss of responding that occurs when the CS is presented repeatedly without the US after conditioning (e.g., see Bouton, 2004; Bouton & Woods, 2008 for reviews). According to trial-based models, responding decreases in extinction because there is an incremental loss of associative strength or increase in inhibition as a consequence of each nonreinforced trial (presentation of the CS). In contrast, the Gallistel-Gibbon model proposes that animals stop responding in extinction when they determine that the rate of reinforcement in the CS is lower in extinction than it was during conditioning. The animal now compares the two rates in the form of another ratio. Because rate is again the reciprocal of time, the animal computes a ratio between the amount of time that has accumulated in the CS during extinction and the amount of time that previously accumulated in the CS between USs during conditioning. When the ratio exceeds a threshold, the animal stops responding. As before, time accumulating in the CS over trials, and not the effects of trials themselves, is what determines learning and performance.

The time-accumulation perspective makes especially interesting predictions about the partial reinforcement extinction effect, or PREE (see Mackintosh, 1974, for one review). In the PREE, the loss of responding that occurs in extinction is slower when conditioning has been conducted with a partial reinforcement schedule (in which nonreinforced trials have been intermixed with reinforced trials) than with a continuous reinforcement schedule (in which all trials are reinforced). According to the time-accumulation view, subjects that have undergone partial reinforcement have learned to expect the US after more accumulated time in the CS. As a consequence, it takes more accumulating time in the CS during extinction for the ratio of CS extinction time / expected time in the US to exceed the threshold. In contrast, more conventional views of the PREE emphasize a rather similar role of trials. For example, according to Capaldi’s sequential view (e.g., Capaldi, 1967, 1994; see also Capaldi & Martins, 2010), the partially reinforced subject learns to expect the US after more nonreinforced trials (“N-length”) than continuously reinforced subjects have. It therefore takes a longer string of nonreinforced trials to stop generalizing from conditioning to extinction (Capaldi, 1967, 1994). The two accounts both emphasize a generalization/discrimination process in explaining extinction and the PREE. But the time-accumulation view emphasizes what might be called the animal’s time expectancy whereas the trial-based account emphasizes what might be called a trial expectancy.

The time-based account of the PREE was tested in a series of experiments using the rat appetitive magazine-entry preparation by Haselgrove, Aydin, and Pearce (2004). In each experiment, partially-reinforced (PRF) and continuously-reinforced (CRF) groups were compared in extinction after first giving them the same rate of reinforcement in the CS during conditioning. In Experiments 1 and 2, reinforcement rate was equated by giving CRF groups a single US on every trial and PRF groups no US on half the trials and two USs on the other half. In Experiments 3 and 4, CRF groups received CSs that were twice the duration of those received by the PRF groups (which were reinforced half the time). Despite the fact that both manipulations equated the groups on reinforcement rate in the CS (i.e., the amount of time in the CS between USs), a PREE was consistently observed. The results thus challenge the time-based view of the PREE, although they did not directly address how the PREE should be explained.

The present experiments continued to examine the time-accumulation and trial-based accounts of the PREE. In Experiment 1, we compared extinction in partially- and continuously-reinforced groups that received equivalent rates of reinforcement in the CS. The results, like those of Haselgrove et al. (2004), challenged the time-accumulation account. Experiment 2 then contrasted the time-accumulation and trial-based accounts more directly by testing whether extinction performance conformed to implications of either view. The results challenged an implication of the time-accumulation account, but confirmed an implication of the trial-based account. Overall, the results favor a trial-based as opposed to a time-based explanation of the partial reinforcement extinction effect.

Experiment 1

In the first experiment, rats received either PRF or CRF during acquisition using a method that equated the accumulated time in the CS between each US in the groups. In each conditioning session, PRF and CRF groups received exactly 160 s of accumulated time in the CS and eight trials in which a CS presentation was paired with a US. However, the PRF group received 16 10-s CS presentations in each session, only half of which were paired with the US. The CRF group received 8 20-s CS presentations in each session, all of which were paired with the US. The groups then underwent extinction. Half the animals in each group received extinction trials with a 10-s CS, and half received extinction with a 20-s CS. The design allowed us to compare the PRF and CRF conditions in extinction under identical conditions while controlling for any generalization decrement that might result from changing the CS duration between conditioning and extinction (cf. Haselgrove et al., 2004). Since all the rats received a US after 20 s of CS time during conditioning, the time-accumulation view predicts no PREE.

Method

Subjects

The subjects were 32 female Wistar rats from Charles River Laboratories (St. Constance, Quebec). They averaged 75–90 days old at the start of the experiment and were housed individually in suspended stainless steel cages in a room maintained on a 16:8-hr light:dark cycle. The experiment was conducted on consecutive days during the light portion of the cycle. The rats were food deprived and maintained at 80% of their free-feeding weights throughout the experiment.

Apparatus

Experimental sessions were conducted in two counterbalanced sets of four Skinner boxes housed in separate sound attenuation chambers (Med Associates, Georgia, VT) and located in separate rooms of the laboratory. Each box measured 30.5 × 24.1 × 21 cm (l × w × h). The front and back walls were brushed aluminum; the side walls and ceiling were clear acrylic plastic. A 5.1 × 5.1 cm recessed food cup was centered in the front wall and positioned 2.5 cm above the floor. The food cups had infrared photobeams positioned approximately 1.2 cm behind the plane of the wall and 1.2 cm above the bottom of the cup. In one set of four boxes, the floor consisted of stainless steel bars 0.48 cm in diameter, spaced 1.6 cm center to center, and mounted parallel to the front wall. In the other set of four boxes, the floor consisted of similar bars, except alternating bars were staggered at 1.6 cm height differences.

The CS was a 3000-Hz tone (80 dBA) that was delivered through a 7.6 cm speaker mounted to the ceiling of the sound attenuation chamber, 28 cm above the grid floor. Illumination was provided by two 7.5-W clear incandescent bulbs mounted to the ceiling of the sound attenuation chamber, to the right of the speaker. A ventilation fan in each chamber provided background noise level of 60 dBA. The US was two 45-mg Noyes precision food pellets (Research Diets, Inc., New Brunswick, NJ) delivered 0.2 s apart at the termination of the CS. The apparatus was controlled by computer equipment and MED-PC IV software (Med Associates, Georgia, VT) located in an adjacent room.

Procedure

Magazine Training

On the first day, each rat was placed in its assigned box for a 20-min magazine training session during which it learned to retrieve food pellets from the food cups. Prior to the start of the session, food cups were baited with two pellets. Each rat received 30 pellets distributed through the session.

Conditioning

The rats were randomly assigned to two groups (ns = 16) that then received four daily conditioning sessions. Each group received the same number of reinforced trials per session, but under different schedules of reinforcement. Group CRF 20 received a continuous reinforcement schedule with a 20-s tone CS. For this group, the CS was paired with the US on each of eight trials. Group PRF 10 received a 50% partial reinforcement schedule with a 10-s tone CS. For this group, the CS was paired with the US on every second trial. Group PRF 10 received 16 CS presentations per session, but due to the 50% reinforcement schedule, there were only eight reinforced trials per session. The number in each group name indicates the CS duration during conditioning. The groups received different ITIs (defined as the interval between successive CSs) in an attempt to allow them to reach a similar level of asymptotic responding. Group CRF 20 received a variable mean ITI of 18 min, whereas Group PRF 10 received a variable mean ITI of 3 min.

Extinction

Following conditioning, each group of rats was further divided into two groups (matched on performance during conditioning), resulting in a total of four groups (n = 8). Groups PRF 10-10 and CRF 20-10 received a series of 10-s tone presentations during extinction, whereas Groups PRF 10-20 and CRF 20-20 received a 20-s tone CS during extinction. There were two extinction test sessions, one per day. The US was never presented. All groups received 16 CS-alone trials with a mean ITI of 7.35 min, which is the geometric mean of 3 and 18, the two ITIs used during conditioning (this should produce equivalent generalization decrement in all groups, e.g., Church & Deluty, 1977).

Data Analyses

The computer recorded interruptions of the food cup photo-beam during the CS and during the equivalent time period that immediately preceded the CS (the pre-CS period). The measure of responding to the CS was elevation scores of the form e= c − p, where c represents the number of responses recorded during the target CS and p represents the number of responses in the pre-CS period. Elevation scores have been used extensively in appetitive conditioning in order to separate CS and baseline responding (e.g., Bouton & Sunsay, 2003). Elevation scores were converted to a consistent one-min base by multiplying the data from 10-s CS duration groups by 6 and multiplying the data from 20-s CS duration groups by 3. Elevation scores and pre-CS scores were analyzed with parallel analyses of variance (ANOVAs) using a rejection criterion of p < .05.

Results

Conditioning

The results of the conditioning phase are shown in Figure 1. The PRF and CRF groups acquired similar levels of responding by the end of the phase. A 2 (Reinforcement: PRF vs. CRF) × 2 (Extinction CS Duration: 10 vs. 20) × 4 (Session) ANOVA revealed a main effect of Session, F(3, 84) = 48.39, MSE = 26.31, p < .001, indicating that responding increase over the course of conditioning. The main effect of Reinforcement and the Reinforcement × Session interaction approached, but did not quite reach, significance, F(1, 28) = 3.17, MSE = 100.10, p = .086 and F(3, 84) = 2.35, MSE = 26.31, p = .078, respectively. There was thus a suggestion that the CRF groups conditioned more rapidly than the PRF groups. No other main effects or interactions approached significance, Fs < 1. On the last session of conditioning, the mean elevation scores (per minute) were 15.4, 16.2, 16, and 15.8, for groups PRF 10-10, PRF 10-20, CRF 20-10, and CRF 20-20, respectively.

Figure 1.

Mean elevation scores of the groups during each session of the conditioning phase of Experiment 1. PRF = partial reinforcement; CRF = continuous reinforcement; numbers before and after hyphens in the group designations give CS duration during conditioning and extinction, respectively.

An analogous 2 (Reinforcement: PRF vs. CRF) × 2 (Extinction CS Duration: 10 vs. 20) × 4 (Session) ANOVA was conducted on the pre-CS scores. The analysis revealed a significant main effect of Reinforcement, F(1, 28) = 19.32, MSE = 33.90, p < .001, indicating higher pre-CS responding in the PRF groups than in the CRF groups. No other main effects or interactions were significant, largest F(1, 28) = 2.38, MSE = 9.76, p = .076. On the last day of conditioning, the mean pre-CS scores per minute were 9.5, 8.3, 4.0, and 2.9 for groups PRF 10-10, PRF 10-20, CRF 20-10 and CRF 20-20, respectively.

Extinction: Response rate

Figure 2 depicts the mean elevation scores (per minute) for each group during consecutive four-trial blocks for the two sessions of extinction. During extinction, the groups differed in the rate at which responding declined. A 2 (Reinforcement: PRF vs. CRF) × 2 (Extinction CS Duration: 10 vs. 20) × 8 (4-Trial Block) ANOVA revealed a main effect of Trial Block, F(7, 196) = 23.76, MSE = 22.20, p < .001, indicating a decrease in responding over trial-blocks. There was also a significant main effect of the Reinforcement factor, F(1, 28) = 8.96, MSE = 141.61, p < .01, indicating more overall responding in the PRF groups compared to the CRF groups. There was also a significant Reinforcement × Trial Block interaction, F(7, 196) = 4.57, MSE = 22.20, p < .001, indicating differential rates of extinction between the PRF and CRF groups. Although there was a trend suggesting less responding in groups extinguished with the 20-s CS, the effect of Extinction CS Duration was not significant, nor was any remaining interaction, largest F(1, 28) = 2.03, MSE = 141.61, p = .17.

Figure 2.

Mean elevation scores of the groups during the extinction phase of Experiment 1. PRF = partial reinforcement; CRF = continuous reinforcement; numbers before and after hyphens in the group designations give CS duration during conditioning and extinction, respectively.

An analogous 2 (Reinforcement: PRF vs. CRF) × 2 (Extinction CS Duration: 10 vs. 20) × 8 (4-Trial Block) ANOVA was conducted on the pre-CS scores. There was a significant effect of Trial Block, F(7, 196) = 9.03, MSE = 7.40, p < .001, indicating a decrease in pre-CS responding during extinction. While the 3-way Reinforcement × Extinction CS Duration × Trial Block interaction approached significance, F(7, 196) = 2.00, MSE = 7.40, p = .056, no other main effects or interactions were significant, largest F(7, 196) = 1.2, MSE = 7.40, p = .31. Pre-CS responding, averaged over the two extinction sessions, was 2.3, 3.3, 2.3, and 1.8, for Groups PRF 10-10, PRF 10-20, CRF 20-10, and CRF 20-20, respectively.

Extinction: Trials to criterion

The Gallistel-Gibbon (2000) model is mainly designed to predict the number of trials it takes individual animals to decide to stop responding. We therefore calculated the trials-to-criterion measure that Haselgrove and Pearce (2003) and Haselgrove et al. (2004) used based on Gallistel and Gibbon (2000, p. 306): The point at which responding declined to 50% of each subject’s pre-extinction rate. Following Haselgrove and Pearce (2003) and Haselgrove et al. (2004), we determined the number of two-trial blocks it took each animal to reach an elevation score that was less than half its average elevation score during the last conditioning session on consecutive two-trial blocks. The mean number of trial blocks required for Groups PRF 10-10, PRF 10-20, CRF 20-10, and CRF 20-20 to reach this criterion were 10.8, 9.1, 4.1, and 5.0, respectively (medians = 10.5, 9.0, 2.5, and 3.5). (One rat in Group PRF 10-10 never reached criterion and was given a maximum score of 16.) A 2 (Reinforcement: PRF vs. CRF) × 2 (Extinction CS Duration: 10 vs. 20) ANOVA on these data revealed a significant main effect of Reinforcement, F (1, 28) = 14.54, MSE = 15.90, p < .01. No other main effect or interaction were significant, Fs < 1. The results confirm that the PREE was obtained in a decision-point measure despite the equation of the groups on the expected time to the reinforcer during conditioning.¹

Discussion

The results of this experiment suggest that a PREE can occur even when the animals have the same temporal expectancy of the US. A PREE was clear in both a traditional rate measure of responding in extinction as well as a trials-to-criterion measure that might better reflect that point at which the animal decided to stop responding. Like similar results reported by Haselgrove et al. (2004), they are highly inconsistent with a time-accumulation view of the PREE. The results extend the previous findings by showing the result after controlling for the match/mismatch of CS duration between conditioning and extinction (cf. Haselgrove et al., 2004).

Experiment 2

The second experiment extended the analysis of the PREE by contrasting implications of the time-accumulation and trial-based accounts. All groups received 10-s presentations of the CS throughout conditioning. However, for a PRF group, one in four of these CS presentations (25%) was reinforced, whereas two control groups received CRF schedules in which every trial was reinforced. One control group received a reinforced trial every time the PRF group received a CS (separated by variable 3-min ITIs), whereas the other received a reinforced trial every time the PRF group received a reinforced trial (i.e., separated by variable 12-min ITIs). The latter CRF condition controlled for the distribution of USs (or reinforced trials) given the PRF group, whereas the former condition controlled for the distribution of CSs.² According to the time-accumulation view, the PRF group should learn to expect a US with every 40 s of CS time, whereas the CRF groups might expect it after every 10 s. According to a trial-based view, the PRF group might learn to expect a US with every 4^th trial, whereas the CRF groups might expect it on every trial. Both the time-accumulation and trial-based views thus predict a PREE.

What made the experiment unusual was the temporal organization of the trials that were presented during extinction. In that phase, all animals received a series of nonreinforced CS presentations that alternated between 30 s and 10 s in duration. Such a series is illustrated in Figure 3. The alternating trial durations allowed us to contrast implications of the time-accumulation and trial-based accounts of the PREE. If animals in the groups are responding based on a temporal expectancy, then one can predict points during the trial schedule in which responding in the CRF and PRF groups should be alike. The lines connecting the PRF and CRF trials in the upper panel illustrate trials on which the time-accumulation account predicts that responding in the PRF and CRF groups should be equivalent—the paired trials represent points in the extinction series when the ratio between accumulating time in the CS / expected time to the US are equal in the two groups. In contrast, the lines connecting PRF and CRF trials in the lower panel illustrate points where the trial-based account suggests responding in the groups might be equivalent. Here the paired trials represent points when the PRF and CRF groups should have the same trial expectancies. If the PREE follows from the Gallistel-Gibbon (2000) model, the extinction response curves should superpose when responding is plotted as a function of time units (upper panel) but not trial units (lower panel). In contrast, if the PREE follows the rules of trial-based models, the extinction response curves might superpose when plotted as a function of trial units (lower panel) but not time units (upper panel). As in Experiment 1, we examined both response rate (elevation scores) on selected trials as well as the number of units required to reach an extinction criterion of 50% of pre-extinction responding.

Figure 3.

The extinction procedure used in Experiment 2. Trial durations alternated between 30-s and 10-s. Top: Time units when the PRF and CRF groups should have expected the US are indicated by arrows (40s and 10 s for the groups, respectively). The lines connect units where the PRF and CRF groups had equal ratios of accumulated time in the CS / expected time to the US. Bottom: Trial units where the PRF and CRF groups should have expected the US are indicated by arrows (every 4^th trial and every trial, respectively). The lines connect units where the PRF and CRF groups had equivalent trial expectancies. See text for more explanation.

Method

Subjects and Apparatus

The subjects were 24 naive female Wistar rats purchased from the same supplier as those in the previous experiment and maintained under the same conditions. The apparatus, CS, and US were the same as those used in Experiment 1.

Procedure

Magazine training

On the first day, rats were trained to retrieve pellets from the food cup with an identical procedure as Experiment 1.

Acquisition

On the next day, the rats were randomly assigned to one of three groups, all of which then received a series of 12 daily conditioning sessions in which four presentations of the 10-s CS co-terminated with delivery of the US. The groups differed with respect to their overall schedule of reinforcement. During each session, Group PRF received a total of 16 CS presentations (separated by a 3-min ITI). Every fourth trial was reinforced; the remainder were not reinforced. Group CRF Massed received four reinforced CS presentations that were separated by the same mean ITI of 3 min. In contrast, Group CRF Spaced received its four (reinforced) trials at the same points in time within the session that Group PRF did. The ITIs consequently averaged 12 min.

Extinction

There was a single session in which all rats received a series of 48 nonreinforced presentations of the CS. The ITI was variable with a mean of 3 min. As noted above, the duration of each CS presentation alternated between 30 s and 10 s, beginning with a 30-s trial. Responses were aggregated over each 10-s interval. In this way, as explained above, we were able to separate the animal’s temporal expectancy of when the US should occur (every 10 s and 40 s for the CRF and PRF groups, respectively) from its trial expectancy on which trial it should occur (every trial and every 4^th trial, respectively).

Extinction data analysis

In addition to reporting and analyzing responding over extinction trials, we compared responding on trials that equated the groups on their hypothetical time- and trial-based expectancies of the US. On the time-accumulation view, the PRF group should expect the US on Trials 2, 4, 6, 8, etc., because 40 s of CS time (one expected time unit) had accumulated by the end of each of these trials. In contrast, the CRF groups should expect the US after each unit of 10 s (i.e., intervals 1a, 1b, 1c, 2, 3a, 3b, 3c, etc., where a, b, and c refer to 10-s intervals within each 30-s CS). Responding on these trials was therefore isolated and compared; if the extinction of responding reflects a time-based expectancy, there should be no difference between the PRF and CRF groups normalized based over time.³ On a trial-based view, the PRF group should contrastingly expect the US every 4^th trial, i.e., on Trials 4, 8, 12, 16, etc. In contrast, the CRF groups should expect it on Trials 1a, 2, 3a, 4, etc. These trials were also isolated and compared; if extinction responding reflects a trial-based expectancy, there should be no difference between the groups on these isolated trials. As before, response counts were converted to response rate per 60 s. For simplicity, we pooled responding in the two CRF groups because they did not differ in either acquisition or extinction (see below).

Results

Conditioning

Two rats (one from Group PRF and one from Group CRF Massed) failed to show any evidence of conditioning during the acquisition phase and were therefore excluded from the data analysis. Both rats had elevation scores that averaged less than 0 throughout the phase.

An initial 2 (Group: CRF Massed vs. CRF Spaced) × 12 (Session) ANOVA compared conditioning in the massed and spaced CRF groups, which had mean elevation scores of 19.4 and 25.2, respectively, collapsing over the phase (the scores were 36.2 and 39.4 on the final acquisition session). Neither the main effect of Group, nor the Group × Session interaction approached significance, largest F(1, 13) = 2.51, MSE = 596.45, p = .14. The groups also did not differ in extinction, where they had mean elevation scores of 5.2 and 6.2 over the phase. A 2 (Group: CRF Massed vs. CRF Spaced) × 12 (Session) ANOVA compared extinction in the CRF groups. Neither the effect of Group, nor the Group × Session interaction approached significance, Fs < 1. As noted above, because the two CRF groups did not differ, the remainder of the analyses collapsed over them.

The results of the acquisition phase with the pooled CRF group are presented in Figure 4. As suggested by the figure, the groups did not acquire responding at the same rate. A 2 (Group: PRF vs. CRF) × 12 (Session) ANOVA revealed a significant main effect of Session, F(11, 220) = 17.47, MSE = 65.44, p < .001, indicating that responding increased over training. The main effect of Group was significant, F(1, 20) = 11.37, MSE = 712.23, p < .01, as was the Group × Session interaction, F(11, 220) = 3.18, MSE = 65.44, p < .01. Thus, the responding of the groups did not change equivalently over time. An ANOVA on the last day of conditioning also confirmed that Group CRF responded more than Group PRF, F(1, 20) = 15.91, MSE = 198.21, p < .01.

Figure 4.

Mean elevation scores of the groups during the conditioning phase of Experiment 2. PRF = partial reinforcement; CRF = continuous reinforcement.

An analogous 2 (Group: PRF vs. CRF) × 12 (Session) ANOVA was conducted on pre-CS responding. The analysis revealed no significant main effects or interactions, largest F(11, 220) = 1.49, MSE = 17.22, p = .14. Mean pre-CS score per minute over the 12 sessions of conditioning were 6.4 and 4.8 for Groups PRF, and CRF, respectively.

Extinction: Response rate

Figure 5 depicts the mean elevation scores per minute for 4-trial blocks over the course of extinction (the scores used responding throughout each 10-s or 30-s extinction trial). Somewhat surprisingly, Group PRF did not differ from Group CRF on the first 4-trial block, F(1, 20) = 1.41, MSE = 113.32, p = .25. However, a 2 (Group: PRF vs. CRF) × 12 (Session) ANOVA revealed a significant main effect of Block, F(11, 220) = 11.31, MSE = 38.60, p < .001, and a significant interaction between Block and Group, F(11, 220) = 2.68, MSE = 38.60, p < .01. Thus, extinction occurred at different rates for the groups. The main effect of Group approached significance, F(1, 20) = 3.23, MSE = 454.20, p = 0.087. Notice that the pattern of more responding in the PRF group is opposite to the pattern observed during conditioning—and is a clear demonstration of the PREE.

Figure 5.

Mean elevation scores of the groups over 4-trial blocks during the entire extinction phase of Experiment 2. PRF = partial reinforcement during acquisition; CRF = continuous reinforcement during acquisition.

Pre-CS responding over the course of extinction was also analyzed with a 2 (Group: PRF vs. CRF) × 12 (Session) ANOVA. The analysis revealed a significant main effect of Block, F(11, 220) = 3.89, MSE = 9.64, p < .001, but no other main effect or interaction was significant, largest F(1, 20) = 2.57, MSE = 9.64, p = .074. Mean pre-CS scores per minute over the 12 four-trial blocks of extinction were 4.4 and 2.3 for Group PRF and CRF, respectively.

Role of time expectancy

According to the time-accumulation view, Group PRF should expect the US after 40 s of CS time and the CRF groups should expected it after 10 s. Trials that equated the groups on these time units were isolated as described in the data analysis section. The trials were then aggregated and averaged (as usual) over 4-trial (4-unit) blocks. The results are shown in the upper panel of Figure 6, which suggests that the CRF group still lost responding more rapidly than the PRF group, whose performance did not change substantially over units. A 2 (Group: PRF vs. CRF) × 3 (Block) ANOVA revealed a significant main effect of Block, F(2, 40) = 6.62, MSE = 73.54, p < .01, as well as a significant Block × Group interaction, F(2, 40) = 6.85, MSE = 73.54, p < .01. The main effect of Group was not significant, F(1, 20) < 1. Simple effect analyses within each group revealed a significant Block effect in Group CRF, F(2, 28) = 16.54, MSE = 85.35, p < .001, but not in group PRF, F(2, 12) = 1.47, MSE = 45.98, p = .27. Thus, when responding in the groups was equated on expected time unit to the reinforcer, responding in the PRF group still extinguished more slowly than that in the CRF group. The PREE was preserved.

Figure 6.

Mean elevation scores of the partially-reinforced group (PRF) and continuously-reinforced group (CRF) in the extinction phase of Experiment 2 as a function time units (upper panel) and trial units (lower panel) expected to reinforcement.

Role of trial expectancy

According to the trial-based view, Group PRF should expect the US on every 4^th trial and the CRF groups should expect it on every trial. Trials that equated the groups on these trial units were isolated as described above. The data were then aggregated and averaged over 4-trial blocks. The results are presented in the lower panel of Figure 6, which suggests little difference between the groups during extinction, except perhaps on the third block, where the CRF group paradoxically tended to respond more than the PRF group. A 2 (Group: PRF vs. CRF) × 3 (Block) ANOVA revealed a significant main effect of Block, F(2, 40) = 9.18, MSE = 87.04, p < .01. However, no other main effects or interactions approached significance, Fs < 1. Thus, when responding in the groups was equated according to their hypothetical expected time units to the reinforcer, the PREE disappeared.

Time and trial expectancies compared

The separate analyses of responding over time-based and trial-based units thus suggest that the partial reinforcement effect depends more on the trial-based expectancy than a time-based one. To compare the two types of expectancies directly, we also conducted an overall ANOVA on the data that included unit type (time-based or trial-based) as a within subject factor. This 2 (Unit Type: Time vs. Trial) × 2 (Group: PRF vs. CRF) × 3 (Block) ANOVA revealed a significant main effect of Block, F(2, 40) = 12.51, MSE = 100.78, p < .001, a significant main effect of Unit Type, F(1, 20) = 6.91, MSE = 47.84, p < .05, and most importantly, a significant Group × Unit Type × Block interaction, F(2, 40) = 5.63, MSE = 59.80, p < .01. The three-way interaction confirms that the strength of the Group x Block interaction indicating a partial reinforcement effect depended on whether the data were expressed as a function of time-based or trial-based units. No other main effects or interactions were significant, largest F(2, 40) = 2.22, MSE = 100.78, p = .12.

Extinction: Trials to criterion

Similar analyses were performed on the trials-to-criterion measure used in Experiment 1 to determine the point at which the animals decided not to respond (Haselgrove & Pearce, 2003; Haselgrove et al., 2004). Overall, the number of two-trial blocks required to reach the criterion of 50% preextinction responding was 12.6 for the PRF group and 4.5 for the pooled CRF group (medians = 14 and 4). The difference between groups was reliable, F (1,20) = 11.78, MSE = 26.17, p < .001. Thus, the PREE was also evident in this estimate of the number of trials it took the rat to stop responding.

Similar analyses then compared the number of time units and trial units (two-unit blocks) that were required to reach the extinction criterion. The means are summarized in Figure 7. For time units, Groups PRF and CRF differed in the number of two-unit blocks it took to reach the 50% criterion (medians = 11.0 and 4.0, respectively). (Three rats in Group PRF did not reach criterion and were given maximum scores of 12.) The difference between groups was significant, F (1, 20) = 10.61, MSE = 6.66, p < .01. Thus, there was once again still a PREE when we equated the groups on units of expected time to the US. For trial units, Groups PRF and CRF were more alike (medians = 6.0 and 4.0); there was no difference between the groups, F (1, 20) = 1.69, MSE = 2.44, p = .21. (Three rats in Group PRF and two in Group CRF did not reach criterion within Group PRF’s maximum number of units [6] and were given maximum scores of 6.) To compare the results with time- and trial-units directly, we also conducted an overall Unit Type (time vs. trial) x Group (PRF vs. CRF) ANOVA on the data. The analysis revealed significant effects of Group, F (1, 20) = 7.20, MSE = 7.61, p < .05, and Unit Type, F (1, 20) = 32.51, MSE = 1.50, p < .001, as well as a Group x Unit Type interaction, F (1, 20) = 13.55, MSE = 7.61, p < .001. The interaction once again confirms that the strength of the PREE depended on whether the data were expressed as a function of time-based or trial-based units.

Figure 7.

Mean number of two-trial blocks required for the partially-reinforced (PRF) and continuously-reinforced (CRF) groups to reach the extinction criterion when trials were blocked in terms of time units (left) and trial units (right) expected to reinforcement in Experiment 2.

Discussion

When we analyzed either response rate or trials to reach an extinction criterion, a PRF group reinforced on 25% of its conditioning trials extinguished more slowly than a group that was reinforced on 100% of its trials. Although this demonstration of the PREE was not surprising, other aspects of the results suggested that it was more consistent with the trial-based approach than the time-accumulation view. Specifically, the PREE persisted when we examined performance as a function of expected time units to the US. In contrast, the PREE disappeared when we examined responding as a function of units of expected trials units to the US. The former result suggests that the animals were not using expected time to the US as the basis of extinction responding; the latter suggests that they were using some representation of the number of trials (e.g., Capaldi, 1967). It is worth noting that the results of equating the groups on the basis of trial units is reminiscent of the results of Mowrer and Jones (1945), who found in an instrumental learning preparation that the PREE disappeared when groups were normalized based on the number of response units that led to extinction. The present results are consistent with the message of Mowrer and Jones, and tentatively confirm an implication of a trial-based account of the PREE while disconfirming an implication of a time-accumulation account.

Because extinction in Experiment 2 was conducted in a single session, it is possible that our conclusions may be restricted to the mechanisms that control extinction when it occurs within such a session. There is evidence that some of the neural and behavioral processes that underlie extinction within a session can be dissociated from those that underlie extinction developing between sessions (e.g., Plendl & Wotjak, 2010). However, we are not aware of any evidence that the PREE is controlled by different within- and between-session mechanisms.

General Discussion

The results of the present experiments challenge the time-accumulation account of the PREE suggested by Gallistel and Gibbon (2000). In Experiment 1, a PREE was evident when we compared extinction in PRF and CRF groups that had the same rates of reinforcement (or temporal expectancy) during conditioning. That PREE was apparent when we looked at extinction performance in terms of rate of responding or trials to an extinction criterion (see also Haselgrove et al., 2004). In Experiment 2, when we used an extinction procedure that allowed time in the CS and the number of trials to accumulate differentially, we failed to eliminate the PREE when we examined response rate as a function of expected time units to the reinforcer, but eliminated the PREE when we plotted response rate as a function of expected trial units to the reinforcer. Once again, the pattern was evident when we examined performance either as response rate over trials representing time or trial units or in terms of the number of units required to reach an extinction criterion. There was no evidence that the animal responded according to the ratio of time accumulated during the CS in extinction over the time in the CS expected before the US. The results thus favor a trial-based account of the PREE over the time-accumulation account proposed by Gallistel and Gibbon (2000).

Although the results are consistent with a trial-based perspective on extinction and the PREE, it should be noted that existing trial-based models of conditioning (e.g., Mackintosh, 1974; Pearce, 1994; Pearce & Hall, 1980; Rescorla & Wagner, 1972; Wagner, 1981, 2003) cannot handle the PREE without modification. Instead, they assume that responding in extinction will largely reflect the amount of associative strength that was acquired in conditioning (see Moody, Sunsay, & Bouton, 2006, for qualifications). Ever since the PREE’s first discovery (Humphreys, 1939), the phenomenon has challenged the idea that the rate of extinction is a simple function of the amount of associative- or habit- strength that was learned during conditioning. However, the PREE has been studied extensively and has been shown to be compatible with an expanded trial-based view. For example, we have already noted that the results of Experiment 2 are directly comparable to the classic findings of Mowrer and Jones (1945), who studied the extinction of instrumental lever pressing. In their experiment, key groups of rats first earned a reinforcer for every lever press, or every second, third, or fourth lever press and then underwent extinction. Although the total number of responses made during extinction was an increasing function of the response-reinforcer ratio used during conditioning, when extinction responding was plotted in terms of the number of “response units” emitted in extinction, this PREE “disappeared” (p. 301). The point of Mowrer and Jones’s report was that something like “associative strength” might well predict the PREE if the proper unit is first considered. The present Experiment 2 analogously found that the PREE disappeared when responding was plotted as a function of expected trial units (as opposed to response units) to the reinforcer. The possible role of the trial unit is consistent with Capaldi’s (e.g., 1967, 1994) sequential theory of extinction, which argues that the animal responds in extinction based on the similarity between (1.) the number of nonreinforced (N) trials that are in immediate memory as extinction proceeds and (2.) the number of N trials that were in memory when reinforcement was delivered in conditioning (N-length). Sequential theory was shown to be consistent with the results of an intensive research effort in the 1960s that was directed toward the PREE (see Mackintosh, 1974, for one review) and is also consistent with more recent research in Pavlovian conditioning (e.g., Capaldi & Martins, 2010). The present results are thus compatible with this broad base of research.

The results are also compatible with the results of recent experiments that were designed more specifically to separate the effects of trials and accumulating time in nonreinforced trials on extinction. Haselgrove and Pearce (2003) reported several experiments that compared the extinction of responding over extinction trials that varied in duration. In two of their experiments, rats received conditioning with 10-s CSs and then extinction with either 10-s or 270-s CSs. There was surprisingly little difference in responding at the start of each trial as a function of CS duration; for example, by the 12^th two-trial block, the 10-s and 270-s CS groups had similar nonzero levels of responding even though they had accumulated a total of 4 and 108 min of exposure to the CS, respectively. Time in the CS was not without effect, however; over all the experiments, the rats responded less in extinction when the CS duration was either increased or decreased relative to the duration used in conditioning. Notice that although the results suggest the animals were sensitive to time, the facilitating effect of changing CS duration is not anticipated by the Gallistel-Gibbon time-accumulation rule, which ignores trials and consequently has no mechanism to detect changes in individual trial duration. In related autoshaping experiments with ring doves, Drew, Yang, Ohyama, and Balsam (2004) found that either doubling or halving the duration of the CS used in conditioning did not affect the number of trials required to stop responding in extinction. This result is not consistent with the time-accumulation view, because the groups differed in how quickly they accumulated CS time over trials. In contrast, quadrupling the CS duration increased the rate of extinction. The results were attributed to the animal learning to discriminate the durations of reinforced and nonreinforced CSs. Animals are sensitive to time in the CS, but responding does not follow the ratio of current CS time / expected CS time to the reinforcer. The number of nonreinforced trials is clearly an important factor.

Other compatible research has studied the impact of nonreinforced trials introduced during acquisition (i.e., in partial reinforcement procedures). Bouton and Sunsay (2003) found that adding nonreinforced trials in the intervals between reinforced trials can hurt the acquisition of conditioned responding when measured either in response rate or on a reinforcers-to-criterion measure. Such results are consistent with either a trial-based or a time-accumulation view. However, adding 40 s of nonreinforced CS time increased the number of reinforcers to an acquisition criterion and decreased response rate more when it was presented in the form of four 10-s trials than when it was presented as a single 40-s trial. Thus, the number of trials, rather than merely the time accumulating in them, mattered. Gottlieb (2004, 2005) studied a second method of creating a partial reinforcement procedure. Instead of adding nonreinforced trials between reinforced trials, he deleted reinforcers that were otherwise scheduled in a CRF condition. Theoretically, this method increases the time in the CS and the interval between reinforcers by the same factor; for example, deleting half the USs doubles the CS time between reinforcers and also doubles the inter-trial time between reinforcers. Consequently, the time-accumulation view predicts no effect of deleting reinforcers on the rate of acquisition in CRF and PRF conditions. (The prediction of trial-based models is less clear, because although the new nonreinforced trials should decrease performance, additional spacing between reinforced trials would potentially increase it.) However, in both the pigeon autoshaping (Gottlieb, 2004) and rat magazine-entry preparations (Gottlieb, 2005; see also Harris, 2011), partial reinforcement created by deleting reinforcers decreased response rate and increased the number of reinforcers required to reach a decision criterion. Gottlieb (2005) also showed that the results were consistent with the published research literature from several conditioning preparations; after performing a meta-analysis and a thorough (though less formal) qualitative literature review, he suggested that “the data overwhelmingly point to the conclusion that higher frequencies of reinforcement lead to superior Pavlovian conditioned responding” (Gottlieb, 2005, p. 330). The inconsistency of the time-accumulation view with the research literature has been ignored in recent summaries of the effect of partial reinforcement on acquisition (Gallistel, 2012). Whether analyzed in extinction or in acquisition with partial reinforcement procedures, the number of nonreinforced trials does appear to play a significant role that is incorrectly denied by the time-accumulation view.

The present results also fit within a broader intellectual context. As noted in the Introduction, several authors have argued that associative learning principles may ultimately be replaced by theories of interval timing, which might provide the true basis of associative learning (e.g., Balsam et al., 2010; Balsam & Gallistel, 2009; Gallistel, 2012; Gallistel & Gibbon, 2000; Gibbon & Balsam, 1981). The present findings are consistent with other findings in the literature in suggesting that this view is premature (e.g., Williams, Lawson, Cook, Mather, & Johns, 2008). In the present rat magazine-entry paradigm, the trial appears to be far more important in both conditioning and extinction than the time-accumulation view recognizes. For example, related research suggests that the effects of even temporal variables such as CS duration and trial spacing may be more consistent with SOP theory (e.g., Wagner, 1981; Wagner & Brandon, 1983), a trial-based model that explicitly recognizes a role for time, than with the time-accumulation view (Bouton & Sunsay, 2003; Moody et al., 2006; Sunsay & Bouton, 2008; Sunsay et al., 2004). Extant time-accumulation models are thus not ready to replace trial-based models. And Bouton and Hendrix (2011) and Bouton, Doyle-Burr, and Vurbic (2012) have suggested that certain timing phenomena themselves might be more amenable to analysis from an associative perspective that models the passage of time with a cascading series of temporal elements that can be associated with the US (e.g., Desmond & Moore, 1988; see also Bouton & Garcia-Gutierrez, 2006; Bouton & Hendrix, 2011; Todd, Winterbauer, & Bouton, 2010). Although temporal variables have an undeniable influence on Pavlovian learning, associative models may be well equipped to accommodate them.

Highlights.

• Two experiments were designed to separate predictions of time-accumulation and trial-based models of the partial reinforcement extinction effect

• Both produced results that favored the trial-based view

• Trials, and not just accumulating time in trials, are important in creating extinction and the partial reinforcement extinction effect