Differential outcome effects in Pavlovian biconditional and ambiguous occasion setting tasks

Abstract

Three experiments with rats explored the differential outcome effect (DOE) using a Pavlovian magazine approach conditioning preparation. Experiment 1 compared groups trained on a biconditional discrimination (AX+, AY−, BX−, BY+) with differential or nondifferential outcomes, and Experiment 2 examined this using an ambiguous occasion setting task (e.g., AX+, X−, Y+, AY−). In both experiments subjects trained with differential outcomes learned the tasks better than subjects trained with nondifferential outcomes. Furthermore, subjects given differential outcome training learned the positive occasion setting component of the ambiguous task more efficiently than the negative occasion setting component although both were enhanced by differential outcome training. Experiment 3 demonstrated that the ambiguous occasion setting task was reversed more readily when the target-outcome relations (as opposed to the modulator-outcome relations) were maintained during the reversal phase. These data suggest that an acquired distinctiveness effect may be responsible for the DOE in Pavlovian learning.

The differential outcome effect (DOE) has been an important phenomenon in the study of instrumental learning. In the first experiment examining this phenomenon (Trapold, 1970), rats chose between two instrumental responses under conditions in which one of these responses, R1, was reinforced in the presence of one stimulus, S1, but the other response, R2, was reinforced in the presence of a second stimulus, S2. When different reinforcing outcomes (O1 or O2) were used to reinforce each correct stimulus-response (S-R) pair, subjects showed better learning on the task compared to a control group that received the same reinforcing outcome for each correct response.

Although this phenomenon has been extensively studied using operant procedures (e.g., see Overmier and Linwick, 2001), there has been considerably less research investigating differential outcome effects using Pavlovian procedures. The point of the present research is to determine if a differential outcome effect can be found in a Pavlovian preparation as well. Given that a wide variety of studies have documented the importance of associations between Pavlovian conditioned stimuli (CS) and distinct unconditioned stimuli (US) (e.g., Colwill and Motzkin, 1994; Delamater, 1995; 1996; Delamater and Holland, 2008; Galarce, Crombag, & Holland, 2007; Kruse, Overmier, Konz, Rokke, 1983), it seems possible that such associations may contribute to a DOE in Pavlovian learning. However, at least one theory of the DOE raises questions as to whether such an effect should be observed in a Pavlovian procedure.

One popular account of the DOE is based on the notion that Pavlovian and instrumental processes combine in their control of instrumental responding. In particular, Trapold and Overmier (1972) proposed that associative chains involving differential Pavlovian and instrumental links help animals efficiently solve the task. For instance, for subjects trained with a differential outcome procedure when S1 is presented this stimulus is assumed to evoke a representation of O1 at the time of response choice. Because the correct response is reinforced in the presence of this O1 representation, then, via Thorndike’s law of effect (Thorndike, 1898), a specific outcome-response (O1–R1) association is learned. Different S-O, and O–R associative chains will be established in this group, but not in a group trained nondifferentially, and this will enable the differential outcomes group to learn the task quickly.

It is of interest to note that a Pavlovian version of this task can readily be created with the understanding that the differential outcome procedure is merely one form of conditional discrimination problem, i.e., a biconditional discrimination. Thus, it can be seen as analogous to an AX-US1, AY-, BX-, BY-US2 task, where target stimuli X & Y play the functional roles of different instrumental responses, stimuli A & B play the functional roles of instrumental discriminative stimuli, and different unconditioned stimuli (US1 & US2) play the functional roles of distinct outcomes. To our knowledge the question of whether learning in such a Pavlovian procedure would be quicker than learning in an analogous nondifferential outcome task has not been empirically assessed.

This question is of some theoretical interest because the mechanisms proposed by Trapold and Overmier (1972) would not seem to provide an advantage to learning in this Pavlovian task. If one assumes, for instance, that differential A-US1 and B-US2 associations are established (in a differential outcome group), and that the US1 and US2 representations, respectively, are active on AX and BY trials, then there would be an opportunity for the establishment of separate US1-X and US2-Y links. However, it is not obvious that such links would necessarily be helpful in solving this task. Given assumptions of SOP theory (Wagner, 1981), for instance, it would seem that even if a US1-X link were assumed to be learned, this link would make it more difficult for X to activate itself in such a way that would be conducive to efficient learning.

On the other hand, another popular theory of Pavlovian learning, the Rescorla-Wagner model (Rescorla and Wagner, 1972), suggests that training with differential outcomes should make a rather large difference in this situation. Consider the difficulties this model faces when subjects are trained with a biconditional discrimination such as that outlined above, but with a single US (i.e., as in the control condition). Without making assumptions about the nature of unique cues in the presence of stimulus compounds (Wagner and Rescorla, 1972; Wagner and Brandon, 2001; Wagner, 2008), this task will be unsolvable according to this model because each individual cue is paired with reinforcement and nonreinforcement equally often. However, it is of interest to note that the task becomes solvable, without requiring unique cues, when a differential outcome procedure is used. Specifically, the Rescorla-Wagner model has no trouble solving the AX-US1, AY-, BX-, BY-US2 task because while separate excitatory X-US1 and Y-US2 associations will develop, separate inhibitory A-US2 and B-US1 associations will also develop, and the combined influence of these excitatory and inhibitory links will permit for appropriate US activation levels on the different trial types. Thus, according to the Rescorla-Wagner model learning with differential outcomes should be easier than learning with nondifferential outcomes. This conclusion applies equally well to the case in which unique cues are also permitted.

A related procedure is also of interest in the present context. The ambiguous occasion setting task (e.g., Holland, 1991; Holland and Reeve, 1991) is another example of a conditional discrimination procedure that differs from the biconditional discrimination task only in that there is one distinctive feature stimulus (rather than two). One could ask if a differential outcome effect might occur in this procedure as well as in the full biconditional discrimination task. One advantage of extending the analysis to this task is that it allows for an assessment of the effects of the differential outcome treatment upon separate excitatory and inhibitory learning mechanisms of the sort described above in solving the task. Consider an AX-US1, X-, Y-US2, AY- discrimination. In this case, stimulus A is referred to as the ambiguous occasion setter, whereas X and Y are considered target stimuli. This task is of interest because while A can be construed as a positive occasion setter for target stimulus X, it concurrently serves as a negative occasion setter for target stimulus Y. By using a differential outcome manipulation it is, therefore, possible to ask whether differential outcome effects might benefit equally the positive and negative occasion setting components of this task or whether the effects might be restricted to one component of the task or the other. While both of these functions could also be learned in the full biconditional discrimination procedure, one can distinguish potential DOE effects between these two functions using the ambiguous occasion setting task. It is of interest to note that some data exists to support the conclusion that both positive and negative occasion setting components are enhanced by a differential outcome treatment in an instrumental learning version of this task (Nakajima and Kobayashi, 2000).

In this paper we report findings from three experiments that investigate differential outcome effects in different Pavlovian conditional discrimination procedures. The first experiment examined this question using a biconditional discrimination task similar to that described above. The remaining two experiments examined this issue in an ambiguous occasion-setting task in order to explore more specific questions regarding the nature of the DOE in these Pavlovian procedures.

Experiment 1

Wilson and Pearce (1989) introduced a biconditional discrimination training procedure that was solved within 10 sessions by rats performing in a magazine approach conditioning paradigm. This is noteworthy given that biconditional discrimination learning is typically much more difficult to achieve, and pilot data in our lab confirmed this. Experiment 1, therefore, used a procedure adapted from Wilson and Pearce (1989) to explore the effects of differential outcome training upon biconditional discrimination learning. In our procedure, different discrete auditory stimuli (a noise or tone) played the roles of the X and Y stimuli described above. These stimuli were of short duration and embedded within longer duration background visual stimuli that played the roles of the A and B stimuli described above. Learning in this procedure was indicated when subjects approached the food magazine more on AX and BY trials than on AY and BX trials. One group was trained using a differential outcome procedure (AX-US1, AY-, BX-, BY-US2) while a second group was trained using a nondifferential outcome procedure (AX-US1/US2, AY-, BX-, BY-US1/US2) where each US was presented on 50% of the reinforced trials. We anticipated that this discrimination would be solved more effectively in animals given differential outcome training if the DOE applies to Pavlovian learning.

It may be noted that DeLong and Wasserman (1981) examined the effects of a differential outcome treatment in pigeons on learning an instrumental biconditional discrimination in a successive symbolic matching to sample task. These authors reported that animals trained with differential outcomes on reinforced trials, where different probabilities of reward served as the basis for this distinction, acquired this discrimination more rapidly than subjects reinforced at an intermediate level for each correct response. At issue in the present study is whether a formally similar result can be obtained using a Pavlovian biconditional discrimination task where differential outcomes are distinguished on the basis of qualitative features of reward.

Method

Subjects

Subjects were sixteen experimentally naïve male Sprague-Dawley rats purchased from Charles River Laboratories. They were housed individually in a separate colony room set to a 16-hr light/8-hr dark cycle. The rats were maintained at 85% of their normal body weights by limiting their access to food. At the start of the experiment the rats free feeding weights ranged from 348–385 gm. Water was available at all times.

Apparatus

Eight identical conditioning chambers were used. Each experimental chamber measured 30.5 cm × 24.0 cm × 25.0 cm and was encased in a wooden, sound and light attenuating outer chamber. Front and rear walls were made of aluminum and the side walls and ceiling were made of clear Plexiglas. The floor consisted of 0.60 cm (diameter) stainless steel rods arranged 2.0 cm apart. Placed at the center of one end wall 1.2 cm above the floor was a recessed food magazine measuring 3.0 × 3.6 × 2.0 cm (L × W × D). One 45-mg pellet (P.J. Noyes Co., Formula A) was released into the magazine during trials scheduled for reinforcement. Sucrose reinforcers (0.25 ml of a 24% solution) were delivered at the same food magazine. An infrared detector and emitter recorded head movements inside the recessed magazine chamber. Access to a lever located 3.0 cm to the right of the magazine and 8.0 cm above the floor, was prevented with a metal cover. A 6-W light bulb, located towards the top portion of the rear wall of the sound attenuating outer chamber, flashed, with equal on/off periods, at a rate of approximately 1.5 cycles /sec when activated. Another 6-W light bulb, located towards the bottom of a side wall of the outer chamber, emitted continuous illumination when activated. Approximately 22 cm behind the front wall of the chamber were two audio speakers. One speaker, when activated, emitted a 1500-Hz pure tone generated by a computer and amplified by a Radio Shack amplifier. The other speaker emitted white noise produced by a Grason-Stadler white-noise generator. The pure tone measured 4 dB and the white noise stimulus 12 dB above background noise. The chamber remained dark during trials except during presentations of the visual stimuli. Fans mounted to the light and sound resistant outer shells of the chambers supplied cross ventilation and produced background noise, measuring 78 dB. A PC and interfacing equipment (Alpha Products) located in the same room as the conditioning chambers automatically controlled and recorded events during experimental trials.

Procedure

Magazine training

Rats were magazine trained for two days with pellet and sucrose reinforcers. During the first 40-minute session, 20 pellet and then 20 sucrose reinforcers (0.25 ml of a 24% solution) were delivered at the same food magazine on a variable time 60-sec schedule. The second session was conducted in the same manner except that sucrose was delivered first, and then pellets within the session.

Conditional Discrimination training

Over the next 28 sessions all subjects were trained on a conditional discrimination task using procedures similar to those described by Wilson and Pearce (1989). Presentations of the visual stimulus A (flashing light) alternated with presentations of the visual stimulus B (steady light). These alternating intervals lasted 2 minutes. Embedded within each 2 minute presentation of the visual stimuli there occurred one 10-s presentation of stimulus X (white noise) and one 10-s presentation of stimulus Y (tone), the order of which was pseudorandomly determined. The target stimulus X was reinforced in A whereas Y was not, but the reverse was true when these stimuli were presented within stimulus B. This created 4 main trial types, AX+, AY−, BX−, BY+. In this study the roles of the white noise and tone CSs were not counterbalanced because our primary interest was in assessing the effects of a differential outcome treatment on learning this discrimination.

In the group receiving differential outcomes, AX+ and BY+ trials were rewarded with different outcomes (pellets for one stimulus compound and sucrose for the other, counterbalanced). In the nondifferential group, AX+ and BY+ trials were rewarded with both outcomes, that is, 50% of AX and BY trials were rewarded with pellets and 50% with sucrose. In each 48-minute session, there were 12 presentations of A and 12 presentations of B. The first target stimulus (X or Y) occurred either 10, 30, or 50 seconds (pseudorandomly determined) after the onset of the visual stimulus. The second presentation of a target stimulus within each 2 minute interval always occurred 30 seconds after the offset of the first target stimulus. Stimuli X and Y were the first targets within each interval 50% of the time. US delivery occurred at the offset of the appropriate auditory target stimulus. The rate of conditioned magazine responses was assessed both during presentations of the X and Y target stimuli as well as during a 10 s pre-stimulus period.

Statistical Analysis

Here, and throughout the manuscript, the magazine entry data were subjected to standard analysis of variance (ANOVA) tests (i.e., split plot factorial ANOVA). Because the statistical power of these tests is known to decline with increasing numerator degrees of freedom (e.g., when assessing many interactions), the type I error rate definition and resulting F tables of Rodger (1974) were used to assess the significance of the various tests conducted here. Use of these methods ensures that statistical power does not decline with increasing degrees of freedom (compare Tables 5 and 6 in Rodger, 1974). A type I error rate of 0.05 was adopted throughout.

Results

Figure 1 displays the course of acquisition for both groups in the conditional discrimination task. The graph represents the mean responses per minute plotted against trials, organized into blocks of four sessions each. The data were combined across the two rewarded conditions (AX and BY trials, referred to in the figure as CS+) and the two nonrewarded conditions (AY and BX trials, referred to as CS−) for the groups receiving both differential outcomes and nondifferential outcomes. The figure illustrates that the two groups acquired the discrimination at similar rates early in training. However, approximately midway through training, responding to the reinforced AX and BY trials in the group receiving differential outcomes continued to increase whereas responding during reinforced trials stabilized for Group Nondifferential.

Figure 1.

Mean magazine responding over 4-session blocks of biconditional discrimination training for groups receiving differential (Gp Diff) and nondifferential (Gp Nondiff) reinforcement in Experiment 1. Data are shown separately in the pre stimulus periods (Pre), as well as collapsed over reinforced (CS+) and nonreinforced (CS−) trials.

A Group (Differential/Nondifferential) × Stimulus (Reinforced/Nonreinforced) × Block ANOVA was used to statistically evaluate these data. This analysis revealed a significant main effect of Stimulus F(1, 14) = 59.98 (MSE = 99.97), and a significant Stimulus × Block interaction, F(6, 84) = 8.75 (MSE = 21.24), indicating that learning had taken place over training. Furthermore, the analysis revealed significant Stimulus × Group, F(1, 14) = 4.76 (MSE = 99.97), and Stimulus × Group × Block interactions, F(6,84) = 2.54 (MSE = 21.24), indicating that the groups differed in this learning. In order to evaluate the source of the three-way interaction, separate Group × Stimulus ANOVAs were performed at each block using a pooled error term (MSE = 32.49) with an adjusted df (Satterthwaite, 1946). Significant interactions were obtained in blocks 6, F(1, 58) = 10.81 and 7, F(1,58) = 8.94, indicating that by the end of training Group Differential had learned the discrimination better than Group Nondifferential.

A separate Group × Block ANOVA was also performed on the pre CS data. This analysis revealed a main effect of Block, F(6, 84) = 3.98 (MSE = 4.71), but no effects involving the Group factor.

Discussion

Experiment 1 demonstrates that a DOE can be found in a Pavlovian conditional discrimination task. These results are noteworthy because they challenge the theoretical analysis for the DOE offered by Trapold and Overmier (1972). It is possible, of course, that the sort of associative chain mechanism hypothesized by these theorists apply to DOE effects in instrumental settings, but their generality to the Pavlovian biconditional discrimination task studied here would seem limited. In the spirit of this approach, however, it is possible that differential A-US1 and B-US2 associations were formed in the present situation, but it is unclear how separate US1-X and US2-Y associations could have been used to help subjects in the differential outcome group learn the task (i.e., respond discriminatively to reinforced and nonreinforced stimuli) more successfully than the nondifferential outcome group. As noted in the general introduction above, the Rescorla-Wagner model fully anticipates that subjects given differential outcome training should learn the biconditional discrimination task more rapidly than subjects given nondifferential outcome training. In order to explain how subjects given nondifferential outcome training could learn the task, albeit, at a reduced level, however, this model would need to hypothesize the existence of unique cues.

Relatedly, DeMarse and Urcuioli (2005) have suggested that different serial compounds can generate differential outcome expectancies that can influence instrumental discriminative performance in some instrumental DOE tasks. Subjects in our task may have represented the stimuli in this manner while learning the discrimination. However, it is not clear from this perspective why subjects trained with differential reinforcement should have developed differential magazine approach CRs on the basis of these serial compound cues more successfully than subjects trained with nondifferential reinforcement. For instance, Group Differential could have learned to associate an A–X compound with US1 and a B–Y compound with US2, and this would establish magazine approach responding in the presence of these compounds more so than in the absence of these compounds. However, by the same logic Group Nondifferential would be expected to associate the A–X compound with both US1 and US2, and the B–Y compound with both US1 and US2. Since magazine approach CRs depend upon activation of either US1 or US2, it is not obvious how this state of affairs could result in an advantage for Group Differential in acquiring the discrimination.

We will take up the more general theoretical issue of how to best construe the DOE in Pavlovian learning later in this paper. For the present purposes, however, we have demonstrated that a DOE effect can occur in a Pavlovian biconditional discrimination learning task but it remains to be seen whether this effect can be shown to occur in other related conditional discrimination tasks.

Experiment 2

While the previous experiment provides evidence for the presence of a DOE in a Pavlovian biconditional discrimination procedure, Experiment 2 investigated whether a similar effect would occur in the closely related ambiguous occasion setting task (e.g., Holland, 1991; Holland and Reeve, 1991). A group trained with differential outcomes would be exposed to the following trial types: AX-US1, X-, Y-US2, AY-. Notice that this task differs from the biconditional discrimination procedure used in Experiment 1 in that the B stimulus is omitted. Because of this, the X and Y target stimuli are presented on their own on some trials and in the presence of the ambiguous occasion setter, A, on other trials. Thus, it is possible with this procedure to separately examine the ambiguous occasion setter’s positive occasion setting function on target X as well as its negative occasion setting function on target Y. When comparing performance in groups receiving differential reinforcement or nondifferential reinforcement, it is, therefore, possible to determine if the positive occasion setting, the negative occasion setting, or both components of this task are affected by the differential outcome manipulation. As noted above, Nakajima and Kobayashi (2000) provide evidence to suggest that learning of both components of this task is enhanced by the differential outcome manipulation in an instrumental learning version of the task. The present study began this exploration in a Pavlovian preparation.

Method

Subjects

16 experimentally naïve male Sprague-Dawley rats were used (purchases from Charles River Laboratories). They were maintained as those in the previous experiment. The rats’ free-feeding body weights at the beginning of the experiment ranged between 321 – 381.

Apparatus

The apparatus was identical to that used in Experiment 1.

Procedure

Magazine training

Subjects received one day of magazine training identical to that of the previous experiment.

Ambiguous Occasion Setting Discrimination Training

The procedures for the present experiment were similar to those used in Experiment 1, except where noted below. In the present experiment, the second background stimulus (B) was omitted. Thus, rather than A and B alternating, the single feature A alternated with 2-min periods of no A (i.e., a darkened chamber). Thus, the four trial types were as follows: AX+, X−, AY−, Y+. Notice that in order to learn this task, subjects must learn that in the presence of the 2-min A stimulus X is reinforced and Y is nonreinforced while the reverse is true in the absence of A. Because our primary interest in this experiment was examining the effects of differential or nondifferential reinforcement upon amgibuous occasion setting, for all subjects A was the 2-min flashing light, X was the 10-s tone, and Y was the 10-s noise. A preliminary study revealed that the identity of X and Y as tone or noise is inconsequential in predicting the course of learning in this task. Stimuli X and Y were reinforced with different outcomes (pellet and sucrose, counterbalanced) in Group Differential, but with both outcomes in Group Nondifferential. Magazine approach responding was once again measured during X and Y presentations and during 10 s pre stimulus periods.

Each 80-min session consisted of 20 alternating 2-min periods of A and no A. In a preliminary study we observed that under some circumstances subjects could use the outcome of the first trial within a given 2-min period to cue the outcome of the second trial within that same 2-min period. In order to eliminate this possibility, in the present experiment there were 8 different trial sequences (presented pseudorandomly) throughout each session. Within any given 2-min period any two-trial combination of X and Y target trials was scheduled to occur (i.e., A: X+ then X+, A: Y− then Y−, A: X+ then Y−, A: Y− then X+, No A: X− then X−, No A: Y+ then Y+, No A: X− then Y+, No A: Y+ then X−). As in Experiment 1 the two trials within a given 2-min period were separated by 30 s, but the first target stimulus could occur with equal probability 10 s or 50 s after the onset of the 2-min interval. Twenty-four sessions were carried out in this manner.

Results and Discussion

In this experiment Group Nondifferential subjects were unable to learn the ambiguous occasion setting discrimination after 24 sessions. Group Differential subjects, on the other hand, learned, relatively rapidly, both positive and negative occasion setting components of the task but displayed superior learning of the positive occasion setting component (see Figure 2).

Figure 2.

Mean magazine responding over 4-session blocks of training on the ambiguous cue occasion setting task for groups receiving differential (Gp Differential) and nondifferential (Gp Non Differential) reinforcement in Experiment 2. Data are shown separately in the pre stimulus periods (Pre, Pre A), as well as to the target stimuli (X, Y) in the presence and absence of the feature stimulus (A). + and * denote reinforcement by different outcomes and – indicates nonreinforcement.

An Occasion Setting Stimulus (reinforced/nonreinforced) × Block × Group ANOVA revealed a significant main effect of Stimulus, F(1, 14) = 10.76 (MSE = 210.38), and a significant Stimulus × Block interaction, F(5, 70) = 11.03 (MSE = 25.70), indicating that learning occurred overall. Furthermore, significant Stimulus × Group, F(1, 14) = 8.04 (MSE = 210.38) and Stimulus × Group × Block interactions, F(5, 70) = 3.88 (MSE = 25.70), suggest that learning was superior in Group Differential compared to Group Nondifferential. In addition, a significant Occasion Setting × Stimulus × Group × Block interaction, F(5, 70) = 2.58 (MSE = 23.69), suggests that the two components of the task were learned to different degrees in Group Differential.

In order to further explore the nature of this 4-way interaction the data were examined following the recommendation of Rodger (1974) in which separate one-way ANOVAs were performed on each group using a pooled error term (MSE = 65.98), and this was followed by Rodger’s post-hoc analyses examining specific differences within each block of training. This analysis revealed that both groups displayed differences in responding across training (F(23,322) = 4.91 for Group Differential and F(23,322) = 3.13 for Group Nondifferential). Subsequent post-hoc tests revealed that significant differences among the conditions (AX-US1 > Y-US2 > X- = AY-) emerged in Group Differential in blocks 4, 5, and 6 of training (ps < .05). In contrast, the only differences that emerged in Group Nondifferential were general increases in responding on all trial types across training. In other words, subjects in this group failed to learn the ambiguous occasion setting discrimination.

One additional analysis was applied to the Pre CS magazine response data. As seen in Figure 2 Pre CS response rates appeared equally low in both groups. A Group × Stimulus (Pre CS during the Flashing Light vs No Flashing Light) × Block ANOVA was applied to these data. This analysis revealed a significant main effect of Block, F(5,70) = 2.74 (MSE = 20.20) but no other main effects or interactions.

The results of Experiment 2 demonstrate that a DOE can be observed using a Pavlovian ambiguous occasion setting procedure. Moreover, this study reveals that subjects trained with differential outcomes learn both positive and negative occasion setting components of the ambiguous discrimination problem more readily than subjects trained with nondifferential outcomes. Different mechanisms may be invoked to explain these effects, and it is the purpose of the next experiment to examine this issue.

Experiment 3

The present experiment was conducted to examine different interpretations of the results reported in the previous studies. One possible explanation for the advantage seen in our groups given differential outcome training is based on the notion of acquired distinctiveness of cues (e.g., Hall, 1991; 1996; Lawrence, 1949; 1950). If the auditory stimuli used throughout these studies as target CSs are, to some degree, similar to one another, then differential outcome training could help through an acquired distinctiveness process (see also Delamater, 1998). In essence, it is possible that by associating the two target cues with different outcomes, these associations could render the two target cues as more distinctive, thus enabling the ambiguous occasion setting discrimination to be solved more readily.

Alternatively, there are at least two other explanations that do not require an analysis in terms of acquired distinctiveness processes. First, as noted above, the Rescorla-Wagner model can be applied to conditional discrimination tasks when differential outcomes are used. In the ambiguous occasion setting task used here, AX-US1, X-, Y-US2, AY-, the A stimulus could develop an excitatory association with US1 and an inhibitory association with US2 while X and Y could, respectively, develop excitatory associations with US1 and US2. This collection of associations will enable this model to explain successful performance in this task.

Another possible explanation for the present findings is based on Rescorla’s modulation account of occasion setting (Rescorla, 1985). Specifically, if an occasion setting stimulus were to act by modulating the activation threshold of the US representation, then an ambiguous occasion setting task should not be solvable if a nondifferential outcome procedure were to be used. This follows from the fact that the ambiguous occasion setting cue would concurrently be trained on negative occasion setting trials to raise the activation threshold for the US representation, but to also lower the activation threshold for the US representation on positive occasion setting trials (see Holland, 1991). Indeed, we failed to find evidence here for animals solving the task when trained with nondifferential outcomes.

However, this mechanism could be especially useful in solving the differential outcome ambiguous occasion setting task used here. This follows from the fact that the ambiguous occasion setter could simultaneously acquire the capacity to lower the activation threshold for one US representation (on positive occasion setting trials) and raise the activation threshold for the other US representation (on negative occasion setting trials). By modulating different US representations, the differential outcome ambiguous occasion setting task should be solvable.

In an attempt to distinguish between these different mechanisms, the present study examined the effects of two different reversal learning procedures on ambiguous occasion setting. The experimental design is indicated in Figure 3. Initially, all subjects were trained using the differential outcome ambiguous occasion setting procedure used in Experiment 2 (i.e., AX-US1, X-, Y-US2, AY-). Then, all subjects were trained on a reversal of this discrimination where the previously nonreinforced trials were now reinforced, and vice versa. Importantly, one group of subjects was reversed in such a way so as to preserve the different target-US relations (Group Targets: AX-, X-US1, Y-, AY-US2). In other words, for this group the X and Y target stimuli were paired with the same distinctive US (on those trials in which they were reinforced) in each phase of the experiment (i.e., X-US1 and Y-US2). In contrast, a second group of subjects was reversed in such a way so as to preserve the modulator-US relations across the two phases (Group Modulator: AX-, X-US2, Y-, AY-US1). In this group the X and Y target stimuli were paired with the alternate US during the reversal phase. This fact would present problems for any acquired distinctiveness mechanism because across the two phases both X and Y would have been trained with both USs, a condition that could lead to acquired equivalence rather than distinctiveness effects (e.g., Hall, 1991; 1996). Therefore, the acquired distinctiveness explanation would anticipate that Group Targets should acquire the reversal task more rapidly than group Modulator.

Figure 3.

Experimental design used in Experiment 3. The target stimuli (X, Y) were followed by (->) one of two reinforcing outcomes (US1, US2) in the presence or absence of the feature stimulus (A). The contingencies were reversed during the reversal phase either by maintaining the target-US or the modulator-US relations.

On the other hand, the modulation (also Rescorla-Wagner) model makes a different prediction. Initially, the A stimulus serves as a positive modulator of the US1 activation threshold (because of the AX-US1, X- trials) and a negative modulator of the US2 activation threshold (because of the Y-US2, AY- trials). Exactly the same modulator-US relations (i.e., A is a positive modulator for US1 and negative modulator for US2) must be learned during the reversal phase for Group Modulator, whereas these modulatory relations must be reversed in Group Target for this group to learn the discrimination. Accordingly, this model (Rescorla, 1985; also the Rescorla-Wagner model, 1972) would lead to the prediction that Group Modulator should acquire the reversal more rapidly than Group Target because the modulator-US relations necessary for learning the task were already learned in the former group but would need to be reversed in Group Target. The present study examined these predictions.

Method

Subjects

32 experimentally naïve male (26) and female (6) rats bred at Brooklyn College from Sprague-Dawley parents (supplied by Charles Rivers Labs) were used. They were maintained as those in the previous two experiments. The free-feeding weights at the beginning of the experiment ranged between 458 – 702 gm (males) and 238 – 371 gm (females).

Apparatus

The apparatus was identical to that used in Experiments 1 and 2.

Procedure

Magazine training

Subjects received two days of magazine training identical to that of Experiment 1.

Ambiguous Occasion Setting Discrimination Training

All subjects were initially trained with the differential outcome ambiguous occasion-setting procedure used in Experiment 2 over 24 sessions. The only difference was that the stimuli and USs used in the present study were fully counterbalanced, i.e., Tone and Noise stimuli equally played the roles of X and Y and all of the particular stimulus-US combinations were balanced across subjects.

Ambiguous Occasion Setting Reversal Training

During the next 12 sessions all subjects were trained on a reversal task. The subjects were matched for their performance in the first phase and assigned to two groups. Group Targets were reversed in such a way so as to preserve the specific target-US relations in phase 1. For instance, if subjects were trained with a Flash: Tone-sucrose, Flash: Noise-, Tone-, Noise-pellet discrimination originally, these same subjects would be trained with a Flash: Tone-, Flash: Noise-pellet, Tone-sucrose, Noise- discrimination during the reversal phase. Group Modulator, on the other hand, was reversed in such a way so as to maintain the modulator-US relations trained originally. Thus, if subjects were trained with a Flash: Tone-sucrose, Flash: Noise-, Tone-, Noise-pellet discrimination originally, these same subjects would be trained with a Flash: Tone-, Flash: Noise-sucrose, Tone-pellet, Noise- discrimination during the reversal phase. The same parameters were used during this phase as were used during original training.

Results and Discussion

Data from one rat in Group Targets was excluded from all analyses because this rat developed a habit of gnawing at the magazine wall (near the photocell) throughout the experiment, and this created abnormally high magazine entry scores (e.g., 3–4 times higher during the Pre CS periods than the average of the other rats). The main conclusions drawn from the analyses did not depend upon whether or not data from this rat was excluded.

Acquisition of the ambiguous occasion setting discrimination proceeded as in Experiment 2, as can be seen in Figure 4 which illustrates mean magazine responding during the various conditioning trial types over 4-session blocks during the acquisition and reversal phases. During the acquisition phase Group Targets and Group Modulator both received the same ambiguous occasion setting discrimination with differential outcomes. Each of these groups displayed superior performance on the positive occasion setting component of the task, and also learned the negative occasion setting component of the task. The acquisition data were analyzed using a Group × Occasion Setting × Stimulus (reinforced/nonreinforced) × Block ANOVA. This analysis revealed a significant main effect of Stimulus, F(1, 29) = 141.60 (MSE = 175.65), as well as significant Stimulus × Block, F(5, 145) = 60.73 (MSE = 33.40), and Occasion Setting × Stimulus × Block interactions, F(5, 145) = 2.00 (MSE = 60.39), to confirm the above impressions.

Figure 4.

Mean magazine responding over 4-session blocks of training and reversal on the ambiguous cue occasion setting task in Experiment 4. Data are shown separately in the pre stimulus periods (A (Pre), No A (Pre)), as well as to the target stimuli (X, Y) in the presence and absence of the feature stimulus (A). + and * denote reinforcement by different outcomes and – indicates nonreinforcement. The first symbol before the / indicates the outcome presented during acquisition and the second symbol after the / indicates the outcome presented during the reversal phase.

The data of most interest were from the reversal phase. It is clear that Group Targets learned to reverse both the positive and negative occasion setting components of the task more rapidly than Group Modulator. A Group × Occasion Setting × Stimulus × Block ANOVA was performed on the reversal data and confirmed this impression. This analysis revealed a significant main effect of Stimulus, F(1, 29) = 5.73 (MSE = 35.83), as well as a significant Stimulus × Block interaction, F(2, 58) = 33.79 (MSE = 18.86). Most critically, the Stimulus × Group interaction, F(1, 29) = 6.41 (MSE = 35.83), was also significant, and indicates that Group Targets learned the reversal better than Group Modulator.

A further analysis of these data was conducted following Rodger’s methods (Rodger, 1974) in order to examine where differences emerged across reversal training. Separate one-way ANOVAs were performed on each group using a pooled error term (MSE = 73.51), and these were followed by separate post-hoc tests to determine where in training differences emerged. In this analysis Group Target displayed differences in responding on the various types of trials across training, F(11,319) = 4.22, whereas Group Modulator did not. Post-hoc tests revealed that in Group Target responding on reinforced trials was greater than responding on nonreinforced trials in block 3 (responding across trial types were ordered as follows: X-US1 > AX- = AY-US2 > Y-), and that responding to target stimulus X was greater than to Y throughout this phase (ps < .05).

Additional analyses were applied to the Pre CS data during both the acquisition and reversal phases. Separate Group × Stimulus (Flash, No Flash) × Block ANOVAs revealed a significant main effect of Block in both phases (F(5,145) = 22.30 (MSE = 8.78) during acquisition and F(2,58) = 12.49 (MSE = 12.12) during reversal), but no other main effects or interactions.

The data from the present study show that ambiguous occasion setting with differential reinforcement was reversed more successfully when the target-US relations were maintained during the reversal phase than when the modulator-US relations were maintained. Although several theoretical approaches to occasion setting, and conditional discrimination learning more generally, will be able to accommodate the basic fact of a DOE in a Pavlovian occasion setting task, these reversal data should serve to constrain the theoretical analysis further. We shall take up this issue more fully in the general discussion section.

General Discussion

The present studies investigated whether a differential outcome effect (DOE) could be obtained in a magazine approach conditioning paradigm, commonly used to study Pavlovian processes. We observed that both biconditional as well as ambiguous occasion setting discriminations were learned more readily in animals receiving a differential reinforcement treatment compared to subjects receiving nondifferential reinforcement. This effect was manifest as more successful biconditional discrimination learning in Experiment 1 and superior learning of both positive and negative occasion setting components of an ambiguous occasion setting discrimination task in Experiment 2 in subjects trained with differential compared to nondifferential reinforcement. Finally, the results of Experiment 3 demonstrated that having been trained on an ambiguous occasion setting task with differential reinforcement, reversal learning of this task was accomplished more successfully in subjects trained with target-US relations (as opposed to modulator-US relations) preserved in the two phases. The present results have a number of implications for our understanding of differential outcome effects in Pavlovian learning, as well as conditional discrimination learning more generally, and these are discussed below.

One of the motivations in conducting this series of studies was to determine if a DOE could be obtained in Pavlovian learning. The present studies were designed as a Pavlovian analogue (AX-US1, AY-, BX-, BY-US2) to the normal instrumental biconditional discrimination task (A: R1-O1, A: R2-, B: R1-, B: R2-O2), but where stimuli (X & Y) played the functional roles of instrumental responses. Converting the task into a Pavlovian procedure preserves the formal structure of the problem as a conditional learning task, but by doing so at least one theory of the DOE would seem to make different predictions. Trapold and Overmier (1972) suggested that Pavlovian S-O and instrumental O–R associations both contribute to successful conditional discrimination learning in the instrumental differential outcome task. However, as noted above, an extension of these ideas to the Pavlovian case will not necessarily result in a DOE. For instance, if we allow for differential A-US1 and B-US2 associations to develop in conjunction with differential US1-X and US2-Y associations, then it is difficult to see how this will lead to an advantage in learning the conditional discrimination task.

At least three other theoretical approaches can be used to explain the occurrence of a DOE in Pavlovian learning. The Rescorla-Wagner model (1972) explains the phenomenon by noting that in a nondifferential outcome training procedure the task should not be solvable without resorting to compound unique cues. On the other hand, when differential outcomes are used, then this would permit for the different feature and target stimuli to enter into both excitatory associations with one US and inhibitory associations with the other US. The one proviso here is that it must be assumed that there exists a high degree of outcome specificity in both excitatory and inhibitory associations, a claim that has been met with a considerable amount of support for excitatory associations (e.g., Delamater, 1995; 1996; 1997; 2007; Delamater and Holland, 2008) but with more modest support for inhibitory associations (Delamater, LoLordo, & Sosa, 2003; Kruse, et al, 1983).

In addition, Rescorla’s modulation theory (1985), as discussed in more detail above, could also be used to explain the DOE in Pavlovian learning established here. It would solve the problem by virtue of the A and B stimuli acquiring separate modulatory functions on the different US representations.

Finally, a theory based on the notion of acquired distinctiveness of cues (e.g., see Delamater, 1998; Hall, 1991; Honey and Hall, 1989) could also apply, in principle, to the DOE in Pavlovian learning. If separate X-US1 and Y-US2 associations were to increase the effective discriminability of these cues, then the discrimination should be made easier for the simple reason that the solution requires animals to learn about these stimuli. This notion has also been entertained in an analysis of the DOE in instrumental learning (e.g., DeMarse and Urcuioli, 1993; Nakajima and Kobayashi, 2000; Urcuioli, 1991).

All three of these theoretical approaches could make sense of the results from Experiments 1–2, but the data from Experiment 3 help to distinguish among these approaches. The accounts outlined above would seem to apply just as readily to a biconditional discrimination task as it would an ambiguous occasion setting task. However, the reversal learning data in Experiment 3 revealed that subjects were able to acquire a reversal of the ambiguous occasion setting task with differential outcomes faster if the different target-US associations (as opposed to the different modulator-US relations) were preserved during the acquisition and reversal phases. As noted above, this finding is exactly what would be expected on the basis of an acquired distinctiveness explanation of the differential outcome effect in these studies. On the other hand, if either the different feature-US modulatory or associative relations were critical, then we would have expected to observe faster reversal learning when the feature-US relations were preserved during the reversal phase. Thus, these findings present problems for the Rescorla-Wagner model (1972) and for Rescorla’s (1985) modulation theory accounts of the DOE in Pavlovian learning.

If we accept that the present data are more in support of an acquired distinctiveness explanation of the DOE in Pavlovian learning, then this would require some further comments regarding the nature of the mechanisms thought to mediate acquired distinctiveness phenomena. One approach to this problem was developed by Honey and Hall (1989). These authors suggested that when different stimuli form associations with the same US, this common associate could mediate an acquired equivalence among the cues. In contrast, when different stimuli form associations with different USs, these distinct associations could mediate an acquired distinctiveness among the cues (see also Lawrence, 1949; 1950).

In the present circumstance, these ideas can be extended to account for some of our findings. Suppose that in our biconditional discrimination task with differential outcomes stimuli A and X both formed relatively weak associations with US1 while B and Y developed weak associations with US2. On AX and BY trials, both elements would be presumed to activate representations of the same outcome (US1 on AX trials and US2 on BY trials). However, on AY and BX trials both elements would be presumed to activate different outcome representations. If subjects could utilize this information, then they could expect reinforcement whenever both elements have been associated with the same as opposed to different outcomes. This sort of rule could not be used in subjects given nondifferential outcome training. This analysis, although appealing at some level, prompts the further question of precisely how such a rule might be conceptualized within an associative system and then utilized.

Another framework in which to think of acquired distinctiveness and equivalence effects was provided by Honey and colleagues (Allman and Honey, 2006; Allman, Ward-Robinson, and Honey, 2004; Grand and Honey, 2008; Honey, 2000; Honey and Ward-Robinson, 2001; 2002; Honey and Watt, 1998). These authors adopted a connectionist network approach to explain biconditional discrimination performance. Specifically, in the biconditional task used here presentation of different stimulus compounds would be assumed to recruit different “hidden units” within a multi-layer network consisting of CSs activating input layer units and USs activating output layer units. Once separate ax, ay, bx, and by hidden units were to be recruited by AX, AY, BX, and BY stimulus compounds, respectively, then such units would be assumed to enter into associations with appropriate output layer units. If differential outcomes are used to reinforce AX and BY compounds, then the corresponding ax and by hidden units would develop excitatory connections with the US1 and US2 output units, respectively. However, in the nondifferential training condition it may be assumed that ax and by hidden units would develop weaker associations, but with both US1 and US2 output units. Given this framework (see also DeMarse and Urcuioli, 2005), it is not obvious to us that differential reinforcement should necessarily make it easier than nondifferential reinforcement to learn the basic task. Additional assumptions would need to be made concerning the mapping of activation of the different US representations into performance.

Other connectionist network approaches (e.g., Gluck and Myers, 1993; Schmajuk and DiCarlo, 1992; Schmajuk, Lamoureax, and Holland, 1998) have in common with Honey and colleagues the idea that a hidden unit layer is critical in finding solutions to complex discrimination learning problems. One way in which acquired equivalence and distinctiveness of cues can be thought of, more generally, is through a process of compression and differentiation of the representations of different stimuli depending upon whether they share common or distinct training histories (see Gluck and Myers, 1993). In particular, if two stimuli have been associated with different USs, then it may be assumed that more distinctive hidden layer representations of these stimuli develop. Conversely, if two stimuli have been associated with the same US, then the two stimuli may be assumed to activate a more similar pattern of hidden layer units – in other words, their internal representations become more similar. It follows from this idea that discrimination learning between the two stimuli should be easier if the stimuli have been associated with different USs, but more difficult if they have been associated with the same US. Delamater (1998) provided evidence for this using a procedure that ruled out outcome-mediated generalization of the sort proposed by Honey and Hall (1989) as a possible explanation. These data together with others (e.g., Allman, Ward-Robinson, and Honey, 2004; Delamater, Sosa, and Katz, 1999; Grand and Honey, 2008; Honey and Ward-Robinson, 2002; Honey and Watt, 1998) offer strong evidence to support the general notion from connectionist models that hidden layer representations of stimuli change as a function of conditioning and contribute to discrimination learning.

It is far from obvious, however, that any particular connectionist model (e.g., Gluck and Myers, 1993; Honey and Ward-Robinson, 2002; Schmajuk and DiCarlo, 1992; Schmajuk, Lamoureux, and Holland, 1998) will explain the present set of findings because there are a number of constraining features in the data. We have demonstrated that (1) differential outcome training produces superior biconditional discrimination learning to nondifferential outcome training, (2) differential outcome training facilitates the learning of both positive and negative occasion setting components of an ambiguous occasion setting task, (3) the positive occasion setting component of the task is learned more rapidly than the negative occasion setting component of the task, and (4) ambiguous occasion setting reversal learning proceeds more rapidly when the target-US as opposed to the feature-US relations are preserved in the two phases. This number of empirical constraints would seem to present challenges to any theory of discrimination learning. In preliminary simulations that we have conducted using the real-time model of Schmajuk and DiCarlo (1992), we have found that although this model can learn the basic ambiguous occasion setting task and show faster learning with differential than nondifferntial reinforcement, the model erroneously predicts that the negative occasion setting component of the ambiguous occasion setting task should be learned more rapidly than the positive occasion setting component, a fact that is at odds with the findings reported here and elsewhere (Holland, 1991; Holland and Reeve, 1991; Nakajima and Kobayashi, 2000). Future theoretical work will be needed to determine how to best conceptualize the present findings within a connectionist framework.

The present data may also have implications for research investigating the neural mechanisms of acquired equivalence and distinctiveness effects and perhaps also the DOE. Honey and his colleagues have observed that the entorhinal cortex (Coutureau, Killcross, Good, Marshall, Ward-Robinson, and Honey, 2002) and the medial prefrontal cortex (Iordanova, Killcross, and Honey, 2007) play critical roles in mediating acquired distinctiveness and/or equivalence of cue effects. If the present data are best described in terms of an acquired distinctiveness or equivalence mechanism, then the same sort of neural manipulations observed to affect performance in other tasks thought to assess these mechanisms should have similar affects on the tasks explored here. In particular, we anticipate that entorhinal cortex or medial prefrontal cortex lesions should abolish the DOE we report here, a prediction that has yet to be examined. Moreover, these structures may also prove to be critical in mediating the DOE in instrumental conditioning (e.g., see Ramirez and Savage, 2007; Savage, Koch, and Ramirez, 2007) to the extent that acquired distinctiveness plays a role there.

One final issue worth some comment is the claim made throughout this manuscript that we are studying Pavlovian as opposed to instrumental learning processes. In the magazine approach paradigm it is true that rats must make an approach response in order to retrieve food reward, and this could introduce an instrumental component to the task. However, it should be recognized that the response requirement in this case (i.e., approach) is considerably less complex than what is required in most instrumental conditioning studies (e.g., lever pressing), and this could emphasize more the Pavlovian relations inherent in this task. This view receives some support from the finding that conditioned magazine approach responding to auditory cues is not eliminated by an omission contingency (Holland, 1979).

In summary, we have provided results from a series of studies documenting the presence of a DOE in Pavlovian learning. These basic results can be understood from a variety of perspectives, however, the more specific findings of a positive occasion setting advantage in the ambiguous occasion setting task, that both components of the ambiguous occasion setting task benefit from differential reinforcement, and that reversal learning occurs more rapidly when target-US relations are preserved more strongly support an interpretation based on the notion of acquired distinctiveness of cues. Exactly how this concept is best understood, however, will require additional theoretical and empirical development.