On The Effects Of Signaling Reinforcer Probability And Magnitude In Delayed Matching To Sample

Abstract

Two experiments examined whether postsample signals of reinforcer probability or magnitude affected the accuracy of delayed matching to sample in pigeons. On each trial, red or green choice responses that matched red or green stimuli seen shortly before a variable retention interval were reinforced with wheat access. In Experiment 1, the reinforcer probability was either 0.2 or 1.0 for both red and green responses. Reinforcer probability was signaled by line or cross symbols that appeared after the sample had been presented. In Experiment 2, all correct responses were reinforced, and the signaled reinforcer durations were 1.0 s and 4.5 s. Matching was more accurate when larger or more probable reinforcers were signaled, independently of retention interval duration. Because signals were presented postsample, the effects were not the result of differential attention to the sample.

Accuracy of remembering typically worsens over time, but how is it affected by reinforcement? Remembering is often studied using delayed matching to sample (DMTS) (Blough, 1959). In each trial of this discrete-trial paradigm, choices (B₁ or B₂) are occasionally reinforced when they match a sample (S₁ or S₂) seen seconds earlier. The typical result is that indices of discrimination performance such as proportion correct, d′ (Macmillan & Creelman, 1991), or log d (Davison & Tustin, 1978) gradually fall as the delay or retention interval between the sample and the choice lengthens. The mathematical form of this reduction over time approximates a number of negatively accelerated, monotonic functions (Rubin & Wenzel, 1996), and is commonly known as the forgetting function. Because remembering performance in DMTS typically is shaped by reinforcing matches (B₁|S₁ and B₂|S₂) but not nonmatches (B₁|S₂ and B₂|S₁), remembering may be viewed as a discrimination established through differential reinforcement (Catania, 1979; White, 1985, 1991, 2001). It is therefore interesting to examine how different reinforcement variables affect the forgetting function.

When correct B₁ and B₂ responses on S₁ and S₂ trials in DMTS are reinforced by different probabilities, durations, or qualities of food, discrimination is overall better (De-Long & Wasserman, 1981; Peterson, Wheeler, & Trapold, 1980; Santi & Roberts, 1985), and the rate of forgetting is less rapid (Brodigan & Peterson, 1976; Jones & White, 1994; Jones, White, & Alsop, 1995). This is the differential outcomes effect. This effect is not to be confused with the signaled magnitude effect. Here, reinforcement for correct B₁ and B₂ responses on S₁ and S₂ trials is the same, but reinforcer magnitude or duration differs according to a signal presented on each trial. In previous studies of the signaled magnitude effect, the signals for long- versus short-duration reinforcers have been a tone versus white noise (Nevin & Grosch, 1990), horizontal or vertical lines on the center key (Jones et al., 1995), or the left or right position of an illuminated houselight (McCarthy & Voss, 1995). In all cases, the result is higher accuracy on trials that signal the longer reinforcer, independently of the retention interval. This generates a forgetting function that is overall higher, but has the same rate of forgetting as when the short reinforcer duration is signaled (Jones et al., 1995).

The same effect on the forgetting function, overall higher accuracy without a change in the rate of forgetting, is observed when the sample is presented for a longer period of time (Nelson & Wasserman, 1978), has a higher fixed-ratio (FR) requirement (Roberts, 1972; White, 1985; White & Wixted, 1999), or when different responses are required on each sample in order for the retention interval to begin (Zentall & Sherburne, 1994). It is therefore plausible, as argued by Nevin and Grosch (1990), that signaling a larger reinforcer improves performance by modifying attentional processes at the time the sample is presented. On the other hand, White (2001) demonstrated that performance at a particular retention interval and not others can be impaired by omitting reinforcers at that retention interval only. This reinforcer effect appears to result from a change in discrimination processes at the time of the choice (White, 1991, 2001). A related mechanism could underlie the signaled magnitude effect.

Although the signaled magnitude effect has been demonstrated several times, we are only aware of one DMTS experiment in which reinforcer probability was signaled. Nevin, Milo, Odum, and Shahan (2003) found that matching at both 0-s and 3-s delays was more accurate when a higher probability of reinforcement (0.8 vs. 0.2) was signaled prior to DMTS trials. Their experiment was not directly comparable to standard signaled magnitude procedures, however, because DMTS trials were accessed by key pecking in a two-component multiple schedule. In more standard procedures, manipulations of reinforcer probability have typically focused on differential outcomes, where reinforcer probability differs for correct B₁ and B₂ responses (DeLong & Wasserman, 1981; Peterson et al., 1980; Santi & Roberts, 1985). In these experiments, accuracy is overall higher with different reinforcer probabilities than same probabilities for correct B₁ and B₂ responses. Reinforcer probability also has been manipulated between conditions, where overall higher reinforcer probabilities for correct B₁ and B₂ responses generate overall more accurate performance (Brown, 2003; White & Wixted, 1999).

In the present Experiment 1, we confirmed the occurrence of a signaled probability effect under the same conditions as a signaled magnitude effect, which was demonstrated in Experiment 2. A novel procedural difference was that, unlike previous signaled magnitude experiments (Jones et al., 1995; McCarthy & Voss, 1995; Nevin & Grosch, 1990), the signal for reinforcer probability or magnitude was only presented after the sample had been extinguished. This change followed Nevin and Grosch's conjecture that the signaled magnitude effect resulted from greater attention to the sample on large-reinforcer trials, as evidenced by a greater number of pecks to the constant-duration sample. A similar account could apply to a signaled probability effect because the version reported by Nevin et al. (2003) involved the signal (the multiple schedule component stimuli) preceding the sample for the DMTS trial. Because the signal in the present experiment was not presented until after the sample had been presented, any signaled probability or magnitude effect found could not be the result of differential attention to the sample.

Experiment 1

Experiment 1 tested whether performance in DMTS could be affected by signaling the probability of reinforcement for correct matching, as Nevin et al. (2003) had suggested. More important, Experiment 1 also tested whether a signaled probability effect would arise when the reinforcer probability signal was shown after the sample had been presented. Reinforcer probabilities were the same for correct red and green responses, but differed from trial to trial according to signals for high and low probabilities of reinforcement.

Method

Subjects

Five adult homing pigeons (Columba livia), numbered N1 to N5 and experienced in DMTS, were housed in individual cages in a room with natural lighting supplemented by incandescent light on a 12:12 hr light/dark cycle. Water and grit were freely available in the home cages. After each experimental session, pigeons were given enough corn, wheat, food pellets, and maple peas to maintain their weight at 85% ± 10 g of their free-feeding weights, which were measured midsummer. Pigeons were within this weight range prior to all experimental sessions.

Apparatus

Five Med Associates Inc. experimental chambers were 29.5 cm high, 29.5 cm wide, and 24.5 cm deep. Each had a grid floor, opaque walls, and opaque black plastic side doors. They were not sound attenuating, so extractor fans provided masking noise. Each chamber contained three plexiglass response keys, 2.1 cm in diameter, recessed 1 cm into the chamber wall and 21 cm above the grid floor. The two side keys were positioned 6 cm on either side of the center key. All keys could be lit red or green and required a force of approximately 0.15 N to activate. In addition, center keys could display white diagonal-cross or vertical-line symbols on a dark background. Symbols spanned the diameter of the center key. When any lit key except one displaying a symbol was pecked, it activated a relay to provide auditory feedback. (Pecks to keys with symbols were inconsequential, and hence feedback was not required.)

Wheat reinforcers could be delivered via a hopper located in an aperture 12.5 cm below the center key. A lamp inside the aperture and above the hopper was lit when wheat was available. The lights, hopper, and recording of key pecks to lit keys were all administered by an IBM®-compatible PC running Med-PC® for Windows v1.15 software under Windows® 95, and connected to Med-PC® interfacing.

Procedure

Each pigeon was tested at the same time (± 30 min) and in the same chamber every day. Each trial began when a red or green sample was presented on the center key. Once the sample key was pecked ten times (FR 10), it was immediately replaced by either a vertical line or a cross on the center key. The vertical line signaled a 1.0 probability of reinforcement for correct matching, whereas a cross signaled a reinforcer probability of 0.2.

After the symbol had been displayed for a retention interval of 1, 3, 6, 12, or 18 s, one side key was illuminated red and the other green. The cross or line symbol remained on the center key while the choices were presented. When one of the side keys was pecked once, all keys were extinguished. Correct responses (pecks to colors that matched the sample) were immediately reinforced by 3.5-s access to wheat, with a probability of 1.0 or 0.2 as signaled. The reinforcer duration was timed from when the hopper was elevated. Incorrect or unreinforced responses resulted in 3.5-s blackout. A dark 15-s intertrial interval followed each trial regardless of its outcome.

Daily sessions consisted of 81 trials, the first of which did not contribute to data analysis. The order and frequency of the different sample colors, prior-trial sample colors, choice color positions (left vs. right), prior-trial choice color positions, retention interval durations, reinforcer probabilities and their associated center-key symbols were fully counterbalanced so that each combination occurred only once over 160 trials. The sequence was split into two 80-trial sessions that alternated between days, with the first trial of each session matching the last trial of the previous day's session. If a pigeon's session did not finish within 50 min, the session was terminated. Each pigeon had 4 months' training with this procedure, followed by 72 sessions when the present data were collected.

Results and Discussion

All 72 sessions of data were analyzed for each pigeon. Data from the first 36 sessions (the first run) were compared to data from the second (the replication) to ensure that any effect was stable. For each pigeon in each replication, responses were summed over 36 sessions, separated according to sample color (red or green), choice color (red or green), choice side (left or right), prior sample color (same or different to the choice color), and retention interval duration (1, 3, 6, 12, or 18 s). There were an equal number of sessions of each of the two trial sequences for each pigeon.

Figure 1 shows the proportion-correct matching accuracy at each retention interval, based on data for all 72 sessions, for trials with signaled reinforcer probabilities of 0.2 and 1.0. A signaled probability effect occurred for each pigeon, but was smaller for Pigeons N1 and N4 than for other pigeons. The proportion-correct measure, however, is constrained by ceiling effects because it is bounded at 1.0 (Nevin & Grosch, 1990; White, 2001). Furthermore, proportion correct is subject to response bias from preferences to choose a particular color and to choose a particular side key. Additionally, the choice on a trial can be biased by the reinforced choice on the previous trial (White, Parkinson, Brown, & Wixted, 2004). In general terms, this is manifest as an overall preference towards the choice that matches the prior trial's sample (D'Amato, 1973; Grant, 1975, 1982; Miyashita, Nakajima, & Imada, 2000; Roberts & Grant, 1976). These biases were treated as per Davison and Tustin's (1978) treatment of response bias. They proposed log d as a measure of performance that was free from response bias. Log d is directly proportional to ℓn α from Luce's (1963) choice theory, and has an almost linear relation with d′ (Macmillan & Creelman, 1991) from signal detection theory (Green & Swets, 1966).

Fig. 1. Proportion-correct matching at each retention interval duration in Experiment 1 (signaled reinforcer probability).

Functions are plotted separately for signaled reinforcer probabilities of 1.0 (filled circles, solid lines) and 0.2 (open triangles, dotted lines). Mean scores are displayed in the main graph, and individual scores are shown in peripheral graphs.

Figure 2 shows each pigeon's discrimination performance at each retention interval, and in both replications, as measured by log d. These data were submitted to a three-factor repeated measures analysis of variance (replication × reinforcer probability × retention interval duration). Figure 2 shows that when a high reinforcer probability (1.0) was signaled, log d was consistently higher than when a low reinforcer probability (0.2) was signaled, F(1, 8) = 23.55, p < .01. This effect was the same in both the first run and replication, F(1, 8) < 1.0, and for all retention interval durations, F(4, 64) = 1.36, in both replications, F(4, 64) < 1.0, as indicated by the absence of statistically significant interactions between the effects of reinforcer probability and these other variables. Experiment 1 therefore demonstrated that a signaled probability effect, analogous to a signaled magnitude effect, could be demonstrated for DMTS, as suggested by Nevin et al.'s (2003) multiple-schedule DMTS procedure. The effect of signaled reinforcer probability on the forgetting function is a change in its overall level, not a change in its slope (particularly when measured by log d, which is bias free and not as prone to ceiling effects). Because the signal on a trial was unpredictable and was presented after the sample had been extinguished, the observed signaled probability effect could not be caused by differential attention to the sample. Would the same be true of the signaled magnitude effect?

Fig. 2. Mean and individual discriminability estimates (log d) at each retention interval duration in Experiment 1 (signaled reinforcer probability).

Data are shown from the first run (top panel) and its replication (bottom panel). Functions are plotted separately for signaled reinforcer probabilities of 1.0 (filled circles, solid lines) and 0.2 (open triangles, dotted lines).

Experiment 2

Experiment 2 revisited the signaled magnitude effect (Jones et al., 1995; McCarthy & Voss, 1995; Nevin & Grosch, 1990). Unlike prior studies, however, the signal for reinforcer duration was presented after the sample had been presented (like the signal in Experiment 1). This ensured that differential sample responding could not occur because the signal for reinforcer duration was unpredictable at the time the sample was presented. Previously, differential sample responding had been implicated in both the signaled magnitude (Jones et al., 1995; Nevin & Grosch, 1990) and differential outcomes (Jones & White, 1994) effects. Additionally, in the present experiment the signal was turned off as soon as the choice stimuli were presented. That is, the signal was not displayed during either the sample or the choice, and therefore the duration of signal presentation exactly equaled the arranged retention interval duration.

Method

Subjects and Apparatus

Five adult homing pigeons, numbered Z1 to Z5, had prior DMTS experience. They were not the same pigeons as those used in Experiment 1. They were, however, treated identically and tested in the same operant chambers. Sometimes pigeons did not participate in the day's experimental session because their weight was outside of the prescribed range.

The only software difference from Experiment 1 was that an earlier version of Med-PC® for Windows, v.1.10, was used. The only equipment difference was that the chambers' plastic side doors were transparent rather than opaque, and partitions were used to obscure chambers from each other in the dark experimental room.

Procedure

The general procedure was similar to that of Experiment 1, but with one important difference and several minor differences. The important difference was that rather than signaling different reinforcer probabilities, the symbol on the center key signaled different reinforcer durations. Specifically, a diagonal cross indicated that correct matching would result in 1.0-s reinforcement, whereas a vertical line signaled 4.5-s reinforcement. Observation by the experimenters confirmed that the short reinforcer duration of 1.0 s was sufficient for all pigeons to obtain at least one grain of wheat. Reinforcers were delivered for all correct matches, whereas nonmatches were never reinforced.

There were six minor differences from the procedure of Experiment 1. First, the signal was turned off immediately prior to the choice being presented, rather than staying lit until the choice was made. Second, there were only four arranged retention interval durations: 1, 2, 4, and 8 s. Third, there were 96 trials per session. Fourth, trials occurred in a random order, without replacement, such that each sample-stimulus/signal/choice-pair combination occurred three times for each retention-interval duration. Because each session was self-contained in terms of the different types of trials, trial sequences were not split over two sessions. Fifth, the sample key required five pecks rather than ten in order to initiate the retention interval. Sixth, there were fewer sessions in Experiment 2. Following six months' preliminary training in this procedure with retention intervals of 1, 3, 6, and 12 s, delays were changed to 1, 2, 4, and 8 s. Training continued under the latter delays for a further 26 days, but some pigeons completed fewer sessions because their weights were outside the prescribed range. There were 26, 25, 26, 14, and 17 sessions for Pigeons Z1 to Z5, respectively. Owing to the smaller number of sessions, data were not separated into first and second halves.

Results and Discussion

Response counts were derived from all sessions for all pigeons. Although this meant that some pigeons contributed more sessions than others did, this has little impact on analyses because comparisons are within-subject. Log d was used as an index of discrimination as in Experiment 1 (percentage-correct scores reveal a similar pattern of results and confirm the findings presented here). The resulting data are shown in Figure 3, and were submitted to a two-factor repeated measures analysis of variance (reinforcer duration × retention interval duration).

Fig. 3. Mean and individual discriminability estimates (log d) at each retention interval duration in Experiment 2 (signaled magnitude effect).

Functions are plotted separately for signaled reinforcer durations of 4.5 s (filled circles, solid lines) and 1.0 s (open triangles, dotted lines).

Figure 3 shows that when long reinforcer durations (4.5 s) were signaled, log d was higher than when short reinforcer durations (1.0 s) were signaled, F(1, 4) = 27.12, p < .01. Although the effect appeared to be stronger at short delays, F(3, 24) = 8.15, p < .001, this was due to floor effects. Whereas mean log d dropped as the duration of the retention interval increased for long-reinforcer trials, F(3, 24) = 23.73, p < .001, it could not do so for short-reinforcer trials because discriminability was already around zero for all delays longer than 1 s. Although these floor effects distort the shape of the forgetting function, they nonetheless indicate keen discrimination between long- and short-reinforcer trials, and hence strong stimulus control by the signal.

Experiment 2 therefore demonstrated that, like the signaled probability effect in Experiment 1, the signaled magnitude effect can be generated when the reinforcer signal is presented only after the sample has been presented. As such, the observed effect could not be due to differential attention to the sample stimulus. A floor effect for short reinforcer trials made it difficult to assess whether the signaled magnitude changed the rate of forgetting. Given that the floor effect caused the forgetting function to become flatter rather than steeper, and given the weight of previous signaled magnitude data (Jones et al., 1995; McCarthy & Voss, 1995; Nevin & Grosch, 1990), we conclude that the underlying process generates a change in the overall level of the forgetting function, not its slope.

General Discussion

Experiment 1 confirmed that a signaled reinforcer probability effect can be generated in delayed matching to sample, as suggested by Nevin et al. (2003). This effect is similar in form to the previously-described signaled magnitude effect, which was demonstrated in Experiment 2. Specifically, overall more accurate performance at all retention intervals results when larger or more probable reinforcers for correct matching are signaled. Experiments 1 and 2 showed that these effects can be generated when the reinforcer signal is presented after the sample has been presented. We therefore conclude that the observed signaled reinforcer effects were not due to differential attention to the sample.

The results are consistent with the view of remembering described by White (2002). In particular, remembering is discrimination at the time of retrieval. Remembering not only entails discrimination of sample and choice stimuli, but also is influenced by reinforcement contingencies. Because the reinforcement contingencies do not change with increasing retention interval duration, they should have the same effect at all delays. Indeed, an analogous effect exists in simpler discrimination tasks where there is no retention interval. For example, Nevin (1974) varied reinforcer frequency and reinforcer duration between components of a multiple schedule. He found that when larger or more frequent reinforcers were signaled, response rate increased. Response rate effectively measures the tendency to emit the reinforced operant (key pecking) and excludes alternative behaviors (nonpecking). Discriminability indices such as log d do the same in DMTS. They measure the ratio of the reinforced operant (matching) to its alternative behavior (nonmatching). The signaled magnitude and probability effects might therefore arise solely through the influence of reinforcement contingencies at the time of retrieval, just as response rates in multiple schedules are influenced by reinforcer frequency and magnitude. This notion is supported by Nevin et al.'s (2003) experiment in which responding in a two-component multiple schedule was reinforced by access to DMTS trials. The variable-interval schedule was the same, but the probability of reinforcement for correct matching differed, between components. Nevin et al. showed that response rates in each component predicted DMTS accuracy in that component.

It might be argued that the low accuracy generated by signaling a low magnitude or probability of reinforcement was the result of disruption by the signaling stimulus. This is not likely, however. First, although they were discriminable by shape, the cross and vertical-line signals were a similar color and brightness. Second, during early training, pigeons were indifferent to high- and low-probability or magnitude signals. Owing to their extensive prior training in a standard DMTS procedure with delays ranging from 1 s to 12 s, the main effect of interpolating signals in the retention interval during preliminary training was a gradual reduction in accuracy on trials where the low reinforcer probability was signaled. This suggests that any effect of the signal must be based not on its stimulus properties, but on its behavioral significance. Furthermore, in a related procedure that followed Experiment 2, briefly reversing the signals began to reverse their effects. Thus the effect of the signal appears to be due to its ability to predict reinforcement, not due to any disruptive properties.

An alternative cause of the signaled probability and magnitude effects is that signaled reinforcement could affect disruptive or facilitatory behaviors during the retention interval. When disruptive events such as food presentation (Jans & Catania, 1980) or houselight illumination (Harper & White, 1997) occur throughout the retention interval, accuracy declines. Such disruptions, however, tend to have little effect at short retention intervals, and a much larger effect at long retention intervals. This results in steeper forgetting functions, rather than functions that are overall lower. According to Harper and White, this is because the effect accumulates as the retention interval lengthens. It is only when the disruption lasts for the same amount of time in each retention interval that a similar drop in performance occurs at all delays. Because it is unlikely that signaled reinforcement would affect a constant duration of disruptive or facilitatory behaviors, we conclude that it is more likely that signaled reinforcement exerts its influence on processes that are independent of the duration of the retention interval.

Nevin and Grosch (1990) and Jones et al. (1995) demonstrated that the rate of responding to the sample was higher when large reinforcers were signaled. They suggested that this implies increased attention to the sample, and thus accounts for the signaled magnitude effects that they observed. The same account does not apply to the present experiments, however, because the signal was unpredictable at the time that the sample was presented.

Another plausible account is that attention to the task as a whole (that is, attention to the task of remembering, on a trial-by-trial basis) increased when reinforcers were larger or more probable. This includes attention to the choice (but note that the signal was turned off during the choice in Experiment 2), and to facilitatory or mediating behaviors during the retention interval. In standard signaled magnitude and probability experiments, but not in the present experiments, attention to the task as a whole might also include attention to the sample. Such an interpretation is consistent with Nevin and Grosch's (1990) and Jones et al.'s (1995), but extends beyond the sample to the entire task of remembering within a trial, and particularly to the choice.

Attention to the task can be understood in terms of the choice between DMTS behaviors and behaviors maintained by reinforcers extraneous to the task. This choice is specific to trials with low versus high reinforcer probabilities in the same way as it is in components of a multiple schedule. Attention to the task accounts for the signaled probability and magnitude effects in that stimulus control of the behaviors contributing to delayed matching is stronger on trials with higher reinforcer probability, compared to stimulus control of behaviors maintained by extraneous reinforcement. Increased attention to the task is also consistent with Brown's (2003) finding that choice latency decreases, and trial-completion rate increases, when reinforcers for correct matching are overall larger or more probable. An effect on processes other than attention to the sample was also suggested by Brown's experiment in which the sample was the same on every single trial throughout the entire experiment, and was therefore immaterial. In spite of the sample's irrelevance, overall reinforcer probability and duration still affected accuracy. Collectively, these findings indicate that attention to the whole task is a function of the reinforcement value for task performance. We therefore suggest that the signaled magnitude and probability effects are driven by the influence of the reinforcer signal on attention to the task as a whole, not simply on attention to the sample.