Relative reward effects on operant behavior: Incentive contrast, induction and variety effects

Abstract

Comparing different rewards automatically produces dynamic relative outcome effects on behavior. Each new outcome exposure is to an updated version evaluated relative to alternatives. Relative reward effects include incentive contrast, positive induction and variety effects. The present study utilized a novel behavioral design to examine relative reward effects on a chain of operant behavior using auditory cues. Incentive contrast is the most often examined effect and focuses on increases or decreases in behavioral performance after value upshifts (positive) or downshifts (negative) relative to another outcome. We examined the impact of comparing two reward outcomes in a repeated measures design with three sessions: a single outcome and a mixed outcome and a final single outcome session. Relative reward effects should be apparent when comparing trials for the identical outcome between the single and mixed session types. An auditory cue triggered a series of operant responses (nosepoke-leverpress-food retrieval), and we measured possible contrast effects for different reward magnitude combinations. We found positive contrast for trials with the greatest magnitude differential but positive induction or variety effects in other combinations. This behavioral task could be useful for analyzing environmental or neurobiological factors involved in reward comparisons, decision-making and choice during instrumental, goal-directed action.

1. Introduction

A key property of motivated behavior is flexibility in the production of goal-directed actions (Toates, 1986; Dunham, 1968; Premack, 1959). Internal and external factors combine to influence flexible responding to changes in reward value (Grigson et al., 1994 ;Cromwell et al., 2005; Papini and Pelligrini, 2006; Liang et al., 2012). Incentive contrast paradigms have been utilized extensively to examine factors involved in the dynamic processes of choice, preference and decision-making (Reynolds, 1961; Eisenberger et al., 1975; Blough, 1980; Flaherty, 1996). One of the initial explorations of contrast used the straight alleyway to explore the depressed (negative contrast) or elated (positive contrast) reactions by rats as they experienced reward magnitude downshifts or upshifts (Crespi, 1942). This ‘Crespi Effect’ relied on a between group comparison with controls not experiencing an alteration of outcome value. Recent work on incentive contrast has found the effect to be more complicated than originally proposed (Binkley et al., 2014; Nyland et al., 2012; Ortega et al., 2011; King et al., 2002). Based on the complex but crucial nature of relative reward processing, it is vital that new paradigms are developed and explored to clarify interactions and overcome obstacles in understanding factors involved in incentive contrast as well as similar effects on behavior (Webber et al., 2011; Cromwell, 2010; Cromwell et al., 2005; Watanabe et al., 2001).

One obstacle toward understanding incentive contrast has been the inconsistency in obtaining the effects using traditional measures of instrumental or operant behavior (Shanab et al., 1969; Campbell et al., 1970; Mellgren, 1972; Seybet and Mellgren, 1972). Contrast effects are more consistent when measuring consummatory actions (Dachowski and Brazier, 1991; Daniel et al., 2008), and the majority of recent work on diverse external influences or biological mechanisms has focused on licking rate or food consumption (Flaherty and Rowan, 1986; Liang et al., 2012; Lopez Seal et al., 2013) To complicate matters more, research findings support the idea that instrumental and consummatory contrast can be dissociated from one another and may rely on separate mechanisms (Flaherty and Caprio., 1976). This makes a translation between work on consummatory (e.g., licking rate) and instrumental (e.g., running speeds) contrast difficult and presents an interesting challenge for designing a paradigm that can measure both phases of behavior accurately and repeatedly.

To possibly overcome these inconsistencies, recent work has investigated instrumental contrast using more ethological behavioral measures such as foraging or general exploration (Pecoraro et al., 1999; Farmer-Dougan and Dougan, 2005). Other work has used operant responses to investigate incentive contrast (Mitchell et al., 2012; Williams, 2002; Beninger and Kendall, 1975); however, the majority of these ‘behavioral contrast’ studies focus on changes in responding when schedules of reinforcement shift (Williams, 1983) instead of after outcome value shifting. An example of this behavioral contrast work: an experimental group is exposed to intermingled sessions of fixed ratio (FR5) responding and extinction and then compared to a control group exposed solely to the FR schedule. How does exposure to the extinction session impact the responding during the rewarded block of trials? The results include a positive contrast effect of prolonged and higher rates of responding in the animals with the mixed experiences (conjoining the reward-extinction blocks; Williams, 2002).

The work on behavioral contrast has been very productive by illuminating critical factors involved in the dynamic properties of operant responding. Despite this progress, the work neglects the influence of shifting outcome value on operant conditioning. Recent work has shown that operant responding can be proportional to the magnitude of reward (Pellegrini and Papini, 2007; Pellegrini et al., 2008). This work on scaling used rates of operant leverpressing between exposure to training outcome (preshift) and testing outcome (postshift). These studies intentionally chose the operant leverpress because it is notoriously difficult to obtain incentive contrast when examining the combination of instrumental and anticipatory responding (Spear, 1965; Weatherly et al., 2005).

Another complicating issue is the existence of different forms of ‘contrast’ that can involve generalization among outcomes and relative effects that are opposite to traditional contrast effects (Weatherly et al., 2001; King et al., 2002; Lupfer-Johnson et al., 2010). For example, positive induction has been proposed as an opponent effect to anticipatory negative contrast (Weatherly et al., 2004). In this case, instead of a diminished response to an outcome because of an anticipated reward outcome upshift as seen in anticipatory negative contrast, a higher level of responding to the lower valued outcome occurs prior to access to the higher valued outcome. Positive induction can resemble secondary reinforcement or higher-order conditioning influencing high rates of responding prior to anticipated future behavior or outcomes (Premack, 1962). Another possible relative effect occurs as a variety effect on behavior and food consumption (Rolls et al., 1981; Lupfer-Johnson et al., 2010). Variety as a higher-order factor can increase food intake in a robust fashion and enhance motivated action within a session to overcome satiety effects (Bouton et al., 2013). There has been little work examining when these different relative reward effects will occur and how to study them in both a complete and rapid manner.

The present study investigated the expression of relative reward processing on a chain of operant responses triggered by a discrete cue that signals upcoming reward magnitude. It offers a rapid technique to test different outcome comparisons using a within subjects design. It extends previous work by others on incentive contrast (Flaherty and Becker, 1982), positive induction (Weatherly et al., 2005) and variety (Melville et al., 1997). The design mimics those used in psychopharmacology and behavioral neuroscience to examine addiction, craving and neural basis of reward (Samson et al.,, 2000; Everett and Robbins, 2005). A series of instrumental actions with the implementation of a cue allows an analysis of reward-proximity on relative reward processing such as incentive contrast. Cues enable the ability to explore expectancies and the relationship between conditioned stimuli and expectations on motivation and learning (Holden and Overmier 2014). Moreover, the design uses the measure of response latency as a novel, possible dependent measure for incentive contrast, and the latencies were obtained from a chain of instrumental responses. The rationale for using latencies arises from the need for a way to measure neural correlates of incentive contrast (Cromwell et al., 2005). In order to do this effectively, we need a precise time-locked indicator of contrast and a single latency measure would be useful in this way. These responses that include the nosepoke, leverpress and the food cup retrieval are oft-used as measures in conditioning experiments on motivation (Dickinson and Balleine, 1994; Corbit and Balleine, 2003).

Finally, we have expertise in recording and monitoring rodent ultrasounds and experimental work has found that rats emit these calls during appetitive responding including seeking behavior and operant responding (Knutson et al., 2002; Webber et al., 2012). Contrast effects have been proposed to be influenced significantly by emotional state (Flaherty et al., 1985; Flaherty and Rowan, 1989; Flaherty et al., 1994). One example is work that shows rats emit 50 kHz USVs during self-administration of drugs of abuse (Knutson et al., 1999) or electrical stimulation of the brain (Burgdorf et al., 2000). Interestingly, incentive contrast effects for ESB as an outcome are very potent (Trowill et al., 1969). We showed in earlier work that USV emission rates are precariously related to incentive contrast effects on running speed in a runway (Binkley et al., 2014). USVs did not follow the behavioral changes of incentive contrast unless a discrete cue was used to signal the upcoming reward value and even then, the relationship was inconsistent across testing days. The current work adds to these findings in terms of clarifying the potential role for emotion as an influence on motivation and incentive contrast. Overall the results of this study could foster investigation of relative reward effects in areas of learning, motivation, psychopharmacology and behavioral neuroscience (Grigson et al., 1993; Cromwell et al., 2005; Webber et al., 2011).

2. Material and Methods

2.1 Animals

Sprague-Dawley male rats (Rattus norvegicus; 4–9 months old; n=10 ) were used in the experiment. Animals were housed individually (65 × 24 × 15 cm cages) and weighed between 200–250 grams at the beginning of testing. All animal general husbandry procedures were carried out by the Animal Facilities Office at Bowling Green State University. Animals were quarantined for 3 days before regular housing procedures and were habituated to the colony room for one week after the quarantine period. The colony room was on an automatic 12:12 hour light: dark cycle beginning at 8 AM. The temperature of the room was 22º Celsius with 40%–50% humidity. All procedures were approved by the Institutional Animal Care and Use Committee at Bowling Green State University (protocol# 10-015).

2.2 Food restriction

Animals were given ad libitum access to food (Harlan Teklad Rat Chow #8604) and tap water during the quarantine period, colony room acclimation period and any days that the animals did not undergo testing. Animals were food deprived and weights obtained the following day served as target for levels of food deprivation and weight loss (87–90% of this baseline weight). Animals were given 5–15 grams of food each day after testing to maintain the restricted weight. This was done to achieve consistent behavioral performance over multiple testing days.

2.3 Equipment

Four standard operant chambers were used in this experiment (Med Associates Inc., VT, 31 × 31 × 25 cm). The food cups were at the center of the wall adjacent to the wall that contained the lever and nosepoke apparatus. A pellet dispenser was secured to the same platform immediately behind the recessed food cup. The dispenser dispensed 45 mg chocolate flavored pellets for training and testing (Bio-Serve Product #F05984, NJ). The operant chamber was connected to a computer that used the software MED-PC (Med. Associates Inc. VT) to run custom written programs that operated the boxes and collected the data. An ultrasound detector was placed on the roof of the operant box to record ultrasonic vocalizations (Petterssen D980 ultrasonic detector, Uppsala, Sweden).

2.4 Behavioral training

Rats were trained to lever press for three chocolate-flavored pellets on a fixed ratio schedule of reinforcement (FR-1). After the rats pressed 30 times per 30 minute session, they advanced to nosepoke training. Animals were trained to enter and hold a nosepoke response (500 ms) in order to gain access to the lever. After the animals reached 30 nosepokes/leverpresses per 30 minute session, then tone training began. Animals were exposed to 4 tone training sessions, and during the sessions the animals were presented with 30 trials of a single tone/reward combinations. There was a variable intertrial interval that lasted from 15–45 seconds (χ́= 30 s). Three tones were paired with three different levels of rewards. For example, some animals experienced the low tone (2 kHz) paired with the availability to press the lever for 1 chocolate pellet (Small reward), the medium tone (4 kHz) with 2 chocolate pellets (Medium reward) and the high tone (6 kHz) with 4 chocolate pellets (Large reward). The first day that animals experienced the associations was considered an acclimation session. During the last 3 sessions animals were exposed to each of the 3 tone/reward combinations. Tone/reward combinations and the order of tone/reward combination training were randomized. The animals reached the training criterion when they demonstrated significant differences in latencies in response to the tone-reward magnitude pairings. Specifically, animals had to produce significantly longer latencies to respond in at least 2 out of the 3 behavioral measures during smaller magnitude outcomes when compared with larger or medium magnitude outcomes.

2.5 Behavioral testing

During behavioral testing animals were exposed to different blocks of reward outcomes across 8 days of testing. Each session consisted of 3 blocks of trials, 2 single outcome blocks bracketing a mixed outcome block. In the single outcome blocks, only one tone/reward outcome was presented. In the mixed outcome blocks, two tone/reward outcomes were presented. In each block, the animals were exposed to 20 trials of each reward trial type (6 session types; see Table 1). The initial single outcome block contained one reward type (Small, Medium or Large) and the mixed outcome block contained two reward types (e.g. Small and Large). The goal of this design was to examine whether an available alternative reward type in the mixed outcome block would alter responding relative to the initial single outcome block. This was achieved by comparing latencies to respond to the same reward type in the mixed outcome block with the initial single outcome block. In positive contrast sessions the larger magnitude outcomes were presented with smaller magnitude outcomes in the mixed outcome block and compared with the presentation of the larger magnitude outcome alone in the initial single outcome blocks. During negative contrast sessions smaller rewards were presented with larger rewards in the mixed outcome block and compared with the smaller reward magnitude trial set. A single outcome block was repeated to examine possible satiation effects. To signal the beginning of a ‘mixed’ outcome session, the initial three trials in the mixed outcome block included the alternate outcome. There was a 45 minute break between each trial block. Sessions which combined the two largest outcome options (2 and 4 pellets) were conducted with only 10 trials for each outcome in each block, and repeated. The data for each session was combined for analysis. This was completed to limit satiation effects during exposure to relatively higher amount of pellets.

Table 1.

This table shows the trial types that were presented in each block for each session type. Sessions are grouped by contrast type. Each animal underwent testing in each session type. Session order was presented in a pseudo-random fashion.

Contrast Type	Session Type	Block 1 Single Outcome	Block 2 Mixed Outcome	Block 3 Single Outcome
Positive Contrast	Large-small	Large	Small & Large	Large
	Medium-small	Medium	Small & Medium	Medium
	Large-small	Large	Medium & Large	Large
Negative Contrast	Small-large	Small	Large & Small	Small
	Small-medium	Small	Medium & Small	Small
	Medium-large	Medium	Large & Medium	Medium

2.6 Behavioral and Ultrasonic Vocalization Analysis

Behavioral latency data were acquired using MED-PC (Med. Associates Inc., VT, USA). Custom-written programs recorded latencies to nosepoke, leverpress and retrieve the food pellet. Latencies were averaged for each animal and grouped according to session type. Nonparametric tests were used because the data violated the assumption of normal distribution. Friedman tests were used to gain omnibus results including all magnitude combinations. Wilcoxon signed-ranked tests were used to make comparisons within specific magnitude combination session types. Data analysis was conducted using PASW Statistics (version 18, SPSS Inc., Hong Kong). Graphs were made using SigmaPlot (Version 12, SPSS Inc., Hong Kong) and R (The R Project for Statistical Computing; http://www.r-project.org). Effect sizes for non-parametric tests were calculated (PS_Dep) for each significant Wilcoxon sign-ranked test. Briefly, the number of positive difference scores (increases in latency) or negative difference scores (decreases in latency) where divided by the number of total ranks (Grissom & Kim, 2012). This gave a score between 0 and 1. A score of 1 represented a unanimous rank winning while a score of 0 represented a unanimous rank defeat. Finally, medians and confidence intervals were calculated for each block and trial type. These are outlined in tables 3, 5 and 7. Upper and lower ranked values were calculated using a procedure designed for calculating confidence intervals in medians rather than means (Conover, 1980).

Table 3.

Medians and confidence intervals for the Large comparison sessions.

	Large-small				Large-medium
	Block 1 Large	Block 2 Small	Block 2 Large	Block 3 Large	Block 1 Large	Block 2 Medium	Block 2 Large	Block 3 Large
Nose Poke
Lower	1.5445	0.1975	0.1935	2.0425	2.315	0.1645	0.2335	1.165
Median	2.33725	0.2915	0.329	2.53325	3.43725	0.264	0.3155	2.5125
Upper	5.172	0.406	0.513	6.5915	4.157	0.4705	0.41	4.7265
Lever Press
Lower	1.345	1.417	1.5625	1.487	1.346	1.5165	1.5405	1.602
Median	1.60775	1.93475	1.8	2.05375	1.5125	1.938	1.85525	1.838
Upper	1.773	5.3135	2.1945	2.4465	1.932	2.501	2.2555	2.959
Food Cup Entry
Lower	0.6485	0.634	0.6525	0.653	0.625	0.6365	0.76	0.634
Median	0.717	0.75475	0.69375	0.693	0.7595	0.808	0.782	0.734
Upper	0.8365	0.8625	0.7585	0.8135	0.7705	0.8445	0.825	0.797

Table 5.

Medians and confidence intervals for the Medium comparison sessions.

	Medium-small				Medium-large
	Block 1 Medium	Block 2 Small	Block 2 Medium	Block 3 Medium	Block 1 Medium	Block 2 Medium	Block 2 Large	Block 3 Medium
Nose Poke
Lower	1.7825	0.262	0.2425	1.638	2.539	0.1825	0.1685	1.8635
Median	2.825	0.3705	0.38725	2.74725	3.684	0.3845	0.31075	2.49875
Upper	4.7825	0.7625	0.4785	5.3345	5.4155	0.5875	0.4675	3.899
Lever Press
Lower	1.196	1.1955	1.494	1.1415	1.421	1.567	1.662	1.7305
Median	1.6575	2.00775	2.80125	2.1125	1.58675	2.159	2.14875	2.00775
Upper	2.3895	2.784	3.685	3.631	2.2515	2.2715	2.2355	2.2025
Food Cup Entry
Lower	0.6675	0.626	0.696	0.5915	0.676	0.7245	0.6815	0.6325
Median	0.802	0.77925	0.779	0.74725	0.82725	0.79575	0.80275	0.8245
Upper	0.8925	1.002	1.009	0.955	0.882	0.8615	0.864	0.8935

Table 7.

Medians and confidence intervals for the Small comparison sessions.

	Small-large				Small-medium
	Block 1 Small	Block 2 Small	Block 2 Large	Block 3 Small	Block 1 Small	Block 2 Small	Block 2 Medium	Block 3 Small
Nose Poke
Lower	2.8855	0.195	0.2235	4.252	2.3455	0.221	0.2095	1.8985
Median	3.19875	0.27425	0.3085	5.34825	3.186	0.313	0.32575	2.91375
Upper	7.18	0.659	0.357	11.8465	4.9145	0.5305	0.545	4.2755
Lever Press
Lower	1.179	1.4185	1.4035	1.7615	1.1805	1.37	1.319	1.309
Median	1.56775	2.1065	1.745	2.31075	1.574	2.36575	2.41075	1.918
Upper	2.016	3.4105	2.349	3.1685	2.2715	2.903	3.1435	3.2965
Food Cup Entry
Lower	0.745	0.6705	0.6655	0.7075	0.7385	0.61	0.6415	0.666
Median	0.83375	0.77825	0.736	0.84625	0.93075	0.73575	0.773	0.81325
Upper	1.0235	1.0485	0.763	1.2865	1.0645	0.9785	0.8995	0.98

Lower Ranked Value:

$$\frac{n}{2}-\frac{1.96\surd n}{2}$$

Upper Ranked Value:

$$1+\frac{n}{2}+\frac{1.96\surd n}{2}$$

Ultrasound vocalization data used for analysis in this study were. WAV files obtained from a digital recorder (Marantz PMD660 Digital Solid State Field Recorder) connected by audio cables directly to the output channels of the BAT detector (D-230 Detector, Pettersson Elektronik AB Sweden). The recorder was placed on top of the operant box over openings made in the Plexiglas cover. USV scoring was completed using spectrographic display of the calls and counted manually (Avisoft Bioacoustics, Berlin, Germany). Detailed spectrographic parameters used for analysis included a 48 kHz sampling rate, 256 FFT length, 2.8 s analytical window and 100% Frame settings with 50% overlap between analytical windows for each file.

3. Results

3.1 Outcome magnitude discrimination

We compared the response latencies for the three behavioral measures using data taken from the single outcome sessions during behavioral testing sessions (1 vs. 2; 2 vs 4 and 1 vs. 4 food pellet trials; see Figure 1A– D) to explore the discrimination abilities for magnitude differences during operant responding. We found significant differences for the food cup entry response latency for 1 vs. 4 pellet trial blocks (Z = −2.8, p = .017; see Figure 1C–D) and a trend for the 2 vs. 4 food pellet sessions (Z = −1.866, p = .059). No other response (nosepoke or leverpress response) was significantly different between these single session trials. Food cup entries were most likely sensitive to outcome discrimination because of close proximity to the reward delivery.

Figure 1. Operant behavior and reward discrimination.

Average response latencies during the initial SOB for each behavioral measure. A. Nose poke latency: no significant effects. B. Lever press latency: no significant effects. C. Food cup entry latency: results showed significantly shorter latencies in bigger when compared to smaller reward trial types (* p < .01). There was a marginal reduction in food cup entry latency in Medium versus small reward trials. D. Combined behavioral measures: this graph shows response latencies for all three measures on the same plot.

3.2 Large reward outcome comparisons

When the large reward outcome is initially presented in the single trial block, subsequent mixed trials with either medium-sized or small-sized outcome should produce positive contrast. This should be measured by faster, decreased response latencies along the instrumental chain of action. We found positive contrast effects for the initial operant of nosepoking. During the Large-small combination set of trials the nosepoke latency was significantly different among trials (X²(3) = 18.4, p < 0.001; Figure 2A and B; See tables 2–3). We found a significantly shorter latency during the mixed session for the larger magnitude trials compared to the identical trials in the initial single outcome session (Z= −2.8, p = 0.005; PS_Dep = 1). A similar result was found for the large magnitude outcome when coupled with the medium-sized outcome (X²(3) = 24, p < 0.001; see tables 2–3). Animals expressed a significantly shorter latency to nosepoke after the tone stimulus in the large magnitude trials during the mixed (large-medium) compared to the single (large only) trial set (Z= −2.8, p = 0.005; PS_Dep = 1). For the Large-medium combination, the medium-sized reward was also significantly faster compared to the large magnitude outcome in the single outcome session (Z= −2.8, p = 0.005; PS_Dep = 1; Figures 2C and D) and not significantly different from the large magnitude trial set in the mixed session. This latter finding suggests that reward discrimination between outcomes during the mixed block is not essential for the expression of positive contrast. For the leverpress action that followed the nosepoke we found a significant alteration in responses among the sessions (X²(3) = 13.1, p = 0.004; see tables 2–3), but not in the direction predicted for positive contrast. We found a longer or slower latency to leverpress during the large reward trials in the mixed block compared to the same trials in the initial single block (Z= −2.19, p = 0.028; PS_Dep = 0.9). The slowing of response continued across the session with a significantly longer latency observed for large magnitude trials in the final single block compared to the same trials in the initial single outcome block. No significant differences were found for the final action of food cup retrieval among the different session blocks or between trials for the identical outcome.

Figure 2. Large reward as the comparison outcome.

Large magnitude comparisons. A. Group data showing the three different responses including nosepoke, leverpress and food cup retrieval. The abscissa provides the different session blocks with the initial single outcome block (SOB) followed by the mixed outcome block (MOB) and the final single outcome block (SOB). The nosepoke response was significantly faster in the MOB compared to either the initial or final SOB (# p < .01). B. An individual example of this positive contrast effect for the nosepoke response. C. Comparing the large outcome to the medium-sized outcome, we found a decrease in nosepoke latency and an increase for lever press for both outcomes during the MOB (# p < .01; Ϯ p < .05). D. Displays responses from an individual animal during the series for the nosepoke. This result represents positive contrast but without discrimination between the outcomes suggesting a possible variety effect (# p < .01)

Table 2.

This table displays individual animal contributions to various effects in Large comparison sessions. The Positive Contrast column displays the number of animals that produced a positive contrast for all three behavioral measures. The Variety column displays the number of animals that produced a variety effect. The Satiety column displays the number of animals that produce a satiation effect. The No Effect column displays the number of animals that did not show one of the 3 predominant behavioral effects. The Group Effect column designates which behavioral phenotype produced a statistically significant group effect.

Session Type	Behavior	Positive Contrast	Variety	Satiety	No Effect	Group Effect
Large-small	Nose Poke	2/10	8/10	0/10	0/10	Positive Contrast
	Lever Press	0/10	0/10	4/10	6/10	X
	Food Cup Entry	0/10	1/10	5/10	4/10	X
Large-medium	Nose Poke	0/10	10/10	0/10	0/10	Variety
	Lever Press	0/10	0/10	6/10	4/10	Satiety
	Food Cup Entry	0/10	0/10	2/10	8/10	X

3.3 Medium-sized reward outcome comparisons

The medium-sized outcome provides opportunities to obtain both positive and negative contrast effects. Positive contrast is predicted for the mixed session combined with the small outcome and negative contrast for the session with the large reward outcome. Positive contrast for the initial nosepoke response was obtained when utilizing the Medium-small combination following the medium single outcome trial set (X²(3) = 24.1, p < 0.001; see figures 3A and B; see tables 4–5). The nosepoke latency was significantly shorter for the medium-sized outcomes in the mixed session compared to the identical trial types in the single session Z = −2.8, p = 0.005; PS_Dep = 1). We found no significant differences between the medium and small reward trials during the mixed outcome. In addition, the small reward trials in the mixed session were also significantly faster compared to the medium-sized reward trials in the initial single outcome session (Z = −2.8, p =.005; PS_Dep = 1). These same behavioral effects were seen during the Medium-large comparison nose poke response (X²(3) = 21.9, p < .001; see figures 3C and D; see tables 4–5). The nose poke latency was significantly shorter for the medium sized outcomes during the mixed session when compared to the same trial types in the single session (Z = −2.8, p = .005; PS_Dep = 1). This was also true of large reward trial types in the mixed when compared to medium reward trial types in the single outcome session (Z = −2.8, p = .005; PS_Dep = 1).

Figure 3. Medium reward as the comparison outcome.

Medium magnitude comparisons. A. Group data showing the three different responses, including latency to nosepoke, lever press and food cup retrieval. Results showed significant decreases in the latencies to nosepoke during both the small and medium reward trial types in the MOB (# p < .01). B. An individual example of this effect found in the nosepoke measure of a single animal (p < .01). C. Group data showing the three different response measures in the Medium-large comparison sessions. Animals again showed significantly faster nosepoke responses during the MOB versus the two SOBs (# p < .01). D. An example of this nose poke latency effect found in an individual animal’s data (# p < .01).

Table 4.

This table displays individual animal contributions to various effects in Medium comparison sessions. The Positive/Negative Contrast column displays the number of animals that produced a positive contrast (Medium –small) or negative contrast (Medium-large) for all three behavioral measures. The Variety/Induction column displays the number of animals that produced a variety (Medium-small) or induction (Medium-large) effect. The Satiety column displays the number of animals that produce a satiation effect. The No Effect column displays the number of animals that did not show one of the 3 predominant behavioral effects. The Group Effect column designates which behavioral phenotype produced a statistically significant group effect.

Session Type	Behavior	Positive/Negative Contrast	Variety	Satiety	No Effect	Group Effect
Medium-small	Nose Poke	0/10	10/10	0/10	0/10	Variety
	Lever Press	0/10	0/10	4/10	6/10	X
	Food Cup Entry	0/10	1/10	3/10	6/10	X
Medium-large	Nose Poke	0/10	9/10	0/10	1/10	Variety
	Lever Press	0/10	1/10	5/10	4/10	X
	Food Cup Entry	0/10	2/10	7/10	1/10	X

3.4 Small reward outcome comparisons

When utilizing the small outcome, we expected that both mixed sessions (small-large and small-medium) would lead to negative contrast in the responses during the instrumental chain of action. In the Small-large comparison session (Figures 4A and B), there was a main effect of trial type on latency to nose poke (X²(3) = 24.1, p < .001; see tables 6–7 ). Animals produced significantly shorter latencies to nose poke during the small (Z = −2.8, p = .005; PS_Dep = 1) and large (Z = −2.8, p = .005; PS_Dep = 1) trials in the mixed session when compared with the small trials in the initial single outcome block. There was also a main effect of trial type on latency to lever press (X²(3) = 16.920, p < .001; see tables 6–7). Animals produced significantly longer latencies to lever press during the Small trials in the mixed session (Z = −2.803, p = .005; PS_Dep = 1), the Big trials in the mixed session (Z = −2.803, p = .005; PS_Dep = 1) and the Small trials in the final single outcome block (Z = −2.497, p = .013; PS_Dep = 0.9) when compared with the Small outcome trials in the initial session.

Figure 4. Small reward as the comparison outcome.

Small magnitude comparisons. A. Group data showing the three different responses, including latency to nosepoke, lever press and food cup retrieval. Results showed significant reductions in the latency to nosepoke during both trial types the MOB when compared to the SOB A(# p < .01). There was also an effect of trial type on lever press. Animals showed significantly longer latencies to lever press over the three trial blocks (Ϯ p < .05). B. An individual example of an animal that produced a significant negative contrast during food cup entry (* p < .05). C. Group data collected during the Small-medium comparison session. Results showed significant decreases in response latencies in the nosepoke measure during the MOB when compared with the SOB (# p < .01). D. An example of this nosepoke latency effect found in an individual animal (# p < .01).

Table 6.

This table displays individual animal contributions to various effects in Small comparison sessions. The Negative Contrast column displays the number of animals that produced a positive contrast for all three behavioral measures. The Induction column displays the number of animals that produced an induction effect. The Satiety column displays the number of animals that produce a satiation effect. The No Effect column displays the number of animals that did not show one of the 3 predominant behavioral effects. The Group Effect column designates which behavioral phenotype produced a statistically significant group effect.

Session Type	Behavior	Negative Contrast	Induction	Satiety	No Effect	Group Effect
Small-large	Nose Poke	0/10	10/10	0/10	0/10	Induction
	Lever Press	0/10	0/10	4/10	6/10	Satiation
	Food Cup Entry	1/10	2/10	1/10	6/10	X
Small-medium	Nose Poke	0/10	10/10	0/10	0/10	Induction
	Lever Press	1/10	1/10	4/10	4/10	X
	Food Cup Entry	1/10	6/10	0/10	3/10	X

When analyzing the nose poke measure in the Small-medium combination (see figures 4C and D; see tables 6–7), we found a significant difference among the sessions (X²(3) =21.9, p<0.001). When comparing between the trial blocks, a positive contrast was obtained for this response with the animals responding faster in the mixed outcome block for the small reward when compared to the identical trials during the single outcome block (Z=−2.8, p = 0.005; PS_Dep = 1). We also found that animals responded faster to the medium-sized outcome in the mixed session compared to small reward outcome in the initial trial block (Z=−2.8, p = 0.005; PS_Dep = 1).

Significant differences were obtained for the leverpress response among the session blocks when the Small-large sequence was compared (X²(3) = 16.9, p<0.01; see figures 4C and D; see tables 6–7). Longer latencies were obtained in the mixed session for the small outcome trials compared to the identical outcome trials during the initial single outcome session (Z=−2.8, p = 0.005; PS_Dep = 1). A similar change was observed between the two single outcome sessions. Longer latencies were obtained in the final single block for small reward compared to the initial block of small reward outcome (Z=−2.5, p = 0.013; PS_Dep = 0.9). Finally, we obtained a significant longer latency to respond for the large reward trials compared to the trials for the small reward outcome during the initial single outcome block (−2.8, p = 0.005; PS_Dep = 1; Figures 4A and 4B).

3.5 Individual Variability in Behavior

During the Large-reward outcome comparisons animals showed positive contrast, variety and satiation effects in the three behavioral measures (see Tables 2–3): During the Large-small session individual animal data revealed that 20% of animals showed a significant positive contrast effect (p < .05 for initial session versus same trial types in the middle mixed session). 80% of animals demonstrated a significant variety effect in nose poke latency (p < .05 for all between session comparisons). 40% of the animals showed a satiation effect in the lever press measure while 50% of the animals showed this same effect in food cup entry latencies (p < .05 for initial session versus final single outcome session). During the Large-medium session 100% of the animals produced a variety effect in nose poke latency (p < .05 for all comparisons). 60% of animals showed a satiation effect in lever press latencies while 20% of animals showed this same effect during food cup entry latency (p < .05 for initial session versus final single outcome session).

During the Medium-reward outcome comparisons animals showed induction/variety and satiation effects in the three behavioral measures (see Tables 4–5): During the Medium-small session individual animal data revealed that 100% of animals demonstrated a significant induction effect in nose poke latency (p < .05 for all single session vs. mixed outcome comparisons). 40% of the animals showed a satiation effect during the lever press while 30% of the animals showed this same effect in food cup entry latencies (p < .05 for initial session versus final single session). During the Medium-large session 90% of the animals produced a variety effect in nose poke latency, while 10% of animals showed the same effect during lever press and 20% with food cup entry latencies (p < .05 for all comparisons). 50% of animals showed a satiation effect during lever press while 70% of animals showed this same effect in food cup entry latency (p < .05 for initial single versus final single session).

During the Small-reward outcome comparisons animals showed negative contrast, induction and satiation effects in the three behavioral measures (see Tables 6–7): During the Small-large session individual animal data revealed that 10% of animals showed a significant negative contrast effect in food cup entry latency (p < .05 for initial session versus same trial type in mixed session). 100% of animals demonstrated a significant induction effect in nose poke latency, while 20% of animals showed that same induction effect in food cup entry latencies (p < .05 for all single vs. mixed comparisons). 40% of the animals showed a satiation effect in the lever press while 10% of the animals showed this same effect in food cup entry latencies (p < .05 for initial versus final session). During the Small-medium session 10% of animals showed a significant negative contrast in lever press and food cup entry latencies (p < .05 for initial versus same trial type in middle session). 100% of the animals produced an induction effect in nose poke latency, while 10% of animals showed this effect in lever press latencies and 60% in food cup entry latencies (p < .05 for all single vs. mixed session comparisons). Satiety effects were found in 40% of animals during lever press measure (p < .05 for initial versus final sessions).

3.6 Ultrasonic vocalizations

Ultrasonic vocalizations were obtained during the sessions. Average number of USVs was relatively lower compared to previous work using other forms of outcomes (i.e., electrical stimulation of the brain). There was no main effect of block type on USV emission in any session (p > .05). Data from positive contrast session types were merged, and data from negative contrast groups were merged in order to examine any positive or negative contrast effects in USV emission. During negative contrast sessions there was a main effect of block on USV emission (X²(2) = 11.474, p = .003). Animals emitted more 50 kHz USVs during the initial single outcome block when compared with the mixed (Z = −2.333, p = .020; PS_Dep = 0.31) and final single (Z = −2.271, p = .023; PS_Dep = 0.31) outcome blocks (Figure 5A). There was no main effect of trial block on USV emission during positive contrast sessions (p > .05) (Figure 5B).

Figure 5. Ultrasounds during reward comparison.

USV emission during positive and negative contrast sessions. A. Negative contrast session: USV emission was significantly lower during the mixed and final single outcome block when compared with the initial single outcome block (p < .05 ☆) B. Positive Contrast session: there were no significant differences.

4. Discussion

To summarize the results, a repeated measures operant response design that used a series of sessions (single-mixed-single) to enable reward comparisons did obtain diverse forms of relative reward effects on behavior. First, incentive contrast was obtained primarily in the positive direction in sessions with the greatest outcome differential and on the initial action of nosepoking that immediately followed the auditory cue. Second, a similar degree of change in responding during the mixed sessions for the different outcomes was found in the conditions designed to elicit both positive and negative contrast. In the case in which negative contrast is expected, the finding resembles positive induction as generalizing of the positive reward value from one outcome to another. In the cases designed for positive contrast, the effect could be working as a new form of induction or as a type of variety effect. The analysis of individual variability of relative effects from this paradigm emphasizes the complex nature of comparing reward outcomes for animals in a trial with interspersed outcomes. The robust nature of the positive induction versus the weak effects and high variability for traditional contrast highlight possible different mechanisms involved in these diverse effects that can compete and outweigh one another. Future work could look into applications for this and related paradigms as a way to investigate internal and external factors involved in relative reward processing. Finally, the results point to a weak effect between relative reward comparisons and rodent USVs. Ultrasounds indicated contrast only when the set of negative contrast sessions were combined. This USV result was not coupled by a significant behavioral result among sessions.

Reasons for obtaining different forms and intensities of relative reward effects rely on procedural design (Weatherly et al., 2005) and include the timing and predictability of the outcome experiences. Timing and predictability were influenced by the insertion of a variable interval delay (χ́ = 30 s) between trials in the present study. This may have led to the greater positive not negative relative reward effects on the different responses. Previous work measuring appetitive action in contrast paradigms typically obtains reliable negative not positive contrast (Gonzalez et al., 1962; Dunham, 1968; Black, 1968). Ceiling effects on appetitive behavior such as running speed have been postulated to cause reduced or absent positive contrast (Shanab et al., 1969; Campbell et al., 1970). These ceiling effects can be abolished with the introduction of a delay in the appetitive task (Mellgren, 1972). Delays could influence contrast effects and work against ceiling effects on movement in numerous ways. For example, delays increase variability in motor parameters (Badler and Heinen, 2006), and enhance variability in motor preparation during anticipatory periods (Wike et al., 1967). Pre-motor neural responses during longer delays also show this increase in variability for firing rates and patterns of activity (Pare and Munoz, 1996). For studies on contrast, this variability could be expressed during elevated bouts of searching following incentive downshifts (Pecoraro et al., 1999). When anticipatory states are produced and animals must perform a series of actions to obtain the outcome, a similar increase in general activity or searching could explain enhanced positive contrast effects for well predicted incentive upshifts. Search behavior effects may be more potent on the appetitive acts compared to consummatory responses (Lopez-Seal et al., 2013).

It is very important to keep in mind that work on incentive contrast can obtain significantly different effects depending upon how contrast is defined. McSweeney and Norman (1979) provided a clear example of this using behavioral contrast. They provided two different ways to define contrast: 1) an interschedule and 2) an intraschedule definition. By these definitions, the present work examines contrast using the intraschedule way in which the comparisons are made against a baseline of responding when a component provides equal rward outcomes or rates of reinforcement. This intraschedule way of examining contrast has been the predominant way contrast with outcome shifts has been studied, but the interschedule way could provide novel insights into contrast as well. This definition focuses on the rate of responding between schedules. When the rate goes up, this reflects positive contrast and vice versa for negative contrast (Hamilton and Silberberg, 1978). The value of interschedule would be its appreciation of the dynamic process possibly independent of a reference point and shifting higher or lower over time depending upon recent experiences (Flaherty, 1996). These two different ways of exploring contrast need to be investigated in more depth possibly in relation to how reference points can be used (Marshall and Kirkpatrick, 2015) or how they may shift along with changes in reinforcement or reward value.

Cues can be another important influence on relative reward effects and could work via manipulating timing or prediction processes (Weatherly et al., 2003; Daniel et al., 2008). Actions can be influenced by changes in their proximity to cues (Rudy et al., 1987). In the present work, the nosepoke response was proximal to the auditory cue and was more sensitive to the relative reward effects. Without cues, activity proximal to the outcome (e.g. runway running speed) has been shown to be the most sensitive to outcome shifts (Mellgren, 1971; Leszczuk and Flaherty, 2000). The power of cues to guide influences of outcome properties most likely relies on the form, associative strength and embedded context of the cue (Meyer et al., 2014; Robinson et al., 2014; Robinson et al., 2014). Influence of cues also relies upon subject variables including homeostatic affective state and other intrinsic variables (Porter et al., 1968; Stidham et al., 1987). With appropriate parameters, even consummatory contrast can be strongly influenced by cues (Daniel et al., 2008). Without the correct parameters when studies involve extensive training or embed differential action expectancies, the power of cues can diminish (D’Amato et al., 1968: Williams and McDevitt, 2011; Holden and Overmier, 2014). Combining visual cues and the runway task actually led to a diminished negative contrast effect (Binkley et al., 2014). These cues led to significantly faster behavioral discrimination but reduced differences in running speeds when the outcome was downshifted. A possible reason for the reduced contrast effect on runway running speeds is that the visual cues were available throughout the complete trial period and predisposed slower running during the total runway behavior. This could disable differences on running speed that interacts with goal proximity. In this situation, it is essentially the time of cue exposure (continuous versus discrete) that could influence the outcome-related instrumental action. Future work will need to explore how the use of cues and manipulating their salience can impact relative reward processes.

When cues and outcome are more easily differentiated, contrast effects may be more consistent. In the present work, contrast rather than induction was observed when the pellet magnitude difference between the outcomes was the largest. It is been shown that animals ‘scale’ responses proportional to outcome differences (Watanabe et al., 2001; Pelligrini and Papini, 2008). It could be that the greater the difference between outcomes, the more likely robust discrimination can occur between cues and outcomes. This could be a key factor in the likelihood of contrast being emitted as an appetitive response. Small quantities of reward are related to significant differences in neural activations linked to cues, delay periods and instrumental action (Cromwell et al., 2003). Similar sensitivity is observed in neural behavior for different qualities or types of reward (Hassini et al., 2001). The present study provides evidence that differences in magnitude outcome are sufficient to produce relative reward effects on operant behavior. Previous work on operant responding varied the type or quality of outcome (e.g., sucrose solution versus food pellets) (Panksepp and Trowill, 1969; Weatherly et al., 1999). Negative and positive contrast were both reliably obtained at high levels on lever pressing rate when using brain stimulation as outcome (Panksepp and Trowill, 1969). It is interesting to note that in this study the rate of operant responding was directly proportional to the amount of high or low electric brain stimulation delivered to the animal. This contingency relationship between effort and outcome value is absent in other work on appetitive action and relative reward processing, but could play a key role in the relationship between approach and instrumental action and relative reward processing. Future work needs to explore how adding an instrumental criterion for different outcomes could influence these different relative outcome processes.

Previous work found that deprivation level and outcome location were key procedural details that determined contrast versus induction (Weatherly et al., 2005). In the present work, food restriction was kept constant in all subjects at a moderate level and outcomes were delivered to an identical food cup location. In general, these procedures favor induction and this form of result was the most often observed and statistically robust. Instead of negative contrast, an increase in the speed of responding for lower magnitude outcome that matched the response of the higher magnitude outcome was obtained. The result resembles positive induction because it includes an opposite direction in responding from predicted negative contrast. The result varies from previous work on induction because positive induction relies partially on anticipation for a future higher value outcome that is well predicted (Weatherly et al., 2001; Weatherly and Moulton, 2001; Weatherly et al., 2001), and in the present study, animals are unable to anticipate exact outcomes from one trial to another. Possibly, this dynamic exposure to alternating outcomes could lead to a form of positive induction that depends more upon outcome generalization than on trial-by-trial outcome anticipation (Meyer and Campbell, 1973). When positive contrast was expected (e.g., Medium-small mixed session) and similar responding to both outcomes occurred, the result could be influenced by the experience with the sequence of the sessions that creates a relative effect independent of the specific outcomes. This form of relative comparison is similar to results on operant behavior postulated to arise from variety of outcomes (Melville et al., 1997). The mixed reward session would be highly discriminable from single outcome sessions and this middle session could be elevated in value merely due to the increase in outcome variety. The study offers this as a possibility, and the finding is somewhat unique. Magnitude is not typically used to examine variety. Flavor, nutrient subtype or quality of food are often used and found to have significant positive effects on food intake and decreased habituation (Treit et al., 1983; Zylan et al., 1996; Rolls et al., 1983). This idea could be explored in more depth to investigate broader contextual variables that could act as units for comparison within a relative reward process.

Finally, we did explore connections between affective state and relative reward comparison by monitoring rodent USVs. The idea that affective state influences incentive contrast is entrenched in the field (Crespi, 1947) and remains a current interest (Mitchell et al., 2012). 50 kHz calls were found during the sessions, and rates of calling were decreased in the mixed sessions in which negative contrast was expected. Still, specific behavioral indicators of negative contrast were lacking in these same sessions. Dissociations between behavioral and USV indicators of contrast have been found (Binkley et al., 2014) supporting the idea that affective state changes occur but do not necessarily translate into similar behavioral profiles. USVs did not reflect positive contrast effects even though behavioral indicators of positive contrast were observed. Positive contrast effects on USV signals may require higher levels of incentive value or greater shifts in outcome valuation. 50 kHz USVs are very sensitive to social reward (Willey and Spear, 2013; Webber et al., 2012) and prior to receiving brain stimulation (Burgdorf et al., 2000). They are also emitted at high intensities when anticipating drug reward (Ahrens et al., 2009). The food pellet reward used along with moderate food restriction may not induce sufficient motivation to lead to outcome modulated USV signals.

Exploring reward comparison is a vital area of research in experimental psychology and behavioral neuroscience and animal models of behavioral pathology (Grigson, 2008). The modified approach of the present study using a chain of operant responses could enable investigation on interaction between external factors including conditioning and learning that play a role in contrast and other forms of reward comparison. It has equal worth in developing ways to explore intrinsic neurobiological mechanisms involved in reward comparison as a rapid, dynamic process. This design could be especially fruitful for investigating interactions between external and internal factors essential for reward processing and motivation.

Highlights.

• Relative reward effects can be diverse.

• Incentive contrast was found for operant responses by the rat.

• Other relative reward effects including induction were observed.

• The type of relative reward effect can depend on outcome differential.

• New paradigms offer in depth study of diverse relative reward effects.