Reward Contrast Effects on Impulsive Choice and Timing in Rats

Abstract

Despite considerable interest in impulsive choice as a predictor of a variety of maladaptive behaviors, the mechanisms that drive choice behavior are still poorly understood. The present study sought to examine the influence of one understudied variable, reward magnitude contrast, on choice and timing behavior as changes in magnitude commonly occur within choice procedures. In addition, assessments of indirect effects on choice behavior through magnitude-timing interactions were assessed by measuring timing within the choice task. Rats were exposed to choice procedures composed of different pairs of magnitudes of rewards for either the smaller-sooner (SS) or larger-later (LL) option. In Phase 2, the magnitude of reward either increased or decreased by 1 pellet in different groups (LL increase = 1v1→1v2; SS decrease = 2v2 → 1v2; SS increase = 1v2 → 2v2), followed by a return to baseline in Phase 3. Choice behavior was affected by the initial magnitudes experienced in the task, demonstrating a strong anchor effect. The nature of the change in magnitude affected choice behavior as well. Timing behavior was also affected by the reward contrast manipulation albeit to a lesser degree and the timing and choice effects were correlated. The results suggest that models of choice behavior should incorporate reinforcement history, reward contrast elements, and magnitude-timing interactions, but that direct effects of reward contrast on choice should be given more weight than the indirect reward-timing interactions. A better understanding of the factors that contribute to choice behavior could supply key insights into this important individual differences variable.

1. Introduction

Impulsivity has been a growing topic of interest in recent years due, in part, to the large number of maladaptive behaviors and disorders with which it co-varies such as: attention deficit hyperactivity disorder (ADHD; Barkley et al., 2001a; Sonuga-Barke, 2002; Sonuga-Barke et al., 1992), substance abuse (Bickel & Marsch, 2001; de Wit, 2008), gambling (Alessi & Petry, 2003; Reynolds, 2006), and obesity (Davis et al., 2010). Additionally, impulsive choice has been hypothesized as a potential risk factor for drug use initiation (MacKillop et al., 2011; Verdejo-García et al., 2008) and predictor of treatment outcomes for drug abuse (Broos et al., 2012; Krishnan-Sarin et al., 2007; Yoon et al., 2007).

As such, the ability to also identify potentially impulsive individuals has gained attention, although impulsivity is a broad and multidimensional construct that may have multiple categories (e.g. Evenden, 1999). One method used to measure impulsive choice is the delay discounting, or impulsive choice, task (Mazur, 1987). This task offers the individual a choice between a relatively smaller alternative that is available sooner (the SS) against a larger alternative that is available later (the LL). The impulsive choice task typically involves assessments of choice behavior at multiple choice points, with adjustments of delay to reward, magnitude of reward, or both occurring over the course of training (Galtress & Kirkpatrick, 2010b; Garcia & Kirkpatrick, 2013; Mazur, 1987; Richards et al., 1997; Roesch & Bryden, 2011; Roesch et al., 2007a; Roesch et al., 2007b). These procedural features present the opportunity for effects not only of the delays and magnitudes themselves, but also the changes in delays or magnitudes.

Indeed, factors such as temporal and reward magnitude sensitivity play an important role in choice behavior (Kirkpatrick et al., 2015; Marshall et al., 2014; McClure et al., 2014; Smith et al., 2015) as well as potential interactions between the two processes (Galtress et al., 2012a; Galtress et al., 2012b; Kirkpatrick, 2014). Moreover, in procedures where reward magnitude is altered within-subjects, strong anchoring effects can occur. Galtress et al. (2012a) tested impulsive choice with either increases or decreases in reward magnitude (1→2→4 pellets versus 4→2→1 pellets). They found that the ascending series resulted in better reward maximizing and stronger adjustments following changes in reward magnitude, whereas the descending series resulted in strong lever biases towards the lever associated with the 4 pellet reward in the initial phase. Thus, it appears that delivering 4 pellets in the initial phase produced an anchor effect that biased choice behavior in subsequent phases.

Procedures that manipulate reward magnitude could induce potential contrast effects not only on choice behavior, but also on timing behavior. Previous studies have shown that changes in reward magnitude and/or value can likewise result in changes to response timing (Balci et al., 2010a; Balci et al., 2010b; Balci et al., 2013; Galtress et al., 2012a; Galtress & Kirkpatrick, 2009, 2010a; Ludvig et al., 2011; Ludvig et al., 2007; Roberts, 1981). Specifically, decreases or increases in reward magnitude shift the peak in responding later or earlier, respectively, in pigeons (Ludvig et al., 2011), rats (Galtress & Kirkpatrick, 2009), and humans (Balci et al., 2013). For example, Galtress and Kirkpatrick (2009) delivered reward magnitudes across phases of either 1-4-1 or 4-1-4 pellets on a peak procedure in which non-reinforced peak trials were intermixed with standard fixed interval trials. They found that increases in reward magnitude sharpened the peak gradient, shifted responding earlier, and increased response rates. Conversely, reward magnitude decreases shifted the peak later and produced lower rates of responding. Furthermore, when these food rewards were devalued with illness-induced lithium chloride exposure or satiety through pre-feeding, there was an overall rightward shift in the peak gradient. These effects persisted over several sessions of training, indicating a long-lasting effect on timing behavior produced by the reward contrast effects.

Given the substantial effects of reward magnitude on timing, it is possible that magnitude manipulations within an impulsive choice task may exert primary effects on choice (through anchoring and/or contrast effects) and also may produce a secondary effect through reward-timing interactions. Galtress et al. (2012a) documented reward-timing interactions within an impulsive choice procedure in addition to their choice effects, finding rightward shifts in peak functions with later start and middle times with decreases in rewards. This provides some direct evidence for these interactions to influence choice and is consistent with findings in more basic timing procedures (Balci et al., 2013; Galtress & Kirkpatrick, 2009, 2010b; Ludvig et al., 2011; Ludvig et al., 2007). However, the Galtress et al. study design did not allow for isolation of the reward-timing interaction component on choice behavior.

The impact of these interactions may also be significant in that temporal perceptions have been shown to have a strong association (Barkley et al., 2001b; Baumann & Odum, 2012; Berlin et al., 2004; Kim & Zauberman, 2009; Marshall et al., 2014; McClure et al., 2014; McGuire & Kable, 2013; Smith et al., 2002; Wittmann & Paulus, 2008; Zauberman et al., 2009) and possible causal relationship (Smith et al., 2015) with impulsive choice behavior. For example, if a rat experiences an increase in magnitude from 1 to 4 pellets for the LL choice, this could lead to subjective shortening of the expected time of reward (consistent with the leftward shift in the peak). In this example, the changes in subjective delay perception may lead to stronger preference for the LL than would be predicted by the change in magnitude alone.

As such, the present experiment sought to assess direct and indirect effects of reward magnitude changes on both timing and choice behavior through a reward contrast manipulation within an impulsive choice task. If individual subjects experience multiple contingencies between different magnitudes, then these interactions are essential to achieve a full understanding of the role of timing and reward processes in choice behavior.

2. Materials and Methods

2.1. Animals

Twenty four male Sprague Dawley rats (Charles River) approximately 80 days old at the start of the experiment were used. Prior to experimental testing, the rats’ weights were reduced to and maintained at 85% of their projected ad lib weight based on a growth curve obtained from the supplier by restricted feeding of standard laboratory chow (LabDiet, Brentwood, MO). They had a mean weight of 315 g (range = 297–350 g) at the onset of experimentation. The rats were pair-housed with free access to water and maintained on a 12:12 hr reversed light-dark cycle, with experimental testing during the dark portion of the cycle.

2.2. Apparatus

The experiment was conducted in 24 operant chambers (Med Associates, St. Albans, VT, USA). Each of the chambers measured 25 × 30 × 30 cm and was housed inside of a ventilated, noise-attenuating box measuring 74 × 38 × 60 cm. Each chamber was equipped with two levers, two optionally back-lit nose poke keys, a house light, a food cup, and a water bottle. The houselight was positioned in the top-center of the right, front wall. Two retractable levers (ENV-122CM) were situated on either side of the food cup at approximately one third of the total height of the chamber with nose poke keys and associated cue lights located directly above each lever. A magazine pellet dispenser (ENV-203) delivered 45-mg food pellets (Bio-Serv, Frenchtown, NJ) into the food cup, where each head entry into the food cup was transduced by an LED-photocell. The water bottle was mounted outside the chamber directly opposite the food cup, and was available through a metal tube that protruded through a hole in the lower-center of the back wall. Med-PC (Tatham & Zurn, 1989), running on two PC computers (one for each set of twelve chambers), controlled experimental events and recorded the time of events with a 2-ms resolution.

2.3. Procedure

The rats were initially split into three groups (n = 8): Group LL Increase (LLI), Group SS Decrease (SSD), and Group SS Increase (SSI). All rats received one pre-training session divided into four blocks for both magazine and lever press training. A 30 min interblock interval (IBI) separated the blocks and sessions lasted a maximum of 14 hr. The session began with a 30 min adaptation period, after which reward deliveries of either 1 or 2 pellets on a random time (RT) 180-s schedule were delivered for a total of 40 food reinforcements. The second block was comprised of four sub-blocks in which only one (either the left or right) lever was extended into the chamber. Responses to the lever were reinforced according to a fixed-ratio (FR) 1 schedule until 5 reinforcements per sub-block were earned, separated by a 10-min IBI. The third block was identical to the second except that responses were rewarded according to a random ratio (RR) 3, and the fourth block according to a RR 5 for five reinforcements per sub-block. Each lever was assigned as either the SS or LL, counterbalanced across rats. Completion of the FR requirement resulted in 1 pellet for group LLI and 2 pellets for group SSD on both levers and either 1 or 2 pellets for the SS or LL, respectively, for group SSI.

The impulsive choice task then followed and was modeled after previous studies from our laboratory (Marshall et al., 2014; Smith et al., 2015), beginning with a 90-min adaption period and lasting a maximum of 14 hr. Four trial blocks began separated by a 90-min interblock interval (IBI) consisting of 50 trials: 8 SS forced choice, 2 SS peak, 8 LL forced choice, 2 LL peak, and 30 free choice trials separated by a 120-s ITI. On free choice trials, both levers were inserted into the box; pressing either lever resulted in the simultaneous retraction of the unchosen lever and illumination of the chosen lever’s corresponding cue light. Food was then delivered according to either a 10- or 30-s fixed interval (FI) schedule for the SS and LL lever, respectively. Forced choice trials were identical to free choice except that only one lever was inserted into the chamber to ensure adequate experience with the outcomes. Peak trials were identical to forced choice trials except that reward delivery was omitted and the lever remained inserted for 90 s, after which both the lever and cue light offset.

The reward magnitudes for the SS and/or LL were manipulated between groups and during Phase 2 (see Table 1). All three groups were returned to their baseline (Phase 1) magnitudes in Phase 3.

Table 1.

The number of pellets received for smaller-sooner (SS) versus larger-later (LL) choices as a function of group and phase. The change in reward magnitude in Phase 2 is marked in bold. The delays to the SS and LL rewards were 10 s and 30 s, respectively.

Group	Phase 1	Phase 2	Phase 3
LL Increase (LLI)	1v1	1v2	1v1
SS Decrease (SSD)	2v2	1v2	2v2
SS Increase (SSI)	1v2	2v2	1v2

Sessions began with a 90-min adaption period followed by a maximum of four trial blocks each separated by a 30-min interblock interval (IBI) consisting of 50 trials: 8 SS forced choice, 2 SS peak, 8 LL forced choice, 2 LL peak, and 30 free choice trials. The sessions lasted approximately 14 hr and consisted of a maximum of 240 pellet deliveries.

2.4. Data Analysis

All responses, stimuli, and different trial types were recorded using unique event codes and associated time stamps with 2-ms resolution in Med-PC IV. Specific dependent measures were obtained from the raw data using MATLAB (The MathWorks, Natick, MA). For all analyses, generalized multi-level linear mixed-effects models (Wright & London, 2009) were conducted in MATLAB 2015A using the Statistics Toolbox functions. Generalized linear mixed-effects models are comparable to repeated-measures regression analyses, but allow for parameter estimation as a function of manipulation condition (e.g., LL magnitude) and individual subjects (Young et al., 2013). Accordingly, such models permit inclusion of both fixed and random effects. Model fitting occurred in two stages: Analyses first determined the model with the best fitting random-effects structure and then determined the model with the best fitting fixed-effects structure that incorporated the aforementioned best fitting random-effects structure. Given the current design, all potential random effects, except for group, are also potential fixed effects; thus, the factor(s) within the best-fitting random effects structure were automatically included as fixed-effects (Young et al., 2013) so that factors that did not vary as a function of subject (i.e., between-subjects factors) could be entered into the model in this second stage. Here, model selection involved determining the model that minimized the Akaike information criterion (AIC), in which the doubled negative log likelihood of the model is penalized by twice the number of estimated parameters. The AIC determines the best approximating and most parsimonious model of the data (see Burnham & Anderson, 1998). Continuous predictors were mean-centered to reduce multicollinearity, and categorical predictors were effect-coded (i.e., codes summed to 0).

2.4.1. Choice behavior

The dependent measure constituted the individual binary choices (SS vs. LL; coded as 0 vs. 1) from free choice trials for each individual rat over the entire experiment. Due to the binary nature of the choices, we employed binomial logistic regression with a logit link function for our multi-level modeling analysis.

For data visualization and additional targeted analyses, we computed a log odds ratio using the individual choices on free choice trials:

$$\mathit{LogOdds}=ln\left(\frac{N_{LL}+.5}{N_{SS}+.5}\right)$$ (1)

in which ln is the natural logarithm, N_LL is the number of LL choices and N_SS is the number of SS choices. The log odds ratio provides a measure of LL choices that is less susceptible to ceiling and floor effects as well as having increased sensitivity to smaller changes in preference compared to percentage choice measures. A value of 0.5 was added to the numerator and denominator to avoid computational problems encountered with exclusive choice of either the SS or LL alternative (Garcia & Kirkpatrick, 2013; Haldane, 1956; Marshall et al., 2014). We conducted steady state analyses using an ANOVA conducted on the log odds LL choices collapsed over the final five sessions of each phase to assess sustained effects of the reward contrast manipulations on choice. Finally, we conducted targeted analyses of behavior at the common 1v2 and 2v2 choice conditions using a one-way ANOVA on the log odds ratios over the last five sessions of the relevant phases with the variable of Group. ANOVAs and post-host tests were conducted using SPSS (version 20).

Due to the long session lengths, we conducted an analysis on the first half versus second half of the session to assess whether there were any systematic changes in choice behavior across the session. We did not find any indication of changes in behavior for any of the groups in any of the phases (ps ≥ .08) so we did not differentiate any of the analyses of choice or timing behavior within sessions.

2.4.1. Timing behavior

Timing behavior was measured in two ways. To assess timing during the choice trials, we computed a median response time measure. The median response is a measure of the middle point of responding during a fixed interval duration (Guilhardi & Church, 2004). We have previously reported this measure as an index of temporal tracking within different choice procedures and found that it is sensitive to different choice contingencies (Peterson et al., 2015). We computed the median responses for SS and LL choice trials separately. The response times during the choice trials were first converted into a response latency. We then removed the choice response so that we were measuring timing once the choice was made. The median response was computed for all trials in which there were at least three responses (in addition to the choice response). The median responses were analyzed over the whole experiment using the same mixed-effects linear regression approach that was used for the choice data. Here, we first tested the median response distributions for normality using JMP (version 11) and did not find any significant deviation from a normal distribution. We then conducted the mixed-effects models in MATLAB using a normal distribution and a linear link function.

We conducted analyses on the steady state mean of the median response times over the last five sessions of each phase with an ANOVA. We also conducted targeted analyses of the mean median response times for SS and LL choices for the common 1v2 and 2v2 pellet conditions by entering the mean median response times over the last five sessions of the relevant phases into a one-way ANOVA with the variable of group. Separate ANOVAs were conducted for SS and LL response times.

The second set of timing measures capitalized on the peak trials. Because there were limited numbers of peak trials in each session, we only measured responding on peak trials over the last 5 sessions of each phase to assess stable steady state timing in the different groups and phases and also for the common 1v2 and 2v2 pellet conditions. In order to quantitatively characterize the peak trial functions, a modified Gaussian distribution was fit to each individual rat’s SS and LL peak trial data from each phase (Guilhardi et al., 2007). The fitted function took the following form:

$$r+A\ast \phi (\mu ,\sigma ),$$ (2)

in which φ(μ, σ) was a Gaussian probability density function with a mean μ and a standard deviation σ, r was the operant (or baseline) level of responding, and A was a scaling parameter for the Gaussian function to approximate the overall response rate. A goodness-of-fit measure (eta squared, η²) was used to evaluate the quality of the fit. The fitting of the modified Gaussian function was conducted using nonlinear fitting tools in MATLAB (The MathWorks; Natick, MA). Three dependent measures were estimated from these fits: (1) the time at which responding reached its maximum rate (i.e., peak time, μ), (2) the spread of the peak (σ), and (3) the rate of responding at the peak time (i.e., peak rate). The peak rate was the value of the fitted function from Eq. 2 at the peak time.

3. Results

3.1. Choice Behavior

3.1.1. Log Odds LL Choices

Figure 1 (left) displays the log odds of LL choices as a function of sessions of training across the three phases. As seen in the figure, the three groups were sensitive to both the initial magnitude and the change in magnitude. Group SSI displayed the highest LL choices in Phase 1 when they were receiving 1 versus 2 pellets and they decreased their LL choices in Phase 2 when the SS magnitude increased to 2 pellets. Groups LLI and SSD displayed strong SS bias in Phase 1 when the two magnitudes were equal (but the outcomes differed in delay) and increased their LL choices in Phase 2 when the LL magnitude was relatively larger. All three groups showed a strong return to baseline in Phase 3.

Fig. 1.

Left. Mean log odds LL choices for each group as a function of session; phase transitions are demarcated by vertical dashed lines. The lines through the data are fits of a four-parameter polynomial curve added for illustration purposes. Middle: Mean (± standard error of the mean, SEM) log odds LL choices for each group collapsed across the last five sessions of each phase. Right. Mean (± SEM) log odds LL choices for each group for the common 1 versus 2 pellet choice point over the last 5 sessions. Group LLI is represented by black filled circles and solid lines, SSD by hatched squares and dotted lines, and SSI by white filled triangles and dashed lines.

To verify the trends in the data, we conducted a multi-level mixed-effects logistic regression analysis (see Data Analysis) by entering each individual choice over the whole experiment into the model. The potential variables for modeling included intercept, group, phase, and session. Group and phase were coded as categorical and used effects coding (−1,0,1). Session was coded as a continuous variable and was mean-centered. The best model, which was determined by the AIC (see Data Analysis), contained random effects of intercept, phase and session and fixed effects of Group × Phase × Session, the lower-order interactions (Group × Phase, Group × Session, Phase × Session), all three main effects, and the overall intercept.

The model revealed a significant overall intercept that was strongly biased towards SS responding, t(113280) = −12.4, p < .001, β = −2.382, 95% CI = [−2.760, −2.005]. There was a significant three-way interaction of Group × Phase × Session which is also evident in the figure. Group LLI decreased their LL choices across sessions in Phase 1, t(113280) = −11.3, p < .001, β = −.069, 95% CI = [−.081, −.057], increased LL choices in Phase 2, t(113280) = 19.0, p < .001, β = .091, 95% CI = [.082, .101], and decreased LL choices in Phase 3, t(113280) = −4.2, p < .001, β = −.022, 95% CI = [−.033, −.012]. Group SSD did not significantly alter their behavior in Phase 1, t(113280) = −1.2, p = .217, β = −.006, 95% CI = [−.015, .003], but increased their LL choices in Phase 2, t(113280) = 19.8, p < .001, β = .078, 95% CI = [.070, .085], and then decreased LL choices in Phase 3, t(113280) = −16.5, p < .001, β = −.072, 95% CI = [−.080, −.063]. Group SSI increased their LL choices over the course of Phase 1, t(113280) = 18.7, p < .001, β = .075, 95% CI = [.067, .083], decreased LL choices over the course of Phase 2, t(113280) = −45.8, p < .001, β = −.168, 95% CI = [−.176, −.162], and increased LL choices over the course of Phase 3, t(113280) = 25.8, p < .001, β = .094, 95% CI = [.087, .094].

The mean log odds for the last five sessions of each phase for each group are shown in the middle panel of Fig. 1 to illustrate the overall steady state effects of reward contrast on choice. A 3 (phase) × 3 (group) mixed ANOVA was conducted on the log odds of LL choices; this revealed a significant group effect, F(2, 21) = 16.0, p < .001, η_p² = .604, and Phase × Group interaction, F(4, 42) = 31.1, p < .001, η_p² = .748. To assess the Phase × Group interaction, separate repeated measures ANOVAs were conducted on each group with phase as a variable. Group LLI, F(2, 14) = 11.2, p = .008, η_p² = .614, SSD, F(2, 14) = 28.2, p < .001, η_p² = .801, and SSI, F(2, 14) = 22.0, p = .002, η_p² = .759, all showed significant changes in choice behavior across phases. In all cases this was due to a difference in choice in Phase 2 compared to Phases 1 and 3, which did not differ.

A final analysis was conducted to assess potential group differences at the common 1 versus 2 pellet choice parameter shown in the right panel of Fig. 1. The data for group SSI are the mean from Phases 1 and 3 and the other groups’ data are from Phase 2. The mean from Phases 1 and 3 was used in group SSI to equate the average amount of training on the task (mean = 38 sessions, averaged across sessions 16–20 and 56–60), as the other two groups’ data were taken from Sessions 36–40. Group SSI showed the highest log odds LL choices, while groups SSD and LLI both showed lower LL preference, despite all groups receiving the same choice parameters. A one-way ANOVA revealed a significant group effect, F(2, 23) = 9.2, p < .001, η_p² = .498, which was due to all three groups differing from each other (p < .05). In addition, Groups SSD and SSI were compared at their common 2 versus 2 conditions (Phase 2 in Group SSI versus the mean of Phases 1 and 3 in Group SSD), and these differed as well, t(14) = 2.4, p = .030, d = 1.164.

3.1.3. Test-Retest Reliability Analysis

Although there were no differences in choice behavior between Phases 1 and 3 at the group level, it is possible that the reward magnitude manipulations altered performance at the individual rat level. To assess this issue, the choice performance by individual rats between the initial and final phases was subjected to a test-retest correlation analysis by comparing individual differences in choice behavior in Phase 1 and 3. Due to the small group sizes this relationship was assessed collapsing across groups. As shown in Fig. 2, there was a significant and very strong test-retest correlation across individuals in Phase 1 versus Phase 3, r = .80, p < .001, indicating that the rank order of individuals remained consistent. This, coupled with the lack of difference between the first and third phases in choice, indicates stable performance at the individual rat level as well as the group level.

Fig. 2.

Test-retest correlation for individual rats in each group in Phase 1 versus Phase 3 when the rats were tested on the same choice parameters. The solid line through the data is the regression fit to all of the rats combined.

3.2. Timing Behavior

3.2.1. Median Response Times

The median response times during SS (top row) and LL (bottom row) choice trials are displayed in Fig. 3 as a function of sessions of training averaged over the last five sessions and for the common 1v2 conditions. There were clear alterations in the median response times for both SS and LL choice trials that generally mirrored the choice patterns. In all three groups, the SS median response times changed in the opposite direction to choice behavior: Groups LLI and SSD decreased their median SS response times and Group SSI increased. For LL timing, Groups LLI and SSI showed increases in LL median response times in Phase 2, which followed the increases in LL choice. Group SSD displayed fairly flat LL median response times across phases, with a slight decrease in LL median response times.

Fig. 3.

Left. Mean median SS (top) and LL (bottom) response times for each group as a function of session; phase transitions are demarcated by a dashed line. The lines through the data are fits of a four-parameter polynomial curve added for illustration purposes. Middle: Mean median (± SEM) SS (top) and LL (bottom) response times for each group collapsed across the last five sessions of each phase. Right. Mean (± SEM) SS (top) and LL (bottom) median response times for each group for the common 1 versus 2 pellet choice point over the last 5 sessions. Group LLI is represented by black filled circles and solid lines, SSD by hatched squares and dotted lines, and SSI by white filled triangles and dashed lines. Response timing was measured during free choice trials.

To verify the trends in the data, we conducted a multi-level mixed-effects linear regression analysis (see Data Analysis) by entering each median response time, with separate analyses on SS and LL choice trials. Group and phase were coded as categorical and used effects coding (−1,0,1). Session was coded as a continuous mean-centered variable. Both SS and LL median response time analyses converged on the same model structure, and this was also the same model structure as the best-fitting model for the choice data. The best models contained random effects of intercept, phase, and session and fixed effects of Group×Phase×Session, the lower-order interactions (Group×Phase, Group×Session, Phase×Session), all three main effects, and the overall intercept.

For SS median response times, the model revealed a significant overall intercept, t(87828) = 67.7, p < .001, β = 5.289, 95% CI = [5.136, 5.443]. The main effects were qualified by a three-way interaction, which is also evident in the figure. Group LLI did not significantly alter their SS median response times in Phase 1, t(87828) = −1.1, p = .266, β = −.002, 95% CI = [−.007, .002], Phase 2, t(87828) = −.7, p = .479, β = −.002, 95% CI = [−.006, .002], or Phase 3, t(87828) = 1.8, p = .073, β = .004, 95% CI = [−.001, .009]. Group SSD did not significantly alter their SS median response times in Phase 1, t(87828) = 1.9, p = .063, β = .004, 95% CI = [−.001, .008], but decreased their response times in Phase 2, t(87828) = −3.1, p = .002, β = −.007, 95% CI = [−.012, −.003], and but then did not significantly change their response times in Phase 3, t(87828) = 1.5, p = .125, β = .003, 95% CI = [−.001, .008]. Group SSI did not significantly change their SS median response times over the course of Phase 1, t(87828) = −.6, p = .531, β = −.002, 95% CI = [−.006, .003], but they increased median response times over the course of Phase 2, t(87828) = 3.8, p < .001, β = .009, 95% CI = [.004, .013], and decreased median response times over the course of Phase 3, t(87828) =−3.0, p = .002, β = −.008, 95% CI = [−.012, −.003].

For LL median response times, the model revealed a significant overall intercept, t(20485) = 56.1, p < .001, β = 16.698, 95% CI = [16.115, 17.281]. The main effects were qualified by a three-way interaction, which is also evident in the figure. Group LLI did not significantly alter their LL median response times in Phase 1, t(20485) = −.6, p = .567, β = −.014, 95% CI = [−.061, .034], but increased their median response times in Phase 2, t(20485) = 2.8, p = .030, β = .059, 95% CI = [.018, .100], and decreased their median response times in Phase 3, t(20485) = −2.0, p = .044, β = −.045, 95% CI = [−.089, −.001]. Group SSD did not significantly alter their LL median response times in Phase 1, t(20485) = −1.0, p = .323, β = −.020, 95% CI = [−.059, .019], Phase 2, t(20485) = 1.0, p = .339, β = .016, 95% CI = [−.017, .049], or Phase 3, t(20485) = .2, p = .838, β = .004, 95% CI = [−.032, .040]. Group SSI significantly increased their LL median response times over the course of Phase 1, t(20485) = 2.2, p = .030, β = .034, 95% CI = [.003, .064], decreased median response times over the course of Phase 2, t(20485) = −5.1, p < .001, β = −.075, 95% CI = [−.104, −.001], and increased median response times over the course of Phase 3, t(20485) =2.8, p = .005, β = .041, 95% CI = [.012, .070].

To examine differences in steady-state behavior (Fig. 3, middle), a 3 (phase) × 3 (group) mixed ANOVA was conducted on the SS median response times; this revealed a near-significant Phase × Group interaction, F(4, 42) = 2.5, p = .058, η_p² = .191, but no group or phase main effects. For the LL median response times, there was a significant main effect of group, F(2,17) = 7.2, p = .006, η_p² = .457, but no effect of phase or any interaction. The group main effect was due to Group SSI having significantly later LL median response times than the other two groups (p < .05).

A final analysis was conducted to assess potential group differences at the common 1 versus 2 pellet choice parameter, shown in the right panel of Fig. 3. The data for group SSI are the mean from Phases 1 and 3 and the other groups’ data are from Phase 2. Group SSI showed the latest median response times for both SS and LL choices, despite all groups receiving the same choice parameters. A one-way ANOVA revealed a significant group effect for LL median times, F(2, 23) = 5.0, p = .017, η_p² = .321, which was due to later median times in Group SSI compared to SSD (p < .05). In addition, Groups SSD and SSI were compared at their common 2 versus 2 conditions (Phase 2 in Group SSI versus the mean of Phases 1 and 3 in Group SSD). There was no difference in SS median response times, but there was a near-significant difference in LL median times, t(14) = −2.1, p = .054 with SSD displaying generally earlier median times than SSI.

3.2.2. Test-Retest Reliability Analysis

To examine whether individual differences in timing behavior were stable over the course of testing, we conducted a test-retest reliability analysis on the median response times for SS and LL trials (Fig. 4). The analysis revealed significant test-retest correlations for the SS median times, r = .59, p = .003. There also was a positive test-retest correlations for the LL median times, r = .38, p = .100, but these did not achieve significance. There was no indication of any test-retest relationship in the LL peak times.

Fig. 4.

Test-retest correlation for the median response on SS (left) or LL (right) choice trials for individual rats in Phase 1 versus Phase 3 when the rats were tested on the same choice parameters. The solid line through the data is the regression fit.

3.2.3. Peak Functions

We were unable to examine the peak trials on a session-by-session basis due to the limited number of observations per session. To quantify the potential effects of reward magnitudes on the peak, the peak functions for individual rats obtained on SS and LL peak trials over the last five sessions of each phase were fit with a Gaussian function (see Data Analysis) and the parameters of peak time, peak rate, and peak standard deviation (SD) were analyzed (data not shown) in a 3 (Group) × 2 (Lever) × 3 (Phase) mixed ANOVA. Peak times on LL trials were later overall than on SS trials, F(1,42) = 2828.7, p < .001, η_p² = 993, but there were no group or phase effects or any interactions (ps > .130). The peak times also closely approximated the SS and LL delays (SS mean = 11.4 s; LL mean = 31.0 s). For peak rate, there was an overall effect of Phase, F(2,42) = 11.3, p < .001, η_p² = .351, Group, F(2,21) = 4.4, p = .025, η_p² = .286, and Lever × Phase × Group, F(4,42) = 3.6, p = .013, η_p² = .256. There also was a near-significant effect of Lever, F(1,21) = 3.7, p = .067, due to generally higher peak rates on the SS lever, and a near-significant effect of Lever × Group, F(2,21) = 3.3, p = .055, which was due to the SSD group showing generally higher peak rates on the SS than the LL lever, whereas the other two groups displayed similar peak rates on the two levers. There were no other interactions (ps > .13). Follow-up ANOVAs conducted on the interaction revealed a significant Phase effect in Group SSD for their SS peak rates, F(2,14) = 12.07, p = .001, η_p² = .633, that was due to higher peak rates in Phase 3 than in Phases 1 and 2 (p < .01) and a Phase effect in Group LLI for their LL peak rates, F(2,14) = 7.3, p = .007, η_p² = .509, due to higher peak rates in Phases 2 and 3 than in Phase 1 (p < .05). For the peak SD, there was a main effect of Lever, F(1,46) = 365.0, p < .001,η_p² = .946, Phase, F(2,42) = 17.3, p < .001, η_p² = .452, but no group effects or interactions.

Additionally, one-way ANOVAs were conducted on each measure for the SS and LL levers to assess group differences at the 1 versus 2 pellet conditions. For the SS lever, there were group effects on the peak rate, F(2, 23) = 3.64, p = .044, η² = .257 only. Tukey post-hoc tests indicated that group LLI showed a significantly lower peak rate compared to group SSI (p < .05). There were no significant group differences in responding on the LL lever. Finally, the timing indices were compared between Groups SSD and SSI in the common 2 versus 2 conditions by comparing Phase 2 in Group SSI versus the mean of Phases 1 and 3 in Group SSD. The only significant difference was in the LL SD, t(14) = 2.19, p = .046, d = 1.093, which was due to lower SDs in Group SSI compared to the other two groups.

3.3 Choice-Timing Correlations

To assess whether the changes in choice were related to the changes in timing behavior across sessions (Figs. 1 and 3, left panels), we calculated a Pearson’s correlation coefficient for each rat for the SS median response and LL median response data versus the log odds LL choices across sessions. This provided an index of correlation between changes in choice and timing over the experiment. We then tested the correlations for the individual rats against zero. Because there were similar correlation patterns in the three groups, we collapsed across group to maximize power. The correlations between SS median response times and choice were significantly negative, mean r = −.11, t(23) = −2.2, p = .041, and the correlations between LL median response times and choice were significantly positive, mean r = .08, t(23) = 2.2, p = .036.

4. Discussion

The examination of reward contrast effects on LL choice behavior revealed large effects of reward magnitude contrast on LL choice (Fig. 1). The contrast effects were apparent in the session-by-session analysis, which showed gradual changes in choice behavior in response to changes in magnitude, as well as in sustained effects on steady state choice behavior over the last five sessions of each phase. Particularly interesting comparisons are the common 1 versus 2 and 2 versus 2 choice points. When 1 versus 2 pellets was experienced in Phases 1 and 3, Group SSI showed greater LL choices than when this was experienced in Phase 2 by the other two groups. A similar pattern was seen in that there were fewer LL choices in the 2 versus 2 condition when it was experienced first by Group SSD. Interestingly, there were no carry-over effects of the magnitude contrast on the return to baseline LL choices in Phase 3, either at the group or individual differences level.

In addition to the noted reward contrast effects, the nature of the change in magnitude (SS decrease or LL increase) mattered as well. These results are consistent with the effects found by Galtress et al. (2012a) which compared choice between an LL with a fixed 60-s delay and an incrementing SS delay that began at 0-s but increased by 15-s following an SS choice. The magnitude to the LL was then systematically either decreased in the order of 4-2-1 or increased in the opposite order across two experiments. Their results showed an increased tendency for the rats to continue choosing the SS at greater delays when the LL magnitude increased relative to decreases in LL magnitude, similar to what was observed here (Fig. 1, middle). These results revealed robust anchor effects of the initial choice parameters on subsequent choice performance, consistent with the differential LL preferences seen at the common 1 versus 2 pellet choice points. The present results further extend these findings by demonstrating additional effects due to the nature of the change in magnitude between the two alternatives. When rats experienced a decrease in the magnitude of the SS (SSD; 2v2→1v2) they preferred the LL more than when they experienced an increase in the magnitude of the LL (LLI; 1v1→1v2), suggesting an asymmetry in the effects of magnitude increases and decreases on choice behavior.

There also were clear effects of reward magnitude on timing that were evident in the median response times on choice trials as a function of sessions (Fig. 3, left). Overall, the median SS responses changed in the opposite direction of choice behavior, such that when rats switched towards the LL, SS median response times decreased, whereas when choices switched towards the SS, the SS median response times increased. This was verified by a negative correlation between LL choices and SS median response times across sessions. On the other hand, LL median response times were positively related to LL choices, such that when LL choices increased, LL median response times also increased. Later median response times are potentially indicative of more precise timing, which has been linked with increased self-control (Marshall et al., 2014; McClure et al., 2014; Smith et al., 2015) and the patterns in the present study are consistent with these previous findings. Furthermore, Peterson et al. (2015) demonstrated that poorer temporal tracking, evidenced by median response times that failed to adjust when the durations changed, was associated with more SS choices, suggesting that temporal tracking within the choice trial is an indicator of impulsive/self-controlled choices, also consistent with the current findings.

In addition to the effects on timing across sessions, there were also some steady-state effects, but these were weaker than what was observed in the choice data. Moreover, the test-retest reliability analysis showed positive correlations between median response times in Phases 1 and 3, but the correlations were lower and only were significant for SS median response times. This suggests that there were some carryover effects of the reward contrast on timing in the return to baseline. In general, the SS median response times mirrored the choice data and the group rankings in the common 1 versus 2 pellet condition were the same for SS median response times and LL choices. Group SSI showed the most noteworthy pattern with later LL median response times overall and in the common 1 versus 2 and 2 versus 2 conditions. This group also displayed the highest rates of LL choices in those conditions. This lends further evidence that greater LL choices were associated with later LL median responses times and may potentially reflect more precise timing.

The analysis of the peak trials did not reveal any robust effects of reward contrast on timing. The only timing index that was affected by the magnitude changes was peak rate on SS peak trials in the common 1 versus 2 pellet condition, which most likely reflects reward value effects on responding during the trial. However, the peak trials did confirm the expected patterns in that the LL peak times were later than the SS peak times, and the LL peak SDs were greater than the SS peak SDs, indicating that the rats did learn to differentiate the SS and LL durations. The weaker patterns in the peak trials may have been due to the more limited number of observations per session which resulted in less stable estimates compared to the median response times. In addition, it is possible that responding on forced choice peak trials may engage somewhat different processes from responding during the choice trials. Galtress et al. (2012a) argued that peak trials within the choice task are not as good of a measure of choice-timing interactions because the choices themselves may affect timing in ways that are not captured in the peak trials. Indeed, studies that have reported robust choice-timing correlations have either measured timing outside of the choice task in either peak (McClure et al., 2014) or bisection (Marshall et al., 2014) tasks, or have used the index of median response times (Peterson et al., 2015) as was done here. Further research should examine choice-timing dynamics and determine how the choices themselves may impact on different timing metrics within the choice procedure.

The primary motivation for the present research was to consider the processes that most likely affect choice behavior at both the global and individual-subject levels. The most widely employed descriptive model of choice behavior is the hyperbolic discounting equation, given in Eq. 3:

$$V=\frac{M}{1+kD}$$ (3)

where V refers to the subjective value of a choice option (SS or LL), M is the magnitude and D is the delay associated with that option, and k is an individual differences parameter, the delay discounting rate. k-values are commonly reported in delay discounting/impulsive choice studies and have been shown to adequately reflect individual differences in choice behavior (see Odum, 2011 for a review). However, both magnitude and delay are treated as veridical in this model even though both are quantities that are likely judged according to Weber’s law. Consistent with this view, there is growing research demonstrating that individual differences in timing and reward discrimination ability predict individual differences in choice behavior (Kirkpatrick et al., 2015; Marshall & Kirkpatrick, 2016; Marshall et al., 2014; McClure et al., 2014). However, these previous studies have not examined the more dynamic factors of reward contrast and reward-timing interactions. The present study demonstrated unequivocal effects of reward contrast on choice behavior, indicating that reinforcement history and magnitude contrast variables likely serve as key processes in determining choices in procedures that manipulate magnitude of reward over time. Interestingly, the reward contrast effects appear much stronger than absolute magnitude effects on impulsive choice, which have often been reported to be weak or unsystematic in choice behavior in rats and pigeons (Green et al., 2004; Richards et al., 1997; but see Grace et al., 2012), suggesting that relative magnitude may play a greater role than absolute magnitude at least in choice behavior in rats. This suggests that future theory development should include these factors, which could potentially be accounted for by the more dynamic reinforcement learning models (e.g., Bush & Mosteller, 1951; Rescorla & Wagner, 1972; Sutton & Barto, 1998) to explain the magnitude valuation component of choice behavior.

While there were some indications of reward-timing interactions within the current procedure, these effects were comparatively small in relation to the direct effect of reward contrast on choice. This indicates that the choices were dominated by reward magnitude effects on choice itself, with a smaller effect potentially occurring through the secondary magnitude-timing interaction route. However, given the present findings, a complete model of choice behavior should include magnitude-timing interactions as a component. In addition, it is possible that some situations may engender larger reward-timing interactions than were seen here. For example, previous work reported robust reward-timing interaction effects on peak timing within the more dynamic PI/FI procedure. It is therefore possible that reward-timing interactions may be more prevalent in dynamic adjusting procedures such as PI/FI or adjusting magnitude tasks (Hackenberg & Hineline, 1992; Richards et al., 1997).

It is clear from the present results that choice behavior is controlled by a complex array of processes. Future theory development should take into account the non-veridical nature of magnitude and delay judgments, as well as the effects of reinforcement history, reward magnitude contrast, and reward-timing interactions on choice performance. Future empirical research should also seek to examine these factors to better understand how they unfold within different types of choice procedures. These efforts should significantly advance the field and provide deeper insights into the factors that lead to impulsive choices that are such critical predictors of other maladaptive behaviors.