Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Feb 1.
Published in final edited form as: Psychol Addict Behav. 2019 Jul 25;34(1):147–155. doi: 10.1037/adb0000495

Reducing Impulsive Choice: VI. Delay-Exposure Training Reduces Aversion to Delay-Signaling Stimuli

Sara Peck 1, Jillian M Rung 1, Jay E Hinnenkamp 1, Gregory J Madden 1
PMCID: PMC6980928  NIHMSID: NIHMS1039520  PMID: 31343195

Abstract

Delay-exposure (DE) training consistently and robustly reduces impulsive choice in rats, but the behavioral mechanisms behind this effect are not yet understood. The present study evaluated if DE training works by mitigating aversion to delay-signaling stimuli – those encountered when rats chose the larger-later reward in impulsive choice assessments. Fifty-seven rats were randomly assigned to 120 days of training with delayed reinforcement (DE), training with immediate reinforcement (IE), or to a no training Control group. Consistent with prior experiments, DE rats made significantly fewer impulsive choices than IE or Control rats. Subsequently, in a separate assessment of delay aversion, rats were given the opportunity to press a lever to temporarily escape from stimuli correlated with long or short time-intervals to food. When these escape opportunities terminated delay-signaling stimuli in the impulsive-choice task, DE rats escaped significantly less than IE and Control rats. When escapes terminated FI-signaling stimuli (a procedure in which there is no response-reinforcer delay), the difference only approached significance. These results support the hypothesis that DE training reduces impulsive choice, in part, by reducing aversion to delay-signaling stimuli.

Keywords: Impulsivity, Delay discounting, Impulsive choice, Delay-exposure training, Delay-aversion, Escape, Time-out

Delay discounting refers to a decrease in the subjective value of a consequence as the time between choice and consequence increases (Odum, 2011). Delay discounting can play an important role in impulsive choice, when defined as preference for a smaller-sooner over a larger-later reward. Specifically, if the value of a larger-later reward is substantially (steeply) discounted, its subjective value will fall below the value of the smaller-sooner reward (Ladd & Petry, 2003). High rates of delay discounting are correlated with a wide variety of maladaptive impulsive choices, including drug and alcohol abuse (Amlung, Vedelago, Acker, Balodis, & MacKillop, 2017; Kirby & Petry, 2004; Madden, Petry, Badger, & Bickel, 1997; Petry, 2001), risk-taking in service of substance use (Odum, Madden, Badger, & Bickel, 2000), over-eating (Amlung, Petker, Jackson, Balodis, & MacKillop, 2016), pathological gambling (Alessi & Petry, 2003; Petry & Madden, 2010; Petry & Casarella, 1999) and suicide (Dombrovski et al., 2011, 2012). Longitudinal studies suggest steep delay discounting is predictive of human drug use (Audrain-McGovern et al., 2009; Barlow, McKee, Reeves, Galea, & Stuckler, 2016; Kim-Spoon, Farley, Holmes, Longo, & McCullough, 2014), and treatment-outcomes research reveals poorer outcomes among those with the steepest discounting functions (Harvanko, Strickland, Slone, Shelton, & Reynolds, 2019; Loree, Lundahl, & Ledgerwood, 2015; MacKillop & Kahler, 2009; Sheffer et al., 2014). Indeed, Coughlin et al. (2018) reported that delay discounting was “the single best predictor of group CBT treatment response” in smoking-cessation therapy.

These correlations suggest that reducing delay discounting might prove beneficial in preventing or reducing maladaptive decision-making (Gray & MacKillop, 2015; Volkow & Baler, 2015). To this end, a wide variety of experimental techniques for reducing delay discounting or impulsive choice have been explored (see Rung & Madden, 2018 for review and meta-analysis). These range from reframing the wording of the alternatives in a delay-discounting task (e.g., Magen, Dweck, & Gross, 2008; Read, Frederick, Orsel, & Rahman, 2005 to providing extended learning experiences to nonhuman animals (e.g., Fox, Visser, & Nicholson, 2018; Mazur & Logue, 1978; Smith, Marshall, & Kirkpatrick, 2015).

One of the most replicated of these learning-based methods of reducing impulsive choice involves providing rats with extended exposure to delayed reinforcement contingencies (Stein et al., 2013). Specifically, adolescent rats given “delay-exposure” (DE) training learn to press a lever to acquire a food reward following a signaled 17.5-s delay. During the delay the lever is retracted and a cue light is illuminated. After this training, which typically lasts 120 sessions, impulsive choice is assessed and compared across groups given DE training, no training, or comparable training with an immediate food reward following the choice. Delay exposure training has produced reliable (Renda, Rung, Hinnenkamp, Lenzini, & Madden, 2018; Rung, Buhusi, & Madden, 2018), long-lasting reductions in impulsive choice (Renda & Madden, 2016; Stein et al., 2013; Stein, Renda, Hinnenkamp, & Madden, 2015).

Although this impulsivity-reducing effect of DE training is well documented, the behavioral mechanism(s) of this reduction are unknown. Rung et al. (2018) investigated if DE training improved the precision of interval timing. Prior research has shown that imprecise interval timing is significantly correlated with impulsive choice (Marshall, Smith, & Kirkpatrick, 2014), and exposure to time-based schedules of reinforcement (e.g., fixed- and variable-interval schedules) reduces impulsive choice (Bailey, Peterson, Schnegelsiepen, Stuebing, & Kirkpatrick, 2018; Fox et al., 2018; Peterson & Kirkpatrick, 2016; Smith et al., 2015; Stuebing, Marshall, Triplett, & Kirkpatrick, 2018). In the Rung et al. experiment, DE training significantly decreased impulsive choice but had no effect on either the precision or accuracy of interval timing.

A second candidate process mediating the effects of DE training on impulsive choice involves a reduction in aversion to delay-signaling stimuli. That is, DE training may increase self-control choices by reducing aversion to the immediate consequence of that choice – the presentation of a cue-light signaling a delay to food. For IE and Control rats, their first encounter with this stimulus occurs during forced-choice larger-later reward trials during post-training tests of impulsive choice. If these rats are strongly averse to that delay-signaling stimulus, their free choices for the smaller-sooner reward may be, in part, influenced by motivation to avoid the delay and/or the associated cue. By contrast, DE rats have completed 120 sessions in which a cue-light signaled a delay reduction to food. That is, during DE training, the response-contingent delay-signaling stimulus marks a delay reduction from approximately 60 s between food deliveries, to exactly 17.5 s to food. This ratio of intertrial interval (I = 60 s) to trial (T = 17.5 s) duration (I:T ratio) of 3.43:1 should readily produce Pavlovian learning of the delay-reduction properties of the cuelight (Gallistel & Gibbon, 2002; Gibbon, Baldock, Locurto, Gold, & Terrace, 1977). Thus, according to this hypothesis, by the time DE rats complete the post-training test of impulsive choice, the delay-signaling stimulus is “good news” (food is getting closer), whereas for IE and Control rats the same stimulus provides either no news about when food is coming (because they have never previously seen such a cue), or provides bad news as it signals the longest post-response delay to food that they have ever encountered. In sum, DE training may reduce impulsive choice by teaching rats that delay-signaling cues are good news (conditioned reinforcers), thereby reducing the default aversion to such cues.

To test this hypothesis, we used an operant escape paradigm to quantify training-related differences in aversion to delay-signaling stimuli. Aversion has long been operationalized as making an operant response to escape from the stimulus. Nonhumans, for example, will make operant responses to escape from painful stimuli or discriminative stimuli signaling a worsening of reinforcement schedule contingencies (e.g., Appel, 1963; Appel, 1961; Brown & Flory, 1972; Dardano, 1973; Everly, Holtyn, & Perone, 2014; Harzem, Lowe, & Priddle-Higson, 1978; Retzlaff, Parthum, Pitts, & Hughes, 2017). In the present experiment, rats were randomly assigned to DE training, IE training, or to a no-training Control group that remained fallow during the training phase. Following a post-training impulsive-choice test, rats completed two tasks in which they could temporarily escape from stimuli signaling either FI schedules or delays to food. The first of these tasks was completed independent of the impulsive-choice assessment, the second imbedded escape opportunities within the assessment of impulsive choice.

Method

Subjects

Fifty-nine experimentally naïve male Wistar rats (Harlan Laboratories, Indianapolis, IN) began the study at approximately 25 days old. Rats were gradually restricted to 85% of their projected growth curve free-feeding weight, at which they remained for the remainder of the study. Rats were individually housed in polycarbonate cages with enrichment (nesting papers and rodent toys) and unrestricted access to water. Home cages were kept in a humidity and temperature-controlled colony room that operated on a 12-hr/12-hr light/dark cycle. Sessions were conducted at the same time each day (during the light cycle), six days a week. Post-session feeding occurred in the home cages at least 3 hours following a session. This study was approved by the Institutional Animal Care and Use Committee at Utah State University (protocol # 2602).

Apparatus

Seventeen identical sound-attenuating operant chambers (Med Associates, St. Albans, VT) were used. Each chamber was equipped with low-profile retractable levers (10.5 cm above the chamber floor), two of which were located on either side of a food receptacle on the front wall, one above the food receptacle and a fourth in the center of the back wall. A stimulus light was placed above each lever. Reinforcement was delivered via a pellet dispenser, delivering 45 mg multigrain food pellets (Bio-Serv, Frenchtown, NJ). White noise was provided throughout all experimental sessions.

Procedures

Rats were matched by weight, and then block-randomly assigned to one of three groups—DE (n = 20), IE (n = 20), or Control (initially n = 19; but 2 were excluded for failure to prefer large over small food reinforcers when both were immediately available; described below). Figure 1 provides an overview of the experiment phases and their approximate durations.

Figure 1.

Figure 1

Open in a new tab

Approximate age of rats at the onset of each phase. Phases denoted with a superscript 1 included an amount discrimination component, and phases denoted with a superscript 2 included front-wall lever training.

Lever Acquisition.

All rats acquired a lever-press response through a modified auto-shaping procedure. Trials began by inserting the rear-wall lever and simultaneously illuminating its cue light. For IE and Control rats, a lever-press produced two pellets immediately and caused the corresponding cue light to extinguish and lever to retract; for DE rats, the contingency for lever pressing was the same, with the exception that pellets were delayed by 17.5 s and the cue light remained on during the delay to food. If a press did not occur within 10 s of activating the lever/light, pellets were delivered noncontingently for the first five trials of a session. Trials were separated by a 98.2-s (IE and Control) or 270-s (DE) inter-trial interval (ITI; blackout). Thereafter, the lever/light remained activated until the lever was pressed, food was delivered, and the ITI separated this event from the next opportunity to press. Sessions ended once 20 reinforcers were obtained or after 2 hours elapsed, whichever came first. Acquisition sessions continued until all rats produced at least 90% of the available food deliveries by lever-pressing, in two consecutive sessions.

Exposure Training Phase.

Following lever acquisition, rats completed 120 sessions of DE- or IE-training, or remained fallow in their home cages (Control). Training sessions consisted of 80 trials, which began every 60 s. Trials were initiated by activating the rear-wall lever/light. For IE rats, pressing the lever once retracted the lever, extinguished the cue light, and produced two pellets immediately. For DE rats, a single press retracted the lever and initiated a 17.5 s delay to the same two-pellet reward. The cue light above the retracted lever remained on for the duration of the delay. Control rats were handled, weighed, and fed at the same times as DE and IE rats.

Front-Wall Lever Training.

Next, all rats learned to press the front-wall levers during 75-trial sessions in which the left, center, or right lever/light was presented (order randomized with the constraint of 25 trials on each lever). Pressing the inserted lever produced two pellets immediately (IE and Control rats) or following a cue-light signaled 17.5-s delay, during which the lever was retracted (DE rats). Trials were initiated every 60 s.

Amount Discrimination.

Next, rats chose between 1- and 3-pellet reinforcers. Choices were made in three 20-trial blocks, each separated by a 7-min blackout period. The first six trials within a block were forced-choice trials (3 on the left lever, 3 on the right; order randomly determined) and the remaining 14 were free-choice trials. Trials were initiated by inserting the rear center lever and illuminating its cue light. Pressing this lever caused one (forced-choice trial) or two side-levers to be inserted with corresponding cue lights lit (free-choice trial). Lever assignment of the 1- and 3-pellet reinforcers was counterbalanced within groups. For IE and Control rats, pellets were delivered immediately after a side-lever press; pellet deliveries were, regardless of pellet amount, delayed by 17.5 s for DE rats. Failure to press an available lever within 20 s terminated the trial and was scored as an omission. Omitted forced-choice trials were repeated until completion. Sessions terminated after 60 trials, or after 2 hours, whichever came first. This phase continued until rats selected the 3-pellet reward on ≥ 90% of trials, for two consecutive sessions.

Impulsive Choice Assessment I.

Impulsive choice was tested using an increasing-delays (Evenden & Ryan, 1996) procedure. Sessions were structured as in the preceding phase except that, for all rats, the delay to the 3-pellet reward increased across the three trial blocks (0, 15, and 30 s) and the 1-pellet reward was always available immediately. Rats completed at least 20 sessions of the impulsive choice assessment and until the following stability criteria were met across the final 5 sessions: (1) area under the percent larger-later choice curve (AUC; Myerson, Green, & Warusawitharana, 2001) did not vary by more than 15%, (2) there was no monotonic trend in AUC, and (3) the larger reward was selected ≥ 80% of the time during the 0-s delay (first trial block). If preference for the larger reward in the first trial-block fell below 60% for two consecutive sessions, rats were placed in remedial amount-discrimination sessions (see Renda et al., 2018 for details). Following these remedial sessions, the increasing-delays procedure was resumed until stability.

FI-Signal Aversion Assessment.

In this phase, sessions were composed of 40 trials, 20 on each of the side levers. A fixed-interval (FI) 5-s schedule was programmed on one lever (the smaller-sooner reward lever in the prior phase) and an FI 40-s on the other. Sessions began with the insertion of one of the side levers, and illumination of the light above it. The first lever press after the FI timer elapsed produced two food pellets on every trial.

On half of trials arranged on each lever, the front, center lever was also presented at trial onset. Pressing this escape lever retracted both levers and extinguished the stimulus light over the side (FI) lever for 5 s; the FI schedule timer was not stopped during this blackout. After the blackout, the FI lever was re-inserted and its cue light illuminated but reinforcers could not be earned for 7 s following a blackout. If the first lever pressed in a trial was a side lever (FI), the escape lever retracted for the duration of the trial. This phase continued for at least 20 sessions and until the rate of escaping showed no monotonic increasing or decreasing trend. Following stability, the longer of the FI schedules was reduced from 40s to 20s and sessions continued until stability was re-established.

Impulsive Choice Assessment II.

To evaluate if group differences in impulsive choice remained, all rats completed a second amount-discrimination assessment, followed by an impulsive-choice retest. Procedures were identical to those outlined previously, except that stability in the impulsive-choice retest was assessed after 10 sessions.

Delay-Signal Aversion Assessment.1

Next, delay-aversion was assessed in the context of the impulsive choice assessment by giving rats the opportunity to escape from the stimulus signaling the delay to the larger-later reward. These escape opportunities were presented during all forced-choice trials arranged in the second and third trial blocks (15-s and 30-s delays to the larger-later reward, respectively). So as to increase opportunities to measure propensity to escape, the number of forced-choice trials was increased to 10 per trial block (5 smaller-sooner and 5 larger-later reward trials).

As shown in Figure 2, escape-opportunity trials began with one side-lever inserted into the chamber; side-lever assignments were as in the impulsive choice retest (the previous phase). If the side lever was associated with the larger-later reward, pressing that lever (1) retracted the side lever, (2) illuminated the delay-signaling cue light above that lever, and (3) inserted the center (escape) lever for 5 s. Pressing the escape lever retracted that lever and extinguished the delay-signaling cue light for 5 s. After this blackout, the cue light was re-illuminated for the remainder of the delay to the larger-later reward. When the escape-opportunity trial occurred on a forced-choice on the smaller-sooner lever, pressing that lever (1) retracted the side lever, (2) delivered one food pellet immediately, and (3) inserted the center (escape) lever. Pressing the escape lever retracted it, but had no other programmed consequence.

Figure 2.

Figure 2

Open in a new tab

Key features of the Delay-Signal Aversion Assessment procedure. Specifically, the larger-later reward (LLR) forced-choice trial in which rats could choose to escape delay-signaling stimuli (bottom alternative), or not (top alternative).

In all other respects, the impulsive choice assessment was as described previously. Rats completed at least 10 sessions in this phase, and until choice and escape responses were judged stable.2 The phase was concluded after 30 sessions if performances did not meet these stability criteria.

Data Analysis

Group differences in non-choice/escape measures were evaluated using non-parametric tests equivalent to ANOVA and t-tests (i.e. Kruskal-Wallis and Mann-Whitney U tests) because some distributions violated normality. Specifically, group differences in days to acquire lever pressing, completed trials, omissions, and latency to press over the final three sessions of training of DE and IE training were evaluated.

For all remaining statistical tests, the final five sessions of each phase were analyzed. Due to skewness and/or non-normal distributions, and the bounded nature of the outcome measures from the impulsive choice assessment (AUC) and delay aversion procedures (proportion trials escaped), traditional group-based analyses such as ANOVA were not appropriate (see Rung et al., 2018 for comparable outcomes with AUC). Instead, beta regressions were conducted for analyses of the impulsive choice and delay aversion outcomes. To accommodate several rats with AUC = 1.0 (i.e., exclusive choice of the large-reward alternative, regardless of delay) AUCs were transformed by subtracting a small, constant value (.00001); similarly, when escape proportions were 0 or 1, a small constant was also added/subtracted. Within-group changes in impulsive choice across the initial test and retest were evaluated using Wilcoxon matched-pairs rank tests (i.e., a non-parametric matched-pairs t-test).

For regression analyses, overly influential observations were screened for using Cook’s d. In instances where overly influential values were found (using a cutoff of 4/N) these values were removed and the model was re-run. Removal of overly influential observations are noted and discussed below.

Pearson correlations were conducted to evaluate the relation between delay aversion (escape from delay-signaling stimuli) and impulsive choice (AUC). Overall and within-group correlations were evaluated. Analyses were primarily conducted in R (R Core Team, 2013); code and output are available in the Open Science Framework (https://osf.io/36wv2/?view_only=647530a9fbd440258c75e04f539fda9c). Beta regressions were conducted using the “betareg” package (Ferrari & Cribari-Neto, 2004); test statistics for overall Group-level effects in the beta regression analyses were obtained using the “car” package (i.e., Fox & Weisberg, 2011); correlations were conducted using the “furniture” package (Barrett & Brignone, 2017); all other tests were conducted using functions available in the base R software.

Results

DE rats took significantly longer to acquire lever pressing than Control rats (K = 8.043, p = .017; no other differences in acquisition were significant. During exposure training, the DE group committed more omissions than IE rats (U = 105.5, p < .001), but completed the same number of trials (U = 200, p > .99) because omitted trials were repeated. DE and IE latencies did not differ significantly (U = 133, p = .072).

The top panel of Figure 3 shows the percentage of larger-later reward choices as a function of its delay. In the initial assessment of impulsive choice (top panels), group was a significant predictor of AUC, χ2(2) =13.1, p =.002. Specifically, DE rats had higher AUC values (i.e., higher preference for larger-later rewards) than IE (z = −2.80, p = .005; d = .78) and Control (z = −3.37, p < .001; d = 1.106) rats. The direction and significance of group differences was the same when overly influential observations were included (n = 4).

Figure 3.

Figure 3

Open in a new tab

The proportion of larger-later reward (LLR) choices as a function of the delay to that reward. The top panel shows results from the Impulsive Choice Assessment I phase; the bottom panels are from the Impulsive Choice Assessment II phase. Embedded graphs depict mean (± SEM) AUC values for the group.

Figure 4 shows the proportion of trials in which the three groups of rats escaped from stimuli signaling the FI 40 s (top panel) and FI 20 s (bottom panel) stimuli.3 Escape frequencies in these phases were correlated (r = .792; p < .001) and group differences were not significant when the stimuli signaled FI 40 (χ2[2] =3.65, p = .16) or FI 20-s schedules (χ2[2] = 4.14, p = .13).

Figure 4.

Figure 4

Open in a new tab

Proportion of trials in which an escape response was made in the FI-Signal Aversion Assessment phase. The top panel shows proportion of escapes from stimuli signaling that the FI 40-s schedule was active; the bottom panel shows proportion of escapes from stimuli signaling that the FI 20-s schedule was active. Group means (± SEM) are shown, in addition to individual subject data.

During the retest of impulsive choice that followed, the main effect of group on AUC was maintained (see bottom panels of Figure 3; χ2[2] =22.6, p < .001). That is, DE rats made fewer impulsive choices (i.e., higher AUC) than IE (z = 3.54, p < .001; d = 0.72) and Control (z = 4.24, p < .001; d = 1.198) rats. The direction and significance of group differences was the same when overly influential observations were included (n = 2). AUC values in the initial and retest of impulsive choice were strongly correlated (r = .945; p < .001).

The top panel of Figure 5 shows the proportion of escape responses made in the Delay-Signal Aversion Assessment phase, when escape opportunities were presented during the impulsive-choice test. Escaping in this phase was not correlated with escapes from FI-signaling stimuli in the prior phase (ps > .95). Instead, in this phase, group was a significant predictor of escapes from the signaled delays to the larger-later rewards; DE rats escaped significantly less than IE (z = 4.48, p < .001, d = 1.41) and Control rats (z = 3.11, p = .002; d = 1.103). Escapes made during smaller-sooner trials were rare (median = 0) and did not differ across groups (p = .364). The bottom panel of Figure 5 shows AUC values during the free-choice trials of this phase. DE rats still had significantly higher AUC values than Control rats (z = 2.20, p = .03; d = 0.92), but the difference between DE and IE rats was not significant (z = 0.01, p = .99; d = 0.596). These comparisons (significance, direction of group differences) were unaffected by the removal of 2 and 4 overly influential values across the models for escapes and AUC, respectively. Rats in the IE (W = −130, p = .01; d = .57), and Control (W = −127, p = .001; d = .78) groups made significantly fewer impulsive choices (higher AUC) in this final phase, when escapes from delay-signaling stimuli were occasionally made available. No within-group change was observed in the DE group (W = −14, p = .81; d = .26).

Figure 5.

Figure 5

Open in a new tab

Mean (± SEM) proportion of trials in which rats in each group escaped from stimuli presented during larger-later rewards to food (top panel). Data are from forced-choice trials in the Delay-Signal Aversion Assessment phase. The bottom panel shows mean (± SEM) and individual rats’ AUC values, separated by group, from free-choice trials in the same phase.

Of interest to the primary hypothesis, across groups, escape frequency within the Delay-Signal Aversion Assessment phase was negatively correlated with AUC in both the initial (r = −0.323; p = .014) and retest (r = −0.314; p = .017) of impulsive choice. The latter of these is shown in Figure 6, where the correlation is visually less compelling than the signifance test suggests. No within-group correlations between AUC and escapes were significant (DE: ps > .30; IE: ps > .65; Control: ps > .12).

Figure 6.

Figure 6

Open in a new tab

AUC from Impulsive Choice Assessment II plotted as a function of the proportion of trials in which escapes were made in Delay-Signal Aversion Assessment. The negative correlation is statistically significant (r = −0.314; p = .017).

Discussion

Delay Exposure training decreased the frequency of escaping from delay-signaling stimuli, when compared to IE and Control rats. The group difference was not significant, however, when escape responses terminated stimuli correlated with FI-schedule contingencies. These findings offer support the hypothesis that DE training reduces impulsive choice, in part, by reducing the aversiveness of delay-signaling stimuli encountered during assessments of impulsive choice.

Five aspects of the present findings deserve comment. First, this is the sixth replication of the finding that DE training reduces impulsive choice in male rats (Renda & Madden, 2016; Rung et al., 2018; Renda et al., 2018; Stein et al., 2013, 2015). As in previous studies, the group difference in impulsive choice was maintained at a follow-up assessment conducted approximately 5 weeks following the first assessment. Also consistent with prior studies, the difference in impulsive choice between IE and Control groups was not statistically significant (Renda et al., 2018; Rung et al., 2018). This lack of a difference suggests that IE training does not increase impulsive choice, perhaps because it merely provides more experience with immediate reinforcement, something that Control rats have also exclusively experienced in the operant chambers, albeit some time ago. Unlike previous DE studies, DE rats’ lever-pressing latencies remained significantly longer than those of IE rats at the conclusion of the Exposure-Training phase. It would appear that convergent latencies are not a necessary DE training criterion for observing subsequent reductions in impulsive choice.

Second, there was no correlation between escaping from delay-signaling stimuli and stimuli signaling FI schedule contingencies. This suggests these procedures are behaviorally distinct, perhaps because a delay-signaling stimulus signals an interval uninterrupted by operant responses on the lever that initiated the delay; by contrast, an FI-signaling stimulus signals an interval until a response on the lever will produce an immediate reinforcer. To the extent that self-control is defined as a choice that produces a delayed reinforcer, DE training may be said to increase self-control, in part, because it reduces aversion to stimuli signaling this response-reinforcer delay.

Third, in the final phase, when escapes from delay-signaling stimuli were available during forced-choice trials, IE and Control rats decreased their preference for the impulsive alternative. Whether or not this reduction in impulsivity is due to these occasional escape opportunities is unknown because procedural changes from the prior assessment of impulsive choice (e.g., more forced- and fewer free-choice trials) were confounded with the escape opportunities. It would be curious if escape opportunities had this impulsivity reducing effect because delay-signaling stimuli were inescapable during free-choice trials, and these are the trials in which impulsive choice is assessed. Perhaps rats in the IE and Control groups more often chose the larger-later reward anticipating that an escape opportunity would be available, which it never was. Alternatively, the reduction in impulsive choice could be the product of maturation. This seems unlikely because the magnitude of the decrease in the present study (21–34% increase in AUC) is large relative to the 1–2% increase observed in our lab over a similar maturation period (Rung et al., 2018). Further evidence against the maturation hypothesis is the high test-retest correlation between the first two impulsive choice assessments in this experiment when escapes were unavailable (r = .95). Future research should further explore the effects of providing the opportunity to escape from delay-signaling stimuli as a technique for reducing impulsive choice.

Fourth, a limitation of this study is that all rats completed the aversion assessments in the same order. The first of these assessments involved exposure to FI schedules, which can reduce impulsive choice in rats (e.g., Fox et al., 2018; Smith et al., 2015). Perhaps owing to the opportunities to escape from the FI-correlated stimuli, and subsequent possible disruptions in timing the FI durations, we observed no reductions in impulsive choice after experience with the FI schedules. Therefore, it is unlikely that prior experience with FI schedules is responsible for the reduction in impulsive choice in IE and Control rats in the final phase, when delay-signaling stimuli could be escaped from on forced-choice trials.

Finally, if the present findings prove replicable and aversion to, and avoidance of delay-signaling stimuli plays a role in impulsive choice, then novel methods of reducing this aversion warrant investigation. One such method would arrange Pavlovian short-delay conditioning trials initiated with a, for example, 18-s conditioned stimulus (CS) and ending with food delivery (unconditioned stimulus, US). Embedding this CS in a long US-US intertrial interval (e.g., 1800 s) would produce rapid learning due to the large delay-reduction to the US signaled by the onset of the CS (an I:T ratio of 100:1; Gallistel & Gibbon, 2002). During a subsequent impulsive-choice assessment the CS would be arranged as the delay-signaling stimulus separating the instrumental choice and the delivery of the larger-later reward. Such research would reveal if instrumental learning/responding is a necessary component of DE training, or if learning the delay-reduction properties of the CS is sufficient to reduce impulsive choice. If Pavlovian learning is sufficient to reduce impulsive choice, then the amount of training needed to reduce impulsive choice should be predicted by the I:T ratio (Gibbon et al., 1977).

Identifying a method for rapidly reducing aversion to delay-signaling stimuli has translational value. For example, delay-signaling stimuli, which often evoke problem behavior in school settings (Hanley, Heal, Tiger, & Ingvarsson, 2007; Luczynski & Fahmie, 2017), could be arranged so that they signal large delay-reductions to US events. Subsequent to this training, children may demonstrate less emotional responding and problem behavior, and more self-control when these specific delay-signaling stimuli are encountered in daily life.

In summary, this study provides further evidence that DE training reduces impulsive choice and, for the first time, offers evidence that this is due, at least in part, to reduced aversion to delay-signaling stimuli. If replicated, this finding would open new avenues for research exploring novel methods for reducing aversion to delay-signaling stimuli, and evaluating the effects of such training on impulsive choice. Given the potential impact of reducing impulsive choice in influencing choices that impact human health (Volkow & Baler, 2015), further explorations of impulsivity-reducing interventions should be a research priority.

Table 1.

Parameter Estimates for the Beta Regressions Predicting AUC by Assessment

Model/parameter Beta Std. error Z p
Impulsive Choice Assessment I
   Intercept (Control) 0.44 0.27 1.63 .10
   Group (IE) 0.28 0.36 0.77 .44
   Group (DE) 1.29 0.38 3.40 <.001
Impulsive Choice Assessment II
   Intercept (Control) 0.48 0.27 1.80 .07
   Group (IE) 0.37 0.36 1.03 .30
   Group (DE) 1.70 .38 4.48 <.001
Delay-Signal Aversion Assessment:
   Intercept (Control) 1.52 .21 7.28 <.001
   Group (IE) .65 .29 2.21 .03
   Group (DE) .65 .28 2.28 .02
Open in a new tab

Note. For all models, the estimates for the intercept reflect AUC values for rats in the no-training Control group. The parameter estimates reflect the log-odds change in AUC, which can be transformed using the inverse logit (Exp(β)/[1+Exp(β)]) to obtain values on the original scale of measurement.

Table 2.

Parameter Estimates for the Beta Regressions Predicting Proportion of Trials Escaped by Assessment

Model/parameter Beta Std. error Z p
FI-Signal Aversion: FI-40
   Intercept (Control) −0.01 0.31 −0.05 .96
   Group (IE) 0.81 0.43 1.89 .06
   Group (DE) 0.56 0.43 1.30 .19
FI-Signal Aversion: FI-20
   Intercept (Control) −0.58 0.31 −1.86 .06
   Group (IE) 0.68 0.43 1.61 .11
   Group (DE) −0.09 0.41 −.22 .83
Delay-Signal Aversion:
   Intercept (Control) −.01 0.33 −0.04 .97
   Group (IE) 0.44 0.45 0.98 .33
   Group (DE) −1.38 0.45 −3.06 .002
Open in a new tab

Note. For all models, the estimates for the intercept reflect proportions of escapes for rats in the no-training Control group. The parameter estimates reflect the log-odds change in escape proportions, which can be transformed using the inverse logit (Exp(β)/[1+Exp(β)]) to obtain values on the original scale of measurement.

Acknowledgments

This research was supported by NIH grants R21 DA042174–01 and R03 DA044927–01. None of the authors have any real or potential conflict(s) of interest, including financial, personal, or other relationships with organizations or pharmaceutical/biomedical companies that may inappropriately influence the research and interpretation of the findings.

Footnotes

Prior to this phase, 17 rats completed 15 or fewer pilot sessions of a modified delay-signal aversion assessment. These rats’ behavior was undifferentiated from the rest of the rats in the Delay-Signal Aversion Assessment, so details of these pilot sessions are not provided.

The criterion requiring that preference for the larger reward in the first trial block (0-s delay to both rewards) was relaxed from 80% or above to 76% or above, to accommodate the behavior of three rats.

Escapes are not shown for FI 5-s trials because only one rat escaped these trials (once, on average, per session).

References

  1. Alessi SM, & Petry NM (2003). Pathological gambling severity is associated with impulsivity in a delay discounting procedure. Behavioural processes, 64(3), 345–354. [DOI] [PubMed] [Google Scholar]
  2. Amlung M, Petker T, Jackson J, Balodis I, & MacKillop J (2016). Steep discounting of delayed monetary and food rewards in obesity: A meta-analysis. Psychological Medicine, 46, 2423–2434. 10.1017/S0033291716000866 [DOI] [PubMed] [Google Scholar]
  3. Amlung M, Vedelago L, Acker J, Balodis I, & MacKillop J (2017). Steep delay discounting and addictive behavior: A meta‐analysis of continuous associations. Addiction, 112(1), 51–62. 10.1111/add.13535 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Alessi S, & Petry N (2003). Pathological gambling severity is associated with impulsivity in a delay discounting procedure. Behavioural Processes, 64(3), 345–354. 10.1016/S0376-6357(03)00150-5 [DOI] [PubMed] [Google Scholar]
  5. Amlung M, Petker T, Jackson J, Balodis I, & MacKillop J (2016). Steep discounting of delayed monetary and food rewards in obesity: a meta-analysis. Psychological Medicine, 46(11), 2423–2434. 10.1017/S0033291716000866 [DOI] [PubMed] [Google Scholar]
  6. Amlung Michael, Vedelago L, Acker J, Balodis I, & MacKillop J (2017). Steep delay discounting and addictive behavior: A meta-analysis of continuous associations. Addiction, 112(1), 51–62. 10.1111/add.13535 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Audrain-McGovern J, Rodriguez D, Epstein LH, Cuevas J, Rodgers K, & Wileyto EP (2009). Does delay discounting play an etiological role in smoking or is it a consequence of smoking? Drug and Alcohol Dependence, 103(3), 99–106. 10.1016/J.DRUGALCDEP.2008.12.019 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Bailey C, Peterson JR, Schnegelsiepen A, Stuebing SL, & Kirkpatrick K (2018). Durability and generalizability of time-based intervention effects on impulsive choice in rats. Behavioural Processes, 152, 54–62. 10.1016/j.beproc.2018.03.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Barlow P, McKee M, Reeves A, Galea G, & Stuckler D (2016). Time-discounting and tobacco smoking: A systematic review and network analysis. International Journal of Epidemiology, 46(3), 860–869. 10.1093/ije/dyw233 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Coughlin LN, Tegge AN, Sheffer CE, & Bickel WK (2018). A machine-learning approach to predicting smoking cessation treatment outcomes. Nicotine & Tobacco Research 10.1093/ntr/nty259 [DOI] [PMC free article] [PubMed]
  11. Dombrovski AY, Siegle GJ, Szanto K, Clark L, Reynolds CF, & Aizenstein H (2012). The temptation of suicide: striatal gray matter, discounting of delayed rewards, and suicide attempts in late-life depression. Psychological Medicine, 42(06), 1203–1215. 10.1017/S0033291711002133 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Dombrovski Alexandre Y., Szanto K, Siegle GJ, Wallace ML, Forman SD, Sahakian B, … Clark L (2011). Lethal Forethought: Delayed Reward Discounting Differentiates High- and Low-Lethality Suicide Attempts in Old Age. Biological Psychiatry, 70(2), 138–144. 10.1016/J.BIOPSYCH.2010.12.025 [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Evenden JL, & Ryan CN (1996). The pharmacology of impulsive behaviour in rats: the effects of drugs on response choice with varying delays of reinforcement. Psychopharmacology, 128(2), 161–170. 10.1007/s002130050121 [DOI] [PubMed] [Google Scholar]
  14. Fox AE, Visser EJ, & Nicholson AM (2018). Interventions aimed at changing impulsive choice in rats: Effects of immediate and relatively long delay to reward training. Behavioural Processes [DOI] [PubMed]
  15. Gallistel CR, & Gibbon J (2002). The Symbolic Foundations of Conditioned Behavior Mawwah, NJ: Lawrence Erlbaum Associates. [Google Scholar]
  16. Gibbon J, Baldock MD, Locurto C, Gold L, & Terrace HS (1977). Trial and Intertrial Durations in Autoshaping. Journal of Experimental Psychology: Animal Behavior Processes, 3(3), 264–284. Retrieved from https://search-proquest-com.dist.lib.usu.edu/docview/614312668?accountid=14761 [Google Scholar]
  17. Gray JC, & MacKillop J (2015). Impulsive delayed reward discounting as a genetically-influenced target for drug abuse prevention: a critical evaluation. Frontiers in Psychology, 6(1104). 10.3389/fpsyg.2015.01104 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Hanley GP, Heal NA, Tiger JH, & Ingvarsson ET (2007). Evaluation of a class wide teaching program for developing preschool life skills. Journal of Applied Behavior Analysis, 40(2), 277–300. 10.1901/JABA.2007.57-06 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Harvanko AM, Strickland JC, Slone SA, Shelton BJ, & Reynolds BA (2019). Dimensions of impulsive behavior: Predicting contingency management treatment outcomes for adolescent smokers. Addictive Behaviors, 90, 334–340. 10.1016/J.ADDBEH.2018.11.031 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Kim-Spoon J, Farley JP, Holmes C, Longo GS, & McCullough ME (2014). Processes linking parents’ and adolescents’ religiousness and adolescent substance use: Monitoring and self-control. Journal of Youth and Adolescence, 43(5), 745–756. 10.1007/s10964-013-9998-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Kirby KN, & Petry NM (2004). Heroin and cocaine abusers have higher discount rates for delayed rewards than alcoholics or non-drug-using controls. Addiction, 99(4), 461–471. 10.1111/j.1360-0443.2003.00669.x [DOI] [PubMed] [Google Scholar]
  22. Ladd GT, & Petry NM (2003). A comparison of pathological gamblers with and without substance abuse treatment histories. Experimental and Clinical Psychopharmacology, 11(3), 202–209. 10.1037/1064-1297.11.3.202 [DOI] [PubMed] [Google Scholar]
  23. Loree AM, Lundahl LH, & Ledgerwood DM (2015). Impulsivity as a predictor of treatment outcome in substance use disorders: Review and synthesis. Drug and Alcohol Review, 34(2), 119–134. 10.1111/dar.12132 [DOI] [PubMed] [Google Scholar]
  24. Luczynski KC, & Fahmie TA (2017). Preschool life skills: Toward teaching prosocial skills and preventing aggression in young children. In Sturmey P (Ed.), The Wiley Handbook of Violence and Aggression (pp. 1–12). John Wiley & Sons Ltd; 10.1002/9781119057574.whbva059 [DOI] [Google Scholar]
  25. MacKillop J, & Kahler CW (2009). Delayed reward discounting predicts treatment response for heavy drinkers receiving smoking cessation treatment. Drug and Alcohol Dependence, 104(3), 197–203. 10.1016/J.DRUGALCDEP.2009.04.020 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Madden GJ, Petry NM, Badger GJ, & Bickel WK (1997). Impulsive and self-control choices in opioid-dependent patients and non-drug-using control participants: Drug and monetary rewards. Experimental and Clinical Psychopharmacology, 5(3), 256–262. 10.1037/1064-1297.5.3.256 [DOI] [PubMed] [Google Scholar]
  27. Magen E, Dweck CS, & Gross JJ (2008). The hidden-zero effect: Representing a single choice as an extended sequence reduces impulsive choice. Psychological Science, 19(7), 648–649. 10.1111/j.1467-9280.2008.02137.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Marshall AT, Smith AP, & Kirkpatrick K (2014). Mechanisms of impulsive choice: I. Individual differences in interval timing and reward processing. Journal of the Experimental Analysis of Behavior, 102(1), 86–101. 10.1002/jeab.88 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Mazur JE, & Logue AW (1978). Choice in a “self-control” paradigm: Effects of a fading procedure. Journal of the Experimental Analysis of Behavior, 30(1), 11–17. 10.1901/jeab.1978.30-11 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Myerson J, Green L, & Warusawitharana M (2001). Area under the curve as a measure of discounting. Journal of the Experimental Analysis of Behavior, 76(2), 235–243. 10.1901/jeab.2001.76-235 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Odum AL, Madden GJ, Badger GJ, & Bickel WK (2000). Needle sharing in opioid-dependent outpatients: Psychological processes underlying risk. Drug and Alcohol Dependence, 60(3), 259–266. 10.1016/S0376-8716(00)00111-3 [DOI] [PubMed] [Google Scholar]
  32. Odum Amy L. (2011). Delay discounting: I’m a k, you’re a k. Journal of the Experimental Analysis of Behavior, 96(3), 427–439. 10.1901/jeab.2011.96-423 [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Peterson JR, & Kirkpatrick K (2016). The effects of a time-based intervention on experienced middle-aged rats. Behavioural Processes 10.1016/j.beproc.2016.11.002 [DOI] [PMC free article] [PubMed]
  34. Petry NM, & Madden GJ (2010). Discounting and pathological gambling. In Madden GJ & Bickel WK (Eds.), The Behavioral and Neurological Science of Discounting (pp. 273–294). Washington, D.C.: American Psychological Association. [Google Scholar]
  35. Petry Nancy M. (2001). Delay discounting of money and alcohol in actively using alcoholics, currently abstinent alcoholics, and controls. Psychopharmacology, 154(3), 243–250. 10.1007/s002130000638 [DOI] [PubMed] [Google Scholar]
  36. Petry Nancy M, & Casarella T (1999). Excessive discounting of delayed rewards in substance abusers with gambling problems. Drug and Alcohol Dependence, 56(1), 25–32. 10.1016/S0376-8716(99)00010-1 [DOI] [PubMed] [Google Scholar]
  37. Read D, Frederick S, Orsel B, & Rahman J (2005). Four score and seven uears from now: The date/delay effect in temporal discounting. Management Science, 51(9), 1326–1335. 10.1287/mnsc.1050.0412 [DOI] [Google Scholar]
  38. Renda CR, & Madden GJ (2016). Impulsive choice and pre-exposure to delays: III. Four-month test-retest outcomes in male wistar rats. Behavioural Processes, 126 10.1016/j.beproc.2016.03.014 [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Renda CR, Rung JM, Hinnenkamp JE, Lenzini SN, & Madden GJ (2018). Impulsive choice and pre-exposure to delays: IV. effects of delay- and immediacy-exposure training relative to maturational changes in impulsivity. Journal of the Experimental Analysis of Behavior, 109(3). 10.1002/jeab.432 [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Rung JM, & Madden GJ (2018). Experimental reductions of delay discounting and impulsive choice: A systematic review and meta-analysis. Journal of Experimental Psychology: General, 147(9), 1349–1381. 10.1037/xge0000462 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Rung Jillian M., Buhusi CV, & Madden GJ (2018). Reducing impulsive choice: V. The role of timing in delay-exposure training. Behavioural Processes, 157, 557–561. 10.1016/j.beproc.2018.04.018 [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Sheffer CE, Christensen DR, Landes R, Carter LP, Jackson L, & Bickel WK (2014). Delay discounting rates: A strong prognostic indicator of smoking relapse. Addictive Behaviors, 39(11), 1682–1689. 10.1016/J.ADDBEH.2014.04.019 [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Smith AP, Marshall AT, & Kirkpatrick K (2015). Mechanisms of impulsive choice: II. Time-based interventions to improve self-control. Behavioural Processes, 112, 29–42. 10.1016/j.beproc.2014.10.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Stein JS, Johnson PS, Renda C, Smits RR, Liston KJ, Shahan TA, & Madden GJ (2013). Early and prolonged exposure to reward delay: Effects on impulsive choice and alcohol self-administration in male rats. Experimental and Clinical Psychopharmacology, 21(2). 10.1037/a0031245 [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Stein JS, Renda CR, Hinnenkamp JE, & Madden GJ (2015). Impulsive choice, alcohol consumption, and pre-exposure to delayed rewards: II. Potential mechanisms. Journal of the Experimental Analysis of Behavior, 103(1). 10.1002/jeab.116 [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Stuebing SL, Marshall AT, Triplett A, & Kirkpatrick K (2018). Females in the forefront: time-based intervention effects on impulsive choice and interval timing in female rats. Animal Cognition 10.1007/s10071-018-1208-9 [DOI] [PMC free article] [PubMed]
  47. Volkow ND, & Baler RD (2015). NOW vs LATER brain circuits: Implications for obesity and addiction. Trends in Neurosciences, 38(6), 345–352. 10.1016/J.TINS.2015.04.002 [DOI] [PubMed] [Google Scholar]

RESOURCES