Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 Jan 1.
Published in final edited form as: J Exp Anal Behav. 2014 Nov 23;103(1):50–61. doi: 10.1002/jeab.115

Working-Memory Training: Effects on Delay Discounting in Male Long Evans Rats

C Renee Renda 1, Jeffrey S Stein 2, Gregory J Madden 3
PMCID: PMC4314439  NIHMSID: NIHMS639275  PMID: 25418508

Abstract

Delay discounting describes the devaluation of a reward as the delay to the receipt of the reward increases. Because steep delay discounting is robustly correlated with a number of behavioral problems (e.g., substance dependence, gambling) and some evidence suggests steep discounting precedes and predicts drug-taking in humans and rats, this study sought to experimentally reduce rats' delay discounting. Human stimulant-dependent participants given working-memory training reportedly decreased their rates of discounting relative to a sham-training group (Bickel, Yi, Landes, Hill, & Baxter, 2011). To evaluate the cross-species generality of this effect, 38 male Long-Evans rats, matched on pretraining delay-discounting rates, were randomly assigned to receive 140 sessions of working-memory training or sham training (which required no memory of the sample stimulus). Large between-group differences in working memory were observed after training; however, posttraining delay-discounting rates were undifferentiated across groups. Potential explanations for these findings are discussed.

Keywords: impulsivity, delay discounting, working-memory training, executive function, Competing Neurobehavioral Decisions Systems theory, lever press, rats

Steeply discounting the value of delayed rewards is correlated with substance-dependence (MacKillop et al., 2011), pathological gambling (Petry & Casarella, 1999), obesity (Weller, Cook, Avsar, & Cox, 2008), and risky behaviors (Chesson et al., 2006; Odum, Madden, Badger, & Bickel, 2000). In addition, evidence from human longitudinal studies (Audrain-McGovern et al., 2009; Brody et al., 2014; Khurana et al., 2013; Kim-Spoon, McCullough, Bickel, Farley, & Longo, 2014) and animal studies (Koffarnus & Woods, 2013; Perry, Larson, German, Madden, & Carroll, 2005; Perry, Nelson, & Carroll, 2008) suggest that steeply discounting delayed rewards is predictive of drug taking. Given these findings, Bickel, Jarmolowicz, Mueller, Koffarnus, and Gatchalian (2012) have suggested that steep delay discounting is a trans-disease process and that therapeutic reductions in discounting may ameliorate discounting-related pathology.

One method by which steep delay discounting may be therapeutically addressed is suggested by the neural substrates involved in impulsive choice. McClure, Ericson, Laibson, Loewenstein, and Cohen (2004) found that the evolutionarily older limbic system is more active when individuals choose a smaller-sooner reward (SSR) over a larger-later reward (LLR). By contrast, the evolutionarily newer frontal cortex and parietal system are more active when the LLR is chosen (see also, Ballard & Knutson, 2009; McClure, Ericson, Laibson, Loewenstein, & Cohen, 2007). Bickel et al. (2007) proposed a framework to characterize the above correlations by parsing neural activity into two distinct systems—the impulsive system (limbic areas, including the nucleus accumbens) and the executive system (frontal cortex and parietal system). The impulsive system disproportionately weights immediate over delayed rewards, whereas the executive system works to reduce this bias. This Competing Neurobehavioral Decision Systems (CNDS) theory posits that maladaptive behavior is the result of a weak executive system, a strong impulsive system, or some combination thereof.

Studies supporting the CNDS theory include those demonstrating that a) separate neural systems are activated when choosing SSRs versus LLRs (McClure et al., 2007; McClure et al., 2004; Tanaka et al., 2004), b) transcranial magnetic stimulation of brain structures responsible for executive-system behavior affects delay discounting (Essex, Clinton, Wonderley, & Zald, 2012; Figner et al., 2010; Sheffer et al., 2013), c) steep discounting and executive dysfunction are independently correlated with many of the same maladaptive behaviors (e.g., Gunstad et al., 2007; Kubler, Murphy, & Garavan, 2005; Petry & Casarella, 1999; Roca et al., 2008; Weller et al., 2008), d) taxing the executive system (i.e., increasing working memory load) increases delay discounting (Hinson, Jameson, & Whitney, 2003; but see Franco-Watkins, Pashler, & Rickard, 2003 for an alternative interpretation), e) poor working-memory ability is correlated with steep delay discounting in humans (Shamosh et al., 2008) and in rats (Renda, Stein, & Madden, 2014; but see Dellu-Hagedorn, 2006), f) overlap analyses of neuroimaging studies that separately assessed working memory and delay discounting revealed large activity clusters in the left lateral prefrontal cortex that were unique to these two processes (Wesley & Bickel, 2013), and g) one study has demonstrated decreased delay discounting following working-memory training (WMT) in human stimulant-dependent individuals (Bickel, Yi, Landes, Hill, & Baxter, 2011). In the latter study, participants were randomly assigned to either a WMT group or a sham-training group. Where the former group completed a commercially available training regimen designed to enhance working-memory performance, the latter completed the same program but were given the correct answers. Pre- to posttraining reductions in the discounting of delayed rewards were observed only in the WMT group.

We sought to evaluate if WMT would decrease delay discounting in male Long Evans rats. Beyond evaluating the cross-species generality of the WMT effect on delay discounting, there were two reasons for conducting this study. First, Bickel et al. (2011) reported that WMT participants' posttraining assessment of working memory was not different from their pretraining assessment. This may have been because the posttraining working-memory assessment was sufficiently different than that of the WMT program, or perhaps because participants completed a maximum of only 15 sessions of WMT. To address the former, rats completed a working-memory assessment that was similar to the training task; the latter was addressed by exposing our rats to 140 sessions of WMT. Second, if WMT could be used to experimentally reduce delay discounting in rats, then this would provide an opportunity to evaluate the causal relation between differences in delay discounting and subsequent propensity for drug-taking (Stein et al., 2013).

In the present experiment, an adjusting-delay procedure was used to quantify pretraining rates of delay discounting. Rats with the most similar pretraining discounting rates were paired. One rat from each pair was randomly assigned to the WMT group; the other rat was assigned to the Sham-training group. WMT was a variation of the titrating-delay match-to-position (TDMTP) task, a commonly used operant preparation to assess working memory in nonhumans (see, Kangas, Vaidya, & Branch, 2010; Porritt & Poling, 2008). We selected this task for two reasons. First, Bickel et al. (2011) hypothesized that the width of the temporal window across which an organism could recall events would be negatively correlated with rates of delay discounting (see also, Yi, Landes, & Bickel, 2009); by significantly increasing the span of time across which rats could recall sample-stimulus information, we expected significant decreases in delay discounting. Second, the medial prefrontal cortex is implicated in delayed match-to-position tasks (see, e.g., Sloan, Good, & Dunnett, 2006). According to the CNDS theory, improvements in this frontal area should produce a stronger executive system, thereby decreasing delay discounting. Following 140 sessions of WMT or Sham training, groups were compared on working-memory performance and delay discounting.

Method

Subjects

Subjects were 38 experimentally naïve male Long-Evans rats (Harlan, Indianapolis, IN), approximately 75 days old at intake. Rats were housed individually within polycarbonate cages in a temperature- and humidity-controlled animal colony room operating on a 12-hr light/dark cycle (light onset at 7:00 am). Free access to water was available in rats' home cages throughout the study. Rats were food restricted to maintain their weights at approximately 85% of their dealer-supplied free-feeding growth curve. Approval for this study was granted by the Institutional Animal Care and Use Committee at Utah State University.

Apparatus

Nineteen identical operant chambers were used (Med Associates, St. Albans, VT). Each chamber was equipped with a white-noise speaker and housed within a sound-attenuating cube. Experimental manipulanda were positioned on the front and rear walls of the chamber. A food receptacle was centered on the front wall (6 cm above the grid floor). A pellet dispenser positioned outside of the chamber delivered 45 mg food pellets (Bio-Serv, Frenchtown, NJ) to the food receptacle. To either side of the food receptacle were two low-profile retractable levers (10.5 cm above the grid floor). One identical lever was centered on the rear wall (10.5 cm above the grid floor). A 28-V DC cue light was positioned above each lever.

Procedures

Figure 1 illustrates the approximate timing and sequence of experimental conditions.

Fig.1.

Fig.1

Open in a new tab

Order of experimental conditions and approximate age of rats. Age varied due to mastery-based criteria.

Pre-training tasks

Lever training

An autoshaping procedure was used to establish lever pressing on the two levers located on the front wall. Levers were presented in a strictly alternating order and rats continued in the 100-trial sessions until they pressed the lever to earn food in ≥ 90% of the trials. In subsequent sessions, the rear-wall lever was inserted at the beginning of each trial. A single press to this lever inserted one lever on the front wall (order alternating between trials) and a single press delivered two food pellets. Initial training ended when the rat completed ≥ 90% of the arranged trials for two consecutive sessions.

Amount discrimination

To ensure that rats could discriminate between 1- and 3-pellets of food, an amount-discrimination task was conducted. Each 60-trial session was partitioned into 15 blocks of 4 trials each. The first two trials within a block were forced-choice trials. These trials began with the insertion of the rear-wall lever and the illumination of its associated cue light. When the lever was pressed it retracted, the cue light turned off, either the left or right lever on the front was inserted (order randomly determined), and the cue light above the lever was lit. For half of the rats, when the left lever was pressed the lever retracted, the cue light turned off, and 1 pellet was delivered to the food receptacle (for the remaining rats, 3 pellets were delivered; pressing the other lever led to the other reward amount). The remaining two trials in a block were free-choice trials in which both the left and right levers (and cue lights) were presented following a rear-wall response. A response to either lever retracted both levers, extinguished both cue lights, and delivered the reward assigned to the pressed lever. Failure to respond on forced- or free-choice trials within 30-s was scored as an omission. Omitted forced-choice trials were repeated. A variable ITI ensured that new trials began every 90 s. Amount-discrimination sessions continued until the 3-pellet reward was selected on ≥ 90% of the free-choice trials for two consecutive sessions.

Assessing delay discounting

An adjusting-delay task was used to quantify delay discounting (Mazur, 1987). Trial structure was the same as that used in the amount-discrimination task but when the lever associated with the larger reward was pressed a delay was imposed between the response and the 3-pellet reward. During the delay, the lever(s) retracted and the cue light above the LLR lever remained illuminated. The delay, initially set at 0 s, adjusted based on each rat's choices in the preceding trial block. Choosing the LLR on both free-choice trials incremented the delay by 1 s, whereas choosing the SSR on both free-choice trials decreased the delay by 1 s. The delay remained constant if both rewards were selected once in the two free-choice trials. The final delay value obtained in a session served as the starting delay for the subsequent session. A programming error occurred during the first 19-25 sessions in which both cue lights accompanied the delay to the LLR. An additional 20 sessions were conducted following the correction of the programming error. Adjusting delays typically stabilize in 30 sessions or less (see, e.g., Craig, Maxfield, Stein, Renda, & Madden, 2014; Mazur, 2012) and have good test-retest reliability when assessed in a fixed number of sessions (McClure, Podos, & Richardson, 2014). Each rat's mean adjusted delay (MAD) over the final nine sessions served as the measure of delay discounting.

Working-memory training and Sham training

Group Assignment

Rats with the most similar MADs were paired. One rat from each pair was randomly assigned to the WMT group and the other to the Sham group.

Interim Training

All rats completed a training phase that shaped the sequence of responses required in the subsequent task. Sessions were composed of 80 trials, which began with presentation of either the left or the right lever (strictly alternating between trials) and the corresponding cue light. A lever press caused the rear-wall lever, and its cue light, to be presented. Pressing the rear-wall lever resulted in a 2-pellet reward and initiated a 20-s ITI, with white-noise accompaniment. To signal the upcoming trial, the white-noise speaker cycled on and off (every 0.25 s) during the final 3 s of the ITI. Failure to respond on any lever within 10 s was scored as an omission and that trial was repeated. Across several sessions, the response requirement programmed on the side levers was gradually increased from a fixed-ratio 1 (FR 1) to an FR 10.

Working-memory training (WMT)

Rats assigned to the WMT group completed 140 WMT sessions in which they earned food by making correct choices in a modified TDMTP task (see, e.g., Kangas et al., 2010; Porritt & Poling, 2008). In this procedure, rats were required to remember a cue over a delay period (i.e., retention interval) in which the cue was absent. The retention interval gradually increased (decreased) as the rat's percent correct was above (below) the accuracy criteria described below. Data from the first 65 sessions of this procedure have been previously reported (see Renda et al., 2014).

Each trial began by inserting either the right or the left lever (i.e., the “sample lever”) and illumination of the corresponding cue light. The sample lever inserted was selected randomly with the constraint that each lever was presented an equal number of times per session and the same sample lever could not be presented in more than four trials consecutively. Upon completion of an FR 10 on the sample lever, the lever was retracted, its cue light darkened, the rear-wall lever was inserted into the chamber (and its cue light illuminated), and the retention interval timer was initiated. A fixed interval (FI) schedule programmed on the rear-wall lever served as the retention interval timer; a single response after the interval elapsed presented the left and right front-wall levers and their cue lights simultaneously (i.e., the “comparison levers”). The FI arrangement on the rear-wall lever was designed to a) reduce the likelihood of mediating behavior during the retention interval (e.g., sitting in front of the correct sample lever) and b) require an operant response during the retention interval, thereby making the task more similar to the NIMH definition of working memory (NIMH, 2010). That is, rats had to actively maintain task relevant information and resist interference during the rear-wall task (for a similar procedure, see Harper, Hunt, & Schenk, 2006). A response on the comparison lever that matched the position of the sample stimulus was scored as correct and resulted in two food pellets; a mismatch was scored as incorrect and did not result in food delivery. After correct or incorrect trials, a fixed 20-s ITI, with white-noise accompaniment, was initiated. To signal the upcoming trial, the white-noise speaker was cycled on and off every 0.25 s during the final 3 s of the ITI.

Limited-hold contingencies were in place such that a response was required within 25 s for the sample lever and 10 s for the rear-wall and comparison levers. Failure to respond before the limited-hold elapsed was scored as an omission. A correction procedure was in place such that omitted or incorrect trials were repeated until the correct comparison lever was selected. During each session, eight 0-s retention interval trials occurred pseudorandomly (with the constraint that only two could occur consecutively). Intermixing these 0-s retention interval trials has been shown to maintain higher accuracy and minimize response bias (see Jones & White, 1994; Sargisson & White, 2001). Sessions ended after 48 (non-correction) trials or 2 hrs, whichever occurred first.

After the first session, the duration of the first retention interval within a session was set equal to the last retention internal experienced in the preceding session. Subsequently, following every eighth trial, percent correct was calculated over the preceding 20 (non-correction) trials; for the first two calculations within a session, responses made in the prior session were used. Based on these percent correct calculations, the duration of the retention interval in the next trial was changed (or not) using the titration rules outlined in Table 1. In general, if local percent correct was very high, the retention interval increased in duration; the opposite was true when percent correct declined. After 65 sessions, the retention interval was reset to 0 s because the accuracy of some rats' performance was declining, due in part to side bias. To continue WMT while better detecting and ameliorating this problem, the retention interval decreased when percent correct began to decrease on any single lever (early detection of lever bias) and increased more slowly across sessions 66-95; the latter restriction was relaxed in sessions 96-140 (see Table 1).

Table 1.

Criteria used to titrate the duration of the retention interval and amount by which the interval was titrated. Different criteria were used in the range of sessions shown in the first column.

Session Increase if overall % correct is Decrease if overall % correct is Decrease if % correct on either lever is Titration increment
1-65 ≥ 90% < 70% < 70% 0.25 s or 2%*
66-95 ≥ 90% < 80% < 80% .06 s
96-140 ≥ 90% < 80% < 80% 0.25 s or 2%*
Open in a new tab
*

whichever was larger.

Sham training

Sham-trained rats also completed 140 sessions, each composed of 48 trials. Events occurring within the trial were, with one exception, exactly as experienced by their MAD-matched WMT rat. For example, if on the first trial the WMT rat received a left sample stimulus and experienced 10-s retention interval, then the Sham rat to which it was matched started the session with a left sample stimulus and experienced a 10-s “retention” interval. However, at the end of the retention interval the Sham rat was presented with only one pseudorandomly selected comparison lever (no more than four consecutive presentations of the same comparison lever and an equal number of left and right lever presentations each session). For the Sham-trained rat, food was delivered after pressing the comparison lever with a probability set to the overall obtained reinforcement rate of its MAD-matched WMT rat. Trials in which the WMT rat failed to respond (i.e., omissions) were not completed by the Sham rat. Sham rats completed the same response requirements on the front- and rear-wall levers and the same limited-hold contingencies were in place. Omitted trials were repeated.

Post-training tasks

Assessing working memory

To evaluate the effects of WMT versus Sham training on subsequent working-memory performance, all rats completed a TDMTP task. This task, outlined by Kangas et al. (2010), was used because it provides a sensitive, continuous metric of working-memory performance that is not subject to ceiling effects. With the following three exceptions, the trial structure was identical to the WMT task: a) the retention interval duration was increased by 1 s following two consecutive correct trials and decreased by 1 s following a single incorrect trial, b) the correction procedure was omitted, and c) no 0-s delay trials were arranged. To ensure that the Sham rats could accurately complete the task, the retention interval was initially set to 0 s and both rats in the WMT/Sham pair completed this match-to-position task until the Sham rat achieved ≥ 85% correct for two consecutive sessions. Sessions ended after 48 completed trials or after 2 hrs, whichever came first. Based on pilot data collected in our lab, 10 working-memory assessment sessions were conducted under the TDMTP task.

Reassessing delay discounting

The amount-discrimination and adjusting-delay tasks (as described above) were repeated, with the latter lasting a fixed 25 sessions.

Reassessing working memory

To determine if the effects of WMT on working-memory performance persist, the working-memory assessment was repeated for 10 sessions using the procedures described above.

Data analysis

To quantify delay discounting, MADs were calculated over the final nine sessions of the pre- and posttraining adjusting-delay tasks for all rats, with lower MADs reflecting steeper delay discounting. Because MADs were not normally distributed, they were natural log-transformed before statistical analyses were conducted. Matched-samples t-tests were used to assess between-group differences in pre- and posttraining MADs. The slopes of lines of best fit were used to evaluate any differences in trend over the final nine sessions. To determine if pretraining MADs were predictive of changes in delay discounting following working-memory (or sham) training (i.e., a rate-dependent effect; see Bickel, Landes, Kurth-Nelson, & Redish, 2014), change in delay discounting scores (posttraining MAD divided by pretraining MAD) was regressed onto mean-centered pretraining MADs. Separate regression analyses were conducted for WMT and Sham groups. T-tests were used to determine if the slope coefficients (b) significantly differed from zero. In the posttraining working-memory assessments, the average retention interval within a session served as the measure of working-memory performance, with higher retention intervals indicative of better working memory. For data collected in these sessions, a mixed-model ANOVA with a within-subject factor (Session) and a between-subject factor (Group) was used to evaluate if retention intervals were higher in the WMT group and if they increased more rapidly across sessions when compared to the Sham group. Bonferroni corrected post-hoc comparisons were made by conducting separate one-way ANOVAs resulting in a criterion alpha value of .005. Another mixed-model ANOVA was conducted to determine if retention intervals changed from the first posttraining working-memory assessment to the reassessment, with Time as the within-subject factor and Group as the between-subject factor.

Results

Following random assignment of rats to the WMT or Sham-training group, there was no significant between-group difference in pretraining MADs in the initial test of delay discounting, t(18) = .98, p = .34 (see Table 2) and the linear trends in grouped MAD values over the last nine sessions were judged to be equivalent and stable (WMT slope = 0.028, Sham slope = 0.027). There were no significant between-group differences in free- or forced-choice omissions or latency to make a response over the final nine sessions of the delay-discounting assessment, p > .25 in all cases (see Table 3).

Table 2.

Pre- and post-training mean adjusted delays (SEM) for WMT and Sham-trained rats.

Group MAD (s)
Pre-training Post-training
WMT 20.63 (3.72) 17.95 (2.50)
Sham 19.34 (3.39) 15.65 (3.18)
Open in a new tab

Note: MADs and SEM were calculated over the final nine sessions of the adjusting-delay procedure. No significant between-group differences were observed. MAD, mean adjusted delay; WMT, working-memory trained rats; Sham, sham-trained rats.

Table 3.

Omissions and response latencies (s) on forced- and free-choice trials (SEM) during the pre- and post-training adjusting-delay task.

Pre-training Post-training
WMT SHAM WMT SHAM
Forced-choice omissions 0.64 (0.47) 0.22 (0.11) 0.32 (0.21) 0.08 (0.03)
Free-choice omissions 0.35 (0.22) 0.11 (0.04) 0.22 (0.18) 0.08 (0.02)
Latency to respond: Forced-choice 1.91 s (0.14) 1.84 s (0.11) 1.77 s (0.15) 1.58 s (0.09)
Latency to respond: Free-choice 1.79 s (0.11) 1.79 s (0.10) 1.73 s (0.13) 1.64 s (0.10)
Open in a new tab

Note: Omissions and latencies to respond to forced- and free-choice trials were averaged over the final nine sessions of the adjusting-delay procedure. No significant between-group differences were observed. WMT, working-memory trained rats; Sham, sham-trained rats.

Figure 2 shows the titrating retention intervals of individual rats assigned to the WMT group. Data are from the final 75 sessions of WMT1. After 140 sessions of training, there were no visually apparent increasing trends and the average retention interval from sessions 131-135 was not significantly different from the average retention interval from sessions 136-140, t(18) = .37, p = .72. Over the final five sessions of WMT, the mean latency to respond on the rear-wall lever for WMT rats was 1.83 s (SD = 1.69), suggesting that these rats did not linger in front of the to-be-remembered sample-stimulus lever during the retention interval. This is consistent with our observations of rats in a pilot study in which no overt mediating behaviors (e.g., pressing the rear-wall lever while orienting body position to the previously inserted sample lever; see Chudasama & Muir, 1997) were recorded.

Fig. 2.

Fig. 2

Open in a new tab

Retention intervals during working-memory training. Each line shows the average retention interval obtained in each session for individual rats across the final 75 sessions of working-memory training. Rats in the sham-training group experienced the same “retention” intervals during these sessions.

Table 4 shows motivational measures collected during the final five sessions of WMT and Sham training. The only significant difference between groups was the latency to press the comparison lever, which was longer in the WMT group, t(18) = 4.40, p < .001; an expected outcome given that only WMT rats were required to choose between two comparison levers.

Table 4.

Omissions and latencies to respond to the sample and comparison stimuli (SEM) during WMT/Sham training and the initial and final working-memory assessments.

Training Initial Assessment Final Assessment
WMT SHAM WMT SHAM WMT SHAM
Sample omissions 1.25 (0.73) 0.16 (0.11) 8.51 (3.44)* 0.96 (0.86) 11.89 (4.84) 2.37 (1.05)
Comparison omissions 0.01 (0.00) 0.01 (0.00) 0.06 (0.05) 0.01 (0.01) 0.05 (0.04) 0.01 (0.01)
Latency to respond: sample 3.84 s (0.81) 3.24 s (0.56) 7.08 s (0.98)** 3.44 s (0.67) 6.65 s (0.81)* 4.19 s (0.74)
Latency to respond: comparison 1.85 s (0.10)*** 1.37 s (0.06) 5.84 s (0.68)** 3.14 s (0.53) 5.85 s (0.62)* 3.69 s (0.54)
Open in a new tab

Note: Omissions and latencies to respond to the sample and comparison stimuli were averaged over the final five sessions of WMT/Sham training and the two post-training assessments of working memory. Significantly different than Sham-trained rats:

*

p < .05,

**

p < .01,

***

p < .001. WMT, working-memory trained rats; Sham, sham-trained rats.

Figures 3A and 3B show individual rats' retention intervals across the 10 sessions of the first posttraining working-memory assessment for rats assigned to the WMT and Sham groups, respectively. Figure 3C represents the average retention intervals for the WMT group (black data path) and the Sham group (gray data path). There was a significant main effect of Session, F(1, 18) = 81.68, p < .001, and a significant interaction between Session and Group, F(1, 18) = 27.75, p < .001; thus, retention intervals increased more rapidly in the WMT group. Post-hoc comparisons revealed significant between-group differences in the average retention interval obtained at every session, p's < .001. Table 4 depicts omission and latency data for the WMT and Sham rats in these posttraining assessments of working memory. In the final five sessions of the initial working-memory assessment, WMT rats had significantly more sample omissions, t(18) = 2.10, p = .05, and significantly longer latencies to respond on the sample and comparison levers, t(18) = 3.04, p < .01 and t(18) = 3.15, p < .01, respectively. No significant difference was observed in comparison omissions over the final five sessions, p > .30.

Fig. 3.

Fig. 3

Open in a new tab

Average retention intervals from each session of posttraining working-memory assessments. Panels A and B show retention intervals obtained during the initial working-memory assessment for individual WMT and Sham rats, respectively. Panel C shows between-subject averages and SEM for the WMT group (black data paths) and the Sham group (gray data paths). Panels D and E show the average retention intervals in the reassessment of working memory for individual WMT and Sham rats, respectively. Panel F shows the between-subject averages and SEM from this final assessment, separated by group. * p < .001; + p < .005

Table 2 shows posttraining MADs for the WMT and the Sham-trained groups. No significant between-group differences were observed, t(18) = .72, p = .48. Likewise, there were no significant between-group differences in free- or forced-choice omissions or latency to make a response over the final nine sessions, p's > .20 (see Table 3). There was no significant correlation between posttraining MADs and the average retention interval obtained during the final session of WMT, r = .22, p = .36; likewise, there were no significant correlations between posttraining MADs and the final retention interval obtained during the initial assessment of working memory for the WMT or the Sham-trained groups, r = .27, p = .27 and r = .36, p = .13, respectively.

In a reanalysis of the data obtained by Bickel et al. (2011), Bickel et al. (2014) reported rate-dependent effects of WMT on delay discounting. That is, individuals with steeper pretraining discounting rates had the greatest reduction in discounting rates following WMT. In the present study, low pretraining MADs (i.e., steeper delay discounting) were not predictive of larger posttraining changes in MADs for the WMT group, b = -.26, t = -1.29, p > .20, or the Sham group, b = -.31, t = -1.81, p > .05.

Figures 3D and 3E show individual rats' retention intervals over the final working-memory assessment (following the delay-discounting assessment) for the WMT and Sham groups, respectively. Figure 3F represents the average retention interval obtained each session for the WMT group (black data path) and the Sham-trained group (gray data path). There was a significant main effect of Session, F(1, 18) = 86.75, p < .001, and a significant Session by Group interaction, F(1, 18) = 5.19, p = .01. Post-hoc comparisons revealed significant between-group differences in the average retention interval from sessions 2-9, p's < .005. With the Bonferroni corrected alpha value of .005, the differences at session 1 and session 10 only approached significance, p = .009 and p = .006, respectively. There were no significant correlations between post-training MADs and the reassessment of working memory for the WMT or the Sham-trained groups, r = .37, p = .12 and r = .34, p = .15, respectively. Table 4 shows omission and latency data for the WMT and Sham rats in the final assessment of working memory. Over the last five sessions, WMT rats had significantly longer latencies to respond to the sample and comparison levers, t(18) = 2.21, p < .05 and t(18)= 2.51, p < .05, respectively. There were no significant differences in sample or comparison omissions over the final five sessions, p's > 05.

A separate ANOVA conducted on the retention intervals obtained at session 10 of the first working-memory assessment and the reassessment revealed a significant Time by Group interaction, F(1, 18) = 30.31, p < .001, but no significant main effect of Time, p = .99. Thus, retention intervals tended to decrease slightly for the WMT rats and increase slightly for the Sham rats from the initial assessment of working memory to the reassessment.

Discussion

The current study examined effects of extended WMT on subsequent working-memory performance and delay discounting in male Long-Evans rats. Although WMT enhanced posttraining working-memory performance relative to the Sham-trained rats, there was no significant between-group difference in posttraining delay discounting. These findings are in contrast to Bickel et al.'s (2011) report that WMT decreased human stimulant abusers' rates of delay discounting by approximately 50%.

What underlies this null effect of WMT on delay discounting is, of course, impossible to say with certainty. Recognizing the speculative nature of what follows, we will discuss four possible accounts for this trans-species failure to replicate the Bickel et al. (2011) findings; perhaps these speculations will prove useful in designing future experiments. First, it is possible that the working-memory ability of the WMT rats did not improve (relative to the Sham-trained rats) by the working-memory training that was provided. One piece of evidence against this is the posttraining retention intervals across which our rats remembered the sample stimulus are, to the best or our knowledge, the highest reported in the titrating-delay match-to-position literature (e.g., Porritt & Poling, 2008, using a similar procedure, reported a mean peak retention interval of 32.85 s, SEM = 4.56, whereas our WMT rats achieved a mean peak retention interval of 68.32 s, SEM = 5.71, in the posttraining reassessment of working memory). More robust evidence that WMT positively impacted working-memory ability would have been obtained had WMT rats performed better than Sham rats in a novel working-memory task (for review of rodent working-memory preparations, see Dudchenko, 2004; Pontecorvo, Sahgal, & Steckler, 1996) and future studies might include such a posttraining test phase. One caution is that effects of working-memory training often do not generalize to improved performance on novel tasks (Ball et al., 2002; Owen et al., 2010; Redick et al., 2013); indeed such was the case in the Bickel et al. study—WMT did not enhance posttraining working-memory performance when the tasks used in testing were different from those used in training.

A second possible account of the trans-species failure to replicate the Bickel et al. (2011) finding has to do with differences in the working-memory tasks used in training. In our WMT phase, the duration over which rats remembered the sample stimulus was increased when accuracy was high, whereas in the Bickel et al. study, the number of stimuli to be remembered was increased (i.e., memory capacity). For example, in the Sequence Recall of Digits test employed by Bickel and colleagues, humans were initially asked to recall a sequence of three digits. With each correct response, the to-be-remembered sequence increased by one digit, up to a maximum of 10 digits; no explicit retention interval was arranged. By contrast, our rats recalled a single stimulus (left/right sample lever presentation) over long retention intervals. Our task was selected because Bickel et al. hypothesized that widening the temporal window across which events could influence behavior was important in influencing discounting; increasing the working-memory retention interval to a duration approximating the delays to reinforcement in the delay-discounting task seemed the most direct translation of this hypothesis. Beyond this, our task was designed to approximate the NIMH definition of working memory — “the active maintenance and flexible updating of goal/task relevant information…in a form that has limited capacity and resists interference.” Rats were required to maintain information about the location of the sample stimulus while completing an interference task on the rear-wall lever during the retention interval. Further, the rats were required to update task relevant information by forgetting the prior sample stimulus with each new trial. Nonetheless, our task did not increase rats' ability to recall multiple stimuli (i.e., memory capacity) and this may underlie our failure to observe an effect of WMT on subsequent delay discounting. Future nonhuman studies should employ a working-memory task that could potentially expand subjects' working-memory capacity (e.g., an odor non-match to sample task; Dudchenko, Wood, & Eichenbaum, 2000).

A third account of the trans-species failure to replicate relates to the nature of the rewards arranged in the two studies. In the Bickel et al. (2011) study, participants completed three delay-discounting tasks in which verbal descriptions of rewards and delays were provided. In two of these tasks the rewards and delays were hypothetical and asymptotic working-memory performance was significantly correlated (or nearly so) with posttraining discounting rates (rho = -.61, p = .02 and rho = -.52, p = .06 when the LLR was $1000 and $100, respectively). In the third discounting task, real rewards and delays were arranged such that participants received the outcome they had chosen on a randomly selected trial. In the latter task, working memory performance was not correlated with posttraining discounting rates, rho = -.37, p = .19. This finding is consistent with the current study (i.e., when real rewards were arranged in both studies, improvements in working memory were not predictive of lower delay discounting). Why the effects of working-memory enhancement would be confined to the discounting of hypothetical events is not immediately obvious.

A final account of the trans-species failure to replicate is that the Bickel et al. (2011) finding is a Type 1 error. Participants that completed WMT in that study did not improve their working-memory ability in a posttraining assessment of working memory and the extent to which participants' working-memory skills improved on the training tasks was not reported. Bickel et al. hypothesized the training tasks were sufficiently different from the posttraining working-memory assessment, such that skills acquired in one task did not generalize to the other. Why these enhanced skills generalized to the delay-discounting task is unclear. A direct replication of this study is needed to address this concern.

In conclusion, the current study failed to provide evidence that WMT reduces delay discounting in male rats. This finding is inconsistent with the human literature and may be due to procedural differences, a species difference, or a Type 1 error. Future studies should examine the effects of increasing rats' working-memory capacity on subsequent delay discounting. There is also a need to replicate the effects of WMT on human delay discounting with real and hypothetical rewards. Experimentally manipulating nonhuman delay discounting is important as it allows an exploration of the possible causal relation between individual differences in delay discounting and maladaptive behaviors (e.g., in nonhuman models of drug self-administration).

Acknowledgments

This research was supported financially by a grant from the National Institutes of Health: 1R01DA029605, awarded to the last author (G. J. Madden). This research was also supported by a thesis grant awarded to the first author (C. R. Renda) by the Society for the Advancement of Behavior Analysis. None of the authors have any real or potential conflict(s) of interest, including financial, personal, or other relationships with organizations or pharmaceutical/biomedical companies that may inappropriately influence the research and interpretation of the findings. All authors have contributed substantively to this study and have read and approved the final manuscript. All authors would like to thank Patrick S. Johnson for his help in the design of the study and collection of pilot data and Kennan J. Liston, Dallin Everett, and Shayne M. Barker for their assistance in conducting the experiment.

Footnotes

1

Spline curves fit to these rats' retention intervals over the first 65 sessions are presented in Renda et al. (2014). No significant differences were observed between the retention interval achieved at session 65 and those reached by session 140, t(18) = .45, p = .66.

Contributor Information

C. Renee Renda, Department of Psychology, Utah State University.

Jeffrey S. Stein, Department of Psychology, Utah State University

Gregory J. Madden, Department of Psychology, Utah State University

References

  1. Audrain-McGovern J, Rodriguez D, Epstein LH, Cuevas J, Rodgers K, Wileyto EP. Does delay discounting play an etiological role in smoking or is it a consequence of smoking? Drug and Alcohol Dependence. 2009;103(3):99–106. doi: 10.1016/j.drugalcdep.2008.12.019. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Ball K, Berch DB, Helmers KF, Jobe JB, Leveck MD, Marsiske M, Vital Elderly Study Group Effects of cognitive training interventions with older adults: A randomized controlled trial. Journal of the American Medical Association. 2002;288(18):2271–2281. doi: 10.1001/jama.288.18.2271. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Ballard K, Knutson B. Dissociable neural representations of future reward magnitude and delay during temporal discounting. Neuroimage. 2009;45(1):143–150. doi: 10.1016/j.neuroimage.2008.11.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Bickel WK, Jarmolowicz DP, Mueller ET, Koffarnus MN, Gatchalian KM. Excessive discounting of delayed reinforcers as a trans-disease process contributing to addiction and other disease-related vulnerabilities: Emerging evidence. Pharmacology and Therapeutics. 2012;134(3):287–297. doi: 10.1016/j.pharmthera.2012.02.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bickel WK, Landes RD, Kurth-Nelson Z, Redish AD. A quantitative signature of self-control repair: Rate-dependent effects of successful addiction treatment. Clinical Psychological Science. 2014 doi: 10.1177/2167702614528162. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Bickel WK, Miller ML, Yi R, Kowal BP, Lindquist DM, Pitcock JA. Behavioral and neuroeconomics of drug addiction: Competing neural systems and temporal discounting processes. Drug and Alcohol Dependence. 2007;90 Suppl 1:S85–91. doi: 10.1016/j.drugalcdep.2006.09.016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Bickel WK, Yi R, Landes RD, Hill PF, Baxter C. Remember the future: Working memory training decreases delay discounting among stimulant addicts. Biological Psychiatry. 2011;69(3):260–265. doi: 10.1016/j.biopsych.2010.08.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Brody GH, Yu T, MacKillop J, Miller GE, Chen E, Obasi EM, Beach SR. Catecholamine levels and delay discounting forecast drug use among african american youths. Addiction. 2014;109(7):1112–1118. doi: 10.1111/add.12516. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Chesson HW, Leichliter JS, Zimet GD, Rosenthal SL, Bernstein DI, Fife KH. Discount rates and risky sexual behavior among teenagers and young adults. Journal of Risk and Uncertainty. 2006;32:217–230. [Google Scholar]
  10. Chudasama Y, Muir JL. A behavioural analysis of the delayed non-matching to position task: The effects of scopolamine, lesions of the fornix and the prelimbic region on mediating behaviours by rats. Psychopharmacology. 1997;134:73–82. doi: 10.1007/s002130050427. [DOI] [PubMed] [Google Scholar]
  11. Craig AR, Maxfield AD, Stein JS, Renda CR, Madden GJ. Do the adjusting-delay and increasing-delay tasks measure the same construct: Delay discounting? Behavioural Pharmacology. 2014;25(4):306–315. doi: 10.1097/FBP.0000000000000055. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Dellu-Hagedorn F. Relationship between impulsivity, hyperactivity and working memory: A differential analysis in the rat. Behav Brain Funct. 2006;2:10. doi: 10.1186/1744-9081-2-10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Dudchenko PA. An overview of the tasks used to test working memory in rodents. Neuroscience and Biobehavioral Reviews. 2004;28(7):699–709. doi: 10.1016/j.neubiorev.2004.09.002. [DOI] [PubMed] [Google Scholar]
  14. Dudchenko PA, Wood ER, Eichenbaum H. Neurotoxic hippocampal lesions have no effect on odor span and little effect on odor recognition memory but produce significant impairments on spatial span, recognition, and alternation. Journal of Neuroscience. 2000;20(8):2964–2977. doi: 10.1523/JNEUROSCI.20-08-02964.2000. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Essex BG, Clinton SA, Wonderley LR, Zald DH. The impact of the posterior parietal and dorsolateral prefrontal cortices on the optimization of long-term versus immediate value. Journal of Neuroscience. 2012;32(44):15403–15413. doi: 10.1523/JNEUROSCI.6106-11.2012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Figner B, Knoch D, Johnson EJ, Krosch AR, Lisanby SH, Fehr E, Weber EU. Lateral prefrontal cortex and self-control in intertemporal choice. Nature Neuroscience. 2010;13(5):538–539. doi: 10.1038/nn.2516. [DOI] [PubMed] [Google Scholar]
  17. Franco-Watkins AM, Pashler H, Rickard TC. Does working memory load lead to greater impulsivity? Commentary on Hinson, Jameson and Whitney. Journal of Experimental psychology: learning memory and cognition. 2003;32:443–447. doi: 10.1037/0278-7393.32.2.443. [DOI] [PubMed] [Google Scholar]
  18. Gunstad J, Paul RH, Cohen RA, Tate DF, Spitznagel MB, Gordon E. Elevated body mass index is associated with executive dysfunction in otherwise healthy adults. Comprehensive Psychiatry. 2007;48(1):57–61. doi: 10.1016/j.comppsych.2006.05.001. [DOI] [PubMed] [Google Scholar]
  19. Harper DN, Hunt M, Schenk S. Attenuation of the disruptive effects of (+/-)3,4-methylene dioxymethamphetamine (mdma) on delayed matching-to-sample performance in the rat. Behavioral Neuroscience. 2006;120(1):201–205. doi: 10.1037/0735-7044.120.1.201. [DOI] [PubMed] [Google Scholar]
  20. Hinson JM, Jameson TL, Whitney P. Impulsive decision making and working memory. Journal of Experimental Psychology: Learning, Memory, and Cognition. 2003;29(2):298–306. doi: 10.1037/0278-7393.29.2.298. [DOI] [PubMed] [Google Scholar]
  21. Jones BM, White KG. An investigation of the differential-outcomes effect within sessions. Journal of the Experimental Analysis of Behavior. 1994;61(3):389–406. doi: 10.1901/jeab.1994.61-389. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Kangas BD, Vaidya M, Branch MN. Titrating-delay matching-to-sample in the pigeon. Journal of the Experimental Analysis of Behavior. 2010;94(1):69–81. doi: 10.1901/jeab.2010.94-69. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Khurana A, Romer D, Betancourt LM, Brodsky NL, Giannetta JM, Hurt H. Working memory ability predicts trajectories of early alcohol use in adolescents: The mediational role of impulsivity. Addiction. 2013;108(3):506–515. doi: 10.1111/add.12001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Kim-Spoon J, McCullough ME, Bickel WK, Farley JP, Longo GS. Longitudinal associations among religiousness, delay discounting, and substance use initiation in early adolescence. Journal of Research on Adolescence. 2014 doi: 10.1111/jora.12104. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Koffarnus MN, Woods JH. Individual differences in discount rate are associated with demand for self-administered cocaine, but not sucrose. Addiction Biology. 2013;18(1):8–18. doi: 10.1111/j.1369-1600.2011.00361.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Kubler A, Murphy K, Garavan H. Cocaine dependence and attention switching within and between verbal and visuospatial working memory. European Journal of Neuroscience. 2005;21(7):1984–1992. doi: 10.1111/j.1460-9568.2005.04027.x. [DOI] [PubMed] [Google Scholar]
  27. MacKillop J, Amlung MT, Few LR, Ray LA, Sweet LH, Munafo MR. Delayed reward discounting and addictive behavior: A meta-analysis. Psychopharmacology. 2011;216(3):305–321. doi: 10.1007/s00213-011-2229-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Mazur JE. An adjusting procedure for studying delayed reinforcement. In: Commons ML, Mazur JE, Nevin JA, Rachlin H, editors. Quantitative analysis of behavior. Vol. 5. Hillsdale, NJ: Erlbaum; 1987. pp. 55–73. [Google Scholar]
  29. Mazur JE. Effects of pre-trial response requirements on self-control choices by rats and pigeons. Journal of the Experimental Analysis of Behavior. 2012;97(2):215–230. doi: 10.1901/jeab.2012.97-215. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. McClure SM, Ericson KM, Laibson DI, Loewenstein G, Cohen JD. Time discounting for primary rewards. Journal of Neuroscience. 2007;27(21):5796–5804. doi: 10.1523/JNEUROSCI.4246-06.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. McClure SM, Laibson DI, Loewenstein G, Cohen JD. Separate neural systems value immediate and delayed monetary rewards. Science. 2004;306(5695):503–507. doi: 10.1126/science.1100907. [DOI] [PubMed] [Google Scholar]
  32. McClure J, Podos J, Richardson HN. Isolating the delay component of impulsive choice in adolescent rats. Frontiers in Integrative Neuroscience. 2014;8:3. doi: 10.3389/fnint.2014.00003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. NIMH. Working memory: Workshop proceedings. 2010 http://www.nimh.nih.gov/research-priorities/rdoc/working-memory-workshop-proceedings.shtml.
  34. Odum AL, Madden GJ, Badger GJ, Bickel WK. Needle sharing in opioid-dependent outpatients: Psychological processes underlying risk. Drug and Alcohol Dependence. 2000;60(3):259–266. doi: 10.1016/s0376-8716(00)00111-3. [DOI] [PubMed] [Google Scholar]
  35. Owen AM, Hampshire A, Grahn JA, Stenton R, Dajani S, Burns AS, Ballard CG. Putting brain training to the test. Nature. 2010;465(7299):775–778. doi: 10.1038/nature09042. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Perry JL, Larson EB, German JP, Madden GJ, Carroll ME. Impulsivity (delay discounting) as a predictor of acquisition of iv cocaine self-administration in female rats. Psychopharmacology. 2005;178(2-3):193–201. doi: 10.1007/s00213-004-1994-4. [DOI] [PubMed] [Google Scholar]
  37. Perry JL, Nelson SE, Carroll ME. Impulsive choice as a predictor of acquisition of iv cocaine self- administration and reinstatement of cocaine-seeking behavior in male and female rats. Experimental and Clinical Psychopharmacology. 2008;16(2):165–177. doi: 10.1037/1064-1297.16.2.165. [DOI] [PubMed] [Google Scholar]
  38. Petry NM, Casarella T. Excessive discounting of delayed rewards in substance abusers with gambling problems. Drug and Alcohol Dependence. 1999;56(1):25–32. doi: 10.1016/s0376-8716(99)00010-1. [DOI] [PubMed] [Google Scholar]
  39. Pontecorvo MJ, Sahgal A, Steckler T. Further developments in the measurement of working memory in rodents. Brain Research: Cognitive Brain Research. 1996;3(3-4):205–213. doi: 10.1016/0926-6410(96)00007-9. [DOI] [PubMed] [Google Scholar]
  40. Porritt M, Poling A. Scopolamine effects under a titrating-delayed-nonmatching-to-position procedure. Psychological Record. 2008;58(1):37–49. [Google Scholar]
  41. Redick TS, Shipstead Z, Harrison TL, Hicks KL, Fried DE, Hambrick DZ, Engle RW. No evidence of intelligence improvement after working memory training: A randomized, placebo-controlled study. Journal of Experimental Psychology: General. 2013;142(2):359–379. doi: 10.1037/a0029082. [DOI] [PubMed] [Google Scholar]
  42. Renda CR, Stein JS, Madden GJ. Impulsive choice predicts poor working memory in male rats. PloS One. 2014;9(4):e93263. doi: 10.1371/journal.pone.0093263. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Roca M, Torralva T, Lopez P, Cetkovich M, Clark L, Manes F. Executive functions in pathologic gamblers selected in an ecologic setting. Cognitive and Behavioral Neurology. 2008;21(1):1–4. doi: 10.1097/WNN.0b013e3181684358. [DOI] [PubMed] [Google Scholar]
  44. Sargisson RJ, White KG. Generalization of delayed matching to sample following training at different delays. Journal of the Experimental Analysis of Behavior. 2001;75(1):1–14. doi: 10.1901/jeab.2001.75-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Shamosh NA, Deyoung CG, Green AE, Reis DL, Johnson MR, Conway AR, Gray JR. Individual differences in delay discounting: Relation to intelligence, working memory, and anterior prefrontal cortex. Psychological Science. 2008;19(9):904–911. doi: 10.1111/j.1467-9280.2008.02175.x. [DOI] [PubMed] [Google Scholar]
  46. Sheffer CE, Mennemeier MS, Landes RD, Dornhoffer J, Kimbrell T, Bickel WK, Vuong M. Focal electrical stimulation as an effective sham control for active rtms and biofeedback treatments. Applied Psychophysiology and Biofeedback. 2013;38(3):171–176. doi: 10.1007/s10484-013-9221-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Sloan HL, Good M, Dunnett SB. Double dissociation between hippocampal and prefrontal lesions on an operant delayed matching task and a water maze reference memory task. Behavioural Brain Research. 2006;171(1):116–126. doi: 10.1016/j.bbr.2006.03.030. [DOI] [PubMed] [Google Scholar]
  48. Stein JS, Johnson PS, Renda CR, Smits RR, Liston KJ, Shahan TA, Madden GJ. Early and prolonged exposure to reward delay: Effects on impulsive choice and alcohol self-administration in male rats. Experimental and Clinical Psychopharmacology. 2013;21(2):172–180. doi: 10.1037/a0031245. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Tanaka SC, Doya K, Okada G, Ueda K, Okamoto Y, Yamawaki S. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nature Neuroscience. 2004;7(8):887–893. doi: 10.1038/nn1279. [DOI] [PubMed] [Google Scholar]
  50. Weller RE, Cook EW, 3rd, Avsar KB, Cox JE. Obese women show greater delay discounting than healthy-weight women. Appetite. 2008;51(3):563–569. doi: 10.1016/j.appet.2008.04.010. [DOI] [PubMed] [Google Scholar]
  51. Wesley MJ, Bickel WK. Remember the future ii: Meta-analyses and functional overlap of working memory and delay discounting. Biological Psychiatry. 2013 doi: 10.1016/j.biopsych.2013.08.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Yi R, Landes RD, Bickel WK. Novel models of intertemporal valuation: Past and future outcomes. Journal of Neuroscience, Psychology, and Economics. 2009;2(2):102. doi: 10.1037/a0017571. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES