Reward value and internal state differentially drive impulsivity and motivation

Abstract

Goal directed behavior is influenced by both reward value as well as internal state and a large body of research has focused on how reward value and internal drives such as hunger influence motivation in rodent models, however less work has focused on how these factors may differentially affect impulsivity. In these studies, we tested the effect of internal drive versus reward value on different facets of reward-related behavior including impulsive action, impulsive choice and, motivation. We varied reward value by changing the concentration of sucrose in the reward outcome, and varied internal drive by manipulating thirst through water restriction. Consistent with the literature we found that both internal state and reward value influenced motivation. However, we found that in high effort paradigms, only internal state influenced motivation with minimal effects of reward value. Interestingly, we found that internal state and reward value differentially influence different subtypes of impulsivity. Internal state, and to a lesser extent, reward value, influenced impulsive action as measured by premature responding. On the other hand, there were minimal effects of either reward value or homeostatic state on impulsive choice as measured by delay discounting. Overall, these studies begin to address how internal state and reward value differentially drive impulsive behavior. Understanding how these factors influence impulsivity is important for developing behavioral interventions and treatment targets for patients with dysregulated motivated or impulsive behavior.

Introduction

Motivational and inhibitory drives for reward are influenced by both internal homeostatic drives as well as the extrinsic value of the reward. For example, how much a person is willing to pay for a pint of ice cream may be influenced both by how hungry they are, as well how much they like that flavor of ice cream. Similarly, the ability to inhibit the urge to eat ice cream may also be influenced by both hunger and the subjective value of the flavor. While a breadth of research has focused on how internal state and reward value influence motivation, relatively fewer studies have investigated how these factors play a role in modulating impulsive behavior. Understanding how internal versus external drives modulate impulse control is important for the interpretation of studies investigating the neural basis of impulsivity, including pathological levels of impulsive behavior.

Motivation can be measured in animal models as the amount of effort or intensity of an action performed to result in a rewarding outcome. Past work across many species has shown that motivation is sensitive to both internal variables, such as homeostatic drives of hunger or thirst, as well as external factors, such as the size or value of a reward (Hull, 1952; Burnett et al., 2016; Dickinson & Balleine, 1994; Minamimoto et al., 2009). Decreases in a subject’s desires for an outcome (incentive value), whether from internal or external factors result in decreased motivation (Balleine and Dickinson, 1998). While there are many demonstrations of how reward value and homeostatic state influence motivational drive, the connections between these factors and impulsivity are less clear and results have varied by study, impulsivity subtype, and species.

Investigating the link between reward value and impulsivity involves consideration of the different subtypes of impulsivity, given that impulsive behavior is multifaceted and has multiple distinct behavioral and neural circuit underpinnings (Strickland & Johnson, 2021; Desrochers, et al., 2022). Two commonly referenced facets of impulsivity are impulsive action and impulsive choice. Impulsive action encompasses premature responding or deficits inhibiting actions, while impulsive choice describes reduced delay of gratification (Dalley & Robbins, 2017). Both have been linked variably to reward value across species (Jentsch & Pennington, 2014). For example, in humans, increased hedonic value of a reward as measured by preference for sweetness is associated with increased impulsive choice, but not with impulsive action (Weafer et al., 2014). In contrast, in rodents, we have previously shown that increasing reward size resulted in increased impulsive action, potentially making inhibition more difficult when the hedonic value is increased (Desrochers et al., 2021). Another study also reports that higher responsiveness to reward is associated with impaired response inhibition in rats, suggesting that external drive may increase impulsive action (Diergaarde et al., 2009).

Internal states also influence impulsivity, as studied most frequently in the context of hunger. The negative mood associated with hunger is most commonly referenced (colloquially referred to as being hangry), however these effects of this internal/homeostatic drive likely also include effects on goal directed behavior such as increased impulsivity, seen in humans as rash decision making and impatience. Hunger can decrease response inhibition, especially in people with higher trait impulsivity (Nederkoorn et al., 2009). Additionally, hunger increases impulsive choice as measured by increased delay discounting for both food and non-food rewards (Skrynka & Vincent, 2019). In rodents, a number of feeding-related hormones have recently been implicated in promoting impulsivity, though many of these use food rewards to measure the impulsive behavior. Melanin-concentrating hormone, a peptide synthesized in the lateral hypothalamus which promotes feeding behavior, increases impulsive action and impulsive choice independent of effects of motivation (Noble et al., 2019). Additionally, the orexigenic hormone ghrelin released from the stomach which initiates meals, also increases impulsive action and impulsive choice when injected centrally into rats (Anderberg et al., 2016). Taken together, the research suggests a potential link between internal homeostatic state and impulsivity which extends beyond the basic food-seeking drive.

In the current study, we investigated the effect of reward value and internal homeostatic state on motivation and two facets of impulsive behavior. We manipulated reward value by providing low or high rewards (water or 5% sucrose, respectively), and we manipulated internal state with satiety through a water restriction paradigm, with or without supplemental water. We measured the two main components of impulsive behavior to gain a better understanding of how reward value and internal drive influence impulsivity. We measured premature responding in a two-choice serial reaction time task (2-CSRTT) to assess impulsive action and a delay discounting (DD) task to measure impulsive choice. We also assessed motivation as a positive control using random ratio and progressive ratio schedules of reinforcement, given previous studies showing effects of reward value and internal state on motivation in these paradigms. Our results indicate that internal state and, to a lesser extent, reward value influence impulsive action, but neither significantly influence impulsive choice. Additionally both internal state and reward value influenced motivation as expected. However, interestingly, we found that this was only the case at low effort requirements, and that only internal state influenced motivation when effort requirements were high. Overall, these studies allowed us to parse the contribution of internal and external factors that contribute to motivation and impulsivity. Understanding what factors contribute to these phenotypes can direct us to neural circuits that may subserve the diversity of pathological presentations of impulsivity and reward-related behaviors.

Methods

Mice

Mice were bred in the Center for Comparative Medicine and Research Facility at Dartmouth College, and all procedures were approved by Dartmouth College Institutional Animal Care and Use Committee. Mice were weaned at postnatal day (PND) 21 into cages of 2-4 same-sex mice. Mice were maintained in a 12:12 light–dark cycle on ad libitum chow and water until experimental testing began. Male and female mice (N=62) of a mixed C57/BI6J x 129 S1 and background were included in the study (see Supplemental Table for group Ns), beginning between 9 and 26 weeks of age.

Behavioral methods

Mice were housed and tested in the DIY Nautiyal Automated Modular Instrumental Conditioning (DIY-NAMIC) cages for behavioral testing as previously described (Lee et al, 2020). Briefly, the DIY-NAMIC system is closed-economy, homecage system which enables high throughput testing of mice in Pavlovian and operant paradigms using an inexpensive, open source, and customizable apparatus. Each DIY-NAMIC box was controlled by an Arduino microprocessor board, and contained three noseports with infrared head-entry detectors and LED cue lights. Liquid rewards were 10ul, unless otherwise noted, and were delivered through a blunted large gauge hypodermic needle located in the center noseport controlled by a solenoid. Each DIY-NAMIC box was created by modifying a standard mouse ventilated homecage (5.5” x 14” x 5”). Mice had access to the noseports 24h/day, and data was logged continuously through Processing software. As previously analyzed in detail (in Lee et al, 2020), mice perform the around 90% of their responses during the dark phase of the light-dark cycle. Mice were singly housed for all behavioral training and testing in the DIY-NAMIC cages. During testing mice were provided with chow ad libitum. Mice were weighed 2-3 times per week, and data were processed during these times to ensure adequate liquid consumption (>0.5ml/day). Mice were assigned to one of three reward conditions: 1) 5% sucrose outcome from the DIY-NAMIC system with no other access to water (high reward value, high internal drive); 2) water outcome from the DIY-NAMIC system with no other access to water (low reward value, high internal drive); or 3) 5% sucrose outcome from the DIY-NAMIC system with supplemental water (high reward value, low internal drive). When supplemental water was provided, it was given ad libitum for 1 hour per day within 2h of the onset of the dark cycle via access to a standard water bottle placed on their homecage. Mice were run across a number of cohorts (see Supplemental Table 1 for paradigms run in each cohort).

Behavioral pretraining

Arduino programs for all paradigms are available online, along with schematics of trial structure: www.GitHub.com/DIY-NAMICsystem. All mice were initially trained on a sequence of four paradigms in the DIY-NAMIC boxes before testing in 2CSRTT, DD, RR, and/or PR as previously described (Lee et al, 2020). These paradigms began with reward port training (P1) for 3-4 days, during which mice were habituated to retrieving a reward from the center port. The center port was illuminated with a blue LED cue light, and remained on until the mouse poked into the port to receive a 10ul liquid reward. Following each trial there was a variable intertrial interval (ITI) averaging 45s. Next, mice were trained to respond to a stimulus light by nosepoking in either of the two side ports (P2) for 3 days. Each trial began with the blue LED cue lights in both side ports illuminated, and a poke to either port resulted in a reward in the center port as described for P1. Third, mice learned to initiate trials by poking in the center port when the center port cue light was blinking (P3) for 3-4 days. Following a response to the center blinking port, the two side ports were illuminated for a trial as described for P2. Finally, mice were trained on a cue discrimination paradigm (P4) for 3 days. Following trial initiation only one of the two side ports were illuminated and only a response to the lit port was rewarded.

Two Choice Serial Reaction Time Task

2CSRTT was used to test impulsive action. Following P4, mice were first introduced to omission trials in which one of the side cue lights was lit for a limited duration of 5s, after which the trial was terminated with no reward if no poke had been made (P5) for 2-3 days. This paradigm was then repeated with a shortened stimulus duration of 1.5 seconds (P5-5) for 2 days. Mice were then tested with a delay window following trial initiation to measure premature responding. A delay window of 3 to 9s was inserted in the trial between trial initiation and the side port cue light illumination. Pokes to the illuminated side port were rewarded, and responses during the delay window were recorded but there was no penalty for these pokes. Delay window lengths began at 3s, and were increased to 6s and then to 9s. Each delay window was run for 3 days before advancing to the next delay.

Delay Discounting

For the DD paradigm mice were first assessed for their baseline side-port preference in the P3 paradigm for 3-5 days. The least preferred noseport (define by having received fewer pokes) was then assigned to yield a large reward (30μl) for the remainder of the paradigm, while the other noseport remained rewarded with 10μl. Mice were run on a paradigm in which the first ten trials of each 24h period, and then two out of every 20 were forced-choice trials in which only one side port would be illuminated to ensure sampling of both ports. The remainder of the trials were free choice with both ports illuminated, and responses to either port were recorded. Following 4-6 days of this paradigm, mice strongly preferred the large port with an average of 93.6±0.3% of their free-choice responses being on the large port. Next, an increasing delay was introduced between a response to the large port and the large reward (30ul) delivery. Delays ranged from 2-12s and were incremented by 2s, with each delay presented for 4 days before advancing to the next delay. Responses to the small reward port were rewarded immediately with 10ul. If there was no response to either side noseport within 10s, the trial was considered an omission and was terminated with the ITI.

Random Ratio and Progressive Ratio

Mice were tested on a random ratio (RR) and progressive ratio (PR) task with 10μl rewards. A response in the blinking center noseport initiated a trial in which one of the two side noseports would become illuminated. The side was chosen randomly for each trial, and only responses to the illuminated noseport were counted towards the RR or PR requirement. There was no time limit to complete a trial, therefore there was no measurement of breakpoint for PR. Mice were run for 4 days on each of three random ratio schedules which required an average of 5, 10, or 20 (RR5, RR10, or RR20, respectively) pokes in the lit port to receive a reward. Mice were tested for 1 day on a progressive ratio paradigm (PR+8 schedule) which started at 1 poke required with each successive trial requiring an increase of 8 pokes to the previous trial’s requirement in order to receive a reward (e.g. 1, 9, 17,…).

Data Processing and Analysis

All stimuli presentation and behavioral responses were recorded with timestamps in text files using Processing scripts. Initial analysis of these text files was conducted using Python scripts to extract relevant data measures, which were then analyzed and graphed in Excel. Statistical comparisons were performed using mixed and one-way analysis of variance (ANOVA) tests using SPSS Software (28.0.1.0) to assess the effect of reward type and internal drive on behavioral responding. Delay length in 2CSRTT and DD paradigms, and effort requirements for RR were analyzed as repeated measures in the relevant ANOVAs. Sex was initially included as a factor in the ANOVAs, and there were two significant effects of sex: on delay length x sex x group interaction in the delay window pokes for the 2CSRTT, and a main effect of sex on trials initiated in the delay discounting paradigm. Otherwise, there were no significant effects of sex, so data was subsequently collapsed across sexes for all analyses. Fisher's least significant difference (LSD) tests were used to determine significance within post hoc tests. The Huynh-Feldt or Greenhouse-Geisser corrections were used as appropriately when the data did not meet the meet the assumption of sphericity.

Results

Increasing internal drive but not extrinsic reward value increases impulsive action

In order to compare the effects of altering reward value and internal drive on impulsive action, we compared three groups of mice, receiving high reward (5% sucrose) or low reward (water), or high reward under water satiety (5% sucrose with supplemental homecage water) in the 2CSRTT (Fig 1A). Internal drive (as manipulated with supplemental water) and reward value (water versus 5% sucrose) both influenced task motivation as measured by the number of trials initiated (Fig 1B). Collapsed across delay lengths there was a main effect of reward condition on the number of trials initiated. Mice receiving the 5% sucrose reward initiated the greatest number of trials across all delay lengths, followed by water reward (p = 0.001 as compared with sucrose-rewarded mice), and then 5% sucrose with supplemental water–rewarded mice (p < 0.001 as compared with sucrose- and water-rewarded mice). This indicates that motivation to perform the task was influenced by both thirst and reward value drives, with satiety influencing task motivation more so than reward type.

Figure 1. Reward value and internal drive influence impulsive action.

A) A diagram of the 2-choice serial reaction time task (2CSRTT) operant paradigm used to measure impulsive action is shown. B) The number of trials initiated is shown during 2CSRTT averaged across the three delay lengths over the three reward conditions. C) The total number of premature responses (port entries during the delay window) normalized by the number of trials initiated is shown over three delay lengths in the three reward conditions. D) The percentage of trials rewarded is shown over three delay lengths in the three reward conditions. Graphs display individual animals (dots) and group averages (bars) ± SEM for the 3 reward conditions: 5% sucrose (N=11, 5M/6F), Water (N=18,11M/7F), and 5% sucrose with supplemental water (N=13, 6M/7F).

Next, we examined the number of pokes made during the delay window as a measure of premature responding. Interestingly, decreasing internal drive through water satiation decreased this measure of impulsive action, whereas decreasing reward value (from 5% sucrose to water did not decrease premature responding. Specifically, there was a main effect of reward group on delay window responses normalized by the number of trials initiated (F_2,36 = 3.938, p = 0.028; Fig. 1B), significant effects of delay length (F_{2, 72} = 39.643, p < 0.001), as well as a significant interaction between delay length and reward group (F_4,72 = 5.260, p < 0.001; Fig.1C). Responding during the delay window was greatest in the low and high reward groups, and the lowest in the group of sated mice (p=0.006 for water vs 5% sucrose w supplemental water; p=0.197 for 5% sucrose vs 5% sucrose w supplemental water). Interestingly, mice receiving the high reward did not have any increases in premature responding compared to mice receiving low reward (p=0.197). This suggests that internal drive but not reward value influenced the premature responding measures in the 2CSRTT.

We also examined the proportion of trials that were rewarded in the 2CSRTT to assess effects on attention and task performance. There was a main effect of delay length (F_2,72 = 23.882, p < 0.001; Fig.1D), with a reduced proportion of correct trials as delay length increased reflecting the increased difficulty and attention required for the task. Interestingly, there was also main effect of reward group (F_2,36 = 4.661, p = 0.016; Fig.1D) on the proportion of rewarded trials. Specifically, pairwise group comparisons showed no significant different between mice receiving low and high reward (p=0.237), but a significant decrease in the proportion of rewarded trials in the mice that received supplemental water (p=0.029 compared to water reward, p=0.003 compared to 5% sucrose reward). This data demonstrates that the internal state influences task engagement in the 2CSRTT more than reward value.

Delay Discounting was not significantly influenced by either reward value or internal drive

Next, we tested the effect of reward and internal state on impulsive choice in the delay discounting paradigm (Fig 2A). Similar to the 2CSRTT paradigm, the number of trials initiated were different among the groups (F_2,29 = 26.979, p < 0.001; Fig. 2B), with 5% sucrose-rewarded mice initiating the greatest number of trials, followed by the water-rewarded mice (p < 0.001), and then mice receiving supplemental water initiated the fewest trials (p = 0.047 compared to water). However, unlike the results in the 2CSRTT paradigm, there was no significant effect of reward group on the preference for the large reward (F_2,29 = 0.889, p = 0.422; Fig. 2C). As expected, there was a main effect of the delay length on preference, with larger delays reducing choice for the large reward (F_3,87 = 14.514, p < 0.001), with no interaction of group and delay length (F_{5, 75}=1.384, p=0.239). Overall, these data suggest that, in contrast to impulsive action, neither reward type nor internal drive influenced impulsive choice in the delay discounting paradigm.

Figure 2. No effect of reward value or internal drive on delay discounting.

A) A diagram of the delay discounting operant paradigm used to measure impulsive choice is shown. B) The number of trials initiated averaged across all 6 delay lengths are shown for the three reward conditions. C) Preference for the large reward port is shown across six delay lengths in the three reward conditions. Graphs display individual animals (dots) and group averages (bars) ± SEM for the 3 reward conditions: 5% sucrose (N=11, 5M/6F), Water (N=12,6M/6F), and 5% sucrose with supplemental water (N=12, 6M/6F).

Internal drive, but not reward value, influences motivation under high effort requirements

Finally, we examined the influence of reward value and internal drive on motivation directly using random ratio and progressive ratio schedules of reinforcement (Fig 3A). Mice completed three random ratio schedules, with the effort requirements to receive a reward increasing from 5 to 20 in three paradigms (RR5, RR10, and RR20). There were significant effects of altering reward value and internal state on motivation (F_2,31 = 54.693, p < 0.001; Fig. 3B). Interestingly, the effects were different across the different effort requirements with internal drive influencing motivation more at higher effort requirements (group by paradigm interaction: F_4,62 = 4.599, p = 0.003). Specifically, at the low effort RR5 schedule, the pattern of motivation matched that seen for task motivation in the 2CSRTT and DD tasks with reward value and internal drive reducing motivation (main effect of group: F_2,31 = 36.860, p<0.001; all pairwise group comparisons: p<0.001). However, in paradigms with higher effort requirements, reduced internal drive but not reduced reward value decreased motivation as measured by the number of rewards received in RR10, RR20, and PR paradigms (F_2,31 >23.789, p<0.001 for RR10 and RR20; F_2,17=6.7, p=0.007 for PR; Fig 3C). Specifically, in these higher effort paradigms, there was no significant difference between the number of rewards received in the water versus 5% sucrose reward groups (p=0.113 for RR10; p=0.881 for RR20, p=0.676 for PR), and there was significantly lower motivation for rewards in the mice with reduced internal drive (p<0.001 for RR10 and RR20; p=0.008 for PR). Overall this suggests that in lower effort tasks, both reward value and internal drive influence motivation, but with higher effort requirements, internal drive rather than extrinsic reward value influences motivation.

Figure 3. Reducing internal drive, but not reward value, decreases motivation at high effort requirements.

A) A diagram of the operant paradigms are shown. The number of rewards earned is shown for the three groups of mice across three random ratio schedules (B) and one progressive ratio (C) schedule. Graphs display individual animals (dots) and group averages (bars) ± SEM for the 3 reward conditions for RR: 5% sucrose (N=11, 5M/6F), Water (N=14,10M/4F), and 5% sucrose with supplemental water (N=12, 6M/6F); or PR 5% sucrose (N=7, 4M/3F), Water (N=7,4M/3F), and 5% sucrose with supplemental water (N=6, 3M/3F). *p<0.05

Discussion

Our studies examined how internal drive and external reward value can differentially influence motivation and impulsivity. We found that increasing both internal drive and reward value increased motivation as measured by task engagement and responding for low-effort rewards. However, when the required effort increased, only internal drive, but not reward value increased motivation. We also saw this effect in premature responding measures of impulsive action in which only internal drive, but not reward value, increased impulsivity. In contrast, neither manipulation influenced impulsive choice as measured in a delay discounting paradigm. Overall, understanding how internal versus external drives govern goal-directed behavior has important implications for understanding the behavioral and neural regulation of dysregulated inhibition and motivation.

While we saw a significant effect of manipulating internal drive on impulsive action, with thirst increasing premature responding in the 2CSRTT, surprisingly we didn’t see an effect of reward value on premature responding. This differs from the interpretation from our previous study that reward value modulates impulsive action (Desrochers et al, 2021). Specifically, we developed a variable-value Go/No-Go task, in which each trial was cued as a large or small reward. Trials within each reward-size included both Go and No-Go trials. We found that mice displayed higher levels of impulsive action on trials that would result in a large reward compared to those yielding a small reward, as measured by more false alarms on No-Go trials. The varied results could be explained by our different choices of assays for impulsive action - 2CSRTT versus Go/No-Go – potentially measuring different subdimensions of impulsive action, namely waiting and stopping impulsivity, respectively. Alternatively, the difference could also be a result of the different ways that we manipulated reward value. In this study, we varied reward value by changing the concentrations of reward, while in our previous study, we varied value by changing the size/volume of the reward. It is possible that the value manipulation of reward volume versus reward concentration has differential effects on impulsive action.

As expected, we saw consistent effects of both internal state and reward value influencing task motivation across all paradigms. This was measured by the number of trials that mice initiated (through a single poke), and increased both as a result of increasing reward value (water vs 5% sucrose) and increasing internal drive (thirsty vs sated). The results from the low effort RR5 paradigm also matched this pattern. However, when the effort requirements were increased in the RR10, RR20, and PR paradigms, a different pattern emerged, with reward value no longer influencing motivation. Internal thirst drive was the only factor that influenced these high-effort tasks with thirsty mice working about twice as hard for a reward compared to sated mice, regardless of the reward value. This suggests while motivation is clearly influenced by both internal and external drives, when effort requirements increase, internal states such as thirst becomes more relevant for motivation than the value of the reward itself.

Unexpectedly, we found no effect of internal state or reward value on impulsive choice in our studies. While the negative result could be due to how we tested impulsive choice in the delay discounting paradigm, the discounting curves looked as expected, with all mice decreasing their preference for the large reward as the delay to receive the large reward increased. Although there were no significant differences between groups, it is possible that mice with high external and internal reward drives (5% sucrose with no supplemental water) showed a trend towards reduced impulsive choice as seen by a less steep discounting curve. Otherwise, although there are some reports of internal state influencing impulsive choice (Skrynka & Vincent, 2019), it is possible that the delay discounting task is not as sensitive to internal state by nature of it being a comparison between two choices always within the same internal state condition. Similarly for reward value, it is possible that the reward size calculation is done relative to the small reward, thus again factoring out any differences across our reward value manipulation conditions within a subject.

Overall, our studies explore how manipulations to internal drive and extrinsic reward value influence motivation and impulsivity. As expected, both factors influence motivation except when the required effort for a goal is high, in which case the homeostatic drive is more influential. While we expected to find greater effects of reward value on impulsivity, it is important to understand that homeostatic drives influence inhibitory control potentially more than extrinsic reward value. This is an important consideration for behavioral interventions and treatments for patients with disorders which include dysregulated impulsivity. For example, it supports the importance of considering homeostatic drives and internal states as a critical factors in abstinence and harm reduction interventions for substance use disorders. Additionally, it is critical to consider the differential effects of internally versus externally motivated drives when studying impulsivity in rodents and human populations.

Highlights.

• External reward value and internal state are both drives to motivate goal directed behavior.

• Both of these factors influence motivation, however internal homeostatic drive dominates under high effort requirements.

• Decreasing internal drive decreases impulsive action more than decreasing external reward value.

• Neither factor differentially influences impulsive choice measured in a delay discounting paradigm.