The influence of reinforcement schedule on experience-dependent changes in motivation

Abstract

The progressive ratio procedure is used across fields to assess motivation for different reinforcers, define the effects of experimental interventions on motivation, and determine experience-dependent changes in motivation. However, less is known about how operant training schedules affect performance on this widely utilized task. Here we designed an experiment to examine the effect of variable ratio versus fixed ratio training schedules of reinforcement on progressive ratio performance while holding other performance variables constant between groups. We found a robust increase in maximum ratio completed between the pretest and posttraining test highlighting a robust training effect on progressive ratio performance. However, it did not matter if the training was under a fixed or variable ratio schedule. Additionally, we show that neither individual rates during training nor extinction responding correlated with maximum ratio achieved during the sessions. Finally, we show that rates during the training sessions do correlate with extinction performance, suggesting that these variables measure a different aspect of performance that does not predict motivation.

Foundational work in the reinforcement field has shown that the schedule under which a stimulus is presented is a critical determinant of behavior (Ferster & Skinner, 1957). This is highlighted clearly under schedules where opposite behavioral patterns can emerge in response to an aversive stimulus, such as in the case of positive punishment, where behavior is reduced as a result of adding the aversive stimulus, versus negative reinforcement, where behavior is increased as a result of removing or avoiding the aversive stimulus (Azrin et al., 1965). Disparate behavioral responses can also be observed under conditions where behavior is under control of the same stimulus in the same fashion (i.e. positive reinforcement, response = sucrose delivery) but the relationship between the response and reinforcer delivery is varied. For example, fixed interval versus variable interval schedules result in different rates of responding even when both schedules result in the presentation of the same stimulus (Catania, 1962; Suboski, 1965). These differences in response rates depend on a variety of factors such as the variability in the relationship between the response and reinforcer delivery (Catania & Reynolds, 1968; Herrnstein, 1970). While there is a large body of work on how these schedules influence ongoing behavior, it is less clear how a history of reinforcement training under fixed versus variable schedules influences future behavior in a controlled fashion (St Peter Pipkin & Vollmer, 2009). The goal of this study was to understand how experience under fixed versus variable ratio reinforcement schedules altered future measures of performance under progressive ratio schedules.

Progressive ratio tasks use increasing ratios, which increase the effort an animal must emit for each subsequent reinforcer delivery. An animal’s willingness to overcome the physical effort, or number of responses, required for reinforcer delivery is thus used to operationally define motivation and thus measure effort-based motivation during these tasks. This allows for researchers to use breakpoint—the maximum ratio at which animals will respond for reinforcer delivery—to assess relative measures of motivation across groups (Hodos, 1961; Hodos & Kalman, 1963; Richardson & Roberts, 1996). Progressive ratio tasks are widely used and applied to basic science questions across behavioral pharmacology and systems neuroscience fields to assess motivation for various reinforcers, such as drugs of abuse, food, water, and others (Gancarz et al., 2012; Hamill et al., 1999; Randall et al., 2012; Randall, Lee, Nunes, et al., 2014; Randall, Lee, Podurgiel, et al., 2014; Reilly, 1999; Rotolo et al., 2019, 2020; Rowlett, 2000; Yohn et al., 2018; Zhang et al., 2007). They are also widely use in preclinical research to understand how environmental and genetic factors contribute to behavioral dysregulation in disease. For example, dysfunction in motivational systems has been linked to several neuropsychiatric disorders including schizophrenia, major depressive disorder, and substance use disorder (Chong et al., 2016; Der-Avakian et al., 2016; Epstein & Silbersweig, 2015; Mannella et al., 2016; Meyer et al., 2016). With this, behavioral analysis in both basic and preclinical science has become an integral tool in studying motivation and understanding how dysregulation in the neural circuits that drive motivation mediates maladaptive behaviors. Therefore, it is imperative to understand the behavioral factors, especially often overlooked factors like training history, that affect widely used measures of motivation such as the progressive ratio task.

Here we sought to understand the effect of training schedule on subsequent progressive ratio performance. As noted above, reinforcement schedules have robust effects on the acquisition, maintenance, and extinction of goal-directed behaviors (Acosta et al., 2008; Quick & Shahan, 2009). Schedule-dependent effects on reinforced behavior can be difficult to parse, as alterations in schedule also often produce changes in total reinforcers delivered, pattern of responding, rate of behavior, and alter the variability between responses and reinforcer delivery simultaneously (Hendry & Van-Toller, 1964; Quick & Shahan, 2009; Reed & Morgan, 2008; Suboski, 1965). To this end, we aimed to specifically test the effect of variability in reinforcer delivery on future responding, by designing a task that allowed us to alter the variability between responses and sucrose delivery while keeping other factors that could also influence future responding constant. Together these experiments have important implications for the effect of experience on measures of motivation and how these can influence the interpretation of results.

Method

Subjects

Forty 8-week-old male (20) and female (20) C57BL/6J mice were purchased from the Jackson Laboratory (Bar Harbor, ME; SN: 000664) and maintained five per cage on a 12-hr 6:00/6:00 reverse dark/light cycle with food and water provided ad libitum. During sucrose administration sessions, animals were food restricted to ~95% of free feeding body weight with water provided ad libitum. All behavioral experiments were conducted during the dark cycle. Four male mice of the total 20 were excluded from the study due to not reaching acquisition criteria. Experiments were approved by the Institutional Animal Care and Use Committee of Vanderbilt University Medical Center. All experiments were conducted according to the National Institutes of Health guidelines for animal care and use.

Apparatus

Mice were trained and tested daily in individual standard wide mouse operant conditioning chambers (Med Associates Inc., St. Albans, Vermont). These were fitted with two back illuminated nose-pokes, cue lights above each nose-poke, stainless steel grid floors, a liquid receptacle with illumination light attached to a syringe pump for sucrose, a white noise generator with speaker, and a 16-tone generator with speaker capable of outputting frequencies between 1 and 20 KHz (85 dB).

Procedure

General Procedural Information

During each daily session (1hr), the initiation of white noise signaled the beginning of the session and remained on throughout its duration. For each task, one nose-poke was designated as the “active poke” that resulted in the delivery of the reinforcer (sucrose) after a predetermined number of responses. The other nose-poke was the “inactive poke” and had no delivered reinforcer, but program consequences that depended on each experiment as outlined below.

Task Design

Training/Acquisition.

Acquisition training was done under a fixed ratio (FR) 1 schedule of reinforcement. Each response on the left nose poke resulted in the nose poke light turning off, the cue light above the nose poke turning on, illumination of the light over the liquid receptacle and delivery of 10 µL of a 10% sucrose solution to the liquid receptacle. These lights stayed on for 5 s after the delivery of the sucrose. Responses on the active nose poke had no programmed consequences during this period. The sessions were 60 min in length and the mice were allowed to respond for up to 50 sucrose deliveries. Acquisition criteria were considered met once the mice had responded for all 50 sucrose deliveries 2 days in a row. Most mice met these criteria within five sessions (mean 4.44 +/− 0.18 S.E.M, median 4), however we set a cutoff of eight sessions. There were four males that did not meet these criteria and were not included in the rest of the study.

Progressive Ratio 1.

Once acquisition criteria were met, mice completed the first progressive ratio session. In this session, the first sucrose delivery was administered after the first press, the second after three presses, the third after five presses, and so on. The full list of values for the progressive ratio is: 1, 3, 5, 6, 8, 10, 14, 18, 25, 31, 38, 45, 55, 78, 100, 140, 180, 245 based on a list from Roberts et al. (1989) but modified to be easier for mice. These sessions lasted 1 hr in length or until animals went 10 min between responses, whichever came first. The sucrose volume (10 µL of a 10% sucrose solution), cues, and timeouts after sucrose delivery were the same for these sessions as during acquisition.

Fixed Ratio vs Variable Ratio Sessions.

Next, mice completed 5 days of 1-hr sessions where sucrose was delivered under an FR5 or variable ratio (VR) 5 schedule of reinforcement depending on group assignment. Mice were divided into groups balanced by sessions to acquisition, as well as the maximum ratio in the first progressive ratio test. This ensured that before starting the fixed versus variable ratio sessions there were no differences between groups in acquisition or measures of motivation. As this set of experiments was designed to test the difference between variable ratio and fixed ratio performance, we wanted to hold anything else constant that could affect future performance on the progressive ratio tasks. These sessions were capped at a maximum of 50 sucrose deliveries to minimize potential differences in intake between groups.

Progressive Ratio 2:

After the 5 days of FR5 or VR5 sucrose sessions, we ran the same progressive ratio test as before with the exception of increasing the time limit to 6 hr or until 10 min between responses elapsed. Importantly, on progressive ratio 1, many of the animals ended the session—via termination due to the 10-min interresponse criterion—before the 1-hr time limit was reached; therefore, changing this parameter did not influence the results and did not preclude us from comparing progressive ratio 1 and 2 sessions to one another.

Extinction.

After the second progressive ratio test, mice completed three additional days of sucrose sessions at FR5 or VR5 (based on the original group assignment, described above) to ensure that they exhibited a stable baseline of responding before moving on to extinction testing. Subsequently, mice completed four consecutive days of one extinction session per day at FR1 with no sucrose delivered, however the nose poke light was turned off and the light over the liquid receptacle was illuminated for 5 s after each active poke.

Statistical Analysis

GraphPad Prism version 8.0 (LaJolla, CA) was used for statistical analyses. We initially analyzed all the data training schedule by sex in two-way ANOVAS and did not see any sex differences, so for clarity and to simplify the story we collapsed the sexes into a single group. For all pairwise comparisons between groups, we utilized two-tailed t-tests, and for all experiments with two categorical variables we utilized two-way ANOVA analysis with repeated measures when multiple within-subjects’ measures were taken. Holm-Sidak post hoc tests corrected for multiple comparisons after one-way or two-way ANOVAs were used, where appropriate. Type I error rate (alpha) was set to .05 for all statistical tests. Data is represented as the mean +/− S.E.M in figures.

Results

Male and female C57BL6/J mice (n = 20 male, 20 female) were trained to self-administer sucrose on a fixed ratio (FR) 1 schedule of reinforcement and 16 males and 20 females met acquisition criteria as described in the methods. Thus, four males were excluded from proceeding in the study. Before beginning VR5 or FR5 sessions, there were no differences between groups before testing in any measures including the average rate of behavior during acquisition (Fig. 1B, t_{( )} = 1.68, n.s.), the number of days required to meet acquisition criteria (Fig. 1C, t₍₃₄₎ = 0.00, n.s.) and the total sucrose consumed during acquisition (Fig. 1D, t₍₃₄₎ = 1.03, n.s.). Additionally, the maximum ratio completed during the first progressive ratio task did not differ between the groups (Fig. 1E, t₍₃₄₎ = 0.26, n.s.).

Figure 1. Behavioral Paradigm for Testing Experience-Dependent Changes in Motivation over Time.

Note. A. Male and female C57BL6/J mice (n = 20 male, 20 female) were trained to self-administer sucrose on a fixed ratio (FR) 1 schedule of reinforcement. Of these mice, 4 males did not meet acquisition criteria and were thus eliminated from the study. The remaining mice (n = 16 male, 20 female) were subsequently tested on a progressive ratio schedule of reinforcement (data shown in Fig. 2). B. Next, animals were randomly split into two groups that went through 5 days of either FR5 or variable ratio (VR) 5 sessions (data shown in Fig. 3). C. Mice were retested on a progressive ratio schedule of reinforcement (data shown in Fig. 4). D. Lastly, following an additional 5 days of FR5 or VR5 sessions, animals then went through extinction where responses on the active lever resulted in no programmed consequence (data shown in Fig. 5).

During FR5 and VR5 training, there were no differences in responding between the groups as seen in the cumulative records from day 5 of the sessions (Fig. 2B for the FR5 group and Fig. 2C for the VR5 group) as well as in the raster plots (Fig. 2D for FR5 group, Fig. 2E for VR5 group) showing the first 500 s of the session where the runs and pauses and variability of the schedule can be seen clearly. The groups did not differ by total responses (Fig. 2F, 2-way ANOVA, main effect of group F_(1,34) = 0.49, n.s.) or total sucrose intake (Fig. 2G, 2-way ANOVA, main effect of group F(1,34) = 1.58, n.s.) across any of the days. Additionally, there were no differences between groups in average rate of responding over the five sessions (Fig. 2H, t₍₃₄₎ = 0.69, n.s.).

Figure 2. Behavior during Acquisition Was Not Different between Groups.

Note. A. Average rate during acquisition. B. Number of days to acquire the task. C. Total sucrose consumed during acquisition. D. Performance during the first progressive ratio task. Data represented as both individual data (points) and mean +/− S.E.M.

After completing 5 days of FR5 or VR5 sessions, mice completed a second PR session (progressive ratio 2) to compare the effects of the intervening training sessions on motivation as measured under a progressive ratio schedule (Fig. 3A). There were not any differences between FR and VR groups on the maximum ratio completed during these sessions (Fig. 3B, t₍₃₃₎= 0.42, n.s.). However, we did find training-dependent shifts in motivation independent of training history (Fig. 3C, t₍₃₄₎ = 5.67, p < .0001) where we saw a significant increase in the maximum ratio in the second PR session as compared to the first. To assess whether the increase from the first to the second test was different between the FR and VR groups, we calculated the training-based change as delta max ratio—the maximum ratio in the second test minus the maximum ratio in the first test. When delta max ratio was compared between groups there was no significant effect (Fig. 3D, t₍₃₃₎ = 0.04, n.s.). As rate of responding is sometimes used as an additional measure for motivation (e.g. Eldar et al., 2011; Zhang et al., 2007), we collapsed the groups and conducted correlations between the average rate of responding during either the FR or the VR sessions and the final maximum ratio (Fig. 3E, R² = 0.02, n.s.) as well as the delta maximum ratio (Fig. 3F, R² = 0.02, n.s.) and saw no significant correlations.

Figure 3. The Probability of Sucrose Delivery Is Differentially Variable between Schedules, but Other Measures of Behavior Are Not.

Note. A-B Cumulative records from day 5 of each task type. C-D. Raster plots showing behavioral responding and sucrose delivery under the, C, FR5 and, D, VR5 schedule of reinforcement. E. There were no differences across the testing in total responses per session. F. No differences in the total number of sucrose deliveries. G. The average response rate between the groups did not differ. Data represented as both individual data (points) and mean +/− S.E.M.

Lastly, mice completed 3 days of additional FR5 and VR5 sessions and then ran 4 days of extinction (Fig. 4A). We saw no between-group differences in extinction responding across all four sessions (Fig. 4B; 2-way ANOVA main effect of group F_(1,34) = 2.51, n.s.), although both groups significantly decreased responding across the four sessions (main effect of day, F_{(2.31, 78.4)} = 82.45, p < .0001). We then averaged the extinction responding on the first day of extinction and generated group cumulative records to see if there were differences in response patterns across the session (Fig. 4C). To quantify whether the slope of the cumulative record was different between groups, we ran a linear regression on each individual cumulative record and compared the average slopes between groups (Fig. 4D t₍₃₄₎ = 1.70, n.s.), and did not see a difference. Again, as extinction responding has been used as an additional measure for motivation (e.g. Acosta et al., 2008), we correlated the rate on the last day of extinction with the maximum ratio obtained during the last PR session and did not see a correlation (Fig. 4E, R² = 0.03, n.s.). However, we did see a correlation with the rate during the FR or VR session and rate on the final day of extinction (Fig. 4F, R² = 2.78, p = .001).

Figure 4. Progressive Ratio Performance Increases with Training but Is Not Affected by the Schedule of Reinforcement during Training.

Note. A. During the second progressive ratio session there was no difference in performance, plotted as the maximum ratio completed, between the animals that self-administered sucrose on an FR5 versus a VR5 previously. B. Training under any schedule significantly increased progressive ratio performance between test 1 and test 2. C. The change in performance between progressive ratio 1 and 2 was not different between the animals trained under each schedule. Average rate of behavior during FR/VR training does not (D) predict performance on the progressive ratio task or (E) predict changes in the maximum ratio as a result of training. Data represented as both individual data (points) and mean +/− S.E.M.

Discussion

In this study, we designed a task to evaluate whether motivation is affected by different training schedules of reinforcement. Here we show that motivation as measured by maximum ratio completed in a progressive ratio task was increased by additional training; however, the schedule of reinforcement (variable ratio 5 vs. fixed ratio 5) did not have a differential effect on this increase in motivation. We additionally show that while response rates during the training session predict response rates during extinction, these rates did not predict final ratios in the progressive ratio task or the change in the maximum ratio observed across animals.

Motivation is often defined as an organism’s willingness to expend effort for a particular reinforcer, often a reward. This conceptualization is based on effort-based decision making where reward valuation and an effort calculation are integrated to produce a cost-benefit construct (Barnes et al., 2017; Chong et al., 2016; Der-Avakian et al., 2016; Epstein & Silbersweig, 2015; Hamill et al., 1999; Randall et al., 2012; Randall, Lee, Nunes, et al., 2014; Randall, Lee, Podurgiel, et al., 2014; Rotolo et al., 2019, 2020, 2021; Stewart et al., 1974; Yohn et al., 2018). Willingness to overcome physical effort (i.e. pressing a lever) that is required to receive a particular reinforcer provides a quantifiable approach for studying motivation in preclinical models (Barnes et al., 2017; Chong et al., 2016; Hamill et al., 1999; Mingote et al., 2005; Salamone & Correa, 2002). Motivation and reward-value are often quantified by the number of reinforcers an animal earns, also known as break-point, and the highest ratio completed in a progressive ratio task (Chong et al., 2016; St Peter Pipkin & Vollmer, 2009). While it is tempting to attribute measures of motivation as trait variables of individual animals, it is important to note that these measures are context- and experience-dependent (Johnson et al., 2019; Mello et al., 2007). Previous work has shown that these measures can change over time (even over a single day), thus motivation is dynamic and can be influenced by a variety of factors in the environment, including factors as simple as training history, as we show here.

Additionally, since responding is a physical activity, training can potentially change the perception of the effort requirement for high-effort activities. In other words, it is possible that the perceived difficulty of responding on a progressive ratio schedule is reduced by prior experience. This has implications for the effects of drug or other manipulations on progressive ratio performance, because a drug could be affecting the primary reinforcing value of a stimulus such as food, or it could affect the tendency to engage in physical activities, including making a response on a lever (Nicola, 2010). Indeed, work has shown that physical exercise outside of the reinforcement context can influence responding for drugs of abuse (Smith et al., 2011, 2018). It is possible that the changes in motivation due to experience here could be due to similar changes in performance that occur with extensive experience; however, in this study nose pokes were used to record operant responding, thus it is likely that other experience-dependent factors are also at play.

In many fields studies focus on how motivation changes after a treatment or other manipulation (Altshuler et al., 2021; Cunningham et al., 2000; Hamill et al., 1999; Heath et al., 2015; Ostlund et al., 2012; Stewart et al., 1974; Zhang et al., 2007). Our results demonstrate that it is critical to ensure that the control and experimental groups have equal experience, as some manipulations may decrease or increase responding in the task or during the training period, both of which could influence subsequent results. For example, drug self-administration studies, a group that is administering saline, or has access to the drug for a shorter access period, will likely have lower overall responses (Algallal et al., 2020; Kawa et al., 2019). Accordingly, a progressive ratio comparison between these groups needs to be interpreted cautiously. Critically, differences in these cases may be due to training history/experience and may not be attributed to addiction of drug-induced changes in motivation as are often suggested (Allain & Samaha, 2019; Kawa et al., 2019).

During operant conditioning tasks, the rate and pattern of responding is a dependent measure of behavior and has been linked to response probability, frequency and amplitude of reinforcement, and motivational states (Eldar et al., 2011; Lotfizadeh et al., 2012). In other words, some have suggested that rates of responding can be used as a substitute measure of motivation. In our current study, we did not see a correlation between rates and maximum ratio in the progressive ratio task, suggesting that these two measurements represent different, independent dimensions of responding. Response rate is often influenced by a host of factors including the schedule of reinforcement as well as the reinforcer. Indeed, altering the reinforcer does not always have logical effects on response rates. A prominent example is the dose- and schedule-dependent effect of drugs on rate (Li et al., 2003; S. G. Smith et al., 1976; Ward et al., 2005). For example, morphine produces an inverted U shaped dose–effect curve when it is presented on a fixed- or progressive-ratio schedule; however, the doses that produce the highest breakpoints do not also produce the highest rates under other schedules (Li et al., 2003; S. G. Smith et al., 1976). Similarly, dose-dependent effects of drugs on rate differ based on whether the drug is presented under a fixed-ratio, progressive ratio, or choice schedule of reinforcement (Li et al., 2003; Ward et al., 2005). Together, this suggests that while rate is an important measurement of behavior, it does not correspond to other effort-based measures of motivation in all cases.

While response rates did not correlate with measures of motivation, we did find a correlation between rate of behavior during training and subsequent extinction responding. Our task resulted in the presentation of consequent stimuli that were previously associated with reinforcer delivery. Conditioned reinforcement tasks such as this are often used in the addiction field as a measure of drug seeking or craving (Acosta et al., 2008; Grimm et al., 2001, 2008). While this may be true in some cases, the relation between response rates across reinforcement tasks suggests that it is possible that some animals respond at higher rates, regardless of schedule. Thus, the behavioral patterns observed during conditioned reinforcement tasks may not represent the latent variables, such as craving and seeking, that have been attributed to them.

Overall, we show a robust training effect on progressive ratio measures of motivation. These studies show that experience with a task is a major variable that controls this behavior and should be considered in studies that utilize this procedure. Additionally, when designing or evaluating experiments that implement progressive ratio procedures, it is important to consider training history across groups in design and interpretation of the results. This will affect study design across many scientific disciplines, including behavioral pharmacology and neuroscience fields.

Figure 5. No Schedule-Dependent Training Effect on Extinction.

Note. A. Responses during the four extinction sessions. Both groups decreased in responding across the 4 days and there were no differences between groups B. Group average cumulative records on day 1 of extinction. C. Average slope of the cumulative record is not different between groups. D. Rate on the last day of extinction did not correlate with the final maximum ratio in progressive ratio. E. Rate during the training (FR/VR sessions) was correlated with rate on the last day of extinction. Data represented as both individual data (points) and mean +/− S.E.M.