Computational Modeling Differentiates Learning Rate From Reward Sensitivity Deficits Produced by Early-Life Adversity in a Rodent Touchscreen Probabilistic Reward Task

Abstract

Background

Exposure to adversity, including unpredictable environments, during early life is associated with neuropsychiatric illness in adulthood. One common factor in this sequela is anhedonia, the loss of responsivity to previously reinforcing stimuli. To accelerate the development of new treatment strategies for anhedonic disorders induced by early-life adversity, animal models have been developed to capture critical features of early-life stress and the behavioral deficits that such stressors induce. We have previously shown that rats exposed to the limited bedding and nesting protocol exhibited blunted reward responsivity in the probabilistic reward task, a touchscreen-based task reverse translated from human studies.

Methods

To test the quantitative limits of this translational platform, we examined the ability of Bayesian computational modeling and probability analyses identical to those optimized in previous human studies to quantify the putative mechanisms that underlie these deficits with precision. Specifically, 2 parameters that have been shown to independently contribute to probabilistic reward task outcomes in patient populations, reward sensitivity and learning rate, were extracted, as were trial-by-trial probability analyses of choices as a function of the preceding trial.

Results

Significant deficits in reward sensitivity, but not learning rate, contributed to the anhedonic phenotypes in rats exposed to early-life adversity.

Conclusions

The current findings confirm and extend the translational value of these rodent models by verifying the effectiveness of computational modeling in distinguishing independent features of reward sensitivity and learning rate that complement the probabilistic reward task’s signal detection end points. Together, these metrics serve to objectively quantify reinforcement learning deficits associated with anhedonic phenotypes.

Plain Language Summary

Exposure to early-life adversity can lead to psychiatric illness, including anhedonia, the loss of pleasure from previously rewarding activities. This article describes findings from rats exposed to a model of simulated poverty on a touchscreen-based assay reverse translated from a task used to characterize anhedonia in humans. We documented the ability of Bayesian computational modeling and probability analyses, identical to those used with humans, to objectively quantify reinforcement learning deficits associated with anhedonia in rats.

Plain Language Summary

Exposure to early-life adversity can lead to psychiatric illness, including anhedonia, the loss of pleasure from previously rewarding activities. This article describes findings from rats exposed to a model of simulated poverty on a touchscreen-based assay reverse translated from a task used to characterize anhedonia in humans. We documented the ability of Bayesian computational modeling and probability analyses, identical to those used with humans, to objectively quantify reinforcement learning deficits associated with anhedonia in rats.

Childhood and adolescent exposure to poverty, trauma, and chaotic environments have been associated with the development of neuropsychiatric illness that can persist into adulthood (1, 2, 3). Early-life adversity affects more than 30% of children in the United States (4) and has been implicated in the emergence and maintenance of mental illnesses including major depression, bipolar disorder, posttraumatic stress disorder, and substance use disorders. Although several psychological domains have been hypothesized to mediate such risk—including cognitive (e.g., executive functioning, memory) and affective (e.g., reward processing, reactivity to social and affective stimuli, emotion regulation) domains—the emergence of anhedonia, a blunted responsivity to previously rewarding activities, has attracted substantial interest (5, 6, 7). Coordinated bidirectional cross-species research efforts, including clinical and preclinical studies, have aimed to characterize key features of early-life adversity and consequent anhedonic phenotypes in humans and model them in laboratory animals to accelerate the development of novel therapeutic strategies (8, 9, 10, 11).

Regarding early-life adversity, one rodent paradigm that was specifically designed to simulate early-life poverty and unpredictable maternal care is the limited bedding and nesting (LBN) protocol. Numerous studies have documented that implementing resource scarcity to dams and their pups elicits a chaotic and unpredictable/fragmented environment, which in turn has been shown to produce depressive- and anhedonic-like behavioral phenotypes in males and associated abnormalities in the reward circuit (12,13). Importantly, corresponding research endeavors of functionally similar early-life adversity phenotypes in the human laboratory have demonstrated that unpredictable maternal behaviors lead to neurodevelopmental deficits in children (14). Accordingly, self-report instruments have been developed to quantify exposure to fragmented and unpredictable environments during childhood, thereby allowing for risk assessments of subsequent psychiatric disorders (15,16).

Regarding anhedonia, one laboratory-based technique that has proven useful in objectively quantifying deficits in reward responsivity is the probabilistic reward task (PRT). This paradigm uses visual discrimination methodology and asymmetric probabilistic reinforcement contingencies such that correct responses to one alternative (rich) are rewarded more often than correct responses to the other (lean). As predicted by signal detection theory (17, 18, 19), healthy control participants reliably demonstrate an adaptive response bias toward the rich stimulus; however, as was originally observed when probing anhedonic phenotypes in patients with major depressive disorder (MDD) (20,21) and subsequently other disorders including bipolar disorder (22) and substance use disorder (23,24), a blunted response bias is commonly observed that is correlated with self-reported anhedonia. Highlighting the richness of the behavioral repertoire that can be derived from this task, PRT studies of patient samples have also probed for possible abnormalities in the probability of specific responses as a function of the immediately preceding trial. For example, compared with healthy control participants, unmedicated individuals with MDD were characterized by a lower probability of selecting the more frequently rewarded (rich) stimulus after correct identification of the rich stimulus in the immediately preceding trial that had not been rewarded (because a probabilistic reward had not been scheduled) (21). Thus, a blunted response bias in MDD was mainly driven by a reduced ability to sustain a preference for the more advantageous stimulus in the absence of immediate rewards.

Due to the ability of the PRT to provide objective quantifications of anhedonic phenotypes, it has been selected as a recommended task to probe positive valence systems in the latest revision (25) of the Research Domain Criteria’s (26) initiative toward advancing medication development for psychiatric conditions that include anhedonia. Given this collective value, and in the spirit of coordinated research efforts in preclinical therapeutics development across species, the PRT has been reverse translated using touchscreen technology for rats (27), mice (28), and nonhuman primates (29), all of which produce outcomes similar to those observed in human participants (18).

Given the parallel development of the LBN paradigm and anhedonia task described above, we evaluated in a recent study whether the LBN protocol in male rats would produce expected deficits in the PRT’s signal detection metrics of reward responsivity. As described in Kangas et al. (30), response bias toward the more richly rewarded stimulus was blunted in 2 independent cohorts of adult rats previously exposed to the LBN procedure during early life compared with control rats reared under standard housing conditions. Moreover, this anhedonic phenotype was shown to be significantly associated with unpredictable and chaotic maternal interaction with the affected pups, as quantified by entropy metrics of dam/pup behavior, thus confirming construct validity.

The purpose of the current work was to test the quantitative limits of cross-species continuity in this LBN/anhedonia platform. Specifically, several recent human PRT studies have highlighted the value of Bayesian computational modeling as an important analytic technique to complement the signal detection end points and elucidate the behavioral mechanisms of blunted reward responsivity in anhedonic patient populations with greater precision (31). Two key parameters that have been shown to independently contribute to PRT outcomes are reward sensitivity (i.e., which captures the immediate hedonic impact of rewards, that is, consummatory pleasure) and learning rate (i.e., which captures the ability to learn from reinforcing consequences, specifically reward prediction errors, which refer to the difference between expected and obtained rewards). For example, in a recent study of adolescents with either a low or high risk of depression based on maternal history (32), expected blunting of reward responsivity in the PRT and neuroimaging correlates were driven by deficits in reward sensitivity but not learning rate. In another study (33), compared with placebo, an 8-week treatment with a kappa opioid receptor antagonist, which has been hypothesized to have anti-anhedonic effects, was associated with higher response biases as well as higher learning rates. Of note, effects on learning rate, but not reward sensitivity, after kappa opioid receptor treatment was a priori hypothesized in light of 1) preclinical evidence that kappa opioid receptor antagonism restores dopaminergic signaling within brain reward pathways [e.g., nucleus accumbens (34)] and 2) computational modeling indicating that dopaminergic manipulations affect learning rate but not reward sensitivity (31,35,36).

Despite its effectiveness in characterizing deficits in human PRT performance, it is currently unknown whether these computational modeling approaches can effectively distinguish reward deficit determinants in laboratory animals. Therefore, secondary analyses were conducted of anhedonic phenotypes in male participants exposed to the LBN paradigm versus outcomes in healthy control participants (30). Specifically, trial-by-trial reinforcement learning was examined using Bayesian computational modeling and probability analyses optimized in previous human studies to determine 1) whether these quantitative approaches are similarly effective in rats at distinguishing reward sensitivity from learning rate and, if so, 2) how these profiles contribute to the LBN-induced anhedonic phenotypes observed.

Methods and Materials

Subjects

Thirty-two male Sprague Dawley rats, offspring of 6 timed-pregnant dams (delivered embryonic day 15; Envigo), were utilized in the current studies, in which primary outcomes were presented in Kangas et al. (30). These rats were subjected to either standard or LBN rearing protocols at the University of California, Irvine. Around postnatal day (P) 100, these rats were transported to McLean Hospital via overnight shipping. Upon their arrival, subjects were quarantined in an isolated, climate-controlled vivarium bay with unrestricted access to rodent chow and water. Following clearance from quarantine, they were transferred to a larger vivarium with otherwise identical housing conditions. Although subjects continued to have unrestricted access to water in their home cage, to establish sweetened condensed milk as a reinforcer, they were food restricted via daily postsession portions of 10 to 15 g of rodent chow. All research assistants and vivarium technicians responsible for conducting the behavioral studies and animal husbandry duties were blinded to the subjects’ group assignment (i.e., control or LBN). The study’s procedures were approved by the Institutional Animal Care and Use Committee at McLean Hospital and in accordance with guidelines from the Committee on Care and Use of Laboratory Animals of the Institute of Laboratory Animals Resources, Commission on Life Sciences (37).

Early-Life Adversity Paradigm

On P2, 6 dams were randomly assigned as either control or LBN. To create litters of 12 with equal numbers of sexes where possible, pups were cross-fostered between litters born within 12 hours of each other. Early-life adversity was imposed using the LBN paradigm, which consists of limiting the nesting and bedding materials in home cages between P2 and P9 as described previously (12,13). For the LBN group, a plastic-coated mesh platform was placed 2.5 cm above the floor of a standard cage. Cob bedding was reduced to cover the cage floor sparsely, and only one half of a single paper towel was provided for nesting material on the platform. Conversely, control dams and litters resided in standard home cages that contained ample cob bedding and 1 whole paper towel that dams shredded for nesting material. Figure 1 shows a representative photograph of control and LBN housing conditions. Both control and LBN cages were undisturbed from P2 to P9 and housed in temperature- and humidity-controlled rooms. On P10, LBN groups were transferred to home cages identical to control conditions. Rats were weaned on P21 and then group housed.

Figure 1.

Representative photographs of the LBN (upper left) and control (lower left) housing conditions, PRT task schematic (upper right), and PRT (lower right). LBN, limited bedding and nesting; P, postnatal day; PRT, probabilistic reward task.

PRT Training and Testing

Upon their arrival and subsequent release from quarantine at McLean Hospital and following the establishment of food restriction conditions, subjects began PRT training and testing protocols. Empirical validation and task optimization of the touchscreen-based rat PRT can be found in Kangas et al. (27). Details of the rat touch-sensitive experimental chamber can be found in Kangas and Bergman (38), and a task schematic and photograph are presented in Figure 1.

Line-Length Discrimination Training

Trials began with presentation of a white line on a black background positioned 3 cm above 2 blue response boxes (5 × 5 cm) that were left and right of center. The line was either long (600 × 123 pixels; 31.5 × 6.5 cm) or short (200 × 60 pixels; 10.5 × 3.25 cm). Long and short line-length trial types varied in a quasi-random order across 100-trial sessions such that there were exactly 50 trials of each type, but a given trial type was not presented more than 5 times in a row. Subjects were trained to respond to the left or right response box depending on the length of the white line (long line: respond left, short line: respond right, or vice versa). Response box designation was counterbalanced across subjects. During the line-length discrimination training phase, each correct response was reinforced with 0.1 mL of 30% sweetened condensed milk that was paired with an 880-ms yellow screen flash and a 440-Hz tone and followed by a 5-second blackout period. Each incorrect response immediately resulted in a 10-second blackout period without reinforcement. A correction procedure (39) was programmed during initial discrimination training in which each incorrect trial was repeated until a correct response was made and was discontinued after <10 repeats of each trial type occurred in 2 consecutive sessions. Concordant with the performance criteria used in previous human PRT studies (20, 21, 22), discrimination training sessions continued without correction until accuracies for both line-length trial types were ≥80% correct for 2 consecutive sessions. After this training criterion had been met, PRT testing commenced.

PRT Testing

On approximately P200, subjects underwent a 5-session testing protocol using 3:1 probabilistic reinforcement contingencies such that a correct response to one of the line lengths (long or short) was reinforced 60% of the time (rich stimulus), whereas a correct response to the other line length was reinforced 20% of the time (lean stimulus). Incorrect responses were never reinforced. The line length associated with the rich and lean contingency was determined for each subject during their final 2 line-length discrimination training sessions by examining their accuracies and designating the line length with a higher mean accuracy as the stimulus to be rewarded on the lean schedule. This method was specifically designed to examine the effects of early-life adversity on response bias generated by responsivity to asymmetrical probabilistic contingencies rather than the amplification of a preexisting inherent bias that is a function of uncontrolled variables.

Data Analysis

To interrogate the effects of LBN on PRT performance beyond the blunted log b response bias metrics reported previously [cf., (30)], we fit a computational model of trial-level performance, which allows the parsing of 2 constructs critically implicated in reward learning tasks: reward sensitivity, which captures consummatory pleasure, and learning rate, which captures the subject’s ability to learn from reward feedback (31,40). This Bayesian model fit 4 reinforcement learning models to the group-level PRT choice data [for details about the mathematical implementation, see (31)]. Model 1 was the stimulus-action Rescorla-Wagner model with separate sensitivities for reward and nonreward events, which postulates that subjects treat nonrewards as actual punishments and assumes that subjects correctly assign the rewards to particular stimulus-action combinations. Model 2 was the basic stimulus-action Rescorla-Wagner model, which assumes that subjects correctly assign the rewards to particular stimulus-action combinations. Model 3 was the belief model, which assumes that because subjects are unsure about the presented stimulus, they assign rewards to both stimuli, with only a certain preference for the actually presented stimulus. Model 4 was the action-only model, which assumes that subjects only learn about the value of each action, independent of the stimuli.

Following prior recommendations (31), models were fitted using an empirical Bayesian random-effects approach and compared through integrated group-level Bayesian information criterion factors. To constrain individual subjects’ parameter inference by an empirical prior distribution, data were fitted concurrently. Finally, to evaluate the robustness of the findings, the Bayesian modeling was run 3 times, and in each run, the model with the lowest integrated Bayesian information criterion value was considered the winning model. All 4 models yielded the 2 parameters of interest—reward sensitivity and learning rate. Unpaired t tests and Cohen’s d were used to evaluate group differences and effect sizes (control vs. LBN) in reward sensitivity and learning rate, respectively, using the winning model. These computational parameters can be disentangled using a mathematical formulation of reward learning, which relies on prediction errors and has been associated with dopaminergic activity (36,41,42). Thus, let us consider an experiment in which rewards are administered stochastically on a select number of trials (in the case of the PRT, 30 rich rewards and 10 lean rewards within a block of 100 trials). We can denote rt = 1 when a reward is received in trial t, and rt = 0 when no reward is dispensed. The variable ρ is used to represent the subjective value that a subject assigns to the reward. Within this framework, a subject has a so-called expectation (Q_t) of the average reward that it might gain on a given trial through a prediction error, which represents the difference δt = ρrt − Q_t (i.e., the discrepancy between the obtained ρrt and expected Qt reward). This prediction error is then utilized to adjust future expectations (43), according to the formula Q_t+1 = Qt + εδ_t, where 0 ≤ ε ≤ 1 is a learning rate. Therefore, the 2 parameters—ρ and ε—could contribute to anhedonic behavior. The larger the ρ, the more sensitive a subject is to the reward. Conversely, ε captures the extent to which reward prediction errors modulate learning, specifically the speed at which reward affects behavior (44). Thus, a high ε points to a large impact of reward feedback of the prior reward feedback on the current decision, whereas a low learning rate reflects a relatively small impact.

As an additional check of model fit, we also generated surrogate datasets for each subject using the winning model’s parameters. The subject-specific model parameters were fed back into the underlying equations to simulate responses that were theoretically plausible under the same task conditions. This simulation process was repeated 500 times for each subject to ensure a robust sample of predicted responses. Then, we examined whether the surrogate response biases reasonably captured the general pattern observed in the empirical data.

In subsequent analyses, to investigate the effects of LBN on PRT performance with more granularity, as in prior human PRT studies (21,22), we computed the probability that rats chose rich- or lean-associated response as a function of the prior stimulus type (rich vs. lean) and whether their correct response was rewarded or not. For example, what is the probability that rats chose the rich-associated response when the preceding rich or lean stimulus had been rewarded or not (because the probabilistic reward was not scheduled)? As in prior human PRT studies that used these metrics, an arcsine transformation was applied to these percentages. Next, 2 mixed analyses of variance (ANOVAs) were performed. In the first analysis, we evaluated the effects of prior rewards by running a mixed ANOVA with preceding stimulus (rich, lean), current stimulus (rich, lean), and response (rich-associated, lean-associated) as repeated measures and group (LBN, control) as the between-subject factor. In the second analysis, an identical ANOVA was performed, but prior nonrewarded trials were considered. For brevity, only effects involving group were followed up and reported.

Results

Model Selection

Of the 4 reinforcement learning models examined (stimulus-action Rescorla-Wagner model with separate sensitivities for reward and nonreward events, basic stimulus-action Rescorla-Wagner model, belief model, action-only model) across the 3 runs, the belief model was associated with the lowest integrated Bayesian information criterion value at the group level and thus yielded the most parsimonious account of the data (see Figure S1 for model outcome comparisons). As detailed above, the belief model assumes that because subjects are unsure about the presented stimulus, they assign rewards to both stimuli, with only a certain preference for the stimulus that was actually presented. As an additional verification of model fit, surrogate datasets were generated for each subject using belief model parameters and fed back into the underlying equations to simulate across 500 iterations the responses that were theoretically plausible under the same task conditions. Figure 2 presents the surrogate response bias as predicted by the belief model, which also captured the general pattern of response bias reasonably well. This suggests that these computational parameters serve as a reliable proxy for understanding and predicting performance on the PRT.

Figure 2.

Real (white bars) and surrogate (gray bars) mean (±SEM) response bias data using the belief model.

Computational Outcomes

Reward sensitivity and learning rate were extracted from run 1, which fit the data best. Figure 3 presents outcomes from Bayesian computational modeling and probability analyses using trial-by-trial reinforcement learning metrics identical to those previously optimized in human PRT studies (31). As the left panel shows, significant group differences were observed in reward sensitivity (t₃₀ = 4.10, p = .0003, d = 1.46). Conversely, however, as shown in the right panel, learning rate did not differ statistically between groups (t₃₀ = 1.67, p = .11, d = 0.59).

Figure 3.

Effects of control (n = 17) and LBN (n = 15) rearing conditions [see (30)] on reward sensitivity (left panel) and learning rate (right panel). Horizontal lines represent group mean (±SEM), and data points represent values for individual subjects. ∗∗∗p < .001. LBN, limited bedding and nesting.

Probability Analyses Effects of Prior Rewards

The preceding stimulus (rich, lean) × current stimulus (rich, lean) × response (rich-associated, lean-associated) × group (LBN, control) model was run on trials following rewarded responses and revealed various main effects and 2-way interactions not involving group. Critically, several interactions involving group were significant, including the preceding stimulus × group (F_1,30 = 5.56, p < .025) and response × group (F_1,30 = 10.06, p < .003) interactions, which were qualified by a 4-way interaction effect (F_1,30 = 4.74, p < .037). To disentangle the 4-way interaction, we performed 2 follow-up ANOVAs and entered current stimulus, response, and group as factors for trials that were preceded by a rewarded rich versus lean stimulus. For both groups, the probability of choosing the rich-associated alternative was significantly higher than the probability of choosing the lean-associated alternative (both ps < .005). However, as shown in Figure 4A, when considering trials preceded by a rewarded rich trial, the only effect that involved group was a significant response × group interaction (F_1,30 = 4.84, p < .036). Bonferroni-corrected simple tests revealed that, compared with the control group, the LBN group displayed a significantly lower probability of choosing the rich-associated alternative after the preceding rich trial had been rewarded (irrespective of the current stimulus type; p < .034) and instead had a significantly higher probability of choosing the lean-associated alternative (p < .038). As shown in Figure 4B, when considering trials preceded by a rewarded lean trial, the only significant effect that involved group was the main effect of group (F_1,30 = 4.97, p < .035), which was qualified by a significant response × group interaction (F_1,30 = 5.99, p < .020). Bonferroni-corrected simple tests revealed that, compared with the control group, the LBN group had a significantly lower probability of choosing the rich-associated alternative after the preceding lean trial had been rewarded (irrespective of the current stimulus type; p < .019) and instead showed a significantly higher probability of choosing the lean-associated alternative (p < .022). In the control group (p < .001), but not the LBN group (p > .39), the probability of choosing the rich-associated alternative was significantly higher than the probability of choosing the lean-associated alternative (p < .001).

Figure 4.

Effects of control (gray bars) and LBN (black bars) rearing conditions [see (30)] on the likelihood of choosing the rich- vs. lean-associated response when the prior trial was a rich and rewarded trial type (A), a lean and rewarded trial type (B), or the prior trial was not rewarded (C). ∗p < .05, ∗∗p < .005. LBN, limited bedding and nesting.

Probability Analyses: Effects of Prior No Rewards

As shown in Figure 4C, the preceding stimulus (rich, lean) × current stimulus (rich, lean) × response (rich-associated, lean-associated) × group (LBN, control) run on trials following correct but not rewarded responses revealed only a significant response × group interaction (F_1,30 = 17.86, p < .001). For both groups, the probability of choosing the rich-associated alternative was significantly higher than the probability of choosing the lean-associated alternative (irrespective of the prior and current stimulus type; both ps < .001). However, Bonferroni-corrected simple tests revealed that, compared with the control group, the LBN group showed a significantly lower probability of choosing the rich-associated alternative after the preceding correct trial had not been rewarded (irrespective of the prior and current stimulus type; p < .001) and instead had a significantly higher probability of choosing the lean-associated alternative (p < .001).

Discussion

Exposure to early-life adversity using a rodent model of simulated poverty induced blunted reward responsivity as quantified by PRT signal detection metrics [see (30) for psychophysical details]. The current work extends the translational value of this research platform via secondary analyses of Kangas et al. (30) that confirmed Bayesian computational models, identical to those optimized for use in human PRT performance (31), and distinguished independent features of reward sensitivity and learning rate that contribute to deficits associated with anhedonic phenotypes. Specifically, reward sensitivity was significantly impaired compared with control subjects. Conversely, learning rate did not differ significantly between the groups. These computational findings are significant considering that youths at increased risk for MDD (32) as well as adults who eventually show poor response to an 8-week antidepressant treatment (45) are characterized by reduced reward sensitivity using the same Bayesian modeling. Thus, the current findings suggest a platform for evaluating novel pharmacological treatments to alleviate anhedonic phenotypes, which are poorly addressed by currently available interventions (46). More fundamentally, we believe that demonstration in laboratory animals and humans of similar effects of given manipulations on precise computational parameters, which dissect behavior in more granularity, offers some of the most compelling cross-species confluence (47). Moreover, confirming correspondence in behavioral mechanisms across quantitative approaches [i.e., signal detection theory (30) and Bayesian analyses (current findings)] should bolster the predictive validity of preclinical models and in turn enhance translational relevance.

Likewise, sequential analyses of choice and consequence during rich versus lean stimulus trial types also served to provide detailed trial-by-trial characterizations of response allocation that, when aggregated, produced the signal detection–derived anhedonic phenotypes. Specifically, they revealed that, although both groups were more likely to choose the response associated with the rich stimulus, compared with control subjects, the LBN group was less likely to select the rich-associated response following a trial in which they were rewarded in the presence of the rich stimulus, rewarded in the presence of the lean stimulus, or not rewarded at all on the preceding trial. This suggests a lack of trial type specificity in their contributions to the session-wide blunting of adaptive response biases in subjects exposed to early-life adversity compared with healthy control subjects.

Conclusions

Taken together, isolating distinct critical mechanistic features in task outcomes across species can identify behavioral mechanisms in reward processing. Ultimately, it is hoped that coordinated bidirectional studies that use these computerized tasks and computational metrics across human and experimental animals will accelerate the development of therapeutic strategies for neuropsychiatric conditions prominently characterized by anhedonia.